Usage Case

The two-sample Kolmogorov-Smirnov Test, or KS test, is used to determine if two different samples originate from the populations with the same or different distributions (Hollander et al. 2014, Massa 2016, Zaiontz 2019).  In short, this means the test can be used to determine whether two populations are distinct or not in terms of their distribution (see example pictures below).

The KS test is a nonparametric test procedure, meaning that it does not require the assumption of a normal distribution (or specific distribution of any kind).  However, it is not directly a nonparametric alternative to any particular parametric test.  It has similar, but not identical, aims to the parametric two-sample t-test and its nonparametric alternative, the Mann-Whitney test, and can be used when the conditions for both are not met.

Hypotheses

H₀: The two samples come from populations following the same distribution.  Or, the two populations have the same distribution.

H_A: The samples come from populations with different distributions. Or, the two populations have different distributions.

Differences from Two-Sample T-Test

At first, this test may seem very similar to the two-sample t-test or its nonparametric alternative, the Mann-Whitney test, in terms of goals.  Those two tests determine if two samples originate from populations with the same mean/median or not.  As we know, the mean and median are both measures of center.  The KS test, on the other hand, tests for a difference in not just the center of the two distributions, but the shapes of the distributions as a whole.  This can be very useful at times because two different distributions with the same mean or median can still differ significantly in shape.  Observe the two distributions below, each with the same mean and median, but vastly different shapes (Fig. 1).

a.    b.

Fig. 1.  Density plots use a smooth curve to show the distribution of a numeric variable.  We see them being used here to show the differences between the two distributions visually.

Even though the mean and median of the two distributions are identical, there are clear differences between these two distributions that could easily be relevant to analysis of the data. Consider the situation above, where the two graphs shown model the proportions of cow herds infected by a certain disease across an agricultural region (each data point is the proportion of cows infected in one herd, so each sample represents multiple herds from each region).  From the first graph, we might think that the disease has a fairly constant rate of transmission throughout the region, as it appears to show a normal distribution.  However, with the second graph, we might suspect that there are other confounding variables that make certain herds more or less susceptible to disease throughout the region, as there are two peaks on the graph.  Testing only for differences in mean or median would not reveal these kinds of differences between the two populations, but testing for differences in the shape of the distribution does.

Conditions

Note that the two-sample t-test would not be appropriate in the above situation because of the notable deviations from the normal distribution found in the second graph. In addition, the Mann-Whitney test is also not appropriate because the shift model is not satisfied.  This is why the KS test is so versatile. It can be used with oddly shaped distributions that do not satisfy either of these requirements.  The only conditions of the KS test are for the data to be collected with random sampling such that each data point is independent of both the rest of the data points in its own sample and all other data points in the other sample (Hollander et al. 2014).

Power

At this point, the KS test may seem like an extremely attractive test to use, especially in lieu of the two-sample t-test and Mann-Whitney test.  Not only does it have few conditions, but it tests for changes throughout the distributions and not just their centers. Unfortunately, there is a catch for using the KS test, as there is usually a downside to a test so general that it does not require any conditions outside of standard data collection protocol.  The statistical power for the KS test is very low compared to the other tests.  Statistical power is the chance of correctly rejecting a false null hypothesis (Massa 2016).  In other words, the KS test is not very sensitive, requiring the presence of more substantial changes than other tests to detect a difference between two distributions.  We will see this characteristic in practice in the example below.  This characteristic is a reason why the two-sample t-test and Mann-Whitney test should be considered before this one when testing for a difference of means/medians (but still, this test is the only option in testing for differences in the shapes of the distributions as a whole).

Example in R

Directions by hand

Literature Cited

Hollander, Myles, et al. Nonparametric Statistical Methods. Wiley, 2014, pp. 190-98.

Massa, S. “Lecture 13: Kolmogorov Smirnov Test & Power of Tests.” University of Oxford,

University of Oxford, 2 Feb. 2016, www.stats.ox.ac.uk/~massa/Lecture%2013.pdf.

Zaiontz, Charles. “Two Sample Kolmogorov-Smirnov Test.” Real Statistics Using Excel,

WordPress, www.real-statistics.com/non-parametric-tests/goodness-of-fit-tests/two-sample-kolmogorov-smirnov-test/. Accessed July 2019.