An omnibus test: the Kolmogorov-Smirnov test
The Kolmogorov-Smirnov statistic:
K-S =
max w | Fhat1(w) - Fhat2(w) |
measures the maximum of the absolute differences between the empirical CDFs. A permutation
test can then be constructed to test the omnibus hypothesis that F1(x) = F2(x),
against the alternative hypothesis that the two CDFs are unequal.
Scoring Systems
Ranking the data helps us avoid the effects of outliers, heavy-tailed
distributions, etc. However, the difference between the distribution of raw data and
ranked data is very apparent.
It turns out that ranked scores are essentially expected values of observations from a
uniform distribution. An interesting question is: is there a way to protect from outliers
and other problems while retaining more information about the original distribution of the
data? The answer is yes if we use different scoring systems.
Normal Scores and Van der Waerden scores
Exponential or Savage scores
General F0 scores