Statistics¶

assess reports three things beyond a count of right answers: a 95% interval around each model's accuracy, a 95% interval around each adapter's gain over the baseline, and a p-value for that gain. All three come from counts alone. Nothing is resampled and there is no random seed, so the same verdicts always give the same numbers.

Accuracy, and how sure it is¶

A model that gets k of n items right has an accuracy of k / n. On a different n items that figure would come out a little different, and the Wilson score interval says how far it could reasonably move: the range the model's accuracy lies in, at 95% confidence.

centre     = (p + z²/2n) / (1 + z²/n)
half-width = z √(p(1 − p)/n + z²/4n²) / (1 + z²/n)

Here p = k / n and z = 1.959963984540054. The lower limit is exactly 0 when no item is right, and the upper limit exactly 1 when every item is.

As a guide to how many items a split needs: at an accuracy of 0.8 the interval reaches about 1.8 points either side with 2,000 items, and about 3.5 with 500.

An adapter against its baseline¶

Every model is scored on the same items, so an adapter is compared with the baseline item by item, not total against total. Each item lands in one of four cells:

Only b and c, the items the two models disagree on, say which of them is better.

• The difference is (c − b) / n, the adapter's accuracy less the baseline's. Above 0, the adapter is ahead.
• Its interval is Newcombe's hybrid score interval for paired data at 95%: method 10 of Newcombe (1998), Statistics in Medicine 17:2635–2650. An interval that lies wholly above 0 says the adapter's lead is more than the choice of items would explain. The result also records φ, the correlation between the two models' answers that the method used.
• The p-value is McNemar's exact test. It asks how often b + c disagreements would split at least this unevenly if neither model were better: min(1, 2 · P(X ≤ min(b, c))) for X ~ Binomial(b + c, ½). It is 1 when the two models never disagree.

Several adapters are each compared with the baseline. Their p-values are not adjusted for the number of comparisons made.

Very small p-values¶

A p-value is worked out exactly, in whole numbers, and only then written as the nearest double. A double holds nothing smaller than about 4.9e-324, so once two models disagree on 1,076 items and every one goes the same way, p is written as 0. The result therefore also holds log10 p, which keeps its value: there, a log10 p of −323.6 says what a p of 0 cannot.

How they are checked¶

Against implementations this project did not write, on twenty tables chosen for their edges: no disagreement, a single item, every item right, every item wrong, and p-values at and past the limits of a double.

The file recorded from statsmodels holds a Newcombe interval too. That one is this project's own formula applied to statsmodels' Wilson limits, so it checks the limits and not the formula.

Proportions and limits agree to within one part in a billion, or to within 1e-15 where that is larger, which is what holds a limit of exactly 0 or 1. P-values agree to one part in a billion wherever a double still holds them in full, and log10 p to one part in a billion or 1e-12.