Some recent cheminformatics papers have been using “win rate” (i.e., how often a model is the best across a set of tasks) to evaluate benchmarking performance. Although it can be a useful proxy for how good a model is, I have some gripes with this metric, particularly if it is presented without additional context.
Firstly, it can hide small differences; a model may achieve the top-ranked performance, but with only a very small margin over the next best model. Secondly, win rate is not a statistical test for ranking; a seemingly high win rate may not translate into clear separation from alternative approaches. Finally, win rate can hide weak points; a model that wins 80% of tasks but performs awfully on the rest appears more trustworthy than it perhaps should be. A more consistent model with a lower win rate may therefore be more useful in practice.
Fortunately there are a couple of remedies for these issues:
- Within each task, compare models with repeated cross-validation results and statistical tests, such as multiple comparison tests [1], like Tukey’s HSD, or pairwise tests with a p-value correction, like Wilcoxon signed-rank tests. Test results should be reported with appropriate effect sizes, such as Cohen’s d for Tukey and rank-biserial correlation for Wilcoxon. These tests enable us to observe mean rankings of the models with significance taken into consideration.
- Subsequently, we can use a Friedman test on model metrics across tasks [2-3]. The Friedman test is for detecting whether average rankings of groups (i.e., models) are significantly different across different settings (i.e., tasks). Because it averages over every task, a model that wins often but fails badly elsewhere is penalised rather than rewarded.
Here’s a worked example with simulated data:
Continue reading
