How should accuracy be reported fairly when a model is allowed to abstain?

Geifman and El-Yaniv's paper on selective classification studies the relationship between prediction risk and coverage when a classifier is allowed to abstain. Its subjects and experiments were not Jev. TypeSafe's confidence-routing documentation, meanwhile, describes an integration pattern that uses judgments and thresholds to route tasks to automatic processing or further confirmation. Points to debate The editors suggest reporting both how much work was handled automatically and how often that automatically handled portion was wrong. Keeping only a small, easy subset of inputs can produce attractive accuracy numbers, but humans still handle the remaining work, which must not disappear from the scorecard. API failures and timeouts should also be reported separately; otherwise, comparisons may cover only samples that returned successfully. One approach is to agree on acceptable error costs and human-review capacity first, then examine the coverage at different thresholds. This suggestion does not imply that high confidence inherently guarantees safety, nor does it replace necessary permission checks and execution rules. When comparing two models, it is preferable to use the same inputs, human-review budget, and definition of an error. If one model handles more difficult cases, looking only at its overall error rate may lead to the opposite conclusion. Discussion questions 1. Are you trying to optimize accuracy across all inputs, or automatic processing under limited human-review capacity? 2. Where do abstentions, timeouts, and failed model calls each appear in your report? 3. At the same acceptable error level, which approach can cover more real tasks? Primary sources Geifman and El-Yaniv: Selective Classification for Deep Neural Networks https://arxiv.org/abs/1705.08500 TypeSafe: Confidence-gated routing https://docs.typesafe.ai/patterns/confidence-routing Editorial note: This official S1 Hub editorial account used AI to help compile public materials. This is not a submission by the original authors, and S1 Hub has not independently tested the work described. Additional evidence and different perspectives are welcome.