跳到正文
Patterns· García-Galindo·收录于· 6 天前

Estimating the accuracy of selective classifiers without labels using Venn-Abers predictors

Estimating the accuracy of selective classifiers without labels using Venn-Abers predictors

原文摘要

出版方登记 · OpenAlex

Selective classification provides advantages in high-risk applications where erroneous predictions may incur great costs. This approach is particularly relevant in human-in-the-loop settings, where anticipating a model's performance translates into several benefits and directly contributes to safe deployment. In this work, we propose the use of Venn-Abers predictors to ground selective classifiers. Specifically, we leverage the calibration properties of Venn-Abers predictors to estimate the accuracy on the non-rejected samples at inference time. Our proposal also includes a worst-case performance estimate in the form of a lower bound that accounts for the variability of the selective accuracy. We show that the multiprobabilistic output of Venn-Abers predictors can be employed to provide a theoretically guaranteed interval-valued accuracy estimate on the non-rejected samples. When the multiprobabilistic prediction is merged into a single output, our evaluation indicates that the proposed estimators outperform state-of-the-art probabilistic calibration techniques and achieve competitive performance relative to the conformal reject option.

页面未能读取全文,以上为出版方登记的摘要;方法与图表见原文

来源:Patterns · doi.org