论文标题
估计通过分发转移的AI医疗设备的测试性能和保形预测
Estimating Test Performance for AI Medical Devices under Distribution Shift with Conformal Prediction
论文作者
论文摘要
估计基于AI的基于AI的医疗设备的测试性能对于评估临床部署之前的安全性，效率和可用性至关重要。由于受管制的医疗设备软件的性质以及获取大量标记的医疗数据集的难度，我们考虑在未经标记的目标域上预测任意黑框模型的测试准确性，而无需修改原始培训过程或对原始源数据的任何分配假设进行修改（即我们将模型视为“黑色盒”的任何分配假设），并将其视为“黑色盒”，并将其视为“黑色盒”的预测输出。我们提出了一种基于共形预测的“黑盒”测试估计技术，并根据几种临床上相关的分配转移类型（机构，硬件扫描仪，Atlas，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hospital，Hosite foripty hosity类型，对三个医学成像数据集（乳房X线摄影，皮肤病学和组织病理学）的其他方法进行评估。我们希望，通过促进黑盒模型的实用有效估计技术，医疗设备的制造商将开发更标准化和现实的评估程序，以提高临床AI工具的鲁棒性和可信度。
Estimating the test performance of software AI-based medical devices under distribution shifts is crucial for evaluating the safety, efficiency, and usability prior to clinical deployment. Due to the nature of regulated medical device software and the difficulty in acquiring large amounts of labeled medical datasets, we consider the task of predicting the test accuracy of an arbitrary black-box model on an unlabeled target domain without modification to the original training process or any distributional assumptions of the original source data (i.e. we treat the model as a "black-box" and only use the predicted output responses). We propose a "black-box" test estimation technique based on conformal prediction and evaluate it against other methods on three medical imaging datasets (mammography, dermatology, and histopathology) under several clinically relevant types of distribution shift (institution, hardware scanner, atlas, hospital). We hope that by promoting practical and effective estimation techniques for black-box models, manufacturers of medical devices will develop more standardized and realistic evaluation procedures to improve the robustness and trustworthiness of clinical AI tools.
