论文标题
旨在减少技术辅助评论中的手动工作量：估计排名绩效
Towards Reducing Manual Workload in Technology-Assisted Reviews: Estimating Ranking Performance
论文作者
论文摘要
Conducting a systematic review (SR) is comprised of multiple tasks: (i) collect documents (studies) that are likely to be relevant from digital libraries (eg., PubMed), (ii) manually read and label the documents as relevant or irrelevant, (iii) extract information from the relevant studies, and (iv) analyze and synthesize the information and derive a conclusion of SR.当研究人员标记研究时，他们可以在相关文档高于无关文档的情况下筛选排名文档。这种练习被称为筛选优先级（即文档排名方法），加快了进行SR的过程，因为标记为相关的文档可以更早地移至下一个任务。但是，该方法在减少手动工作量方面受到限制，因为要屏幕的文档总数保持不变。为了减少筛选过程中的手动工作量，我们研究了SR的文档排名的质量。这可能会在排名相关研究中的位置上表明研究人员所在的位置，并让他们决定在哪里停止筛选。在对不同排名模型的SR文档排名进行了广泛的分析之后，我们假设“主题宽度”是影响SR排名质量的因素。最后，我们提出了一项估计主题广泛性的度量，并证明拟议的度量是一种简单而有效的方法，可以预测SRS文档排名的质量。
Conducting a systematic review (SR) is comprised of multiple tasks: (i) collect documents (studies) that are likely to be relevant from digital libraries (eg., PubMed), (ii) manually read and label the documents as relevant or irrelevant, (iii) extract information from the relevant studies, and (iv) analyze and synthesize the information and derive a conclusion of SR. When researchers label studies, they can screen ranked documents where relevant documents are higher than irrelevant ones. This practice, known as screening prioritization (ie., document ranking approach), speeds up the process of conducting a SR as the documents labelled as relevant can move to the next tasks earlier. However, the approach is limited in reducing the manual workload because the total number of documents to screen remains the same. Towards reducing the manual workload in the screening process, we investigate the quality of document ranking of SR. This can signal researchers whereabouts in the ranking relevant studies are located and let them decide where to stop the screening. After extensive analysis on SR document rankings from different ranking models, we hypothesize 'topic broadness' as a factor that affects the ranking quality of SR. Finally, we propose a measure that estimates the topic broadness and demonstrate that the proposed measure is a simple yet effective method to predict the qualities of document rankings for SRs.
