论文标题
一项大规模的随机对照试验对同行评审讨论中的放牧
A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions
论文作者
论文摘要
同行评审是学术界的骨干，人类构成了这一过程的基石，负责审查论文并做出最终的接受/拒绝决定。鉴于已知人类决策容易受到各种认知偏见的影响，因此重要的是要了解同行评审过程中存在哪些偏见并设计管道，以使这些偏见的影响最小化。在这项工作中，我们专注于审视者之间的讨论的动态，并研究其中的放牧行为的存在。在此，我们旨在了解审稿人和更多的高级决策者是否会受到讨论中提出的第一个论点（在审稿人的情况下）在与他人讨论之前对论文形成独立意见时的第一个论点的影响。具体而言，与ICML 2020的审查过程（一个大型的顶级机器学习会议）结合使用 - 我们设计并执行了一项随机对照试验，目的是测试讨论引发者对论文结果的讨论的有条件因果效应。
Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisions. Given that human decision making is known to be susceptible to various cognitive biases, it is important to understand which (if any) biases are present in the peer-review process and design the pipeline such that the impact of these biases is minimized. In this work, we focus on the dynamics of between-reviewers discussions and investigate the presence of herding behaviour therein. In that, we aim to understand whether reviewers and more senior decision makers get disproportionately influenced by the first argument presented in the discussion when (in case of reviewers) they form an independent opinion about the paper before discussing it with others. Specifically, in conjunction with the review process of ICML 2020 -- a large, top tier machine learning conference -- we design and execute a randomized controlled trial with the goal of testing for the conditional causal effect of the discussion initiator's opinion on the outcome of a paper.
