论文标题
如果可以的话，请抓住我：在同行评估中检测战略行为
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
论文作者
论文摘要
我们考虑各种同行评估任务中的战略行为问题，包括考试或家庭作业的同行评分以及招聘或晋升中的同行评审。当同行评估任务具有竞争力（例如，当学生在曲线上分级时）时，可能会激励代理商进行虚假报告评估以提高自己的最终地位。我们的重点是设计用于检测此类操作的方法。具体而言，我们考虑一个设置，在该设置中，代理评估其同行和输出排名的子集，后来汇总以形成最终排序。在本文中，我们研究了此问题的统计框架，并设计了检测战略行为的原则测试。我们证明我们的测试具有强大的错误警报保证，并在实际设置中评估了其检测能力。为此，我们设计并执行了一个实验，该实验从受试者中引起战略行为，并释放可能具有独立关注的战略行为模式数据集。然后，我们使用收集的数据进行一系列真实和半合成评估，这些评估证明了我们测试的强大检测能力。
We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-assessment task is competitive (e.g., when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering. In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour. We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings. For this, we design and execute an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest. We then use the collected data to conduct a series of real and semi-synthetic evaluations that demonstrate a strong detection power of our test.
