学术文献库

Machine learning has a recognised need for large amounts of annotated data. Due to the high cost of expert annotations, crowdsourcing, where non-experts are asked to label or outline images, has been proposed as an alternative. Although many promising results are reported, the quality of diagnostic crowdsourced labels is still unclear. We propose to address this by instead asking the crowd about visual features of the images, which can be provided more intuitively, and by using these features in a multi-task learning framework through ensemble strategies. We compare our proposed approach to a baseline model with a set of 2000 skin lesions from the ISIC 2017 challenge dataset. The baseline model only predicts a binary label from the skin lesion image, while our multi-task model also predicts one of the following features: asymmetry of the lesion, border irregularity and color. We show that multi-task models with individual crowdsourced features have limited effect on the model, but when combined in an ensembles, leads to improved generalisation. The area under the receiver operating characteristic curve is 0.794 for the baseline model and 0.811 and 0.808 for multi-task ensembles respectively. Finally, we discuss the findings, identify some limitations and recommend directions for further research. The code of the models is available at https://github.com/raumannsr/hints_crowd.