The Melanoma Classification Benchmark

We provide the first open access melanoma classification benchmark for both non-dermoscopic (MClass-ND) and dermoscopic (MClass-D) images to date. The benchmark was created with lesions of white-skinned individuals from Western populations (derived from the ISIC (dermoscopic) / Med-NODE (clinical) databases). Binary artificial intelligence algorithms trained for this target group can now be compared to 157 dermatologists for dermoscopic images and to 145 dermatologists for non-dermoscopic images of melanoma in terms of sensitivity, specificity and the receiver operating characteristic. MClass allows comparability between algorithms of different publications and provides a reference standard for binary melanoma classification of white-skinned Western populations.

The Melanoma Classification Benchmark (MClass) test-sets, individual results per dermatologist per image, file names and ground truth per image may be freely downloaded here:

100 dermoscopic images; 80 melanomas and 20 nevi (.zip): MClass-D

Results per image and per individual dermatologist (Excel-sheet): Download

File names and underlying ground truth of diagnosis (Excel-sheet): Download

100 non-dermoscopic images; 80 melanomas and 20 nevi (.zip): MClass-ND

Results per image and per individual dermatologist (Excel-sheet): Download

File names and underlying ground truth of diagnosis (Excel-sheet): Download

The article describing the performance of dermatologists on both test-sets must be cited: 

Titus J. Brinker, Achim Hekler, Axel Hauschild, Carola Berking, Bastian Schilling, Alexander H. Enk, Sebastian Haferkamp, Ante Karoglan, Christof von Kalle, Michael Weichenthal, Elke Sattler, Dirk Schadendorf, Maria R. Gaiser, Joachim Klode, Jochen S. Utikal; Comparing artificial intelligence algorithms to 157 German dermatologists: the melanoma classification benchmark; European Journal of Cancer, Volume 111, 2019, Pages 30-37, ISSN 0959-8049, https://doi.org/10.1016/j.ejca.2018.12.016.