Commons:WMF support for Commons/Upload Wizard Improvements/Logo detection

As part of our work to improve the current user experience with UploadWizard, we defined a tool to automatically detect logos when uploaded on Commons through UploadWizard, in order to facilitate their evaluation by the community. A need for machine detection tools was raised in several discussions and user interviews we had in the past with the community, and logos are the second reason for media deletion.

Given as input an image file, we detect whether it's a logo by training an EfficientNetV2 classifier to predict logo and out-of-domain probability scores. The experiment overall showed promising results for eventual integration in the ecosystem.

We report below the model that performed best against a test dataset, and observe that it's accurate enough to reliably fulfill the task.

Evaluation

• source: available Commons images
• # images: 47,976 - half belonging to Category:Logos, half random
• accuracy: 96.9
• AUC precision/recall: 98.8
• AUC ROC: 99
• loss: 10.2
• best training epoch: 8

Metrics

• accuracy
• area under the curve (AUC), computed separately for each class and then averaged across classes, see also en:Receiver operating characteristic#ROC curves beyond binary classification
  • AUC precision/recall
  • AUC ROC
• model's loss function, i.e., categorical cross-entropy

How likely is this a logo?

The gallery below displays 50 images and their logo probability score. Images are randomly sampled from the test dataset described above.

• 
  99.82 %
• 
  99.86 %
• 
  99.70 %
• 
  98.60 %
• 
  99.92 %
• 
  99.87 %
• 99.09 %
  99.09 %
• 
  0.32 %
• 
  0.22 %
• 
  2.53 %
• 
  0.99 %
• 
  0.77 %
• 
  0.32 %
• 
  0.45 %
• 
  1.71 %
• 
  0.04 %
• 
  0.51 %
• 
  95.43 %
• 
  99.98 %
• 
  0.27 %
• 
  1.38 %
• 
  64.14 %
• 
  0.11 %
• 
  3.97 %
• 
  0.28 %
• 
  99.77 %
• 
  0.09 %
• 
  99.99 %
• 10.42 %
  10.42 %
• 
  99.79 %
• 
  99.47 %
• 
  0.24 %
• 
  1.05 %
• 
  99.86 %
• 
  0.54 %
• 
  0.92 %
• 99.92 %
  99.92 %
• 
  0.62 %
• 
  0.53 %
• 
  99.92 %
• 
  99.99 %
• 
  0.23 %
• 
  99.93 %
• 
  98.68 %
• 
  99.97 %
• 
  99.72 %
• 
  0.06 %
• 
  83.41 %
• 99.91 %
  99.91 %
• 
  99.38 %