Commons:Structured data/Computer-aided tagging

This page is presently inactive and retained primarily for historical interest. This feature is disabled and will be removed in the future.

Computer-aided tagging is new technology. Sometimes suggested tags will be wrong or inappropriate. This is expected behavior.

The computer-aided tagging tool is a feature in development by the Structured Data on Commons team to assist community members in identifying and labeling depicts statements for Commons files. There are tens of millions of carefully curated files on Commons, but the structured data tool is new. With this feature, existing files can have their contents easily, quickly, and – if used with care – accurately described. To contribute, editors won’t need to know how Wikidata works or speak a particular language. This new feature prompts users with suggestions for "tags", using a computer vision model, for human review. Commons users will be able to visit a Special page on Commons and see suggested depicts tags, which can be selected to be confirmed or ignored. Tags will never be automatically added without human involvement.

Computer-aided tagging helps populate files with structured data, in turn these files can be found using general search terms in Special:MediaSearch in a manner that was previously not possible. This helps users easily find media that otherwise isn't easy to come across using the old search, which often relies on specific information in file descriptions or category placement to find files. If specific information is lacking, it can be hard-to-impossible to find a lot of media on Commons through the standard search. For example, Peter_iredale_sunset_edited1.jpg shows up in a search for "beach" using Special:MediaSearch thanks to the "beach" depicts statement added by computer-aided tagging; it does not show up at all in a search for "beach" using regular search.

Computer-aided tagging is a stand-alone MediaWiki extension and is not a core part of Commons itself, and ties into Commons using Special:SuggestedTags. On the back-end, the tool will use Google Cloud Vision for depicts suggestions. Wikimedia already uses the Google Cloud Vision service in Wikisource OCR, and this will work similarly. This tool is opt-in for registered, auto-confirmed users. It is not on by default for any user group, and is unavailable to new and unregistered users.

Updates on CAT/SuggestedTags usage, September 2020

To date (Updated on 14 February 2022):

1. 5,809 total users have made edits via the Computer-Aided Tagging tool

   • 962 of these users did so via mobile web

2. 341,957 total files have had edits made via Computer-Aided Tagging

   • 41,563 of those files have Computer-Aided Tagging edit on mobile web

3. 72% of files with CAT edits had those edits done by the same user who uploaded the file
4. Approximately 10,000 files edited by CAT so far were purely manual edits
5. We’re averaging about 20 new users a week currently

Charts for this data are updated every Monday on the CAT usage report analytics page

CAT specificity

We’re working on possible techniques for improving the tool’s ability to accurately identify specific elements of photos, but it’s important to keep in mind that the Google Vision algorithm already does fairly well in many topic spaces.

Upcoming tweaks to the queue for general images

Although most usage of the Computer Aided Tagging system comes from users editing their own uploads (72%), there is a separate queue for “popular” images. Based on recent feedback from the Commons community, we’re exploring ways to prioritize this queue differently. Particularly, we’re considering a system that would focus more on files that do not have curated categories yet.

Google Cloud Vision

All information that passes through Google Cloud Vision will also be public. Dumps will be available of completely anonymous data that lists the Commons File, its suggested tags, and which tags were accepted. Google Cloud Vision is completely isolated from Wikimedia Commons, the feature is separate from the core Commons experience.

Although there are open source computer vision platforms available to start from, any such package would require resources or specialized expertise to provide an industry-standard experience with computer vision that the Wikimedia Foundation is unable to itself provide at this time. The team recognizes that Google Cloud Vision is not open source software. There will not be any non-free or proprietary code written by the Foundation for this project; all contributions will remain open source.^{[clarification needed]} Google will not have access to any private, non-public, personal information, there will be no direct communication between users and Google's service.

Architecture and workflow

Design of information flow in computer-assisted image tagging. The "machine vision" provider on the far right requests and sends potential tags for images; there is no personal information exchanged and the provider is isolated from the rest of the system and Commons.

Registered, auto-confirmed users will be able to opt-in through their preferences or uploading files. After some time has passed, the user will be contacted through their notifications that their uploads are ready for tagging at Special:SuggestedTags. Users who have opted-in can visit Special:SuggestedTags at any time to view files ready for tag processing. Anonymous users, new users, and users who have not opted-in will not be able to access Special:SuggestedTags.

The concepts that are available for tagging are ones that translate from Google Knowledge Graph IDs to Wikidata IDs. At 2.1 million triplets, the list is too long to catalog here, but is available for download as freebase-wikidata mappings.

Development stage

All originally planned features for the tool are now deployed and available for use. The development team will continue with tweaks, and possible new features in the future.

• Usability study
• Tracking task for building the tool

Implementation and usage notes

• No personal information is sent to the computer vision platform provider. At launch, this new feature will only use the Google Cloud Vision system, which will be accessed via a middleware layer that hides all user data. Commons images are sent to Google servers from Wikimedia Foundation servers. There will be no direct communication between the user and external services. No personal information (IP, username, etc.) is sent to Google servers. The middleware that contacts Google servers is a Wikimedia project and is open source. No part of Google's service or code will be part of Wikimedia infrastructure.
• Suggestions from the computer vision will not be added to an image file’s structured data until a user has verified them: This service is provided as a means to augment human activity, not replace it. All suggestions from the computer vision service are stored in a separate, specialized database. Suggestions are not saved as structured data on the Commons file until a human user confirms them.
• Users can opt in to receive notifications alerting them that their recent uploads have suggested tags. In the last step of the UploadWizard upload process, users have an option to enable notifications that will inform them when recently uploaded files have passed the waiting period and have tags available for confirmation. This option can also be found in User Preferences under Notifications.
• User contributions that confirm suggested depicts tags are licensed as CC0. This data is equivalent to adding Wikidata to an image, and as such must be contributed under the same CC0 license that Wikidata uses. Clear license notices will inform users that all contributions made via the computer vision tool will be licensed under CC0.
• Analysis of images on Commons: The feature will analyze only images, and provide suggested “depicts” tags based on the content of those images.
• Certain types of images will be excluded: Some types of imagery on Commons are not well-suited for this type of system. Small images (less than 100px wide), artworks (identified via the Artwork template), book page scans, and other files will not be included.
• Newly uploaded files will be analyzed, but not during upload: Commons users continuously monitor new files for vandalism, copyright violations, and relevance to the project. Files that don’t meet the criteria are marked for deletion. The new computer vision feature will only analyze new files after a waiting period has passed, and will not analyze files marked for deletion.
• All tag confirmations show up as regular structured data edits with an edit summary tag that identifies their origin from the computer vision tool: This enables all the usual curation and moderation workflows so changes can be improved, edited, or reverted. It also helps us measure the revert rate and ensure that edits made using CAT are not more frequently reverted than the average edit.
• Problematic tags can be blocked from being suggested: There is a blocklist of tags that will not be suggested by the tagging tool. The official blocklist currently exists within the configuration file for Commons and cannot be edited directly by the community, but suggestions can be made on the blocklist talk page.

Userbox

You can use this userbox in your user page.

{{User Computer-aided tagging}} links talk view

This user uses Computer-aided tagging tool for tagging images.

This was a failed project

As early as 13 February 2020, experienced Commons users were complaining that the bulk of tags added using this tool were, as one put it, "way too vague, irrelevant or even detrimental". After numerous such complaints over the next several years, on 16 June 2023 the Sr. Director in the WMF Product department acknowledged that "We understand that the accuracy and utility of the tags generated by this tool have been called into question." After some study, on 14 September 2023 they announced, "we will be deactivating the tool on September 20, 2023, after completing the necessary code changes."