User:Rhododendrites (WMF)/Suggested Edits

Hi everyone. I've been doing some research on the Suggested Edits feature of the Wikipedia Android app, focused on its impact on Commons. I'd like to share some findings and open discussion about it.

Background

Suggested Edits (see also the MediaWiki documentation) is a feature of the Wikipedia (not Commons) Android app which encourages new users to make small but meaningful contributions with minimal training. On Commons, there are two tasks, both of which contribute structured data: Depicts statements (sometimes called "tags") and captions.

In September 2020, I noticed an unhelpful edit to one of my uploads made through Suggested Edits. I did some research, doing a cursory evaluation of recent edits in the log. I opened a thread on the administrators' noticeboard expressing my concerns, followed by a more substantial thread on the village pump. Sometime later, a few people from the Wikimedia Foundation reached out to talk more about it. We had a few conversations, both on- and off-wiki. Eventually they offered to bring me on as a short-term contractor to continue those conversations and research in a more structured way. So I have a financial COI here, but my goals/opinions/intentions have not changed. I still think -- as I did before -- that Suggested Edits has a lot of potential for both positive contributions and new user engagement, but it needs some work. My primary goal here is to improve the quality of contributions through that feature.

Data

For information about the sample and rubrics used, see User:Rhododendrites (WMF)/Suggested Edits/data.

Findings

Captions

• Overall quality was a mean of 3.715/5, with a median of 4, indicating a reasonably high rate of positive contributions.
• There were very few instances of too much detail, inappropriate content, and value judgments.
• Copyright violations did not seem to be an issue. When users did copy or translate descriptions, they were plain and descriptive such that it's highly unlikely to clear the threshold of originality.

• The most common issues were complete inaccuracy (11.5%), partial inaccuracy (6%), and vagueness (22%). Edits which are unlikely to be good faith account for more than half of the complete inaccuracy (7.5%).

Depicts

• The average number of statements added was 1.9, with a median of 1. This is skewed by a small number of edits with a very large number of statements.
• Overall quality was a mean of 3.23/5 with a median of 3, indicating that the average contribution is fairly positive, albeit less so than with captions.
• There were very few instances of value judgments.
• There were fewer edits unlikely to be good faith (3.5%) as compared to captions.

• The most common issues were missing information that should be easily available (51%), redundancy (15.5%), partial inaccuracy (14%), complete inaccuracy (13%, of which 3.5% were unlikely good faith), incorrect links (9.5%), and wrong statement type (9%).

Observations

• Overall quality of these samples was better than the quality of the much smaller, informal spot check I did last year.
• Overall quality is better for captions than depicts. Whereas depicts reduces the amount of bad faith edits by restricting users to searching for Wikidata items rather than presenting an open text box, this has downsides where (a) appropriate Wikidata items do not exist or are not accessible; (b) incorrect but similarly labeled items appear in search results; (c) users do not know what to search for.
• Many of the issues have to do with inaccuracy, vagueness, and missing information that should be easily available. This could be helped by making existing categorization visible and improved visibility of descriptions and filenames.
• Redundancy is not a very large issue, but clarity is needed regarding the plans and intentions for structured data on Commons and the community's guidelines and help pages on the topic. There are areas of mismatch between the two.
• One of the major limitations is likely outside the scope of the app/feature: missing items, aliases, and labels on Wikidata.

Recommendations

Primary recommendations

The two primary suggestions for the app development team:

• Make categories visible (but not hidden categories), especially for depicts statements.
• Add clearer instructions for both tasks.
  • "Tags" (depicts statements) are not clearly defined. There is a screen which appears on first use that explains that tags "help make images easier to search for" but doesn't say what they are. We are looking for something specific rather than the very general sense of tag people are used to. Once past that screen, users see only the image, description, and the option to add a tag. The only instructions available are by tapping the "i" symbol, which brings the user outside of the app to the web page mw:Wikimedia Apps/Suggested edits, which is not new user friendly. There are arrows to go forward and backwards, but it's unclear when they should be used. Some examples may help before getting started, too.
    • Sample instructions, which should appear both at first use and should be easily accessible at any point in the future without exiting the app: "Add tags for people, places, and things you see in the picture. Look at the image description and categories for clues. Add tags for all main subjects of the image, being as specific as possible. If you're not sure a tag is correct or you don't see a tag that fits, skip it. No tag is better than an incorrect tag." (Note that these instructions assume categories are made visible and that our guidance for depicts statements does not change). Even something which simply asks "What do you see in the image? If you can't tell, please skip the image" would help.
  • Captions are more self-evident, as what we expect is fairly consistent with the kinds of captions people are used to. There should nonetheless be clearer instructions telling users to skip if they aren't sure, and to be specific but concise. Some examples may help before getting started.
  • It is worth noting that with a recent app release, a new form of Suggested Edits has been included: adding images to articles. The manner of onboarding here is significantly more engaging. Not only is there a welcome screen introducing the task, but pop-ups which point to parts of the screen, instructing the user how to use the tool. Within this context, there are a variety of ways we could onboard the user for the two Commons tasks. For example, an introductory screen which explains the general purpose of tags, a pop-up by an image asking what a viewer sees in the image, pop-ups pointing to the description and categories for clues, and -- importantly -- a sample drop-down search results with multiple pop-ups showing a false positive, using other possible names, and telling the user to skip it if they're not sure. The new tool also includes screens after each selection asking about the user's experience or why they chose what they did. This seems ripe for questions based on these findings.

Other recommendations

Captions:

• It is very hard to caption without a description, but descriptions are not available in every language. It may help to only present captions when users have selected a language that matches one of the descriptions. I have only selected English, but several of the images presented to me had a non-English description. Without category visibility, this makes writing a caption difficult.

Depicts:

• It is not clear that the description field can scroll when it extends beyond three lines. Some sort of down arrow in the corner of that area may help.
• When you start to search for an item, the drop-down results can completely block the description and image, which is a problem if you need to cross-reference either while searching. Perhaps the drop-down could be positioned just below the description instead.
• Allowing use of the priority ("prominent") modifier may help not just the quality of data but also to contextualize the exercise (to get users thinking about what they can see).
• It would help to indicate that the arrow can mean "skip if you're not sure".

General:

• There is some confusion among the Commons community about structured data: its purpose, long-term plans, what is and is not possible, ideals for specific types of structured data, how the data relates to existing Commons mechanisms, etc. Most relevant for this exercise is "what constitutes a good depicts statement?" These questions would be helpful to address by clear documentation on Commons (written for a non-technical audience, with a clear venue to ask questions) and a conversation between stakeholders in the future.

Discussion

There are at least two distinct audiences for these findings: the Wikimedia Foundation teams involved with the development of the Suggested Edits feature, and the Wikimedia Commons community.

I've presented this to Foundation staff separately, and would like to use the talk page to focus here on any community questions, comments, concerns, or suggestions about these features.