Commons talk:Structured data/Archive 2020

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search

Postcards

File depicting postcard which depicts another postcard

Hello I want add structured data to images of the c:Category:Postcards. How do I add the property "Postcard"?

What is the better choise? The goal is to search for any thematic on postcards (only). IMHO the P31 is the better choise, because sometime also a postcard is depicts on a postcard. For example this image: Postcard at a postcard. -- sk (talk) 16:42, 29 January 2020 (UTC)

We do not use Instance of since we would have "instance of" "file", which "depicts" "postcard". --Jarekt (talk) 21:09, 29 January 2020 (UTC)
@Jarekt: I don't understand your answer. Can you make an example at one postcard? How should we set the fact: "This is a postcard!" ? Maybe you can set it at this File:Dortmund Postkarte 001.jpg as an example for me. Thanks. -- sk (talk) 22:18, 29 January 2020 (UTC)
So do we say that the file depicts the postcard, and then the postcard, in turn, depicts its subject matter? - Jmabel ! talk 00:52, 30 January 2020 (UTC)
Photograph of the painting depicting Philippe Lenoir
The subject of the item is the file or image which depicts the postcard and the postcard depicts the girl and the postman. in this case I would save both lavels of depicts in depicts (P180). Alternatively if the postcard is notable enough we might create wikidata item for it and store postcard metadata there. We can see that in File:Philippe Lenoir by Horace Vernet.jpg is a photograph by User:Rama of the painting Philippe Lenoir (Q29837745). SDC should have digital representation of (P6243)=Philippe Lenoir (Q29837745) and afterwards the properties of the painting should go to Wikidata's d:Q29837745 page and properties of the photographs (author, photo date, photo license, etc.) should go SDC. The painting page d:Q29837745 already have a lot of depict statements and I would not duplicate them on SDC. I would also treat P6243 as one of the photo depicts and not replicate it. --Jarekt (talk) 03:33, 30 January 2020 (UTC)

Ok, if I understand you right, then I use digital representation of (P6243)=postcard (Q192425) at all postcards in Commons. At the moment this are more estimated 220.000. If in the postcard a postcard I use depicts (P180)=postcard (Q192425). I am right?

Are this examples right? Maybe I misunderstand something. -- sk (talk) 05:16, 30 January 2020 (UTC)

At the moment we have only one postcard if I search with "haswbstatement:P6243=Q192425". The result is File:Bird's-eye_View_of_Brattleboro,_VT.jpg. There I see the digital representation of (P6243) and also an Qualifier manufacturer (P176)=Detroit Publishing Company (Q3391589). Is this the right way? --sk (talk) 05:25, 30 January 2020 (UTC)
@Sadads: Hello Sadads, please read the text. I ping you, because you edit the sturcture data of the postcard File:Bird's-eye_View_of_Brattleboro,_VT.jpg. Maybe you can help me. Thanks. -- sk (talk) 18:03, 31 January 2020 (UTC)
At the very least we need to model that the image isn't just "depicting" a postcard you never see the postcard as a postcard, rather because of the way it has been digitized, the image best "depicts" a number of other things, but the file is a "digital representation of" that original object. This is an important distinction in cultural heritage metadata, and I think folks like @Multichill: and others may be able to better describe it. Sadads (talk) 00:27, 3 February 2020 (UTC)

Feedback request: geo-coordinate input designs

Tracked in Phabricator
Task T227116

The team is making progress wrapping up geo-coordinates support, and would like your feedback (if you have any) on the proposed designs. Please check out the Phabricator task for mockups, rationale, and other input. Thanks! Keegan (WMF) (talk) 21:24, 8 January 2020 (UTC)

Keegan (WMF), I will look through the phabricator task, but you might also cross post it at Commons talk:Geocoding, Template talk:Location and Commons_talk:Structured_data/Modeling/Location, as all those places is where most related discussions happen. Also a quick impression: seems quite promising, however the name of the property should be coordinates of the point of view (P1259) instead of "Geolocation". I would be OK with renaming the property to any of the aliases, but we do need to clarify that is camera location and not the object location that is expected. Also any GUI for adding geocoordinates should also capture camera's heading, the way current GUI does. --Jarekt (talk) 01:57, 9 January 2020 (UTC)
Thanks for the suggestions, I'll wander over there to post later today. Keegan (WMF) (talk) 17:19, 10 January 2020 (UTC)
Many of us don't think in arcseconds, so being able to see (and set?) the precision in terms of metres would be useful. --bjh21 (talk) 11:50, 9 January 2020 (UTC)

Just a quick feedback:

  • We must be able to paste the coordinates in a single field, the same way we do it on Wikidata. Having to fill in separately latitude and longitude is not at all user friendly when you copy-paste coordinates.
  • Otherwise, the selector is a nice improvement. I hope it will be ported to WD.
  • Will the map be displayed in read mode? Phab description says so, but it's not clear on the mockups. Seeing the map in read mode on WD is very useful to spot errors.

Ayack (talk) 18:15, 9 January 2020 (UTC)

  • "We must be able to paste the coordinates in a single field, the same way we do it on Wikidata. Having to fill in separately latitude and longitude is not at all user friendly when you copy-paste coordinates."
    Keegan (WMF) this is a must.
    Another thing, longitude and latitude is not a daily base information, depending on the location, will be hard to remember with this N/S is longitude or latitude, - or +. i.e.: 41° 08′ 40.65″ N, 8° 36′ 56.99″ W
    And, when we make a mistake as it is right now, we need to redo all of it, if we forgot one digit, puff, start over.
    The precision should be automatic or by a qualifier, because the great majority of volunteers will not know how precise that info is, and may lead to an incorrect information.
    The sum all that and this is the last user friendly experience to include one data.
    About the layout:
    The way that you put at phabricator, is kind hard to image at a wiki page, but, looks okay to have an option to expand and see the map, but you also could include a link to geohack allowing other ways to see a map, as we do right now.
    -- Rodrigo Tetsuo Argenton m 17:57, 10 January 2020 (UTC)

Where should we leave the feedback. There or here? I am leaving it hear. Is that just a display function or does it allow to geocode images? If so, I would propose:

  • to provide more layers/maps such as OSM or tourist maps of mapy.cz, which provides important information, which helps to identify photo site?
  • does that tag photographer possition geo or object position?
  • would it be possible to set the direction from the coordinates of a photographer, to indicate the direction of making a shot?

--Juandev (talk) 15:13, 9 February 2020 (UTC)

WMF done?

So WMF is done with this project? Any continuation project? It seems to me that the infrastructured build is used by incredibly small amount of users. --Juandev (talk) 15:04, 9 February 2020 (UTC)

My understanding is that funding stopped on 31 December, and nothing else would be done.--Ymblanter (talk) 18:27, 9 February 2020 (UTC)
Pinging Sandra just in case: @Spinster: --Ymblanter (talk) 18:38, 9 February 2020 (UTC)
I see a lot of assumptions here. Keegan can probably provide an update. Multichill (talk) 18:58, 9 February 2020 (UTC)
Thanks for the ping.
@Ymblanter: while it's true that the final grant funding ended on 31 December, the team was never planning on (then or now) simply stopping work and walking away the next day. In fact, the development team's name was changed this past year from the Multimedia team to the Structured Data team, as the WMF has made an internal committment to continue this work.
I've updated the development timeline. The team is still working in a few main areas around version one of SDC, namely remaining non-text properties support, constraints, and some remaining Lua bits. The team will move on later to do more work on search - what this work is exactly remains to be scoped out so I'm unsure at this time, other than it is a priority. There is also potentially another round of grant funding to work on more structured data, but I won't know more about that until early March (and for sure I'll be posting about it).
For sure, there's more work remaining, and the team is still here. Keegan (WMF) (talk) 19:18, 10 February 2020 (UTC)
Thanks, it is good to hear--Ymblanter (talk) 21:27, 10 February 2020 (UTC)

Unable to add structured data to files

Hi! I am currently unable to add structured data to files. Instead, I get the following error message: Invalid value "" for integer parameter "baserevid". Steps to reproduce:

  1. Select "Structured data" tab on a file page.
  2. Add any Wikidata item to the "Items portrayed in this file / depicts" field.
  3. Click "Publish changes".

I first noticed this problem on 2020-02-12 at ca. 22:00 UTC. Best regards, ––Apalsola t • c 10:45, 13 February 2020 (UTC)

Now this seems to work again. ––Apalsola t • c 17:30, 13 February 2020 (UTC)

Tool

I've found the tool Special:SuggestedTags. A nice tool, but IMO too much overhead - similar to overcats. Is the tool really useful? It does not respect Commons:Depicts#Generic_items. --XRay talk 17:33, 28 February 2020 (UTC)

@XRay: Please see discussion at Commons:Village pump#Misplaced invitation to "tag" images (especially Commons:Village pump#Why this matters) and Commons:Village pump#Depicts. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 23:01, 29 February 2020 (UTC)
Thank you. A lot to read. --XRay talk 06:08, 1 March 2020 (UTC)

Modeling

Could you link modeling discussions from this page? I think its a task, which I still ongoing. --Juandev (talk) 15:15, 9 February 2020 (UTC)

I mean from Get involved page. I am sorry, I thought I am in the Get involved talk page. --Juandev (talk) 15:16, 9 February 2020 (UTC)
This is a wiki. Knock yourself out. Multichill (talk) 18:58, 9 February 2020 (UTC)
If this pile page is maintaind by someone, I am just proposing it.--Juandev (talk) 07:14, 5 March 2020 (UTC)

Discussion about the place of structured data within Wikimedia Commons

Please see this discussion, note that I don't agree with its content, but I wanted to bring it to the attention of the people who work with structured data the most. --Donald Trung 『徵國單』 (No Fake News 💬) (WikiProject Numismatics 💴) (Articles 📚) 11:14, 11 February 2020 (UTC)

Thanks for the notice. This seems to conflate the Data: namespace and tabular data with Structured Data on Commons. I've added some links about the Data namespace. Keegan (WMF) (talk) 18:42, 11 February 2020 (UTC)
Yes, Data namespace is not related to Structured data, it is a way to store some file types not as blobs but in more useful format. --Jarekt (talk) 05:16, 18 February 2020 (UTC)
Linked discussion now in the archive. --Juandev (talk) 07:24, 5 March 2020 (UTC)

6 million files or more than 10% of all files have statements now

The bots have been active and we have more and more files with statements. We just passed the 10% milestone (6M out of 59,5M). Multichill (talk) 21:50, 28 February 2020 (UTC)

Yes, several of my files got redundant information in this enterprise as well, cluttering up my watchlist with useless stuff. Everything was already fine on the pages, no information was added, it's just a nonsensical occupational therapy for underemployed bots. I couldn't care less but about lazy programmers, who are incapable to program their spider bots in the right way. Licence: Already there, perfectly fine, no need for redundant doubling. Author: Already there at the right place. Date and time: already there at the right place. It's just cluttering of watchlist with useless redundancies. Grüße vom Sänger ♫ (talk) 08:09, 29 February 2020 (UTC)
Great, a clueless complainer. Maybe you should have invested some time in reading what this project is all about.
Special:Preferences#mw-prefsection-watchlist and tick the bot "Hide bot edits from the watchlist". Multichill (talk) 10:14, 29 February 2020 (UTC)
It obviously is about adding redundant clutter to existing files. I thought it is about tagging images with useful new stuff. Grüße vom Sänger ♫ (talk) 10:21, 29 February 2020 (UTC)
WOW, it look optimistic. But still for those, who does not know the code, there are not enough "good" tools to tag. Nonexistand usefull software is not just blocking people from adding structured data, but also from adding any kind of metada at all. Here I point out that still in 2020, when we know there are batch photographers we still work on model filling all metada separetly for each image.--Juandev (talk) 07:39, 5 March 2020 (UTC)

File:Cottages at Garneddwen - geograph.org.uk - 750937.jpg

Could someone let me know what on earth we are supposed to do with this? There is no obvious link to the alleged "structured data" linked in the template. In any case, it should not be assumed that all images must have camera location and location. Sometimes both are appropriate, sometimes neither. I don't see why we should have to go elsewhere to fix mistakes. I had an idea that this would be a disaster in practical terms, and I am beginning to tire of obstructions to well-known ways of working that suddenly get destroyed. Rodhullandemu (talk) 14:10, 8 March 2020 (UTC)

Der Fehler wurde mit diesem Edit von Multichill eingefügt, als alle Daten aus der Datei gelöscht und durch irgendeine dahergelaufene Vorlage ersetzt wurden. The error was put in there with the mentioned edit by Multichill, that deleted all data from the file and introduced some random template. Grüße vom Sänger ♫ (talk) 14:29, 8 March 2020 (UTC)
Thanks, if an "experiment" fails, it should be reverted until it works. Have so reverted to a usable version. Rodhullandemu (talk) 14:41, 8 March 2020 (UTC)
Can you revert the edits that actually break things, if any? To not remove valid data. Jura1 (talk) 14:44, 8 March 2020 (UTC)
If there is valid data elsewhere, I suggest it should be imported properly. Rodhullandemu (talk) 14:59, 8 March 2020 (UTC)
I uploaded over 1,8 million Geograph files. One of these is used for testing if a template functions correctly. I hope you're proud of yourself now. Multichill (talk) 18:33, 8 March 2020 (UTC)
Luck of the draw, but it is the 13th chime of the clock that invalidates all the others. In any event, what is the point of moving the ability to edit image data away from the image page and into a completely different project? If the idea is to deter Commons editors, that is going to happen. Rodhullandemu (talk) 18:48, 8 March 2020 (UTC)

Bug in adding structured data?

Tracked in Phabricator
Task T247118

I just tried to add structured data to the photograph I uploaded, File:Mouth of the Heart River 1.jpg. There are two items which need to be there: d:Q1743985 (Peace River, a town in Alberta) and d:Q2220 (Peace River, a river). If I choose from the dropdown menu, whatever I choose I get Q2220 selected (right now, I have two times Q2220 added to the structured data). Moreover, if I just insert "Q1743985" to the prompt and choose the only item which shows up, I still get Q2220 added. I have not found a workaround so far.--Ymblanter (talk) 14:38, 3 March 2020 (UTC)

The same here: File:Hay River above Alexandra Falls 1.jpg, I do not manage to get the Hay River (river) and get instead Hay River (town) which is a wrong label. Apparently, if there are omonymous Wikidata entries, the interface does not process them properly.--Ymblanter (talk) 11:57, 4 March 2020 (UTC)
Ok, nobody cares. As of now, I stop adding any structured data to my uploads.--Ymblanter (talk) 07:28, 5 March 2020 (UTC)
@Ymblanter: I've tried and it works to me. So maybe some temporary bug or something in your browser javascript? --Juandev (talk) 07:44, 5 March 2020 (UTC)
Thanks, this is really strange but at least encouraging. I will try to experiment with new uploads and see whather the old ones can be handled properly.--Ymblanter (talk) 07:47, 5 March 2020 (UTC)
Now, I have reviewed your edits for the first image and what I can see in the history it looked it worked properly. So when you add in Q1743985 it could be seen in the preview and then when you add Q2220 twice in the row it could be seen again. So I would say and unknown temporary bug.--Juandev (talk) 07:49, 5 March 2020 (UTC)
I just tried again, and it does not work with File:Hay River above Alexandra Falls 1.jpg. This probably means something is on my side. I will try again in the evening from a different computer.--Ymblanter (talk) 08:03, 5 March 2020 (UTC)
No, it did not help. For example, File:Hay River below Alexandra Falls 2.jpg from the other computer has a wrong WD item.--Ymblanter (talk) 19:17, 5 March 2020 (UTC)
I just had the same problem with "Columbus" on this image. I had to add by QNumber to get it to add the right one. Something is off with the widget. —TheDJ (talk • contribs) 08:30, 6 March 2020 (UTC)
Great, at least I am not alone.--Ymblanter (talk) 09:01, 6 March 2020 (UTC)
@Ymblanter: I think I ran into that here on 24 February 2020‎. I was busy doing something else so I didn't look deeper into it. Did you happen to file a bug in phabricator? Multichill (talk) 17:41, 6 March 2020 (UTC)
No, I have not, I wanted to make sure this is not on my side. I will try to open it now (never done this before).--Ymblanter (talk) 18:26, 6 March 2020 (UTC)
Now filed.--Ymblanter (talk) 19:06, 6 March 2020 (UTC)
I should have noticed this discussion… 1234qwer1234qwer4 (talk) 19:37, 9 March 2020 (UTC)

Wikidata & Commons event in Ulm, Germany

Hello all,

I wanted to let you know about the Wikidata Wochenende, a week-end dedicated to working on Wikidata-related projects that will take place in Ulm, Germany, on June 12-14. We are especially looking forward to welcome Commons editors, people working on Wikidata-powered templates or Structured Data. The event, held in German, will be the occasion for you to meet other people working with Wikidata, learning new skills and sharing yours.

If you're interested, you can read more about the details and the funding possibilities here. Please help me sharing the information to people who could be interested. Cheers, Lea Lacroix (WMDE) (talk) 08:49, 11 March 2020 (UTC)

@Lea Lacroix (WMDE): I assume this is canceled now? Multichill (talk) 14:30, 15 March 2020 (UTC)
The page says "Die Veranstaltung findet unter Vorbehalt der Gesundheitslage statt. Wir beobachten die Situation. Im Falle einer Verschiebung des Workshops informieren wir hier.", so probably not yet, but chances are indeed big.--Ymblanter (talk) 21:10, 15 March 2020 (UTC)
Yeah, basically we started announcing the event with the hope that in June, things get better. Of course, we're monitoring the situation carefully, and will announce it in the (quite likely :( ) case that we have to postpone it. Lea Lacroix (WMDE) (talk) 07:46, 16 March 2020 (UTC)

Structured data about Tabular data?

Tracked in Phabricator
Task T235332

As far as I can see, the Structured data efforts have a strong focus on things in the File namespace, but is there also some tiny corner concerned with COM:Tabular Data? I think I had seen this discussed a while back but could not find anything relevant right now, so posting anew. -- Daniel Mietchen (talk) 02:37, 17 March 2020 (UTC)

@Daniel Mietchen: I don’t see any reason to store anything about tabular data pages in Wikibase. Traditional file description pages use wikitext, which is easy to use for humans, but hard to process for machines. Tabular data is structured in itself, so the page content itself can be easily read by machines, there’s no need to store anything separately in Wikibase. —Tacsipacsi (talk) 00:30, 18 March 2020 (UTC)
There have been some ideas about making it possible in WDQS to query those datasets. I think that would be useful, but more likely to happen from the WDQS side than the Common Structured Data side. —TheDJ (talk • contribs) 09:33, 18 March 2020 (UTC)
@TheDJ: Thanks, that makes sense. But probably we should have a working SPARQL endpoint for SDC before working on it… —Tacsipacsi (talk) 00:53, 19 March 2020 (UTC)

"Depicts tradition"

https://commons.wikimedia.org/w/index.php?title=File:Lunar_New_Year_in_Seattle_2020_-_Oolleemm_troupe,_Korean_folk_dance_22.jpg&diff=407971934&oldid=394068302: didn't we have an understanding that until further notice, depicts statements are supposed to be confined to the relatively concrete and clear? Depicting tradition (Q82821) seems to me to be nothing of the sort. - Jmabel ! talk 00:34, 30 March 2020 (UTC)

Its kindof bizarre: I think since the tool is suggesting the concept, people are shortcutting to that as "good enough" without figuring out the right thing. I have made some progress on culling tradition (occurrence, mode of transportation, architecture), and a number of the other vague concepts through Petscan + Quickstatements. Some of the clusters of really bad "tags" in depicts are starting to look much healthier -- but it took a fair amount of work to make sure that we are provide accurate depicts statements. Sadads (talk) 10:10, 31 March 2020 (UTC)
Right. The question is why the tool is suggesting the sort of tags we agreed not to use for now. Those should simply be off limits. - Jmabel ! talk 16:15, 31 March 2020 (UTC)
I'm working on getting the tool's blacklist posted this week, with the list comes the opportunity to add more properties to it as the community requests them (using the talk page or some other light-weight process). Keegan (WMF) (talk) 16:33, 31 March 2020 (UTC)

Using "color" for dominant color in an image?

So this is a bit of a modeling question: when we have dominant colors in an image, i.e. any of these identified by the Computer Aided Tagging tools as red, should that be stored as color (P462)? I think it would make sense to leave this facet around for dominant color, just like we would on Wikidata, or for a depicted item here. Being able to query by dominant colors would allow for some really neat things, like generating Category:Photomosaics, Sadads (talk) 10:14, 31 March 2020 (UTC)

Computer-aided tagging blacklist posted

I've published the blacklist page, with the initial included properties: Commons:Structured_data/Computer-aided_tagging/Blacklist.

Requests/suggestions can be made on the talk page using whatever kind of process the community would like. The team will patch in new additions as they come up. Keegan (WMF) (talk) 17:39, 1 April 2020 (UTC)

Bug in haswbstatement search

I have found a couple concepts that are currently tagged in files like land vehicle (Q1802779) that have become redirects on Wikidata (see https://www.wikidata.org/w/index.php?title=Q1802779&redirect=no), however, because of the lack of q number in the interface, if you try to search for "Land vehicle" it misses those items. We probably need some type of maintenance report so that these can be fixed in the future, and/or a change in the way search indexes (so it searches for all the redirects as well). Sadads (talk) 23:43, 2 April 2020 (UTC)

Thanks, I'll pass it along and get a bug report up as needed. Keegan (WMF) (talk) 21:31, 3 April 2020 (UTC)

P2701

Is P2701 going to be added to images by bots or is it considered redundant? 1234qwer1234qwer4 (talk) 18:30, 6 April 2020 (UTC)

Not on top of my list, but given that we have Category:Images by file format, we'll probably add it at some point. Probably just like the category first to the bit less used formats. Should probably done in a single edit per file with a lot of other metadata. Multichill (talk) 19:06, 6 April 2020 (UTC)

haswbstatement search via API

Hi, I checked many places .... I want to do this reuqest:

https://commons.wikimedia.org/w/index.php?sort=last_edit_desc&search=haswbstatement%3AP180%3DQ7378&title=Special:Search&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns6=1&ns12=1&ns14=1&ns100=1&ns106=1

via an API, i.e. I would like to have a json.

In wikidata I do:

curl 'https://www.wikidata.org/w/api.php?action=query&list=search&srsearch=haswbstatement:P180=Q7378&format=json'

so I would expect:

curl 'https://commons.wikimedia.org/w/api.php?action=query&list=search&srsearch=haswbstatement:P180=Q7378&format=json'

but this is not working ... I check a while and I really do not find a solution. Any ideas?

@DD063520: The default namespace of list=search is always 0, i. e. the main namespace (Gallery on Commons). This is unlike Special:Search, which has a configurable and usually more useful default. This works: https://commons.wikimedia.org/w/api.php?action=query&list=search&srsearch=haswbstatement:P180=Q7378&srnamespace=6&format=json --Lucas Werkmeister (talk) 00:43, 9 April 2020 (UTC)
@Lucas Werkmeister: Thank you!!!!!!!!! Perfect!
@Lucas Werkmeister: Sorry for bothering again .... For some specific images we would like to extract the structured data section ... any format is fine (if it is RDF even better), is there an API for that too? I couldn't find it .... — Preceding unsigned comment added by DD063520 (talk • contribs) 11:52, 10 April 2020 (UTC)
@DD063520: You can feed the page IDs from the search into either action=wbgetentities (JSON only, but allows you to get several entities per request) or Special:EntityData (any format), e. g. Special:EntityData/M15925090.ttl. (Side note: pings only work if you add a signature ~~~~ in the same edit.) --Lucas Werkmeister (talk) 13:32, 10 April 2020 (UTC)
@Lucas Werkmeister: , wonderful! : ) — Preceding unsigned comment added by DD063520 (talk • contribs) 07:41, 16 April 2020 (UTC)

P180 depicts Flores hawk-eagle

My watchlist is full of structured data updates, all of which claim to be Flores hawk-eagles. As an example, this edit on the edit summary says "Created claim: depicts (P180): railway (Q22667)", but on my watchlist it shows up as "Created claim: depicts Flores hawk-eagle (P180): railway (Q22667) Tag: Computer-Aided Tagging". What's going on? -mattbuck (Talk) 16:53, 21 April 2020 (UTC)

@Mattbuck: Someone briefly changed the English label for depicts (P180): d:Special:Diff/1161800693. --bjh21 (talk) 17:13, 21 April 2020 (UTC)
That explains it, thanks Bjh21. -mattbuck (Talk) 17:17, 21 April 2020 (UTC)

Structured data and Wiki Loves Monuments

I've been adding structured data to files uploaded as part of Wiki Loves Monuments. Over the years about 2,4 million files have been contributed to Wikimedia Commons. About 2,1M of these files now have a statement. On Commons:Wiki Loves Monuments/Structured data I documented what kind of data is added to these files. Recently my robots have been focused on adding basic relatively easy to extract data like source of file (P7482), creator (P170), copyright status (P6216), copyright license (P275), inception (P571), coordinates of the point of view (P1259) & coordinate location (P625).

I'm probably going to focus a bit more on depicts (P180) and location of creation (P1071) now (see also User:ErfgoedBot/Depicts monuments.js). I use the monuments template to get the identifier and based on the identifier I find the relevant item. So for example File:Haarlem - Nieuwe Gracht 62.JPG has {{Rijksmonument|19594}} and that gives me Nieuwe Gracht 62, Haarlem (Q17254716). This I've already done for most of the Netherlands, but I'm open for suggestions for other countries to do to. Any countries that have good coverage on Wikidata and a decent amount of photos here I should work on? For adding the location of creation (P1071) I can use the item too. I have three options:

  1. Adding the same as depicts (Nieuwe Gracht 62, Haarlem (Q17254716)). That would only be correct in this case if the photo would have been taken inside
  2. Adding the street which is in located on street (P669) (Nieuwe Gracht (Q17195901)). This happens to be correct for this photo, but often this won't be correct
  3. Adding the municipality which is in located in the administrative territorial entity (P131) (Haarlem (Q9920)). This should be correct almost always, we do loose a bit of detail.

Given the numbers (Netherlands alone is over 400.000 photos) manual review would take forever. I'm leaning towards adding the municipality because it's always correct. Another option is to add both the street and the municipality and leave it up to the user to remove one of the two. Removing is much faster than having to look up things to add, but it's still a lot of effort. Any opinions? Multichill (talk) 21:24, 25 February 2020 (UTC)

Multichill, Why location of creation (P1071) instead of regular location (P276) property? Is that related to properties of the photograph vs. properties of photographed object, ir is it something else? I would vote against adding location properties which duplicate located on street (P669) or located in the administrative territorial entity (P131) properties. I think those are more specific and clear. I can see a need for location of creation (P1071) / location (P276) if it is more specific than other location info. I wonder if there is some way of specifying item ID of the monument, the way we use digital representation of (P6243) for artworks. Depict statements are not that great for that, since if there is a cat or car in a photo than we might be depicting cat or specific car model as well. Some of those location statements should be already on Wikidata. Would we duplicate those or rely on Wikidata copy? Sorry, I seem to have more questions than answers. --Jarekt (talk) 16:22, 26 February 2020 (UTC)
@Jarekt: location of creation (P1071) is the right property to indicate where a work was made. location (P276) indicates where a work is at some point in time. See File:De Moulin Rouge in Parijs bij avond, Bestanddeelnr 254-5695.jpg for an example where I used the different location properties.
I'm under the impression that multiple concepts get mixed up in the questions. Please have a look at Commons talk:Structured data/Modeling/Location#Types of locations. I hope that makes it clearer. Multichill (talk) 17:27, 27 February 2020 (UTC)

Structured data for WLM in the UK

Based on your criteria (good coverage on Wikidata and a decent amount of photos) you could try working on the UK photos - though bear in mind that the Wikidata entries come from four separate official listing databases (England, Scotland, Wales, Northern Ireland), which use different listed building/scheduled monument IDs. MichaelMaggs (talk) 19:14, 26 February 2020 (UTC)
@MichaelMaggs: As long as each has a unique property / template pair, that shouldn't be a problem. For each source, can you provide:
  • property - Property id on Wikidata
  • template - The template here on Commons
  • designation - The heritage designation (P1435) to filter by on Wikidata (optional)
The rest of the fields listed at User:ErfgoedBot/Depicts monuments.js I can figure out myself based on this info. Multichill (talk) 17:27, 27 February 2020 (UTC)
Thanks Multichill. Here are the UK WLM Campaign -> Wikidata mappings that are currently known to me. They should I think work, but I believe that there was some updating and fixing last year which I wasn't involved with.
For quick reference, you will find the UK winning images at Commons:Wiki Loves Monuments 2019 in the United Kingdom/Winners.
As I've decided to step down from organising the UK contest, I won't be in a position to follow up or help out further with this, I'm afraid. It's likely that the contest will run again this year, as normal, either with a new lead volunteer or perhaps one of the staff at Wikimedia UK. If you need more details User:Nev1 at WMUK should be able to put you in contact with the right person. All the best, MichaelMaggs (talk) 15:32, 6 March 2020 (UTC)
@MichaelMaggs: thanks for the pointers. I added Wales and Northern Ireland. I already had England. Scotland I can't handle right now because it uses multiple templates based on {{Historic Scotland listing}}. That's something I should look into supporting at some point. Multichill (talk) 19:28, 6 March 2020 (UTC)
@Multichill and MichaelMaggs: I'm so sorry I missed this conversation. Is there anything needed at this stage? Richard Nevell (WMUK) (talk) 11:13, 24 April 2020 (UTC)

Structured data for WLM in the Czech Republic

Great work @Multichill: ! Try Czech Republic, there should be a good coverage with decent amount of pictures. --Juandev (talk) 07:30, 5 March 2020 (UTC)

Thank you. I agree about the good coverage and amount of images that's why Czech Republic is on the list. Multichill (talk) 19:31, 6 March 2020 (UTC)

Abuse filter for labels

IMO the modifying of label (and descriptions) should show a hint, if more than 50 % of the characters will be removed or if emojis are added. --XRay talk 09:47, 25 April 2020 (UTC)

stability of proposal of SD?

Same set of images, same category, different proposed SDs.

(I cannot see in the history, what the bot proposed - another flaw). I assume, that users do not add missing SDs, but only delete unsuitable SDs. So why does the first object use the WD-item associated with the Commons category (which seems to be ok), and the later doesn't. Is the algorithm stable? Or does it produce arbitrary results? --Herzi Pinki (talk) 12:34, 28 April 2020 (UTC)

Hello @Herzi Pinki: . The Suggested Tag feature does not use categories to suggest depicts statements. Categories are displayed within the tool as a guide to help users understand the context and content of the image so they can choose tags accordingly, but the suggestions come from a Machine Vision analysis tool that looks at the content of the image itself. If you'd ever like to see a log of what tags were suggested for an image, append ?action=info to any File Page URL and scroll to the bottom of the page to see Suggested Labels RIsler (WMF) (talk) 23:40, 28 April 2020 (UTC)
@RIsler (WMF): Thanks. Thanks for the info about action=info. So the assignment of SD Hügelgräberfeld Eggforst, Hermagor (Q37897818) in the first image, which is the by far very best describing the subject (which is underneath the soil), was done by the user. Not by the automatism.
BTW, the link https://www.wikidata.org/wiki/Special:EntityPage/M89480645 on the info page yields a bad request. --Herzi Pinki (talk) 05:40, 29 April 2020 (UTC)

automatic update of SDs

If the mechanism proposes SD based on categories and the categories turn out to be wrong or are changed for other reasons, who is in charge to synchronously fix also associated SD? --Herzi Pinki (talk) 12:36, 28 April 2020 (UTC)

As mentioned in the reply above, categories are currently not used to suggest structured data, they're only displayed for reference. RIsler (WMF) (talk) 23:42, 28 April 2020 (UTC)

duplicate entries

see File:Hügelgräberfeld_Eggforst_01.jpg, motif grave field (Q2593777) is a duplicate. Does this make sense? What is the meaning of adding a motif twice? (e.g. if an image shows three buildings, shall there be three different motif-entries building (Q41176)?) Shouldn't there be a check to add each motif only once? Can some bot care for the cleanup? best --Herzi Pinki (talk) 12:57, 28 April 2020 (UTC)

Tracked in Phabricator
Task T251289

so this is not the case here. created CR: Can the software prevent situations like this and can some bot clean up the mess? best --Herzi Pinki (talk) 15:37, 28 April 2020 (UTC)

On SPARQL

@Jheald, Jarekt, Multichill, Jean-Frédéric, and Jura1: plus all others interested in the SDC query service.

Disclaimer, my personal knowledge and understanding on this topic is pretty limited, so bear with me here as we work through this.

As you may be aware, the Wikidata Query Service (WDQS) is not run by the Structured Data team at the Wikimedia Foundation. The WDQS team underwent some changes towards the end of last year, and they're still working to get back up to speed with Wikidata and looking towards the future of WDQS support. During this time, Ramsey has still been working to get the SDCQS up and running. Unfortunately due to the aforementioned other issues, the progress has been slow (as you're certainly aware). Ramsey's still looking into ways to move things along, and something that might be helpful is if we can hand the WDQS team some scoping for the basic features needed here; the "minimal viable product" as it's known in the business.

I think that there are two main areas around surfacing data that need addressed from the beginning. Please correct me if I'm wrong, or if this is missing a point or two or three:

  • Maintenance and administration use-cases
  • Exposing data connections for other tools to build on

Support for Commons queries will be built up gradually as resourcing allow the project to scale up. Assuming these points, what's the basic support needed here? Can a simpler system that provides search on key-value pairs would suffice for their use cases for now while the services are being scaled? Feel free to provide examples, if you have them. And please ask questions, I'll do my best to address them or get Ramsey to answer where he can as possible. Keegan (WMF) (talk) 20:50, 13 March 2020 (UTC)

I think the priorities would be that
  1. https://sdcquery.wmflabs.org/#SELECT%20%2a%20%7B%20%3Fa%20%3Fb%20%3Fc%20%7D%20LIMIT%201 runs again
  2. and gets fed regularly from live data (maybe not as quickly as Wikidata; which I think was never the case).
  3. Also, output as "image grid" didn't work, as no link to the file was included (previous comments)
  4. Federation with Wikidata would be good to have too (I don't think that was set up either, but I might be wrong), i.e. run queries on https://sdcquery.wmflabs.org/ that combine data with data from Wikidata.
  5. Federation on Wikidata should work too, i.e. run queries on https://query.wikidata.org/ that include data from Commons Structured Data.
  6. The mwapi service for requests to the MediaWiki api should be up too.
  7. Some users use another interface than the web interface, maybe having that up too would be good for them.
The above is (my view) by order of priority. Supposedly https://sdcquery.wmflabs.org/ will be replaced with some other url. The advantage of having Blazegraph running is that any maintenance query or output could be handled. Jura1 (talk) 10:25, 14 March 2020 (UTC)
I agree with many points Jura1 raised and would like to add some other ones:
  • We need to be able to run a lot of constraint related queries, most of them are the same as the constraints on Wikidata.
  • Federation with Wikidata is a must. If I want to check copyright status of a photo of an an sculpture the copyright for the photo will be on Commons while information on the sculpture will be on Wikidata. Also many constraint queries might require access to Wikidata. Another example query would look up all the wikidata items used in some property which are redirects.
  • display of results as image grid or on a map would be great
  • the queries connecting to wikidata should be able to look up items which link to given file through image (P18) property (or other similar ones)
  • Another capability would be ability to limit the search to files in some category or with some template, ideally federation with SQL server, or something like in this query.
--Jarekt (talk) 03:50, 15 March 2020 (UTC)
@Keegan (WMF): the MVP is a working SPARQL endpoint. Not much you can strip off to make it more minimal.
https://sdcquery.wmflabs.org/ was a nice prototype. From my point of view you need:
  1. Production infrastructure to replace it
  2. Regular RDF dumps which you need to feed to the query engine (bootstrap)
  3. Incremental RDF feed to keep the data up to date
That's probably the most minimal you can go. If you try to strip down any of these points you either get really poor performance or outdated data. If you're already doing this, it's a minimal step to enable all the bells and whistles we have on https://query.wikidata.org/ . Actively disabling these is probably even more work than just leaving these enabled. Just put up a disclaimer that some things might still be broken. Multichill (talk) 14:18, 15 March 2020 (UTC)
+1 to what most people have said here. Also, the current way of using 'haswbstatement' in the search engine is already a pretty good 'simpler system' that just does key/value-pairs. Husky (talk to me) 00:24, 16 March 2020 (UTC)
Tracked in Phabricator
Task T221921

I think this thread is prompted by this recent comment by User:Gehel on phabricator ticket T221921:

Some of the use cases described here are already supported by search (wbstatement keywords, etc...). We are not going to work on a new SPARQL endpoint before we have a scaling strategy for the current WDQS. It looks like the remaining use cases described here might be better served not by a SPARQL endpoint, but by a more specific service.

In my view the comment above is based on a misperception. WDQS is currently pulling about 5 million queries a day [1], and struggling to keep up with managing them together with data update. I do not believe there will be anything like the same massive external demand for Commons SPARQL Query Service (CQS) in the short term, because it does not serve the same generic content;nor IMO would it be the purpose of a naive SPARQL endpoint will appropriate to support primary end-user search and discovery, such as the holy grail of faceted search -- there wouldn't be either the scaling or the responsiveness, IMO.
What CQS is IMO invaluable and irreplaceable to support is a fairly small subset of Commons and SDC power users, who will use it to understand
(1) how properties are being used, in particular the interactions between different peoperties, and how different properties are being used together, to grow and refine the data model in the right way. At the moment we are flying blind and it is massively holding back the critical & urgent task of understanding and refinement of the data modelling.
(2) queries to drive maintenance of properties and statements that are not being used appropriately. Usually identifying the bad statements requires finding particular combinations of statements, which SPARQL is perfect for.
(3) queries to identfy gaps in data that can be filled for particular subsets -- to be compared with other sources (eg categories, descriptions), that power users that then use to fill statements in particular sub-areas in a targeted way, (cf Wikidata: identify gaps, fill with quick=statements; or Wikidata: data extract from source, compare with existing over some specific group of items, add missing data -- *key* workflows for Wikidata work.
(4) prototype proofs-of-concept for faceted search approaches and other tools and demos (cf Crotos). As demonstrations of what may become possible, not for volume use. (And if they did get over-exposed, it would be potentially possible that such demos could have their use throttled - eg by requiring an API key & limiting it).
In the early years of Wikidata, there was a system called WDQ by Magnus that allowed retrieval of particular combinations of property-value pairs, similar in some ways to what Keegan suggests in his intro above. (cf [2]). WDQ got Wikidata off the ground (and perhaps was more resource-efficient). But I believe such a stopgap would be a waste-of-time dead end for CQS, because:
(1) SPARQL is far more elegant, easy to read, and easy to understand -- and is *what the community knows*. It's crazy to split between 1 system for Wikidata and another for CQS.
(2) A SPARQL solution is available out-of-the-box for Wikibase.
(3) Many many many queries will rely on information both from Commons and from Wikidata. In SPARQL this is a built-in feature, using federation. Any other stopgap would have to develop such linkage from the ground.
In my view a SPARQL CQS system is desperately needed now, or even better last year, and putting roll-out on hold is insane for the project. Jheald (talk) 12:03, 16 March 2020 (UTC)
+1 on the scaling issue. Wikidata/WDQS has some specific issues, not least continuous import of academic paper items, often huge; and stars & whetever other catalogues are being raided. I think there's perhaps much less scope for this on commons. The JSON -> RDF export is known to be less efficient than it might be, and this is particularly associated with large JSONs - the academic papers with very many authors and/or citations; the chess players with 1,000s of ELOs; cities with hundreds of population statements. I see less scope for such hugeness in commons structured data; and it's probable that in the timescale commons structured data ramps up, WMDE will deliver JSON-RDF improvements. Finally I think less demand for reports from a commons structured data report service, than for WDQS. Obvs, all famous last words.
I'm basically where Jheald is on points (1) - (3) immediately above, endorse his "insane" and prepend "absolutely expletive". --Tagishsimon (talk) 17:18, 16 March 2020 (UTC)
Investing in any system which is not capable of verifying that property constraints are followed, would be a waste of time. --Jarekt (talk) 19:52, 16 March 2020 (UTC)

Thanks for everyone's feedback so far. We are fully aware of the significance of the query service. Our desire is to provide something useful for the hard-working volunteers contributing the data, and we acknowledge this is taking a very long time and it is frustrating for us as well.

We're discussing this internally and exploring a variety of possible solutions. We hope to provide some ideas for moving forward soon.

In the meantime, work on remaining SDC front-end features will continue. The actual Structured Data team that is assigned to solely work on this project is mostly a front-end team (which is why much of the recent work has been front-end). The backend infrastructure work is distributed among several specialized teams that split their time and resources across multiple critical projects. We'll keep you updated as things progress. Thanks again. RIsler (WMF) (talk) 20:31, 19 March 2020 (UTC)

Update

Hello all,

I'm sorry about the delay, extenuating circumstances pushed back some of the discussions needed to move forward. The good news is that the discussions have been had, and the importance of SPARQL has been conveyed and accepted.

Copying over what's been put in the Phabricator task:

  • The work to create a SPARQL endpoint for Commons has been re-prioritized [moved up in importance]. Our teams will be working on it over the next few months and the search team is currently estimating the work involved.
  • The first release will be a beta endpoint that will be updated via weekly dumps. Caveats will include limited performance, expected downtimes, and no interface, naming, or backward compatibility stability guarantees.
  • We do plan to move this to production, but we don't have a timeline on that yet.
  • The SPARQL endpoint for Commons will be restricted via a light and unobtrusive form of authentication, so that we can contact abusive bots / users and block them selectively (as a last resort) when needed. More details on this to come.
  • We want to emphasize that while we do expect a SPARQL endpoint to be part of a medium to long term solution, it will only be part of that solution. Even once the SPARQL endpoint is production-ready, it will still have limitations in terms of timeouts, expensive queries, and federation. Some use cases will need to be migrated, over time, to better solutions — once those solutions exist.

Two additional points not on Phabricator:

  • CBogen, the team's new program manager, and I plan on providing updates on this at least every two weeks on the task and here.
  • Constraints for SDC have been deployed, and should be functional when the endpoint is stood up (@Jheald: ).

Thanks all, we'll keep you posted. Keegan (WMF) (talk) 15:32, 30 April 2020 (UTC)

Issues with SDC mass uploads

@SandraF (WMF) and Keegan (WMF): ,I do not know if that is on anybody's radar but phabricator:T246746 and phabricator:T245349 issues are big handicaps for working with SDC. I am using QuickStatement tool for a lot of SDC statements, I am logged is as user:JarektBot and account I am using so my edits are marked as bot edits and do not flood peoples watchlists. Unfortunately I am due to issues reported in phabricator:T246746/phabricator:T67494, my bot edits are not marked as such. This might not be SDC issue, but it is an annoyance to many and I am trying to avoid annoying people with my SDC edits. Phabricator:T245349 / phabricator:T237991 is also a big problem. Since phabricator:T221921 is still unsolved we are relying on maintenance categories to know which file still need statements, see Category:Structured Data on Commons tracking categories, often added with help of Module:SDC tracking. Unfortunately adding SDC statement does not trigger a page refresh, and does not remove files from the category. The only solution is to run "touch" / purge operations on those files which is much slower than adding statements. Any chance, one of you can reprioritize some tasks to get those done? --Jarekt (talk) 17:46, 28 April 2020 (UTC)

@Jarekt: I'll check with Ramsey and see where we are with these tasks. Keegan (WMF) (talk) 18:14, 28 April 2020 (UTC)
A ticket's been made to investigate the cache issues, the bot edits require a further look into existing tickets. There should be further updates in the tickets on Phabricator in the near future. Keegan (WMF) (talk) 16:36, 4 May 2020 (UTC)
Thank you. Unmarked bot edits to SDC seem to be an irritant to a lot of people. I will try to complete the task of adding OTRS IDs to SDC, but I will not start any more tasks until the issue is resolved of I master some other ways of mass editing. --Jarekt (talk) 17:09, 4 May 2020 (UTC)

Help needed from javascript speakers

See MediaWiki_talk:Gadget-PermissionOTRS.js#Add_P6305_SDC_statement. --Jarekt (talk) 14:28, 8 May 2020 (UTC)

Constraint violations database reports

@CBogen (WMF), Keegan (WMF), and Jheald: I do not know if this was discussed anywhere else, but I was wandering about ways we are or are going to track SDC constraint violations. I know we are waiting for phabricator:T230314 and SPARQL database queries, but I was wandering about other parts of the system. Each property has a page on Wikidata, like creator (P170) and that page is a central point for all the links to pages related to the property: that is where we store constraints and that is where we have link to d:Wikidata:Database reports/Constraint violations/P170 (this link is actually on the talk page). So here are some questions:

  • Are we going to have some constraints relevant only to SDC which are different from Wikidata constraints? If so how are we going to model that?
  • Some types of constraints do not require SPARQL database query (as explained in phabricator:T230314 by User:Lucas Werkmeister). I am not sure if any of them apply to SDC, but if so are there any pages on Commons or Wikidata showing SDC constraint violations for such cases?
  • Maybe we need some system of pages on Commons which list properties related to Commons, where we can discuss SDC issues related to those properties and can use as a hub for links to constraint violations database reports, etc. I do not know if Wikidata property pages allow sitelinks but if they do than we could connect them. --Jarekt (talk) 17:54, 4 May 2020 (UTC)
FWIW, I know little to nothing about the mechanics of this topic itself. I'll see what I can find out if there's a question related directly to the developers. Keegan (WMF) (talk) 16:40, 11 May 2020 (UTC)
Keegan, Thanks for replying. I guess what is confusing is that on Wikidata constraint violations infrastructure is a bit of a patchwork of MW software, user controlled bots, and occasional extensions and javascripts. As a result, it is really hard to transplant it to Commons. Who does what is not easily transparent, either. So I was trying to start a discussion about how we are going to bootstrap such infrastructure. I do not know a whole lot about it either, but I am trying to wrap my head around some of those issues. If I had a single question to the developer team would be to document what exactly will be provided by MW software once SPARQL queries are live and phabricator:T230314 finalized. That way, we will know what work has to be done by the community, if we want constraint violations infrastructure similar to the one we are use to on Wikidata. --Jarekt (talk) 17:00, 11 May 2020 (UTC)
Sounds good, I'll make sure that's shared and happens. Keegan (WMF) (talk) 17:46, 12 May 2020 (UTC)

By the way, I’m not sure if Ivan A. Krestinin’s C++ bot (which updates the constraint violation pages) actually uses Wikidata Query Service. Unfortunately the source code is still not public (as far as I know). —Tacsipacsi (talk) 01:22, 5 May 2020 (UTC)

Wikidata Wochenende taking place online

Hello all,

A few months ago, I told you about the Wikidata Wochenende (de), a week-end dedicated to working on Wikidata-related projects where we would love to have Commons editors, for example working on Wikidata-powered templates or Structured Data. The event, initially planned in Ulm, will take place entirely remote on June 12-14. It's going to be a mix of hackathon and workshops, where people can connect, work on their projects, and learn more from others.

If you're speaking German and interested in attending, please register in the next few weeks. Cheers, Lea Lacroix (WMDE) (talk) 08:37, 19 May 2020 (UTC)

Structured Search

Hey everyone, i've released a tool that some of you might find of use. It's called Structured Search and provides another user interface to the Commons search engine. I developed this tool for two reasons:

  • To provide a friendlier user interface to show the richness and beauty of all the wonderful free content that is available here.
  • To showcase the possibilities of Structured Data on Commons.

I've made it so that it's easy to search for other images that have structured data: try clicking on any image on the first page that you get, then look for ‘depicts’ statements in the image detail pane. There are also options to search for categories and exporting queries to PetScan. For those of you who want to access the tool from a regular Commons search results page i've made a little userscript.

Check it out here. Husky (talk to me) 20:52, 24 May 2020 (UTC)

@Husky: thanks for making this at the hackathon!
Following up on this search prototype on labs, the structured data team is working on designing a prototype on-wiki. More information is on the Commons:Structured data/Media search page, please look it over and leave your feedback about the design and implementation. I'm posting links to this page in various places on Commons throughout the next day or so. Keegan (WMF) (talk) 18:03, 28 May 2020 (UTC)

Redirect and structured data

This does not seem to make sense: https://commons.wikimedia.org/w/index.php?title=File:Official_Portrait_of_President_Donald_Trump.jpg&action=history . Bug? --2A02:810D:6C0:2FB0:5AF:A0FD:D85D:5063 09:03, 26 April 2020 (UTC)

See headline. Structured data for a redirect does not make sense IMHO --2A02:810D:6C0:2FB0:4088:F34E:3A4:AE61 18:21, 26 April 2020 (UTC)
Answered at Commons:Forum#Strukturierte_Daten_bei_Weiterleitung. Multichill (talk) 18:33, 26 April 2020 (UTC)
Can't find information there. --2A02:810D:6C0:2FB0:B8BD:D76B:D60C:7311 19:10, 17 May 2020 (UTC)
Where can this misbehaviour be reported? I think this is a bug. --2A02:810D:6C0:2FB0:3893:1DC7:E319:A1DC 19:14, 21 June 2020 (UTC)

Could structured data be used to document file usage outside of Wikimedia websites?

I have an idea of something that might be useful, create the statement "Published" for files that originated (as in were first published) on Wikimedia Commons. Currently if you use a Wikimedia Commons media file on another Wikimedia website it will list it where it is currently being used, but one needs a special template to show that a file was also used on another website.

But could Structured Data on Wikimedia Commons (SDC) to document where Commonswiki files are used as well? Maybe a bot could "scan" other websites to automatically detect where files are used and then a human needs to confirm that the file is indeed used there (this could theoretically also be used to find more copyright infringements), maybe when a statement is added a bot could automatically get more information from the linked website (article title, date of publication, access date, author, Etc.) and perhaps even automatically add an Internet Archive link. Are any of these things possible? --Donald Trung 『徵國單』 (No Fake News 💬) (WikiProject Numismatics 💴) (Articles 📚) 13:00, 21 June 2020 (UTC)

I'm not a technical person so I can't contribute much to the idea. What I can offer is that structured data about a file is stored locally on Commons as a JSON blob and is not actually attached to a file in the way that EXIF or other metadata might be, so I'm not sure how helpful SDC itself might be for such a feature. Keegan (WMF) (talk) 16:23, 24 June 2020 (UTC)
Donald, We could create SDC property to store URL's where given image is also published on the web. However writing a bot to scan the web and populate that field might be more tricky:
  1. Someone volunteer would actually have to write it
  2. We would have to narrow it down to images first published on Commons (we do not want that for images of Mona Lisa for example), but even than comparing images at different resolutions and different crops might get tricky. So it would have to be by-hand filled property, the way we do it now. --Jarekt (talk) 19:16, 24 June 2020 (UTC)
The bot idea isn't really a priority. Regarding the property it could be used to identify which websites use Wikimedia Commons files more often than others and how popular some images are outside of Wikimedia websites.
Maybe in the future we can have a "Property proposals" page akin to Wikidata's, once Structured Data on Wikimedia Commons (SDC) is more integrated into the Wikimedia Commons community. --Donald Trung 『徵國單』 (No Fake News 💬) (WikiProject Numismatics 💴) (Articles 📚) 19:25, 24 June 2020 (UTC)
Donald you can propose a new property at d:Wikidata:Property proposal/Commons.--Jarekt (talk) 04:44, 29 June 2020 (UTC)

SPARQL update

Tracked in Phabricator
Task T221921

Copying over an update on the SDC SPARQL endpoint:

  • The initial data has been loaded and federation is working
  • The next work is around authentication, automated data reloads, and necessary UI updates.

If work continues at this pace without surprises it should be done in about four weeks or so. If something unforeseen shows up, that could hamper the timeline. Another update to come in a couple of weeks. Keegan (WMF) (talk) 15:50, 26 May 2020 (UTC)

 Thank you. for update--Jarekt (talk) 16:17, 26 May 2020 (UTC)
NICE —TheDJ (talk • contribs) 07:25, 27 May 2020 (UTC)
  • I don't have the public (or a private) URL right now. The Phabricator ticket and this page will be updated with a link the moment I have something for consumption. Keegan (WMF) (talk) 18:31, 11 June 2020 (UTC)
15 June
30 June

At this point it looks like we're a little over two weeks out from release:

  • OAuth code is ready but still needs to be merged
  • Deploying configuration changes to the UI in order to customize the title, logo, examples, etc for the Commons Query Service proved to be a little bit more complicated than expected, but is nearly ready
  • The "wd" prefix needs to be changed for Commons for clarity purposes. It is taking longer than expected to do this in the dumps, so we are temporarily doing this at load time, which delayed things a bit.

Sometime next week the data should be loaded in, and after that comes approximately one more week of testing. We're almost there. Keegan (WMF) (talk) 17:56, 30 June 2020 (UTC)

8 July

The timeline has been moved back by about a week, with a goal of having the SDC SPARQL endpoint running for users by 24 July, but sooner if possible.

  • After testing, the instance is being moved to a bigger server to assure enough space for dumps.
  • Login/logout functions are mostly all that remain in actual engineering of the endpoint.
  • Other commitments this week took some time away to getting things completed a little sooner and contributed to the delay.

I have not been made aware of any other potential blockers to release, we should be on track to wrap all this up. Keegan (WMF) (talk) 18:33, 8 July 2020 (UTC)

Some aspects of the "look" of SDC statements make no sense to me

Tracked in Phabricator
Task T256933
Comparison of the SDC statements found in File:"1912 - Mission Moderne" Wallraf-Richartz-Museum & Fondation Corboud.jpg with the same statements shown using Wikidata "look"

Most statements we are adding to files, like author, cameras geo-coordinates, date, copyright status, OTRS number, etc. will have a single value. Some always, like cameras geo-coordinates, and some in most cases. That is why I do not like the "look" of the statements adopted by SDC GUI, with prominent search window inviting to add a second value and much less prominent actual value of that statement. The look used at Wikidata just makes much more sense to me: there is a property and its value and the search/input window only shows up when you are adding a second value (rare case) or correcting the value. To think of it the only property that this GUI makes sense for if "depict" statement. Any chance we could move the value of the statement to a more prominent location and hide the confusing search window for the properties which expects (in most cases) a single value? I do like the current look for adding "depict" statements. --Jarekt (talk) 04:42, 29 June 2020 (UTC)

I agree removing the prominent search fields is a good idea. However, pay attention that the Wikidata-like interface features reference sections that are not present on Commons. Probably removing them won’t affect the UI much, though. —Tacsipacsi (talk) 15:06, 29 June 2020 (UTC)
phabricator:T253053 proposes activating references on Commons. In most cases it is probably not necessary, as the metadata is provided by the uploader, but there might be some images where they are useful. Also I am not arguing for GUI with exactly the same look as Wikidata, but current Wikidata look is much closes to the ideal look than SDC's look. Another thing is that I think SDC's look should be much closer to the iconic look us use for last 15 years for {{Information}} template. --Jarekt (talk) 16:03, 29 June 2020 (UTC)
Keegan (WMF), I understand, and many other of my phabricator tickets are much more urgent to me. However it is something that always bothers me and I started this discussion (as opposed to writing phabricator ticket) in order to see how other feel about it. --Jarekt (talk) 02:15, 1 July 2020 (UTC)
I agree as well. We should probably make a phabricator ticket for this to put it on the backlog. Ainali (talk) 06:54, 1 July 2020 (UTC)
Same for me. Each time, I get the feeling that there is no statement. Ayack (talk) 07:36, 1 July 2020 (UTC)
+1 --El Grafo (talk) 08:51, 1 July 2020 (UTC)
When I think about it, yes, this actually happens for me, too. 1234qwer1234qwer4 (talk) 17:06, 1 July 2020 (UTC)
OK I created Phabricator:T256933 please add comments there as well. --Jarekt (talk) 03:14, 2 July 2020 (UTC)
@Jarekt, 1234qwer1234qwer4, El Grafo, Ainali, Ayack, Multichill, Christian Ferrer, and Tacsipacsi: so when I said the "near future" I clearly meant 48 hours . I'll update when the patch is live here. Keegan (WMF) (talk) 18:57, 2 July 2020 (UTC)
@Jarekt, 1234qwer1234qwer4, El Grafo, Ainali, Ayack, Multichill, Christian Ferrer, and Tacsipacsi: the design change is live, please have a look at your favorite example file to see the change. Keegan (WMF) (talk) 16:53, 15 July 2020 (UTC)
Nice! I think it looks really good. Ainali (talk) 17:06, 15 July 2020 (UTC)
Wow, thanks for the quick reaction! 1234qwer1234qwer4 (talk) 17:43, 15 July 2020 (UTC)
That is much better. Thanks --Jarekt (talk) 01:26, 21 July 2020 (UTC)
… and another +1 from me ;-) --El Grafo (talk) 07:57, 22 July 2020 (UTC)
This section was archived on a request by: 𝟙𝟤𝟯𝟺𝐪𝑤𝒆𝓇𝟷𝟮𝟥𝟜𝓺𝔴𝕖𝖗𝟰 (𝗍𝗮𝘭𝙠) 16:45, 5 September 2020 (UTC)

Reverse Wikidata connection?

In Wikidata we have with property image (P18) - I think that depicts (P180) from Commons structured data must be available from within Wikidata. And instead of setting qualifier media legend (P2096) for each language for each usage of one file in Wikidata, we might need to forward this information from Commons structured data. ·Carn 12:52, 16 July 2020 (UTC)

Dumps for the Commons SPARQL query service

Apparently the two issues preventing dump updates for the Commons SPARQL query service were resolved beginning last month. My question is when we will be getting an updated dump to work on and what update frequency is planned. I am asking as a user who plans to update structured data to all his files till sooner than later. Since I would like to use the service to track completeness and consistency of data added, you can imagine why I am asking. :) Cheers --[[kgh]] (talk) 07:52, 8 September 2020 (UTC)

Dumps run every Sunday now, see https://dumps.wikimedia.org/other/wikibase/commonswiki/ . On Monday Commons:SPARQL query service gets updated. In the future recent changes updates will be implemented. Search is mostly up to date if you need to check something.
Do you have all your files under this account or some other account? Multichill (talk) 13:58, 8 September 2020 (UTC)
Thanks a lot for your reply and clarifying. This is great news. I will update the respective page accordingly.
I have indeed uploaded most of my files with another account. Since I did the edits on late Sunday and constructed the query with filters the results were destined not to show this week.
Thanks again. Cheers --[[kgh]] (talk) 19:39, 8 September 2020 (UTC)
In the meantime I updated Commons SPARQL query service a bit. Thanks again for the help. --[[kgh]] (talk) 18:38, 11 September 2020 (UTC)
Checkmark This section is resolved and can be archived. If you disagree, replace this template with your comment. [[kgh]] (talk) 18:38, 11 September 2020 (UTC)

WCQS authentication with a script

The Wikimedia Commons Query Service (WCQS) beta has been announced today. It requires authentication with a Wikimedia Commons account to be used, unlike the Wikidata Query Service (WDQS) which is publicly available without authentication. Has anyone already figured out how to do the OAuth authentication with a script? I am sure that this is possible, but I have not done something like that yet and would greatly acknowledge if someone would share code or give hints :-) Thanks, MisterSynergy (talk) 19:46, 22 July 2020 (UTC)

Make it possible to change statements

After the quick response to the look I wonder what the opinions are about another aspect of the current interface: It's impossible to change statements. For depicts (P180) that makes perfect sense to me. You want user to add more depicts (P180) and discourage users from modifying existing. For other statements it doesn't make a lot of sense. For example if I want to fix coordinates I end up having to add new coordinates and removing the old coordinates. Wouldn't it be to introduce the ability to modify statements for all properties except depicts (P180)? Multichill (talk) 11:05, 18 July 2020 (UTC)

  • I don't see that being unable to edit makes sense even for "depicts". In my experience on my own photos, at least a third of the depicts (P180) that are added are wrong. If someone marks a railway bridge as a road bridge, it should be possible to modify that. - Jmabel ! talk 17:11, 18 July 2020 (UTC)
Multichill I see what you mean. I can edit string, URL and number statements, but I can not edit item or coordinate statements. Date qualifiers I can edit but date statements I can not. strange. --Jarekt (talk) 19:39, 21 July 2020 (UTC)
I recall what seemed like a sensible explanation at the time for designing statements being essentially one-and-done, you have to delete and start over, but I do not remember the details. Let me see what kind of conversation I can rustle up. Keegan (WMF) (talk) 20:22, 21 July 2020 (UTC)
I would be very interested. 1234qwer1234qwer4 (talk) 20:06, 1 August 2020 (UTC)

Upload media button suppression

Not a structured data issue

Maybe disambiguation categories should not have an Upload media button/link (“great” UI design, b.t.w.) to avoid what would be an 100% chance of miscategorization upon upload…? -- Tuválkin ✉ ✇ 01:13, 13 August 2020 (UTC)

Well, first, this does not seem to be the correct place for this kind of discussion, as it has nothing to do with the structured data approach. Second, the function of this link is not linked to the current page - it simply doen't matter which page you're on when you click it. So why should it not be displayed? -- H005 06:36, 13 August 2020 (UTC)
@Mike Peel: , as it is from the {{Wikidata Infobox}}.
H005: Actually, the link does use the category to prefill the upload process (see Commons:Upload_Wizard/Fields_prefilling).
Jean-Fred (talk) 20:41, 14 August 2020 (UTC)
@Tuvalkin, H005, and Jean-Frédéric: I've modified {{Wikidata Infobox/sandbox}} so that it doesn't show the upload link where instance of (P31)=Wikimedia disambiguation page (Q4167410) or Wikimedia disambiguation category (Q15407973), is that better? Also see the broader discussion at Template_talk:Wikidata_Infobox#Infoboxes_on_disambiguation_pages. Thanks. Mike Peel (talk) 11:21, 15 August 2020 (UTC)
@Jean-Frédéric: Thanks! @Mike Peel: Sounds great. @H005: Loud and clear… -- Tuválkin ✉ ✇ 12:29, 15 August 2020 (UTC)
Well, actually this could have been avoided if you had stated loud and clear that you are talking about the infobox and not, as I assumed, the standard sidebar on the left, which also provides this functionality ... ;-) -- H005 06:31, 21 August 2020 (UTC)
@H005: It does? But not in Monobook, at least. There’s an "Upload file" link under "Participate", but it’s there always, regardless of namespace, and it doesn’t lead to a custom upload form pre-filled to the current cat. Your hostility was noted, tho. Funny how paladins of the one WMF project that gets the most buzz, the bigger slice of the financing, and has most distinct pedigree (it originated from a Google project, doncha know, not some rabblesome grassroot wiki) — have yet such a thin skin. -- Tuválkin ✉ ✇ 16:53, 23 August 2020 (UTC)
"... but it’s there always, regardless of namespace, and it doesn’t lead to a custom upload form pre-filled to the current cat." Well, that was exactly my point, wasn't it?
Btw, interesting to note that you indeed see "hostility" and "thin skin" elsewhere, allocating them into plain stereotypes to top. -- H005 06:17, 24 August 2020 (UTC)
  • @H005: Your point was 100% moot, based as it was on your misundersting of my question (which would have been more clear, granted, should I had mentioned {{Wikidata Infobox}}, though others managed to infer my intent) and apparently triggered by the need to retort at my grumble about the lousy UI. Said retort, hostile and unwelcoming («Well, first, this does not seem to be the correct place»), shows how hasty your reaction was — you thin-skinnedly jumped to the conclusion that I merely bumbled into the wrong venue to ask about basic page UI, unlike others who correctly deduced that since I posted my question here (unclear as it was) then it was somehow related to SD. It was, in a way, and now {{Wikidata Infobox/sandbox}} is fixed. Again my thanks to those who did positive stuff to sort this matter. -- Tuválkin ✉ ✇ 19:03, 24 August 2020 (UTC)


Misleading authorship claims

There are currently a bot or bots adding claims about authorship which are false.

For example, in this edit, a bot adds the claim "Property / creator: Some value without a Wikidata item".

Note the explicit statement "without a Wikidata item".

The image is my creation, and there is a Wikidata item about me. This can be verified programmatically, in several different ways: by traversing the category tree (in this case to Category:Andy Mabbett) to determine the corresponding QID; or by a reverse lookup of the user name (note that the user name is known; and added as a qualifier in "Property / creator: Some value without a Wikidata item / qualifier - URL: https://commons.wikimedia.org/wiki/User:Pigsonthewing "; or can be obtained by obtaining the username from the {{Information}} template's |author= parameter); or in some cases from the presence of a {{Creator}} template which "knows" the relevant QID.

In the case of another bot, this error has been fixed; although some cases persist

This issue is ongoing, and is happening to hundreds if not thousands of my images, and, no doubt, many more thousands of others'.

I don't know if there is a plan, yet, to fix these errors, but until such a plan is executed satisfactorily, author information should not be removed from the {{Information}} template, and such claims in authorship data should not be relied upon. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 11:03, 27 August 2020 (UTC)

I think this is a mis-rendering of a PropertySomeValueSnak. PropertySomeValueSnak is defined to mean that the property has some value without saying anything about this value. So I think the bot is correct: those pictures have an author, even if the bot isn't clever enough to understand who that author is. The problem is that the Web interface renders PropertySomeValueSnak as "Some value without a Wikidata item", which isn't what it means. It looks like this text comes from MediaWiki:Wikibasemediainfo-filepage-statement-some-value, but presumably any change should be made upstream since it would apply to all Wikibase instances, not just Commons. The diff uses MediaWiki:Wikibase-snakview-snaktypeselector-somevalue instead. Curiously the en-GB translation of that is just "some value", so the diff looked fine to me. --bjh21 (talk) 11:40, 27 August 2020 (UTC)
I've submitted a bug report about the incorrect wording on the file page: phab:T261404. It looks like MediaWiki:Wikibase-snakview-snaktypeselector-somevalue has been overridden here on Commons, so I've placed an edit request on MediaWiki talk:Wikibase-snakview-snaktypeselector-somevalue. Pinging Multichill, who set the current text following discussion on Commons talk:Structured data/Modeling/Author. --bjh21 (talk) 12:39, 27 August 2020 (UTC)
I updated the message. Still might confuse people, but better than "unknown value". Not sure if we can make it a wikilink so we can point it to some page explaining the concept and how it's used on Commons? Multichill (talk) 19:54, 27 August 2020 (UTC)
@Multichill: Thank you. That seems to be working for making diffs say "some value", though caching seems to mean that diffs that have already been rendered, like the ones linked above, still have the old text. Do you have any idea how translated versions of this message are meant to get updated? For instance, if I switch to French I get "valeur inconnue", which indicates we're still getting the default text in non-English languages. Meanwhile, GZWDer has closed phab:T261404 as invalid, so MediaWiki:Wikibasemediainfo-filepage-statement-some-value may need to be overridden locally as well. --bjh21 (talk) 21:21, 27 August 2020 (UTC)

unnecessary message about heading 360°

got the message There is a discrepancy of 360 degrees between the above camera heading (set to 0) and the ones stored at SDC (set to 360). Please reconcile them. on [3] (now gone, not shown any more). As far as I remember, 360° is the same as 0°. Any value of x+n*360 is equivalent to x in this case. Can somebody please add code to her bot to move all degrees to [0°,360°[ and suppress the message in this very case? best --Herzi Pinki (talk) 06:54, 20 September 2020 (UTC)

Beta Commons SPARQL query service is available

As you might have seen if you're subscribed to the Wikidata or Commons mailing lists, the beta endpoint for the Wikimedia Commons Query Service is now available.

The WCQS is essentially a clone of the Wikidata query service, with the same features and limitations as well. The search team hopes to move the WCQS to a different production server in the future, but for now they're returning focus to supporting and building out future plans for the Wikidata Query Service.

I've started a basic documentation page with the release notes from the launch announcement email. I'd be grateful for any support in building up the support pages around the service, including the main page and the examples page.

I plan on placing a notice about the service's availability on the Village Pump tomorrow, barring any unforeseen changes.

Thanks to you all for expressing the pressing need for this service. Keegan (WMF) (talk) 20:31, 22 July 2020 (UTC)

@Keegan (WMF): Would it be possible to add https://wcqs-beta.wmflabs.org/sparql as allowed service url for https://query.wikidata.org so it would be possible to use it with existing tools like listeria via federation? (Now it is possible from wcqs-beta.wmflabs.org only)--Zache (talk) 13:58, 6 September 2020 (UTC)
@Keegan (WMF): I have real life use case for this too. It would be nice to be able read dates of the photos directly from SDC instead of them to be saved as P18 qualifiers in Wikidata. --Zache (talk) 11:25, 17 September 2020 (UTC)
I'll see about getting a Phabricator ticket filed for the request. I have no idea if that's something that will occur before moving the service out of beta or not, so we'll find out. Keegan (WMF) (talk) 15:55, 18 September 2020 (UTC)
Current WCQS endpoint has very limited connection to WDQS, see phabricator:T261716 for details. It is mostly useful for small subset of realistic use-cases. If we going to propose connecting WDQS to WCQS than perhaps high volume connection should be baked in from the start. --Jarekt (talk) 19:14, 18 September 2020 (UTC)
@Keegan (WMF): Also one quick-fix solution could to add wbgetentities and wbgetclaims to MWAPI which would be just a configuration change) as the MWAPI service and API itself are already in production. It would not allow the high volume queries because of the timeouts, but it would be an enabler for use cases where the number of the results is low. --Zache (talk) 09:12, 22 September 2020 (UTC)
Also my current implementation in fiwiki w:fi:Malline:Wikidata-galleria/Torkkelinmäki. This fetches list of photos related to are of target item and generates a photo gallery. It will read the wiki code of the target page and checks if the photo is already used in the page and shows only a new photos. Second thing what it does is that reads the commons categories of the photos and tries to get the year of the photo from there. If it the update of the page is too slow idea is to change it so that listeria would subst the slowest templates or pre-fetch data using SPARQL so it would be slow only when the ListeriaBot is updating the page. With this the saved wikicode would be fast. --Zache (talk) 14:44, 23 September 2020 (UTC)

Add an update button

  1. User updates e.g., {{Location}}.
  2. User starts seeing warnings.

There should be a button to "Update all Structured data", that would re-read the page. Jidanni (talk) 17:57, 23 September 2020 (UTC)

More than half of all files have one or more statements

Hi everyone, A nice milestone: Half of all files (32,8M out of 64,5M) have at least one statement. Special:MediaSearch is a nice frontend for the existing search. I hope work on the search backend will soon start showing some results too. Multichill (talk) 16:47, 28 September 2020 (UTC)

It's a great milestone. I was planning to post about it at the end of the week but you beat me to it :) One major clarification here: Special:MediaSearch is using an entirely new search backend, so that work is well underway and showing results in the new search. It integrates categories, structured data and wikitext from Commons, and Wikidata to find results, all built from the ground up on the backend using ElasticSearch. The backend ranking documentation information still needs to be published, but that should be done relatively soon. Keegan (WMF) (talk) 20:15, 30 September 2020 (UTC)
@Keegan (WMF): are you sure about that? When I open Special:MediaSearch and search for "Haarlem" it's hitting https://commons.wikimedia.org/w/api.php?action=query&format=json&uselang=en&generator=search&gsrsearch=Haarlem%20filetype%3Abitmap%7Cdrawing&gsrlimit=40&gsroffset=0&prop=info%7Cimageinfo%7Cpageterms&inprop=url&gsrnamespace=6&iiprop=url%7Csize%7Cmime&iiurlheight=180&wbptterms=label&mediasearch=true . It does append "mediasearch=true" to the url, but the only effect seems to be a "Unrecognized parameter: mediasearch." warning. Multichill (talk) 08:32, 1 October 2020 (UTC)
@Multichill: I am 100% sure. ?mediasearch=true serves a warning as it's not a supported parameter, but it allows the backend to use the new search code which is not otherwise enabled (as it being enabled would immediately impact Special:Search and other search API calls like the media inserter in visual editor). Keegan (WMF) (talk) 16:20, 1 October 2020 (UTC)
This may be a nice milestone, but I don't think what this actually means is that useful, as the vast majority of the statements currently seem to be copyright information that is also searchable through incategory search. Search for depicts statements only returns 2,77M results. 𝟙𝟤𝟯𝟺𝐪𝑤𝒆𝓇𝟷𝟮𝟥𝟜𝓺𝔴𝕖𝖗𝟰 (𝗍𝗮𝘭𝙠) 12:51, 5 October 2020 (UTC)
Most files also contain now the author as well as dates/coordinates. Which allows already nice thing to do
Hope this helps to get some further ideas. --Schlurcher (talk) 13:24, 5 October 2020 (UTC)
The SPARQL query examples offer additional utility as well Keegan (WMF) (talk) 18:40, 5 October 2020 (UTC)
@Schlurcher: : "which reminds me that I have only 2 with proper coordinates" I have many more with proper coordinates, but your SPARQL also shows only two of them when I replace 'Schlurcher' with 'H005'. Any idea why? -- H005 22:13, 5 October 2020 (UTC)
@H005: I have checked a couple of your files. There are properly geo-coded in the file namespace, but for most the data has not been transferred to the SDC namespace. As such, they will not appear in this Query yet. Unfortunately, the bots that transfer these data currently only support the decimal notation. I'm just about to update my bot to support the hour,minutes,seconds notation that you use. So you can expect some progress once I have tested the new code. I will use your files to test the new code, so keep a look out. But even then we still have a backlog of 3.5 Million files Category:Pages with local camera coordinates and missing SDC coordinates, but we made progress it was 6M when we started this year. Also the data for the tool refreshes only every week, so have some patience after the edits occurred. --Schlurcher (talk) 07:39, 7 October 2020 (UTC)
@Schlurcher: Thanks for your reply! I am aware of course that the coordinates have to be in SDC. Take a look at this file as an example: File:Aquila_nipalensis_-_20100905.jpg. It has proper coordinates in SDC, and yet it is not shown in the SPARQL query. -- H005 10:11, 7 October 2020 (UTC)
@H005: . Thanks for providing an example. Indeed it should be included in this query. The SDC coordinate was added on 4-Oct-2020. I think the database behind the SPARQL queries gets updated each Monday. So, I am suggesting to wait till 13-Oct-2020 to see if the issue persists. --Schlurcher (talk) 12:53, 7 October 2020 (UTC)
Thanks for the info, I've taken the liberty of adding that to Commons:The Commons Log. --El Grafo (talk) 15:04, 5 October 2020 (UTC)

Importing data via LUA to other wikis?

Two related questions:

  1. Modules like Module:Wd, Module:Wikidata and Module:WikidataIB can pull fields from Wikidata to include on other wikis. Is something similar possible to pull structured wikibase statements from commons?
  2. Have the author name strings and license information from {{information}} been put into wikibase statements yet?

I would like to be able to automatically call information from commons to format up attributions under figures (example manual implementation). T.Shafee(Evo﹠Evo)talk 02:46, 13 October 2020 (UTC)

Ad 1: No, it isn't possible yet (phab:T238798). --Matěj Suchánek (talk) 10:41, 13 October 2020 (UTC)
But you can try it out on Commons, Commons:Structured data/Lua. - Premeditated (talk) 10:46, 13 October 2020 (UTC)
@Evolution and evolvability: only local, see {{Geograph from structured data}} and usage for an example.
Import of data is still ongoing, but quite a bit of data has already been imported. See for example Commons:Wiki Loves Monuments/Structured data for what kind of data has been mass imported for Wiki Loves Monuments related images. Multichill (talk) 12:44, 13 October 2020 (UTC)
Aha, thank you all for the info. Useful to know & I'll keep an eye on the topic. T.Shafee(Evo﹠Evo)talk 00:18, 14 October 2020 (UTC)

Is this really how it's supposed to work?

File:Jacksonville, Oregon - First Presbyterian Church 01.jpg tells me the file depicts Oregon. I would think that we would have some notion of place distinct from depicts, especially when dealing with a place as large as a country, state, or province. I get that we might need to say something depicts a historic district, but for a state that feels wrong to me. - Jmabel ! talk 02:45, 2 August 2020 (UTC)

I agree. By this notion we could also say this file depicts the world, if not the whole Universe. I would draw the line if not at district, as sugested above, but at the municipal level. But we should also delve into the question why "Oregon" was picked in the first place. The edit after all was not done by a bot. Was it a language barrier? Why was Oregon picked out of "en|First Presbyterian Church, California and Sixth Streets, Jacksonville, Oregon, U.S." What would have been a better pick? "First Presbyterian Church", "California and Sixth Streets", "Jacksonville". How can we help to make the right pick? --Wuselig (talk) 06:10, 2 August 2020 (UTC)
No, it's not how it's supposed to work. I removed that claim. Preferably it should not depict a generic "church building" either, but the specific church. But since it doesn't seem to have an item on Wikidata yet, it's good for now. Ainali (talk) 07:49, 2 August 2020 (UTC)
That raises an interesting question of how properties are updated to be more specific when a new Wikidata item is created. There seems to be no way of flagging this unless you are familiar with the relevant items. Rodhullandemu (talk) 08:02, 2 August 2020 (UTC)
Well, the best way to at least ease this process is to add further information, especially the geocoordinates. This way it will be easier to find related media when the item has been created some day. I've just added those coordinates and also the P131 property (which by the way, is the correct property for towns, countries etc., and not "depicts" as it was before). -- H005 08:50, 2 August 2020 (UTC)
@Rodhullandemu: The properties themselves should not be updated. But i guess you meant the statements on individual files, and yes, that is somewhat of a problem. But I guess actual usage, and discovering some files have values that are to broad, along normal wiki processes is the way forward. Ainali (talk)
@H005: I don't think P131 is particulary good. Take a look at Commons:Structured_data/Modeling/Location (and the talk page) to see current thought on modeling location. Ainali (talk) 10:47, 2 August 2020 (UTC)
The fact that it is not mentioned there does not make it a bad thing. At least I cannot see any disapproval of its usage. It is clear that the exact geocoordinates are the preferred statement, but if these are unknown, or just as an additional fact I think it is reasonable (and imho common) to use P131. One might argue of course that the image itself has no physical location. But without P131, how actually would you currently find, say, all images of churches in Istanbul? You would have to query for each coordinate whether it belongs to Istanbul or not. To my knowledge there is no affordable way to accomplish that. (But I'd be happy to learn that I am wrong.) -- H005 16:07, 3 August 2020 (UTC)
I disapprove. For one, P7108 is better than P131. But your other argument is a very easy query: all churches in Istanbul. As soon as the Structured Data Query Service is out of beta it will not time out when combined in a search for pictures that depict those churches. Remember that all data in Wikidata are available as well, we don't need to duplicate it here. Ainali (talk) 21:52, 9 August 2020 (UTC)
(quick clarification about wcqs) Beware that a query consistently timing-out on the current beta system is very likely to timeout in production as well esp. the one requiring a lot of data being transferred between commons and wikidata through federation. Hopefully this is not the case with your query as it can be optimized to not time out using query hints. DCausse (WMF) (talk) 11:53, 1 September 2020 (UTC)
The "depicts: Oregon" statement was added in whose summary includes: "Tags: Mobile edit Mobile app edit Android app edit Suggested Edits edit". Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 19:10, 9 August 2020 (UTC)

And now this: https://commons.wikimedia.org/w/index.php?title=File:Syttende_Mai_2019_-_Ballard_-_12.jpg&diff=438007434&oldid=351176238. Just how does a Syttende Mai parade in Seattle "depict" Norway? Pretty roundabout. I will remove that statement, but I am increasingly going from being neutral on "depicts" to seriously hostile. It seems that the vast majority of "depicts" that are added to my photos are either trivial ("depicts people" here) or actively wrong ("depicts Norway" here). That has not been the case at all with edits made by the more longstanding Commons approaches, which are over 90%, maybe over 95%, on the mark. - Jmabel ! talk 16:57, 9 August 2020 (UTC)

It did guess the country right, but should have suggested flag of Norway (Q83149).
Please resist becoming a grumpy old man. Now that we have SPARQL at least it's easier to find these very generic cases so that we can fix them: Query. Multichill (talk) 17:54, 9 August 2020 (UTC)
"Tags: Mobile edit Mobile app edit Android app edit Suggested Edits edit" Can anyone see a pattern, yet? Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 19:11, 9 August 2020 (UTC)
If something isn't working, TURN IT OFF. I've lost count of the number of my images, especially QIs and VIs, that have been bastardised by people who aren't experienced enough to get the subtleties. If my name's on it, sorry, I don't want to look like a dick. And WRT "grumpy old men", yes, I admit that, but I live in Liverpool where most of the population seem to be bent on exposing me to that virus. Doomed! Rodhullandemu (talk) 21:59, 9 August 2020 (UTC)
@Pigsonthewing: @Multichill: I don't think I'm being at "grumpy old man". "Old," yes, "grumpy," not. Rodhullandemu put it well: if my name's on it. I don't want to look like a dick. I have over 50,000 photos on here, and so far my experience of "depicts" is that it's taking up a detectable amount of my time removing false information from my photos. In exchange, it is adding something that seems to me to be largely redundant to categories and usually inferior in terms of the information provided.
If I am being grumpy, maybe it's because three years ago I had what I thought were very coherent suggestions as to how structured data could be integrated with the existing category system and how editing that structured data could be integrated via serialization/deserialization into our wikisource editing, and they weren't merely rejected out of hand but I was told by someone who I am sure has far less data modeling experience than I that I didn't have enough data modeling experience to usefully participate in the discussion. So I shut up and got out of the way for a while but, sorry, at this point I feel that we are getting something right at the lower end of my expectations for what would happen. Adding dates works fine, as does tracking copyrights. If there's really an advantage to having those in a wikibase, fine, though I suspect that the reason they are going fine is that they were already easily scraped from our {{Information}} template, etc. "Depicts" so far seems to me like a student exercise, and not one that would earn an "A". "Captions" seems like something even less than that. - Jmabel ! talk 03:04, 10 August 2020 (UTC)
Also, you write, "It did guess the country right, but should have suggested flag of Norway (Q83149)." No, it presumably identified the flag correctly, and rather than correctly offer that flag as what was depicted, it drew the wrong conclusion that the presence of a Norwegian flag meant Norway was being depicted. By almost the same logic, the presence of a Honda would constitute a depiction of Japan. - Jmabel ! talk 03:12, 10 August 2020 (UTC)
@Jmabel: I didn't make those comments. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 09:24, 10 August 2020 (UTC)
@Pigsonthewing: I'm so sorry! I scanned the comments at a single indent, saw your sig at the end & didn't notice the block wasn't all yours. - Jmabel ! talk 15:53, 10 August 2020 (UTC)

So let's go to a recent one that's less of a mess, but I still don't see the point: https://commons.wikimedia.org/w/index.php?title=File:Boats_on_Cozy_Cove_off_Hunts_Point,_WA_03.jpg&diff=438935500&oldid=434515050.

The depicts say it is depicts a "body of water", "boating", "water", and "sky". Leaving aside the point that anything that depicts a "body of water" also depicts "water", how is this even one eighth as useful as the categories, which tell you it is "Wetherill Nature Preserve", "Lake Washington", "Boats in Washington (state)", "July 2020 in Washington (state)". Nothing actively wrong here, its just that the usefulness of this completely escapes me. - Jmabel ! talk 15:37, 12 August 2020 (UTC)

Actually your argument (also in previous posts) is a bit like claiming "electronic devices are inferior than paper" because you are looking at three example where somebody wrote a bad essay on a computer.
If somebody assigned that image to the categories "Water", "Body of water", "Sky" and "Boating", would you argue against the concept of categories? -- H005 11:25, 14 August 2020 (UTC)
@H005: No. That is a completely bogus analogy. If we lay aside "technical" categories and structurd-data tags (what type of camera was used, what is the copyright status, etc., well over 90% of the categories other people add to my photos are basically on the mark, either things worth noting that I missed or refinements of categories that I used. The structured data that is being added is overwhelmingly either useless (like here) or actively wrong. There is obviously something in how this is being done that is failing to provide useful content in proportion to Commons' longstanding category approach.
If somebody assigned that image to the categories… I'd revert them and tell them on their talk page that they need to learn more before trying to add categories to other people's work. But it almost never happens. Should I be saying something analogous to people who add structured data like this? I still can't really work out the intent of structured data depicts. Since Rodrigo basically told me to go away when I tried to participate in the early discussion of how depicts was to be used, I did just that. But the end product is not impressing me a whit. So back to my original question: Is this really how it's supposed to work? Or are those considered just as bad in the guise of depicts as they would be as categories? - Jmabel ! talk 17:51, 14 August 2020 (UTC)
I agree, it's not the mechanism, it's the application of the AI. People seem to be unquestioningly accepting these "suggestions" and many of them seem to be new users with an insufficient insight to know what's right and what's wrong. Far better to just turn it off and go back to the drawing board. Rodhullandemu (talk) 18:13, 14 August 2020 (UTC)
@Rodhullandemu: do you think I brought this up in the wrong place? I haven't been making much distinction of how useless or actively bad tags get added to my photos, just noting that they do. - Jmabel ! talk 20:44, 14 August 2020 (UTC)
Though I really like structured data, I think we should turn off whatever tool prompts the "Computer-Aided Tagging" until it works better. I have checked many of them, and they are at best useless.
By the way if the structured data could show showed descriptions next to the labels, it would make it much easier to check whether the value is right.-Zolo (talk) 08:53, 16 August 2020 (UTC)
Just dialing back a bit to make sure we are all talking about the same things:
  • Depicts is only one aspect of what Structured allows us to model − such as source, author, copyright etc. − this is worked upon at Commons:Structured data/Modeling)
  • The current guidelines on Depicts is Commons:Depicts, and is quite explicit
    It has also been suggested (for the purposes of good coverage in the search function) to "tag" more generic items that have your specific item as an instance or subclass. In the M87* example above, you could also imagine adding Depicts: supermassive black hole (Q40392) and Depicts: black hole (Q589), because M87* (Q3841190) is an instance of a subclass (supermassive black hole (Q40392)) of black hole (Q589). This would contrast with how we use categories on Commons, where we try to prevent overcategorization. These generic "tags" should not currently be added if more specific depicts statements already exist. If this guideline changes, these more generic items may later be inferred from the relationships in the structured data on Wikidata.
  • Many (don’t know which proportion) depicts: statements are added by the Computer-aided tagging tool, and via “Suggested Edits” in the mobile apps. I can’t say for sure about “Suggested Edits” ; but it’s definitely the case that the Computer-aided tagging tool suggests, and thus encourages, very broad 'tags' (one just needs to drop by Special:SuggestedTags to get a feel for it).
My guess/assumption is that when Jmabel (and others) witness “overwhelmingly useless or wrong” depicts statements, then these come from one of these two tools − this is also, I believe, what Pigsonthewing meant above.
I have high hopes for SDoC in general ; but as I already said this on the Village pump back in February: I think it is increasingly clear that this Computer-aided tagging tool is a net-negative when it comes to the good will of Commons contributors towards SDoC. Quite understandably, many contributors come to equate “Depicts” or even “the whole of SDoC” to “the garbage suggested by some computer vision algorithm and approved by drive-by users”. This is highly damaging, and I wish the WMF would have taken notice by now.
Jean-Fred (talk) 15:25, 17 August 2020 (UTC)
Sorry Joe, I'm not saying you are a grumpy old man (I wouldn't dare), I meant we all risk becoming grumpy old men over time, especially if we have to deal with low quality.
I'm all in favor of trying new things and accepting that it's not perfect at the start, but it should improve over time. I get the feeling that is not really happening when looking at the recent edits. Several things:
How do we measure the quality of these edits? We could look at the edits in a certain time window and how many got removed again. What would be an acceptable percentage?
How do we improve the quality of these edits? I could imagine that we actively start informing users who use this tool about what is correct and not. More a technical improvement would be to only suggest users to work on files that have no depicts (P180) at all. This would at least remove some frustration.
What do we do if we consider the quality too low? We give it some time (at least a couple of months), we measure it, we agree on the minimum quality and that it's too low, what do we do? I really hope it doesn't get to this point, but if the tool is doing more harm than good, we're better of disabling it.
Multichill (talk) 16:07, 23 August 2020 (UTC)
@Multichill: I'm going to paraphrase what I said above: if somebody assigned equivalent categories to my photos, I'd revert them and tell them on their talk page that they need to learn more before trying to add categories to other people's work. Is it acceptable to do an equivalent here? Is someone already trying to do that? But most important: are these super-vague tags even considered wrong by the people behind Structured Data, or do they consider it a success to note that a picture depicts people, or that the sky is visible? - Jmabel ! talk 16:44, 23 August 2020 (UTC)
Wow. this tops it all. I agree "suggested edits" is not working in its current form at all. There should at least be an introduction shown directly on the edited page when a user is about to edit structured data for the first time. 1234qwer1234qwer4 (talk) 11:33, 24 August 2020 (UTC)
@1234qwer1234qwer4: “Wow” is enough, no need for all the words you wrote after  — Ltrlg (talk), 15:42, 24 August 2020 (UTC)
@Ltrlg: i am sorry. yes, wow. 1234qwer1234qwer4 (talk) 16:58, 24 August 2020 (UTC)

I concur that the problem apparently is the AI tool that encourages users to apply over-generic tags such as "water" or "blue sky". I believe it should be deactivated unless it has been re-worked. Where is the place to discuss this? The structured data approach still works fine to me, just the tagging tool doesn't. -- H005 17:26, 24 August 2020 (UTC)

I am still waiting for someone to answer my original question: are these "depicts" tags like "sky" and "person" considered a success or a failure? If somebody assigned equivalent categories to my photos, I'd revert them and tell them on their talk page that they need to learn more before trying to add categories to other people's work. Is it acceptable to do an equivalent here? - Jmabel ! talk 00:22, 25 August 2020 (UTC)

May be we should just establish a policy governing this.--Ymblanter (talk) 11:18, 25 August 2020 (UTC)
I would appreciate this, it would be very helpful. Abittaker (WMF) (talk) 20:19, 26 August 2020 (UTC)
Failure.
Come on, let's get rid of this shitty system. It's not working, it damages Commons, and the "temporary" trial of captions and depicts statements being forced into the upload wizard with no understanding of workflow for newbies, needs ripping out. --Fæ (talk) 11:21, 25 August 2020 (UTC)
@Fæ: I know you and I consider them bad, but my question is whether the people who built this, are advocating it, and are behind this gamified way of getting people to add tags consider it success or failure when things like this are added to "depicts". I still don't understand their intent (especially because I was told I didn't understand enough to participate in the discussion of how "depicts" would be used), and I'm trying to use this question to clarify that. And I am very frustrated that it appears none of them will give me a straight answer. - Jmabel ! talk 14:54, 25 August 2020 (UTC)
@Jmabel: As far as I remember, this is the kind of decision they wanted the community to make. WMF/WMDE provide the tools, we decide how to use them. So maybe it's time to head over to COM:VPP, have an RfC with a vote and then ask them to stop this nonsense. If it's not possible to deduce from depicts=Airbus A380 (Q5830) that what we see here is an aircraft, the whole idea of having our meta data structured has pretty much failed. I was hoping we could one day replace intersection categories like Category:Aircraft with 4 jet engines with a simple StructuredData-based query that would show me all pictures that depict an aircraft that is described at wikidata as having 4 engines of a type that is described at wikidata as being jet engines. I don't know if that's what people were thinking they were promising, but that's the message I received back then. --El Grafo (talk) 15:18, 25 August 2020 (UTC)
You don't have to ask "them", it's not their call. We say it's a stinky failure, so per El Grafo, RFC time because, apparently, the WMF does not have to gain consensus to damage the project but the community does have to gain consensus to repair the damage. --Fæ (talk) 16:12, 25 August 2020 (UTC)

Hello, everyone. Here are some clarifying points about what systems currently exist for adding depicts statements and their usage patterns. As of now, there are four vectors for users to add depicts statements via WMF-built tools - a.) manually via the Structured Data interface either on UploadWizard or File Pages b.) machine-vision suggestions from Special:SuggestedTags c.) manual edits via Special:SuggestedTags and d.) manual edits via the Android app. Three of those four functions are completely manual - the interfaces simply provide users with the ability to add their own values, with no suggestions at all other than the usual auto-suggest dropdown that appears in the search box when the user starts typing a term (and this is powered by Wikidata). Although Special:SuggestedTags initially launched with machine-vision suggestions only, after implementing the manual addition functionality months ago, much of the activity from the Special:SuggestedTags feature is actually manually done now. Roughly speaking, depending on the day, anywhere from a quarter to more than half of the current edits from SuggestedTags are actually completely manual additions, and it is common for users to do a mix of selecting suggested items as well as adding their own.

Of the four methodologies WMF has built, the Android app feature is the newest and the one that most commonly leads to users adding data for files that are not their own uploads (all four methods allow this to some degree, but the features on Commons are mostly used on the user’s own files). The Android team plans to refine their feature based on feedback and usage. RIsler (WMF) (talk) 21:39, 25 August 2020 (UTC)

I'm getting the impression here that you haven't actually read this thread. The problem isn't really the mechanism, the problem is (a) the AI that suggests overly-broad and thus unhelpful additions, which make a mockery of our careful and curated category system, although that itself is still a work n progress, and is always going to be. What happpens at the moment is not categorisation but tagging, which by definition lacks structure. That's an epistemologically invalid position to adopt. Perhaps we could talk to someone who actually understands how data should work and its relationships to knowledge? Forgive me if I seem unduly harsh, but sometimes I feel that WMF are a clueless bunch of amateurs when they stray into things they simply don't understand, for example getting the T&S balance right. #epicfail Rodhullandemu (talk) 21:56, 25 August 2020 (UTC)

I am still waiting for someone to answer my original question, and RIsler (WMF) I believe your remark above does noting to answer it (but please point out if you answered and I missed something): are these "depicts" tags like "sky" and "person" considered a success or a failure? If somebody assigned equivalent categories to my photos, I'd revert them and tell them on their talk page that they need to learn more before trying to add categories to other people's work. Is it acceptable to do an equivalent here?

I reckon no one can definitely answer that question. Technically, it is correct to apply the "sky" and "person" tags, as those are depicted and there is no more specific (subclass/instance) item on Wikidata that could be used. But, on the other hand, it is questionable if this adds any value. Of course it might be possible that someone would be searching for photos of a building where you also see the sky, or persons in front, or whatever. But these are rare cases and usually it is just unnecessary and confusing to apply these tags, unless it is the main intention of the image to show the sky etc.
Problem is: there is no strict rule when such a tag is reasonable. But these grey zones have ever been a problem with categories, too. And we should overcome them by making users aware of the problem, i.e. being careful with applying too common tags such as sky, flower, house, car, if they are not a focus/intent of the image. -- H005 05:19, 26 August 2020 (UTC)
As I wrote above, if someone were routinely adding categories like this, we would tell them to stop. If someone had a gamified tool that routinely encouraged people to add categories like this, we would certainly demand that the tool be shut down. But depicts is not categories. - Jmabel ! talk 16:05, 26 August 2020 (UTC)

I'm still waiting for RIsler (WMF) - or anyone - to answer the points raised in February, in Commons:Village pump/Archive/2020/02#Misplaced invitation to "tag" images. Those being the points he said were not being ignored; and including (but not only) "requests to show where there is consensus for the tool to operate, or to use depicts statements in the manner it is [and] requests to explain how the tool, or the invitation to tag, can be turned off." Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 08:25, 26 August 2020 (UTC)

How about this? I can't make head or tail of it. I'm guessing it should be reverted, but separate from that: why would this edit have been "suggested"? - Jmabel ! talk 00:01, 1 September 2020 (UTC)

  • @Jmabel: I'm pretty sure not all of the "suggested edits" are actually suggested. See my Wikidata edits, in which I've just manually added some descriptions but which were tagged as "suggested" for some reason (I guess the tag is applied to any mobile Wikidata edit currently). 1234qwer1234qwer4 (talk) 10:01, 1 September 2020 (UTC)
    • So we have a semi-automated tool that gives an actively misleading edit summary. Shouldn't that be changed? - Jmabel ! talk 14:01, 1 September 2020 (UTC)
      Arguably, the tool does submit the file for consideration of the user, which is “suggested” ? Agree it’s not very clear (especially as it’s the same as the CV tool where [at least some] values are definitely suggested by the tool. Jean-Fred (talk) 16:26, 1 September 2020 (UTC)
Hello @Jmabel: . On the mobile apps, there are currently no machine vision features so "Suggested Edit" is as @Jean-Frédéric: mentioned above - it refers to a file/item that the app suggested as a candidate for manual editing. The machine vision/computer aided tagging feature will only have edits tagged with "Computer-Aided Tagging" and are currently only available via web interface (desktop or mobile). RIsler (WMF) (talk) 17:01, 1 September 2020 (UTC)
Then can we reword it so it doesn't give a misleading edit summary? Explaining here is good, but it's not like every Commons editor is going to learn the subtleties of a bunch of different tools to understand what each one actually does and correctly decipher a bad edit summary. - Jmabel ! talk 00:27, 2 September 2020 (UTC)
@Jmabel: There's nothing wrong with the edit summary; if you mean the tag, we could just create a page like Commons:Suggested Edits explaining the feature and link it from the label of the tag apps-suggested-edits. 1234qwer1234qwer4 (talk) 10:30, 2 September 2020 (UTC)
@1234qwer1234qwer4 and Jmabel: That was a good suggestion − I started a small stub at Commons:Suggested Edits and linked it from the tag. Jean-Fred (talk) 15:41, 7 September 2020 (UTC)
@Jean-Frédéric: Thank you. I suggest we also link Commons:Structured data/Computer-aided tagging from the computer-aided-tagging tag label, as you did on MediaWiki:Tag-apps-suggested-edits with the suggested edits tag. 𝟙𝟤𝟯𝟺𝐪𝑤𝒆𝓇𝟷𝟮𝟥𝟜𝓺𝔴𝕖𝖗𝟰 (𝗍𝗮𝘭𝙠) 19:42, 7 September 2020 (UTC)

Being relatively new in this corner of the wikiverse, I'm somewhat suprised by the vitriol in this thread. Calling people from WMF amateurs is entirely unwarranted hostility. It is amusingly wrong to boot, considering the definition of amateur (as opposed to professional) is "working without being paid", and they happen to be the only one in this thread that do not fit that definition. I also find the notion of "my files" bewildering, but maybe that's just because I mostly work on texts, where you are lucky when you can still recognize some phrase you enjoyed writing after a year or two have passed. The nature of photographs as atomic creative works naturally leads to some differences in their treatment, and these differences are reflected (and sometimes enhanced) by policies that give far more deferences to creators than for other media. But even though that means this project probably works very well for your free backup needs, it isn't flickr or Google Photos, and licensing your works implies giving up control to a certain degree.

I also seem to remember seeing some borderline offensive comments directed at someone suspected of having limited knowledge of English when I first read this thread a few days ago. I can no longer find that, so maybe someone's better nature got the upper hand after cooling down a bit. But I do want to point out that occasionally having to try to accomodate non-native-level speakers comes with the privilege of having a project that is notionally language-agnostic, but in practice happens to accomoate your language proficiency rather exclusively.

As to the actual matter: the number of actively wrong tags seems to be rather low, even though I do not doubt your ability to find examples confirming your believe that everything is just terrible. I don't see a reason why these particular snippets of data should be different from all other snippets of data that can be edited here and in other projects. That openness does allow vandalism and incompetence, but isn't exactly a new problem. If, in fact, tags currently are more problematic than what you're used to, the most obvious explanation would seem to be that this system is successful in broadening the ability to participate, whereas before open participation was just marketing, with obscurity of UI and a bewildering set of informal practices fortifying individual fiefdoms against actual participation by anyone but the most motivated newcomers with an unusual amount of free time on their hands. The vast majority of these tags, then, seem to be (as already pointed out above) of the "somewhat broad, but not wrong" variety. Contrary to someone above, I find it rather believable that someone might be looking for an image of, say, "a castle on a coast". Considering the mechanics of training such AI systems, these tags must be the result of some (manually annotated) training set of images with those tags occuring with roughly the same frequency as they are being suggested now. Browsing flickr, for example, you will indeed see taggings that seem rougly in line with what this system is currently suggesting to users.

That does not make it equivalent to your fancy categories, of course. But true-if-overbroad tags do get to claim to be better than nothing. I am sure it would be possible to adjust the system and have it make some more specific suggestions. Those would obviously imply a higher risk of being wrong, and it would be interesting to learn if people are capable of continuing to make the right calls under those circumstances. Maybe it could work like an actual game or duolingo, becoming more difficult when you get a lot right. Ideally, it would notice when someone is excellent at identifying insects or medieval churches, and adjust accordingly.

That thought leads directly to how the system is useful even in its current incarnation: it serves as a sort of first-level triage, categorising incoming images with little metadata into broad categories. That information then allows some tretrapodologist to jump into the ball pit that is an endless stream of images of x-legged (I'm not going to look up ancient greek here) creatures to categorize, without ever having to interact with those yukky quadropodologists and their sorry excuse for a superphylum. Given an image of some old building and no further information, only a tiny majority of people will be able to correctly identify it, or its localtion, era, and architectural style. To somehow get that picture to one of the three people who know where in Vietnam it is, requires first identifying it as being roughly typical for that country. That task is slightly easier, thankfully, and so is the previous level of placing it somewhere in South-East Asia. Conceptually, I am somewhat convinced that this is the exact mechanism long being used here, where laypeople would probably categorize something as 'a painting', and watchful eyes lurking in the shadows of that category quickly shoving it into, say, expressionism. --Matthias Winkelmann (talk) 06:38, 2 September 2020 (UTC)

  • @Matthias Winkelmann: Your own «vitriol» and «offensive comments» are noted. Please, then, do embrace gamified edits by uninvolved, uninvested human drive-by editors that leave all the strategic work to AIs and funnel both donations and decision-making to the tip of the (“professional”) pyramid, soon to be unemcumbered from those pesky peddelers of «informal practices» and «fancy categories». Your wall-of-text shows you have a basic grasp of what might be going on (atomic contributions v. collaborative text creation et c.) and therefore how much your slings and arrows spectacularly miss cannot be due to lack of intelligence, but only either to a huge amount of bad faith, or to disinterested contrarianism, or to mere fanboy shilling. As for your denouncing of COM:OWN, you’re again missing the mark by the whole 180° — understandably, again, since you presume to get a good grasp of the situation after a cursory read. Concerning your quip in supposed defense of people «suspected of having limited knowledge of English», põe-te neste a ver se eu deixo. -- Tuválkin ✉ ✇ 13:00, 2 September 2020 (UTC)
@Tuvalkin: Yes, I freely admit to have limited knowledge of the history here. That is probably why this discussion seemed abnormal to me, whereas regulars will no longer notice these things, in the way that fish don't have a concept of water. I figured there must be psychological dynamics at play here that go beyond the immediate issue, since that seems more likely than the alternative, a staggering high prevalence of sociopathy. Point being: while such old baggage with never be forgotten, it is irrelevant for the specific issue. Allowing it to continue poissoning your interactions is entirely unproductive, To the extend that you are apparently invested in some conflict of WMF vs. You, you are liable to extend your animosity even to new people who are entirely blameless for whatever happened before they joined the WMF. These will, over time, obviousy reciprocate in kind, and then it's helical or spiral stairs or staircases downwards from there on. So I was hoping people might reflect on their feelings for just a second, and maybe considering to readjust their default to slightly less hostily. As to me being a "fanboy": I would not use that label myself, and since we already agreed that I have had almost no previous exposure to any of the parties involved, it should maybe give you pause to see how quickly you managed to turn a naive mind into someome you consider a "fanboy" of your opposition. --Matthias Winkelmann (talk) 22:39, 3 September 2020 (UTC)
  • @Matthias Winkelmann: On my own photos, the number of absolutely bad depicts that have been added (totally unrelated; "close but no cigar") strongly outweighs the number that are genuinely on the mark (accurate and non-trivial), with the "basically useless" -- sky, people, tree -- outnumbering either. - Jmabel ! talk 16:06, 2 September 2020 (UTC)
  • @Matthias Winkelmann: ditto for what JMabel said. Also, "quickly shoving it into, say, expressionism" really does demonstrate your relative newness. Had we a plethora of experts with the eyes of Argus, this might happen "eventually", but it's not guaranteed. So don't you think it's far better not to make a mess in the first place, rather than hoping someone will chance by and clean it up later? Rodhullandemu (talk) 16:33, 2 September 2020 (UTC)
@Rodhullandemu: Your interpretation of "quickly" is somewhat bad-faithy. It was supposed to be high praise for the work of these experts. And I still don't see how uncategorized images are less of a mess than having them in even overly broad categories. --Matthias Winkelmann (talk) 22:39, 3 September 2020 (UTC)
No. In fact, absolutely not. I cut my teeth on this project here for over two years, nearly three, curating categories of Scotland and trying to rationalise the structures that existed when I started into something usable. That is a model that many others have followed since. So, it works. What does not work is later additions that ignore that structure when it is entirely possible, since I have in a samall way made it so, that it requires little intellectual effort to assign new iages to the existing system. And that is where this so-called "AI" sytems fails, because although it may know something about images, it knows nothing at all about the context into which they should be placed. You're expecting an awful, no terrible, and unreasonable commitment, from volunteers who have rhe choice as to where and how they direct their talents and efforts. I've said it before, and I'll say it again: this is a failure and should have been strangled at birth. There are far to many people coming here with ideas that, as it turns out, cause more harm than they benefit Commons. Rodhullandemu (talk) 23:00, 3 September 2020 (UTC)
@Matthias Winkelmann: I would have to agree that there has been more hostility than there should have been (unfortunately, I get too used to it to notice it anymore :-( ).
I think you make some good points. Part of the issue, I think, is that it’s not about “uncategorized images are assigned overly-broad tags − and from there can tagged more accurately” (which means it’s both “better than before” and “first step towards even better”) − but « really precisely categorized pictures are being assigned overly-broad tags”.
I have, personally, always been a supporter of SDoC ; what personally really bothers me with the direction depicts is going towards (in particular with the CV tool) is something I have not really voiced before, and something different from what you have addressed.
The SDoC team seems to be of the opinion that making a file tagged with German Shepherd (Q38280) returned for searches for "dog" or "pet" is too difficult / impossible (because, as I understand it, the Wikidata ontology is too messy/unwieldy/smth). Hence, the solution would be to have this file tagged with German Shepherd (Q38280) and dog (Q144) and pet (Q39201) and so on and so forth − and the SDoC would provide tooling (including, explicitly, the CV tool) to help with the backlog of tags to add.
I have a hard time expressing how much this bugs me. First off, it is contradictory with Wikimedia Commons practice (we categorize to the most precise category), as well as the policy we started at COM:DEPICTS. But it is also contradictory to long-standing Wikidata practice − and, really, to the core concept of a graph database (Westminster Abbey (Q5933) does not have instance of (P31): cathedral (Q2977)). It is also somewhat contradictory to what was "promised"; at least implicitly − I’m all for rescaling expectations when hitting reality but personally, I feel borderline misled − and I think very few in either the Commons or Wikidata community would have been enthusiastic at this perspective.
(Also, I have yet to read a proper explanation of it, rather than one-off remarks here and there on-wiki or mailing-list which often merely assert “we tried it, discussed it, does not work, never will”. There have been interesting suggestions by @Jheald: in particular about it (the “shadow-tags” concept), that do not seem to have been really considered (or if it has, it was never properly summarized). There also seem to be some level of misunderstanding in the discussions − when SPARQL is being dismissed because the WDQS would alledgedly only get “a few dozen queries per day” while the actual figure is several million…)
But not only that, the whole idea strikes me as “sending a human to do a machine's job” − we can’t figure out a technical way to leverage Wikidata’s ontology to create a search-engine that returns german shepherds when looking for dogs, so we’re going to ask thousands of volunteers to do millions of manual edits for that purpose. I find that reasoning deeply upsetting, because, frankly, my time is worth more than that. Sure, the WMF should be mindful of its engineering resources and not throw billions at an impossible endeavor ; but it also, to a degree, has “volunteer resources” − in the sense that the WMF, while not “controlling editors”, is capable to some level of “nudging volunteers towards a particular task”. It’s an incredible power, which also not be used lightly − and I don’t feel it’s being wisely used in this particular case.
Jean-Fred (talk) 11:05, 4 September 2020 (UTC)
Well put Jean-Fred and it reflects my view. Thanks for taking the time to writing it down.
A long time ago I wrote User:Multichill/Next generation categories:
I would rather see development time spend on better search than on putting data in (like with the CV). Sure the subclass tree isn't always as clean, but that's search, search is messy. If no plans exist to make this work in search we might as well stop importing data now because than it's just a giant waste of time. Multichill (talk) 15:33, 4 September 2020 (UTC)
I concur that it is about "really precisely categorized pictures are being assigned overly-broad tags." Also, that we have not discussed the value (positive or negative) of trivial tags: e.g. "sky" on a landscape, "people" on a downtown street scene. - Jmabel ! talk 15:37, 4 September 2020 (UTC)
I also agree with Jean-Fred who very eloquently and precisely described my feelings on the matter. Current software limitations should not drive editorial decisions. If current search can not identify German Shepherd (Q38280) as a dog (Q144), so be it. Futher searches perhaps will and we can wait. I also share Matthias Winkelmann's concerns about the tone of some earlier conversations. WMF team have my full respect and appriciation, our priorities do not always allign at 100%, but I think overall they are doing fine job. Commons:Structured data/Computer-aided tagging (CV tool) would be a great tool for uncategorized images. For categorized images I would combine it with a tool which can also understand some of our category structure and propose tags based on categories as well. Files in Category:Albert_Einstein should propose Albert Einstein (Q937) in addition to human (Q5) as one of the depict options, if CV determined there are people in the image. --Jarekt (talk) 17:46, 4 September 2020 (UTC)
I appreciate all the discussion that's gone on in this thread. There's some good points raised that have been discussed and some new ones to address about the purpose of computer-aided tagging in relation to depicts and surfacing content on Commons. This is a Friday afternoon going into a holiday weekend in the States, so it'll be sometime next week before I can get together some responses. There's also upcoming updates to MediaSearch and the Computer-aided Tagging information page in general that should help tie these topics together throughout the month. Keegan (WMF) (talk) 19:35, 4 September 2020 (UTC)
Small update, I met with Ramsey and we talked through some ideas he has to work on getting depicts/suggests to be more considerate of existing categories. There are a few options, including category/entity matching from Wikidata, leaving out images with (x) number of categories already applied, plus some other ideas. We'll work some of this into the forthcoming stats-and-usage update to the CAT page. The challenge isn't so much in finding potential solutions as it will be finding resources to build the potential solutions–budgeting and allocation aren't normal right now, so we'll have to see what can happen here. Keegan (WMF) (talk) 18:13, 11 September 2020 (UTC)
@Jean-Frédéric: A quick update, since the opinion you referred to is a few years old (back when everything was still conceptual and it wasn't even possible to add statements yet.) We are actually still trying to solve this, but I can at least try to explain some of the things that make it hard to solve.
First, technical limitations: in order to find something with a "German Shepherd dog" statement when you search for "dog", "dog" needs to be indexed for that file as well. As soon as the "German Shepherd dog" statement is added, we could traverse certain properties ("instance of", "subclass of", ...) and add all of the related things to the search index, but there are an awful lot of those (most of which will never end up being searched for anyway, like "Wikidata metaclass" or "organisms known by a particular common name") or could cause confusion. Of course, we'd also need to index all labels & aliases in all languages for all of those things. And if one of those changes, or a relationship changes, all relevant files would need to be updated. TL;DR: too much data, too much updates - not feasible.
We're currently investigating an alternative where we wouldn't index things ahead of time, but look them up on-demand (find matching Wikidata entities for "dog", traverse "instance of", "subclass of" ... downward until we find "German Sherpherd dog" along with many others, find all files that have one of those entities). Based on a few quick tests, those results seem quite promising. But that is a lot of computationally intensive things that all have to happen sequentially, so it might very well end up not working out...
Second, ontology challenges. I've already mentioned some above, but there are a few more:
- Some entity trees are massive. E.g. "passenger car"'s "instance of" and "subclass of" trees have tens of thousands of car models. This also feeds into technical limitations again.
- Some trees are misleading: "German Shepherd dog" is "subclass of" "dog", which is "subclass of" "pet". Except that a file of a "police dog" (Q39235) or "working dog" (Q1806324) might also depict:"German Shepherd dog". When searching for "pet", a "police dog" should not show up, but if "German Shepherd dog" infers that it is a pet.
As mentioned already, "search is messy," and even if we do manage to get something like this going, it will not be without flaws, and we can only hope that the benefits generally outweigh those.
Sidenote/question: while I think we all agree that tagging random images with "sky" is not desirable, how do people feel about an image of an electric car charging receiving the statement "electric car" in addition to "Toyota Prius c"? Or about a photo of a police dog getting both "German Shepherd dog" and "police dog" (which has no direct relationship to the former.) Or similarly, another photo of a "German Shepherd dog" also receiving a "pet" statement (even though that one does have a direct relationship)?
I wonder whether the relationships within Wikidata even matter at all in this discussion: are generic tags simply a problem when/because they're useless (like "sky"), or also when they aren't useless but could be inferred from another (more detailed) statement (e.g. "electric car" vs "Toyota Prius c".) I'm curious to understand how much grey area people are ok with, and where (if at all) we can draw a line between helpful "emphasis" of potentially inferable data and overly broad duplication of data. Mmullie (WMF) (talk) 15:01, 30 September 2020 (UTC)

And another: https://commons.wikimedia.org/w/index.php?title=File:Washington_State_Capitol_-_looking_west_down_path_to_Temple_of_Justice.jpg&diff=489452628&oldid=427854252 Useful or not? None of these are inaccurate, but they bury the lede (that the picture shows the Washington State Temple of Justice). Surely that is more important than it depicting "sky". - Jmabel ! talk 14:51, 14 October 2020 (UTC)

Future direction of depicts and search?

@RIsler (WMF) and CBogen (WMF): I would really appreciate it if you can respond to the points raised by JeanFred. What direction are you going to take? What can and can't we expect in the future? Today I discovered File:2020 Structured Data Across Wikimedia proposal.pdf. Is this the plan? Multichill (talk) 19:25, 28 September 2020 (UTC)

Hi. @Multichill: Keegan, Carly, and I are working on a new page/update that should address all of the above. RIsler (WMF) (talk) 22:21, 29 September 2020 (UTC)
I've posted an update page specific to this topic at Commons:Structured_data/Media_search/Future–it's on its own page so that it can have dedicated discussion space. Keegan (WMF) (talk) 18:34, 5 October 2020 (UTC)

Notifications/watchlist improvements related to SDoC work

Hi @RIsler (WMF) and Keegan (WMF): ,

I have noticed a trend where users are getting annoyed at Structured Data edits flooding their watchlists/inboxes. Some of the relevant Phabricator tasks have been mentioned in various places but I thought I would recap them here:

  • phab:T262750 − Allow to exclude bots from watchlist notifications (ie e-mail) & phab:T247433 − Disable watchlist notifications for bot edits made on Commons structured data
  • phab:T174349 − Have a way to exclude Tagged edits (in watchlist)
  • phab:T209589 − Have a way to exclude a bot-flagged account (in watchlist)
  • phab:T265573 − Allow to mark Structured Data on Commons edits as minor

I imagine that these may not be within the remit of the SDoC team, but is there any chance that they could be prioritized? It is clear to me that these issues are eating up quite some good will from Commons contributors towards SDoC, and should be addressed. For some, that ship may have sailed (by the time T262750 is looked into, MultichillBot and SchlurcherBot might be done with editing all of Commons :-þ) but others still stand: as new SDoC tools are developed/get more traction), their general acceptance is very much linked to not pissing off other contributors (especially long-time contributors).

Thanks, Jean-Fred (talk) 08:24, 15 October 2020 (UTC)

Hello @Jean-Frédéric: ,
Thanks for assembling this all in one place; I see T265573 is newly authored by you, that's an interesting idea for toolbuilders. Ramsey and I have talked about the other tickets in the past, as you say they're outside of our remit but we're certainly curious to see where we can help beyond the existing ability to turn off bots in the watchlist and mute specific bot notifications through preferences. I'm interested in your new ticket and what toolbuilders have to contribute to the idea, it could be quite useful outside of the bot area of work. Keegan (WMF) (talk) 20:46, 15 October 2020 (UTC)
@Jean-Frédéric: , thanks for raising the issue again. I also think that control of the watchlists is very important to keep Commons community exited about SDC in a positive way. I had a lot of trouble with my bot using QuickStatements which somehow was not marking my edits as bot edits. That was an annoyance to a lot of people and I stopped adding SDC using QS. Other tools might have similar issues. --Jarekt (talk) 02:23, 16 October 2020 (UTC)

Extension

Is the extension that powers Structured data on Commons WikibaseMediaInfo? If so, may I add this information on the page? I had to find it through Special:Version because the link is not here, unless I'm mistaken. -Luk3 (talk) 11:20, 18 October 2020 (UTC)

By all means, add the link wherever you might find appropriate. Keegan (WMF) (talk) 17:42, 19 October 2020 (UTC)

Not doing the same thing twice

Just now, while categorizing it, I noticed that this photo was has wrong gelocation in its EXIF metadata, about 4 km off to the west, which is displayed in Flickr and got extracted into Commons. I removed the faulty data in the filepage’s wikitext and, on a whim, I did the same in the file’s Structured Data. But frankly, I cannot see myself doing this kind of duplicated work for each and every file I curate. -- Tuválkin ✉ ✇ 00:33, 22 September 2020 (UTC)

Perhaps eventually a bot will be deleting redundant coords where they agree, but instead adding warnings where they don't. Jim.henderson (talk) 22:27, 5 October 2020 (UTC)
  • Deleting the wikitext I entered to syphon it away to feed a Google Code vanity project? It better not. -- Tuválkin ✉ ✇ 03:01, 15 October 2020 (UTC)
    • This is a familiar thing. We who are engaged in mass photography know that the Foundation still has old thinking, which is to process one file separately, while we would need them to start thinking about processing groups of files at once.--Juandev (talk) 10:30, 20 October 2020 (UTC)

Medium

Hi All,

Could you please share your thoughts on this property proposal? I think having a property like this could make searching and categorising images somewhat easier. --Adam Harangozó (talk) 22:34, 20 October 2020 (UTC)

Structured copyright and licensing for search indexing

The development team is making progress on building new features for Special:MediaSearch. Among these are the the filter for copyright and licensing. Talking it over with search team engineers, it's estimated that if we try to index text-based copyright and license data, it will be at least two to three months, followed by the work to get that information into search. In addition, and perhaps most importantly, that information would be mostly in English. If the search team were able to rely on structured copyright and licensing, the information could be provided in every language available, immediately. Currently there is one bot, SchlurcherBot, doing the work to add structured copyright and licensing where available to files with the {{Own}} template only.

Following this, the search team was wondering if there is any interest from the community in speeding up the process of adding copyright and licensing information to files. Specifically, could bot throttles be increased? Could more bots be run to help with this task? Another idea or ideas? The team is interested to hear any thoughts on how to get this accomplished more efficiently and in a multilingual manner. Keegan (WMF) (talk) 20:35, 20 July 2020 (UTC)

@Keegan (WMF): What "text-based" copyright and licensing data are you trying to index? Pretty much all copyright information is applied by templates, and those templates generate machine-readable data that are exposed through the Mediawiki API (action=query&prop=imageinfo&iiprop=iiextmetadata). I would expect that indexing those would be reasonably simple. The templates also generate category memberships, and the categories are linked to Wikidata, so you can find files with a given licence by two steps from Wikidata. Is there some more complicated information that you're looking for that's not captured by our templates, and if so how do you expect bots to find it? --bjh21 (talk) 23:45, 20 July 2020 (UTC)
Templates are text-based–they are wikitext markup. What you're describing is what we're trying to avoid because a) scraping and indexing all the files will take months, as mentioned, b) the results will mostly be in English, and c) it's wholly inefficient when the goal in the long-run is to have this information in structured data anyway. Essentially, doing the work twice and the first time doing it in a very complicated way that takes a lot of time. Porting the data in to Wikibase is much more efficient and achieves the goal of multilingualism in search, and is something that is technically possible immediately once the information is there. Keegan (WMF) (talk) 19:13, 21 July 2020 (UTC)
@Keegan (WMF): I think adding SDC properties will take a while. I am working since March (using QuickStatements) on adding Wikimedia VRTS ticket number (P6305) and digital representation of (P6243) to files while keeping track of what is done and what still needs to be done using categories, since we still do not have SPARQL query endpoint (phabricator:T221921). However phabricator:T237991 bug (SDC changes do not trigger page refresh) make things quit slow and phabricator:T247433 issue (bot edits are not marked as ad bot edits) piss off a lot of people. I think adding SDC during upload (phabricator:T245861) would go a long way so this is not an open ended problem. Also maybe approach taken by SchlurcherBot and User:BotMultichill to write everything in python might be a better way at this point. --Jarekt (talk) 02:15, 21 July 2020 (UTC)
I certainly see the potential in at least having two bots running, with perhaps more permissive throttling, to achieve the goal. I'm curious as to who else might be interested in automating the task. Keegan (WMF) (talk) 19:13, 21 July 2020 (UTC)
I agree. I guess if we clone the bot and find multiple volunteers to run it, we could complete the process much faster. I might look into it. --Jarekt (talk) 19:31, 21 July 2020 (UTC)

@Schlurcher: , What do you think? Would it be possible to increase the throttle speed of your bot? Is the process something that can be easily cloned and run by others? --Jarekt (talk) 14:36, 22 July 2020 (UTC)

@Keegan (WMF) and Jarekt: Are you asking me to run my bot faster or slower? I'm confused. Throttling is normally associated with slower edits, but likewise you are asking for multiple bots. I'm currently trying to stay within my approved 30 edits per minute (maybe going to 50). With the current rate, it will take indeed forever to achieve this task. The code I use runs completely un-throttled and the edit rate is mainly defined by the number of parallel workers that I activate. So it is fairly easy to scale, I would say. --Schlurcher (talk) 15:09, 22 July 2020 (UTC)
Throttling is about keeping the site up and running. It isn’t to protect the bots or to make contributions lists shorter, but to prevent the servers from crashing. Both lowering the throttle and introducing more bots has the same risks, so before doing anything, I’d like to see an okay from whoever is responsible at WMF for keeping Commons up. —Tacsipacsi (talk) 15:33, 22 July 2020 (UTC)
@Tacsipacsi: oh absolutely. Right now we're in the sharing ideas phase, before implementing anything that involves bots and throttles I'd expect both the bot approvals to be consulted here and site operations to be consulted on the Foundation's side. There are no plans to just run off and go do stuff. Keegan (WMF) (talk) 16:57, 22 July 2020 (UTC)
Schlurcher, My understanding was that WMF team represented here by Keegan (WMF) was hoping to begin to rely more on SDC data. At the current rate it might be couple years before your bot will manage to visit most pages on Commons and we were brainstorming to see how can we get there faster. One idea was to allow higher rare of edits by your bot and other idea to clone it and run it by multiple people. My preference would be option #1, assuming that higher edit speeds do not affect performance of the site, as pointed out by Tacsipacsi. I usually try to follow the advice of en:Wikipedia:Don't worry about performance essay, but perhaps we should be careful. I wonder how much of an impact on a site it would be to run Schlurcher's bot at lets say 10 times the current speed and if there is some site performance indicator which can be monitored in order to know if it is safe to increase editing speed at given time or not. There are links to a lot of resources at mw:Wikimedia Performance Team page. --Jarekt (talk) 17:09, 22 July 2020 (UTC)
@Jarekt and Keegan (WMF): I have now performed some limited tests. If I pull in resources from Microsoft Azure, I can achieve significant improvements in bot speed. Even with the 12 month free 1vCPU machine that Microsoft currently offers, I am able to achieve ~100 edits per minute, which would more then double/tripple my current rate. So if there is interest to increase bot speed there are definitely options on my end. The question would be how we plan to proceed which this. I'm willing to consult the bot approvers here to site alignment. Please let me know. --Schlurcher (talk) 13:25, 29 July 2020 (UTC)
Schlurcher at the rate of ~100 edits per minute you might be able to do about 4M per month, so it would take over a year to visit most of the files on Commons. It might be unclear how to model some files, but if we could get most of "own works" using {{Information}} template that would be great. Keegan (WMF) you should check with whoever is responsible for making sure the commons website is not experiencing performance issues, what rate is acceptable for this unique task. To provide some background, according to [4] in 2020 we are doing 6-9M edits per month and in Febuary 2020 we had 6M bot edits [5]. In that light 4M/month would be significant fraction, but it is unclear how many edits per month is considered safe. That way, if we want to do this "by the book", we would go back to bot approvers and ask for increased edit speed, we can state that such speed should not affect performance. --Jarekt (talk) 16:31, 29 July 2020 (UTC)
@Jarekt, Schlurcher, and Multichill: I've exchanged emails with a database administrator. Basically, it's fine to slowly scale up bot operations–keeping API:Etiquette in mind, specifically Manual:Maxlag_parameter of course–lowering the throttle first, then add a clone, then another, etc. and we'll see how it goes. Overall the addition of the revisions to the table isn't a problem so much as the potential for the bots running into replication lag at that scale. So yeah, we're okay to slowly ramp up from the operations side whenever the community is ready, I'll just need to let them know so they can keep an eye on things. Keegan (WMF) (talk) 17:27, 5 August 2020 (UTC)
Keegan (WMF) thank you for checking on this and thanks for links for API:Etiquette. I never run into that page and it is a good to know. If User:Schlurcher's bot can handle more clear cases of own work with CC license, which are probably majority of the photographs on Commons, I could start looking into modeling of PD works and other more messy cases. By the way I was recently working on adding Wikimedia VRTS ticket number (P6305) statements to 1M files with {{PermissionOTRS}} template, which should be mostly done by now, so I might switch to copyright statements. --Jarekt (talk) 00:56, 6 August 2020 (UTC)
Thanks for checking and providing relevant information. I have implemented a check for maxlag parameter (with conservative 2 seconds) into my bot's code now. I have also made a request for community input. The discussion can continue there: Commons:Bots/Requests/SchlurcherBot9‎ --Schlurcher (talk) 21:07, 6 August 2020 (UTC)
Keegan did you guys forget that we installed CommonsMetadata extension? See for example the output of today's image of the day. That was meant to ease the transition from the old template system to structured data on Commons.
I would use that extenstion to expand the search index.
The extension can be updated to understand structured data on Commons and Wikidata so that it will provide all the needed (multilingual) information. That shouldn't be too hard and can be done in small steps.
Of course more structured data should be added. I can fire up the bots again. Multichill (talk) 18:45, 24 July 2020 (UTC)
Talking about the license stuff, see https://w.wiki/Y4w . Gives the Wikidata item, Creative Commons URI, Commons category and Commons template for each Creative Commons license. Not complete now, but it will be in the future. Multichill (talk) 18:43, 27 July 2020 (UTC)
We considered using CommonsMetadata, but that would probably mean something along the lines of adding new field to the search index, building a hook in CommonsMetadata to expose the data on save, building & running a script to revisit existing pages & populate the index, etc. It'd also need code specific to licenses to query the search index (new keyword?). It's basically an awful lot of work (+bugs +maintenance) for something that is already a solved problem with structured data. And given that bots are already populating that data, we were really hoping we could use that instead :) Mmullie (WMF) (talk) 11:36, 31 July 2020 (UTC)
I'm not 100 % sure I understand what this is about, but just want to point out that we have many rather complicated cases of copyright such as File:Weltpostdenkmal (Bern) 06.jpg: An image with different copyright status for the object depicted (public domain for different reasons in source country and US, reflected by using {{PD-old-auto-expired |deathyear=1915}}) and for the photo of the object (CC-BY-SA 4.0), using {{Art Photo}}. One of the characteristics of Wikimedia Commons is that we strive to not just say "this image is PD", but exactly where and why it is PD (or freely licensed), to give potential users confidence. This is expressed in the copyright templates we use. Gestumblindi (talk) 15:26, 2 August 2020 (UTC)
@Gestumblindi: yes, they "where" and "why" is preserved. This task is about being able to filter and find files in search based on copyright or licensing type. The details of the copyright and licensing are unchanged. Keegan (WMF) (talk) 17:40, 5 August 2020 (UTC)

I only had a brief look at this discussion before. Now time for a proper reply. I see that the CommonsMetadata is off the table. That's fine with me. Copyright and licensing data is for me not the most interesting data to add. I like the properties that answer what (depicts (P180)), where (location of creation (P1071) / coordinates of the point of view (P1259)) and when (inception (P571)) more because these provide context. That doesn't mean I haven't added the copyright related properties. I have been adding that and also copyright information for a while mostly focused on Wiki Loves Monuments related files (which should be own work). I have kept track of my progress at User:Multichill/Structured data progress. Of the about 2,4M files uploaded as part of Wiki Loves Monuments about 2,3M files have copyright status (P6216) and copyright license (P275). I also linked up some of the license templates and categories with Wikidata, see for example Creative Commons Attribution-ShareAlike 3.0 Netherlands (Q18195572).

I think we can agree on the fact that we want to add structured data to every file preferably by adding as much data as possible in a single edit. That's where it gets harder. We have a lot of different subsets of files that are relatively consistent within the set, but quite inconsistent from a global perspective. Bots can keep on pushing the bulk. I'll fire up some more instances to clear out the easy bulk edits. Semi-automatic edits like using petscan and QuickStatements should be made easier so that more people can help out with the edge cases.

The reason Schlurcher and I have been focusing on own work files is because for these files we have consensus about how to model it. For other subsets like {{PD-art}} that's not the case. We can't mass convert files if we don't have consensus on how it should be modeled. Commons talk:Structured data/Modeling/Copyright has been quiet. Take for example the template {{Licensed-PD-Art}}. I have some ideas on how to model it, but not complete so files like this recently uploaded file don't contain any copyright information in the structured data.

Besides the bulk work going on in the background, I would like to focus on our best images. I'm currently mass adding Commons quality assessment (P6731) to the relevant files and I'm planning to add more tracking categories. Experience from these files can be applied to the other files. Multichill (talk) 18:47, 9 August 2020 (UTC)

I did some improvements on my bot code and I have been running for the last couple of days slowly increasing speed. According to Wikistats I'm currently at about 300 edits/minute and did about 275.000 edits in the last 24 hours. I'm currently working on the most used user templates because these get skipped by my normal bot. Multichill (talk) 15:43, 23 August 2020 (UTC)
@Multichill: the point may have been raised thousands of times before, sorry for that, but shouldn't we remove the data from {{Information}} at the same time whenever possible ? Duplicating things does not sound optimal maintenance-wise. --Zolo (talk) 11:43, 26 August 2020 (UTC)
I would strongly oppose removing copyright data from wikitext. - Jmabel ! talk 16:01, 26 August 2020 (UTC)
Copyright data are not retrieved from structured data by the {{Information}} template, so I would certainly not advocate removing them for now. But the bot also adds info about the author, date and source example. I think it would make a lot of sense to remove the content from the wikitext, when we can make sure the template shows something equivalent using structured data. -Zolo (talk) 17:10, 26 August 2020 (UTC)
I strongly oppose removing anything from the wikitext. Copyright and license information in structured data is incomplete like other Informations. Attribution is missing, user license templates are not be considered, and so on. --XRay talk 19:09, 26 August 2020 (UTC)
@Zolo: I don't think we're ready for that. Removing wikitext doesn't improve structured data and is very scary for a lot of people invoking a lot of strong responses. I rather focus on getting data in a structured format on existing files and work on new uploads with only or as much structured data as possible (like this file). That way we can improve step by step and people can slowly get used to it. Multichill (talk) 19:24, 26 August 2020 (UTC)

For the umpteenth time: my proposal three years ago for how wikidata content could be serialized into wikitext and deserialized back was roundly ignored. If someone at some point wants to discuss it seriously, I'm still open to that conversation. - Jmabel ! talk 23:43, 26 August 2020 (UTC)

Because it sounded very far fetched and impossible to implement. Of course you can prove me wrong by actually doing it instead of trying to get others to do it. Multichill (talk) 19:58, 27 August 2020 (UTC)
@Multichill: I develop software for a living. It's not what I volunteer to do for free in my spare time. - Jmabel ! talk 00:14, 28 August 2020 (UTC)

@Keegan (WMF) and Mmullie (WMF): bots have been running for a while and now we have more than 10 million files with basic copyright info. That seems enough to be to start development on this. What are the plans and what is the expected timeline? When doing search on this you first have to look at copyright status (P6216) for public domain (Q19652), copyrighted, dedicated to the public domain by copyright holder (Q88088423) or copyrighted (Q50423863) (only look for the truthy statements). If it's one of the first two, a re-user can do whatever they like. When instead you run into copyrighted (Q50423863), you have to go over to copyright license (P275) to see what license applies. How to use licenses hasn't really been modeled on Wikidata yet so I hope you have another data source for that. Multichill (talk) 20:05, 28 August 2020 (UTC)

The primary use case for copyright/licensing here is in search filters for Special:MediaSearch. The filters should be live at some point towards the end of September, so that's when we'll start seeing benefit as it relates to search. More information and plans about the next phase of MediaSearch are coming within the next week or two. Keegan (WMF) (talk) 17:23, 31 August 2020 (UTC)

Update

Commons is now approaching 22 million files with licensing data, adding it at a rate of nearly a million files per day. The end result for all this will be much faster and more efficient search in the very near future (more to come on that next week), so thank you very much to those of you engaged in this work and helping make it happen. Keegan (WMF) (talk) 17:49, 16 September 2020 (UTC)

Structured licensing part mostly done

Several bots have been quite active over the last couple of months. About 80% of all files on Commons now have copyright license (P275). Creative Commons licenses are by far the most used licenses with 49M out of 65M files using them, for these licenses the coverage is currently above 99,99%. The remaining files are either in the public domain or using slightly more obscure licenses. My bots have now gone into maintenance mode for copyright license (P275) meaning that the missing copyright license categories like Category:Creative Commons Attribution-Share Alike 4.0 missing SDC copyright license get checked twice per day. See Category:Structured Data on Commons tracking categories for all the tracking categories. I'll probably take a bit of a break from this before I start working on the remaining public domain part. Multichill (talk) 18:27, 1 November 2020 (UTC)

Admin help needed

We have some protected files that robots can't edit. Most of these are linked from Commons:Cascade-protected items and subpages of Commons:Auto-protected files. Admins can help by manually adding the missing statements. Adding copyright license (P275) and copyright status (P6216) removes most of the tracking categories. We also have quite a few files that are fully protected. That's probably from before partial protection? Best option is to lower the protection for the wikitext to auto confirmed users and keep moving and re-uploads for admins (example). Who wants to help? Category:GNU Free Documentation License, version 1.2 or later missing SDC copyright license is a good category to work on. Multichill (talk) 16:30, 10 October 2020 (UTC)

@Multichill: I changed a few files in Category:GNU Free Documentation License, version 1.2 or later missing SDC copyright license but it takes a while, so I added SDC to the files using ACDC tool. Any other categories full of protected pages? --Jarekt (talk) 01:59, 11 October 2020 (UTC)
@Jarekt: thanks! The files on Commons:Cascade-protected items are a bit spread over different categories, see for example this search. Looks like Category:Creative Commons Attribution-Share Alike 1.0 Generic missing SDC copyright license has plenty of them and each file seems to have a list of different licenses. Maybe I should start listing the missing SDC copyright license categories here for which the bots are done? I already completed quite a few licenses. Multichill (talk) 10:17, 11 October 2020 (UTC)
@Multichill: yes posting categories which are done by the bots would be good that way me and hopefully others might look at categories which need more manual attention. Another approach could be to find some way of mass downgrading protection on a lot of protected files to restrict renames and re-uploads but allow changing categories, SDC or wikitext. I do not know how to do it for a lot of files and perhaps should discuss such move first. --Jarekt (talk) 21:53, 11 October 2020 (UTC)
@Jarekt: I will post categories in the future. In the meantime I could use some help with these protected files. Multichill (talk) 16:15, 19 October 2020 (UTC)
@Jarekt: Category:Creative Commons Attribution-Share Alike 3.0 Unported missing SDC copyright license is easier and you should be able to do it with ACDC. Multichill (talk) 20:03, 19 October 2020 (UTC)
✓ Done --Jarekt (talk) 01:24, 20 October 2020 (UTC)
Thanks Jarek, can you apply the same trick to Category:Creative Commons Attribution-Share Alike 1.0 with different SDC copyright license, Category:Creative Commons Attribution-Share Alike 2.0 with different SDC copyright license & Category:Creative Commons Attribution-Share Alike 2.5 with different SDC copyright license? These currently contain the same 124 files using {{Cc-by-sa-2.5,2.0,1.0}} or some later version. When done, some of them should still be in Category:Creative Commons Attribution-Share Alike 3.0 with different SDC copyright license and a small amount in Category:Creative Commons Attribution-Share Alike 4.0 with different SDC copyright license. Multichill (talk)
✓ Done--Jarekt (talk) 03:37, 25 October 2020 (UTC)
@Multichill: Can you easily run your bot on all fully protected images from your main account? We can then discuss if you should. Last time I checked, there were only about 500 fully protected images. This should be a fairly small batch run and might be better than asking admins to do this individually. This was also (to some extend) suggested, when I asked how my bot can internationalize all fully protected images (see Commons_talk:Bots/Requests#Admin_Bot_Request) --Schlurcher (talk) 16:08, 20 October 2020 (UTC)
Technically yes, it's just a matter of adding a sysop username in Pywikibot. I'm quite reluctant to run a bot under my main account. Looks like gadgets are doing the trick and I also noticed quite a few files that are fully protected that should probably be lowered. Multichill (talk) 17:05, 20 October 2020 (UTC)
I wonder is there a way to lower them in some batch mode. Limit reupload and rename but allow description edits. --Jarekt (talk) 18:18, 24 October 2020 (UTC)

User help needed with last remaining files missing SDC copyright license

For various reasons bots are unable to process some files. Quite of then this is because of slightly non-standard (messy) wikitext. I'll post sets of files here that need attention. This should probably be done by the more experienced users. Two possible ways of solving a file:

I'll add new sets every once in a while. Please leave a comment and cross out the line when you've done a set of files. Multichill (talk) 17:23, 20 October 2020 (UTC)

@Multichill: I will have a look. I was away from my pc for a few days and doing cleanup on phone is a pain and too risky. I want to help not make things worse! :-) Files are being transferred from ja.wiki and other wikis so new files will show up non stop. --MGA73 (talk) 10:48, 1 November 2020 (UTC)
@MGA73: Thanks for helping out. Yes, uploads keeping coming in. What you can do is use the search, sort it by date and go in a couple of pages:example. Multichill (talk) 12:02, 1 November 2020 (UTC)
  • @Jarekt: It seems that most of the times where the bot can't add the data it is because there is a mess on the page. So fixing the problem like here and here should do the trick. I noticed those 2 files in categories with problems so it seems your edits was not enough to make "the system" happy. --MGA73 (talk) 17:47, 1 November 2020 (UTC)

Stripping p571 inception dates

[6] [7]

Thoughts? What does "inception" mean here? It's not the image (Commons resource) creation date, but it is the creation date for the object illustrated. Andy Dingley (talk) 23:12, 19 October 2020 (UTC)

The creation date of the object illustrated. It is not the creation date of the copy (ie. scanning date) or upload date. --Zache (talk) 18:09, 20 October 2020 (UTC)
Please note that I removed these statements as they were incorrectly added as Gregorian calendar dates. They should and will be added back as Julian calendar dates. --Schlurcher (talk) 18:19, 20 October 2020 (UTC)
  • That's nonsense, there's no Julian / Gregorian issue here, they're only dates approximate to a year.
Clearly what has actually been happening here (look at the other changes in this run) is that the values are being sanity checked for photograph creation dates. But if Zache is correct (and I would agree with him about "inception") these medieval dates were correct. However Commons just doesn't do "inception dates", it does photograph creation dates. Andy Dingley (talk) 00:03, 21 October 2020 (UTC)
Just for transparency, I have performed these changes. And if you look at my and my bot's talkpage you will see that several people complained about the Julian / Gregorian issue, and they are correct. With your comments I see that there are also a lot of incorrect dates like 204-04-12, which should probably be 2004-04-12. Given that I will not simply add back in the Julian dates, but also try to sort these out. --Schlurcher (talk) 07:12, 23 October 2020 (UTC)
  • This is just wrong. Place de la Concorde didn't come into existence in 1935. That's a photograph creation date, not an inception date, by any reasonable meaning, or by the meanings that Wikidata has tried to place on it. This is just the wrong property being used here (and it's a very widespread bot-generated bulk error). Andy Dingley (talk) 01:53, 26 October 2020 (UTC)
  • What is wrong about using inception (P571) in this way? Commons holds information about the media; information about the object(s) shown on the media is to be found on Wikidata. -- H005 07:01, 26 October 2020 (UTC)
  • In what way? The problem is that this 'bot is using it in two ways at once, a recipe for confusion.
It doesn't matter much what a property means, provided that this meaning is clearly visible, and that it's consistent. But this is being inconsistent, thus making the property useless. Andy Dingley (talk) 10:40, 26 October 2020 (UTC)
Could you please provide an example of such an inconsistency? -- H005 13:54, 26 October 2020 (UTC)
  • What does "inception date" mean? Is that the creation of the subject, or the creation of the image of the subject?
The Julian / Gregorian issue raised by the 'bot operator implies that p571 is the subject's inception date instead
The links at the top of this post are examples of a 'bot auto-stripping dates (correct dates for the creation of the artefacts) because medieval dates are seen as inappropriate for photographs (p571 is being treated as the photo creation date).
The 1935 date for Place de la Concorde is the photography date.
Here's a Roman bronze statue being dated to an inception of 2013
For a map of 1555, the 'bot is claiming this is the artefact date, the opposite of the first two examples.
What's been happening is clear. The 'bot is simply taking the date field from the image description and using that as the inception date. This is not a correct behaviour.
This is Commons. Commons has almost no ability to ever identify a correct inception date for an artefact - maybe for some specific imported collections, where individual museums have given us more-structured metadata which the content. But that's rare, and would also need per-source work to use it. In general, the Commons date field holds any one of four values: the image creation date (this isn't the inception date, so shouldn't be used), the upload date (commonly done, isn't either date, but could be detected fairly easily and thus ignored), an artefact inception date (rare, hard to recognise as such) or finally, simple garbage - mostly truncated years. So the 'bot here is clearly only blindly treating the date field (whatever its meaning) as the p571 date. That's not workable. Although Wikidata would no doubt like these p571 inception dates, they're just not available from Commons.
We need to do the following:
  • Stop adding p571 dates from Commons
  • Delete all p571 properties sourced from Commons by this 'bot. They are just unreliable.
  • See if we can identify any p571 dates correctly from any assets. As noted, this is probably only possible for a few museum collections and identified sources and metadata formats.
Andy Dingley (talk) 15:01, 26 October 2020 (UTC)
Re: "Commons has almost no ability to ever identify a correct inception date for an artefact", that is absolutely false. I am exactly as able to do that research when I am editing Commons as I when I am editing Wikipedia or Wikidata. I would guess that Commons has (for example) far more inception dates for ships and buildings than does Wikipedia. - Jmabel ! talk 17:28, 26 October 2020 (UTC)
  • That is not Commons. That is research outside it. If Wikidata wants an inception date for the Place de la Concorde, that's how it would have to be done. Commons (as a system bounded by itself) can't do it. How would you run a 'bot (which is the process we're discussing here) in order to examine the metadata for each image and from that produce new statements for Wikidata? Andy Dingley (talk) 00:56, 27 October 2020 (UTC)
@Andy Dingley: You are combining two different concepts here. 1.) First is making a copy of the work like scanning the page of the book or photocopying the painting with the camera. 2.) second is a creation of the new work (ie. photographing the statue or Place de la Concorde) In case of #1 date is usually the creation date of the artefacts in the image and in case #2 the date is the date of the creation of the photograph. Afaik it is pretty useful as it is. --Zache (talk) 22:45, 26 October 2020 (UTC)
  • You're talking there about just a subset of Commons content, which only applies for cases where we photograph a physical artwork. Nor am I combining them, I'm complaining that the 'bot is confusing their values into one property.
We can only recognise that inception date (I think we agree on the property's appropriate meaning) for a small set of Commons content (i.e. the property only has any meaning for a small subset), and it's particularly hard to extract it, because (as discussed above) only a few museum collections are ever going to make this available to us.
For the Place de la Concorde photo, we certainly can't do it. 1935 is the photograph's date, not anything to do with the square itself. What's an inception date for the square? 1795? 1830? or that for the buildings? the sculpture? the obelisk? Am ancient Egyptian date for the obelisk's original manufacture? We can't judge this without knowing the scope of that property and what its subject is (the subject would not be the Commons image or the photographic negative, that's the point) and there's nothing in the Wikidata representation of a Commons item to indicate subjects within an image. Nor can we tell this from any metadata we've been given in relation to this photo.
Overall, Commons just shouldn't be involving itself in inception dates for objects in the real world like this, other than artworks that we are representing by mechanical reproductions. Wikidata ought to hold such dates for a notable topic like the Place, but it would have to do so as a composite set, and it shouldn't be populating the values from Commons. Andy Dingley (talk) 00:52, 27 October 2020 (UTC)
Please see Commons:Structured data/Modeling/Date on how we aligned to model the date parameter, which should be used for storing the time photograph was taken, document created, painting or other artwork completed.. According to this the differentiation from Zache is very relevant. Please also note that your repeated interpretation of the English word inception is misleading. The property we use is P571. The German translation of P571 is way more elaborative and does not fit to your definition of inception above. So the question should be what we should store in property P571, not how we define inception. As such it is a modelling problem and was discussed Commons talk:Structured data/Modeling/Date --Schlurcher (talk) 07:54, 27 October 2020 (UTC)
Artefacts in the photo are currently defined using depicts (P180) on the photo and the artefacts inception (P571) values are stored to artefacts wikidata item and not directly to the photo. (an example: File:Place_de_la_Canourgue-en-hiver.jpg and Belleval hotel (Q3145748), place de la Concorde (Q189503). Example for the use with wcqs-beta) --Zache (talk) 07:57, 27 October 2020 (UTC)
@Zache and Schlurcher: IMO inception (P571) was not a good property for use on SDC. It would be far better to create a new property, described and defined specifically for the needs of Commons. Jheald (talk) 21:27, 1 November 2020 (UTC)
I certainly agree with what Jheald said here and said I so quite a while back. At the time, I was simply told I was wrong. - Jmabel ! talk 23:41, 1 November 2020 (UTC)
I strongly disagree. The property is in use extensively on Wikidata for creative works and that is exactly what we have here. The label is just a bit odd. Multichill (talk) 08:51, 2 November 2020 (UTC)
@Multichill: The difference is that on Wikidata it is clear what inception (P571) means: viz. the date the picture was painted. But on Commons there are many different dates that could be associated with a file -- the date the underlying work was made, the date that perhaps a derivative work was made, the date it was photographed, the date the photograph may have been digitised, the date the image was uploaded. For these reasons IMO it would be better to leave inception (P571) on Wikidata, and have new properties, precisely defined and more narrowly named, for use here. Jheald (talk) 10:29, 2 November 2020 (UTC)
While I'm usually hesitant about specialized properties, given that there are over 60 million of images on Commons, we are not talking about something horribly specialized. - Jmabel ! talk 16:39, 2 November 2020 (UTC)
@Jheald: you can apply that same reasoning to a lot of other properties used for creative works like for example creator (P170) and copyright status (P6216). How to model multiple works is described at Commons_talk:Structured_data/Modeling#Summary_multiple_works. Multichill (talk) 18:22, 2 November 2020 (UTC)

Proposal: File page is the master record about a media file & Proposal: Restrict structured data edits to named accounts

Two proposals related to structured data: Commons:Village pump/Proposals#Proposal: File page is the master record about a media file Commons:Village pump/Proposals#Proposal: Restrict structured data edits to named accounts. Multichill (talk) 18:12, 2 November 2020 (UTC)

Files captured with a camera before that camera was released

Thought it could make sense to show it here: I added to the examples a query Files captured with a camera before that camera was released which I found interesting, and likely surfaces many errors.

I only had a brief look at the results but it’s not necessarily a mistake − there are several possibilities:

Hope this is interesting to others,

Jean-Fred (talk) 17:04, 17 November 2020 (UTC)

Similar to your second point, there are also cases where a camera was used to reproduce an older (analog) photograph, and the date refers to the original photograph. Gestumblindi (talk) 20:17, 17 November 2020 (UTC)

Model discussion progress

Hey,

one year has just past and I would like to ask whether there is any progress in Model discussions. I had to quit them because it seemed to me so complicated (and I would say it seemed complicated also to others). But on the other hand, it would be nice to come into the decision and establish some models.--Juandev (talk) 20:45, 21 November 2020 (UTC)

Most of the discussion happened on Commons:Structured data/Modeling and subpages − several models are outlined there as well. Jean-Fred (talk) 22:09, 21 November 2020 (UTC)

Search preference survey

I've posted a quick survey for users to take about which search experience they prefer on Commons. Please take a moment to look it over and participate if possible, it will be open for about three weeks. Keegan (WMF) (talk) 21:07, 17 December 2020 (UTC)

Deletion of Wikidata objects and effect on structured data on Commons

Hello, if an Wikidata object is deleted, what is the effect on structured data on Commons?

  • Will any Wikidata object only be deleted if it is not referenced in structured data on Commons?
  • Or will it be deleted anyway and the reference in Commons will become invalid?
  • Or will the references in Commons be deleted as well? If so, by whom and when? (automatically by a bot, manually by the user, who deletes the object, ...)?

Also see d:Wikidata:Project_chat#Deletion_of_Wikidata_objects_and_effect_on_structured_data_on_Commons_(SDC) Wikidata project chat (diff)

Thanks a lot! --M2k~dewiki (talk) 16:35, 20 December 2020 (UTC)

Answer: for example https://commons.wikimedia.org/wiki/File:Leroy_Leone_Schauspieler.jpg has assigned deleted object d:Q104146999. So, SDC entries, where the wikidata object is deleted, become invalid and return an error massage. --M2k~dewiki (talk) 15:33, 28 December 2020 (UTC)

What is correct way to describe source of the file from internet?

I tried to check how to store the the source of the file as a structured data. Least reference URL (P854) has been used but the documentation says that is for references only and not for commons files. Also source of file (P7482) has been used for files taken by users (example) so can P7482 be used also like this for internet files?

Ie. like this: File:Senaatintalo_-_N25992_-_hkm.HKMS000005-km0000p5tq.jpg:

--Zache (talk) 20:20, 28 December 2020 (UTC)

@Zache: described at URL (P973) & operator (P137), see Commons:Structured data/Modeling/Source and File:Domenico Ferri - Vue composite des monuments parisiens (P425) - P425 - Musée Carnavalet.jpg as an example. Multichill (talk) 11:22, 29 December 2020 (UTC)
Ok, thank you! --Zache (talk) 06:33, 30 December 2020 (UTC)
@Multichill and Zache: It would be useful to specify the actual download URL for the image itself, as well as the description page. What is the current recommended form for that? Jheald (talk) 19:22, 30 December 2020 (UTC)
Why would this be useful? Don't think we ever did that before so what's the need to start doing it in structured data? Multichill (talk) 19:44, 30 December 2020 (UTC)
Maybe we didn’t really do that (I did personally try to do it), but e.g. Commons:First steps/Quality and description#Good file descriptions asks for it. —Tacsipacsi (talk) 23:35, 30 December 2020 (UTC)
Retrieved from "https://commons.wikimedia.org/w/index.php?title=Commons_talk:Structured_data/Archive_2020&oldid=708246459"