Commons:Bots/Requests/OgreBot 3

OgreBot 2 (talk · contribs)

Operator: Magog the Ogre (talk)

Background:

• As of late, the file upload bots have been doing an especially lazy job of properly transferring files from Wikipedia, and their owners have not made fixes when requested. As a result, various reptitions, improperly transcluded templates, etc. are added to a page which serves only to confuse the reader (ex: [1] [2]).

Cleanup of files moved from Commons via bot. I have already coded most of the following functionality, and I hope to be able to add the rest as well:

Extended content

Remove the following templates if they take no parameters: {{Template other}}, {{Tl}}, {{Tls}}, {{Tlx}}, {{Tlp}}, {{Tlsx}}, {{Tlsp}}, {{Mbox}}, {{Tmbox}}, {{Lan}}, {{JULIANDAY}}, {{Red}}, {{En}}, {{Max}}, {{Max/2}}, {{Str left}}, {{Str right}}, {{Date}}, {{Cite book}}, {{Citation/core}}, {{Citation/make link}}, and a few other related templates. Remove the following templates if they are located in the license field and if they take no parameters: {{FULLROOTPAGENAME}}, {{Ns has subpages}}, {{!}}, {{!!}}, {{!)}}, {{(!}}, {{Description missing}} (unnecessary; see below) If the description field reads {{LangLink|No original description}}, then that description itself will be removed so that {{Description missing}} is automatically transcluded from the {{Information}} template. If any of the citation templates are detected but not used elsewhere on the page, they can be added to User:OgreBot/Pages moved to Commons with citation templates. This is because the bots don't bother to actually copy the citations, so someone will have to move the information back over by hand. If any of the OTRS permission templates are detected but not used elsewhere on the page, they can be added to User:OgreBot/Pages moved to Commons with missing OTRS templates. Remove the {{Cc-by-sa-3.0-migrated}}, {{License migration not eligible}} where migration information is already present as the parameter of another template (normally this is GFDL templates or self/GFDL, but occasionally custom templates as well). Remove {{PD-old}} where {{PD-old-100}} is already present on the page right next to it. Combine {{PD-art}} and {{PD-old}} into {{PD-art|PD-old}}, {{PD-old-100}} into {{PD-art|PD-old-100}}, etc. After this transformation, it can then be changed to just {{PD-art}} (for 70, as old-70 is the default), {{PD-art-100}}, etc. Combine {{Should be SVG}}, {{SVG}}, etc. into a single template and move it out of the licensing field into the other_versions field of the Information template (assuming one exists). Perform similar alterations on templates in Category:Image cleanup templates. (not yet coded) Remove nested {{Information}} templates (e.g., [3], [4]) Remove the text <!-- Templates "Template:...", "Template:..." [etc.] were used in the original description page as well , but do not appear to exist on commons. --> where the templates listed are known to be harmless (e.g., w:Template:File other, w:Template:Copy to Wikimedia Commons, w:Template:Main other) because it is an eyesore on the text of the page.(minor edit) Remove Permission = See below, Permission = [just a rehash of the templates under the licensing section], and similar. (minor edit) If there is a self-created license template or {{Own}}, then the text "original uploader was" will be removed from the author field as long as it unambiguously does not goof up the text. Internationalization of text using {{Original upload log}}, {{int:license-header}}, {{Original description page}}, {{Transferred from}}, {{User at project}}, and {{Original uploader}}. Will be interpreted to only do replacements if and only if text unambiguously meant it. == {{int:license}} == → == {{int:license-header}} == (minor edit) (not yet coded)add {{Original description page}} in such cases. Fix of language description templates to properly include content within when there is an equals sign present e.g., |Description={{en|2+2=4}} (Displays simply as English:) → |Description={{en|1=2+2=4}} (Displays correctly as English: 2+2=4). Remove {{GFDL}} where another GFDL template is already transcluded. This includes GFDL-with-disclaimers templates, as this is often a faulty move by the bot. (not yet coded)|Date=2000年1月1日 (Chinese, Japanese date formats), |Date=2000년-1월-1일 (Korean) → |Date-2000-01-01, and similar (e.g., |Date=2000年, |Date=2000 1월 1일) in those languages. The date field already internationalizes those for local languages. Fix {{Multilicense replacing placeholder new}}, {{Multilicense replacing placeholder}}, and {{Self}} (they all point to the current uploader, whereas they should point to the original uploader at the local project; this "fix" will only be done if the author field unambiguously points to the original project uploader as the only author, so as to avoid misfires with derivative works on Commons).

• Lastly: I would like permission to be able to add functionality to the bot when I am requested. The list above is only what I currently know I want to implement.
• An important note: each change to the page will be classified as either "major" or "minor". If all the changes to a page are minor, then the edit will not be performed. This will prevent nuisance edits from the bot.

How the bot will find the pages

• The bot will go through the upload history from a certain date onward, and edit only the description pages of those files. The bot will pull the text from the SQL on the Toolserver, so as not to bog down Wikimedia's servers unnecessarily. Alternatively, I am considering a run which will edit every page which transcludes {{Original upload log}}, because they are the most prone to error; however, I will not do this unless I have success on the first run of the bot, because there are over 300,000 transclusions and that would take up a lot of time and resources.

Bot's tasks for which permission is being sought: See above.

Automatic or manually assisted: Automatic, unsupervised. A few test runs will be done first, with the number of files being edited on each test run to increase, once I'm more sure there are no bugs or omissions.

Edit type (e.g. Continuous, daily, one time run): Originally meant to be a one-time run, but could perhaps run on a daily basis or more often to continually fix ugly uploads.

Maximum edit rate (eg edits per minute): Unthrottled. In fact, I plan to run several concurrent instances of the program in order to be able to make edits faster, in view of the fact that there will be a lot of pages edited.

Bot flag requested: (Y/N): Y

Programming language(s): PHP

Magog the Ogre (talk) 04:21, 27 May 2012 (UTC)

Discussion

• Please make a test run. --EugeneZelenko (talk) 14:46, 27 May 2012 (UTC)

   Done Magog the Ogre (talk) 20:40, 28 May 2012 (UTC)

I don't know if such edits (replacing e.g. Name by Name at German Wikipedia) were done in error or not, but they should be stopped. I only agree if at en.wikipedia is there before (example). In this and other cases, the way of attribution chosen by the author was changed. --Leyo 21:15, 28 May 2012 (UTC)

You are correct; I am removing that. Magog the Ogre (talk) 04:49, 29 May 2012 (UTC)

 Done - I think you'll find the recent batch of updates much more agreeable (albeit after I fixed several kinks in the peachy framework that parsed templates incorrectly). Magog the Ogre (talk) 07:03, 29 May 2012 (UTC)

Thank you.

What about adding CommonsHelper to {{Transferred from}} or at least converting [http://tools.wikimedia.de/~magnus/commonshelper.php CommonsHelper] to [//toolserver.org/~magnus/commonshelper.php CommonsHelper] (see diff)? However, IMO the information template should only contain information that is relevant for a reader or reusers. The fact that a file was first uploaded to Wikipedia and then transferred to Commons by User:XY is not relevant in this sense. Hence, it is sufficient that this information is contained in the original upload log and in the text next to the thumb. --Leyo 08:00, 29 May 2012 (UTC)

OK, I can remove the http:, although I will make it a minor edit. In which cases do you want it to remove the http:? For toolserver and wikimedia links? Also, I would be willing to remove the transferred information, in the case that a) there is other relevant information present (so it doesn't end up with an empty field and leave a {{Nosource}} included), and b) it doesn't include who transferred the file, unless it is User:Boteas, because that's also irrelevant. I would not feel comfortable removing the information altogether; perhaps it could be moved down into the original upload log field, except in situation (a) mentioned above (again, to avoid the nosource template). Magog the Ogre (talk) 17:36, 29 May 2012 (UTC)

Removing http: should not be the only change in an edit. I agree on case a), but not on case b), especially when the transferring user is mentioned next to the thumb (i.e. in newer transfers). --Leyo 23:14, 29 May 2012 (UTC)

I guess I see your point - if the initiator is listed in the upload history (which often won't be the case for CommonsHelper; File Upload Bot gets it). However, will the community support it, or will I get a ton of blowback for removing it? Magog the Ogre (talk) 00:15, 30 May 2012 (UTC)

• Overall I like the idea of more standardize the cleanup after the transfers, but edits such like this one, are not wrong, but the main problem in this case is the uploading tool en:WP:FTCG. I will leave a talkback for the developer to this thread. mabdul 19:11, 29 May 2012 (UTC)
  • I didn't know about those templates, and I will fix FtCG to use them by default! This, that and the other (talk) (developer of For the Common Good) 05:13, 30 May 2012 (UTC)
    • Alright, well it's definitely still got some kinks, but I don't want to run it any further without the flag and/or permission(amended below). I would like to be a bit more aggressive in what the bot removes as well; I will provide diffs shortly to give examples. Magog the Ogre (talk) 22:00, 31 May 2012 (UTC)

The bot is not adding {{Transferred from}} template correctly. See [5], [6], [7] etc --Sreejith K (talk) 11:34, 1 June 2012 (UTC)

Thank you. Magog the Ogre (talk) 12:13, 1 June 2012 (UTC)

Above I stated I wasn't going to run it anymore. However, while I'm still adding features, I'd like to keep running and in order to test it. It's only making a limited number of edits which I am supervising. This doesn't seem to me like it's a problem; correct me if I'm wrong. However, I would still like the bot flag when possible. Magog the Ogre (talk) 00:42, 2 June 2012 (UTC)

---

If there no any objections, I think bot status should be granted. --EugeneZelenko (talk) 14:35, 6 June 2012 (UTC)