User talk:Zhuyifei1999/Archive 49

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search

Exitcode: 137, ffmpeg got killed :(

Hey Zhuyifei1999,

While uploading some videos, I got the following.

An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/5bcf45513414e779/dl.mp4 -threads 16 -row-mt 1 -qmin 19 -qmax 19 -vcodec libvpx-vp9 -tile-columns 4 -auto-alt-ref 1 -lag-in-frames 25 -speed 4 -f webm -ss 0 -an -pass 1 -passlogfile /srv/v2c/output/5bcf45513414e779/dl.mp4.vp9.webm.log /dev/null Exitcode: 137

An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/d69924b2b48e1732/dl.mp4 -threads 16 -row-mt 1 -qmin 19 -qmax 19 -vcodec libvpx-vp9 -tile-columns 4 -auto-alt-ref 1 -lag-in-frames 25 -f webm -ss 0 -ab 128000 -ar 48000 -acodec libopus -pass 2 -passlogfile /srv/v2c/output/d69924b2b48e1732/dl.mp4.vp9.webm.log /srv/v2c/output/d69924b2b48e1732/dl.mp4.vp9.webm Exitcode: 137

An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/01778e455716d81f/dl.mp4 -threads 16 -row-mt 1 -qmin 19 -qmax 19 -vcodec libvpx-vp9 -tile-columns 4 -auto-alt-ref 1 -lag-in-frames 25 -speed 4 -f webm -ss 0 -an -pass 1 -passlogfile /srv/v2c/output/01778e455716d81f/dl.mp4.vp9.webm.log /dev/null Exitcode: 137

Solution to this problem : https://stackoverflow.com/questions/19548027/ffmpeg-closes-with-return-code-137

Warm regards,
Eatcha (talk) 15:02, 23 July 2019 (UTC)


Just got:

An exception occurred: WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).

Eatcha (talk) 17:26, 23 July 2019 (UTC)

Wait, don't say I crossed the limit https://github.com/toolforge/video2commons/commit/3fa1bdff180442f501f793d7776c01246d55ecc2#diff-8e3399f1ba3c09fc774a32a1f3eddc54 ·  :) -- Eatcha (talk) 17:37, 23 July 2019 (UTC)

After hypervisor migration the load was quite calm and I only bothered to pool only a single host. Since around 13:20 the host was suddenly overloaded with a lot of tasks and started to struggle. OOM killer was invoked 43 times at 14:12:02, 14:27:45, 14:28:21, 14:29:52, 14:31:45, 14:32:19, 14:32:49, 14:33:27, 14:42:02, 15:32:17, 15:32:55, 15:33:35, 15:33:35, 15:33:35, 15:34:13, 15:34:56, 15:42:04, 15:43:02, 15:44:57, 16:21:39, 17:07:28, 17:20:14, 17:20:47, 17:21:26, 17:43:01, 19:50:23, 21:12:02, 21:14:12, 21:44:59, 21:54:57, 21:55:32, 21:56:09, 21:56:32, 21:56:33, 21:56:56, 21:57:33, 21:58:13, 22:06:05, 22:19:08, 22:22:59, 23:08:12, 23:18:01, 23:29:30. In 39 cases ffmpeg was killed and in 4 cases python was killed.
I have pooled two other instances to share the load and started depooling this overloaded instance so no further tasks get scheduled on this overloaded instance. --Zhuyifei1999 (talk) 23:57, 23 July 2019 (UTC)

Structured Data - testing other statements

You can now test using other statements for structured data on the file page on Test-Commons. Some datatypes are not yet available, such a coordinates, but further support will be extended soon. You can find more information about testing on the SDC talk page. The team looks forward to your feedback. -- Keegan (WMF) (talk) 16:41, 24 July 2019 (UTC)

FlickreviewR overwrite in 2012

File:Menfi beach.jpg

I don't know exactly what happened there, if original was unavailable from Flickr at the time or maybe a bug, or if there are any logs to find out if the same thing happened to other files. - Alexis Jazz ping plz 01:34, 28 July 2019 (UTC)

Unfortunately, I'm not Bryan, so I have no idea what was going on that early... --Zhuyifei1999 (talk) 01:40, 28 July 2019 (UTC)
Woops! - Alexis Jazz ping plz 01:58, 28 July 2019 (UTC)

A barnstar for you!

The Technical Barnstar
For the wonderful tool "Video2Commons". You are the reason that we can upload 8K and 4K videos from sites like YouTube and Vimeo with just some clicks. Thanks for creating this tool. Eatcha (talk) 06:03, 25 July 2019 (UTC)
@Eatcha: I'm wondering, would you be interested in improving it? --Zhuyifei1999 (talk) 04:15, 26 July 2019 (UTC)
@Zhuyifei1999: Sure! but how?, are talking about the issues at https://github.com/toolforge/video2commons/issues or something else ? I'm actually sure it would be much faster if you supply bestvideo[ext=webm] and bestaudio[ext=webm] for YouTube , which means no need of conversion if webm format is available (YouTube has 2 variants of highest quality format video one in webm and other in mp4), Since version 2015.04.26 it's deafult to prefer MP4 over webm. I also figured out that Video2Commons is the best thing we have to review Videos,(after conversion cheksum test fails, bot can only verify the url which can be changed) if video2commons automatically appends a review with a time stamp that should work much better. Best, Eatcha (talk) 04:54, 26 July 2019 (UTC)
  • I mean, would you like to contribute code and/or maintaining it? You know how to access toolforge and you know how to code :) Though, the transcoding servers are not on toolforge
  • The current 'download format spec' for the default parameters is bestvideo[vcodec=vp9][filesize<4294967296]+bestaudio[acodec=opus]/bestvideo[ext=webm][filesize<4294967296]+bestaudio[ext=webm]/bestvideo+bestaudio/best, with prefer free formats option on. I thought this would handle all cases but apparently there are still exceptions I have not looked into deeply. Things to consider:
    • What if the site is not YouTube?
    • What if the resolution of the largest webm version is smaller than that of the largest non-webm version? It can happen when the video has just been uploaded to YouTube.
    • What if the largest version is larger than 4GiB and can't be uploaded?
  • The review can be faked given that the user has full control of the file description page for the video. Do you have a solution for this? --Zhuyifei1999 (talk) 17:46, 26 July 2019 (UTC)
  • If you are worried about the desciption-box that user gets to edit before the upload starts then the most simple solution is to use Tags like PHP7 (Why not use Tag: ReviewedVideo2commons ?) and make the review template uneditable(don't show in the description-box of video2commons ?) and add a time stamp using {{subst:#time: H:i:s}} for easy verification if needed? -- Best, Eatcha (talk) 18:08, 27 July 2019 (UTC)
Good point regarding using tags. Would you like to make a patch for this? --Zhuyifei1999 (talk) 20:49, 27 July 2019 (UTC)
Unfortunately I don't have the required rights(edit editinterface required) to modify/create tags, and Yes the tagging part is most important to review videos. I can't do much to help, just the simpler python adjustments required like here and the template part(to include time-stamp). Where in the source-code you are adding the tag ? I can't find it anywhere in the source, is it added from somewhere else ? -- Best, Eatcha (talk) 18:23, 28 July 2019 (UTC)
The tag 'video2commons' is added by OAuth extension like any changes from a non-consumer-only OAuth consumer. The upload is done as part of the backend sequence using a patched pywikibot upload code (which I no longer any interest to invent more time in to get it merged). The backend will do a call to youtube-dl and that's probably the best place to see if we should add any additional tags. Data from the frontend is not to be trusted besides authentication data.
BTW: This code is like 3.5 years old... it's ugly :( especially the backend, where I was just piecing stuffs together. Given that py2 is gonna die very soon I should really spend some time to port to py3 and rewrite some parts.
As for the tags, I can create them ;) The code is core complicated than the tags :P --Zhuyifei1999 (talk) 21:09, 28 July 2019 (UTC)
Just a question, what if a user doesn't port to 3? Will their bot really stop working, any method to save them all?Any guess how many bots are vulnerable? And I'm starting to port+ format the python code now ), where should I send the pull request? Certainly not the master, you must check the diff to prevent a temporary disaster. Best, Eatcha (talk) 05:12, 29 July 2019 (UTC)
Is there any bot-like ClueBot-NG(it runs on cluenet servers) that runs completely on tool-forge, Cluenet servers are used thus disclosing the identity of author Christopher Breneman and others. I'm planning an auto-categorization for my next summer vacations but I don't want to reveal my true identity. I connect to Wikimedia using a self-made proxy (not open-proxy, can't be blocked by proxy bot) to avoid getting identified but if I use my own servers like Cluenet my real identity gets exposed? What's the GPU quota/RAM per tool and can it be increased? my web shells can't read the stats properly. I'll track admins, LR, and rollbackers for the data required. Best, Eatcha (talk) 05:43, 29 July 2019 (UTC)
Or it(my next summer vacations project for commons ) can be an automatic copyvio detection tool using Vision API, but I'm considering auto-categorizing as Vision API is not free it's costliest for web (see this ) detection service. :( -- Best, Eatcha (talk) 05:51, 29 July 2019 (UTC)

(Edit conflict)

Regarding py2 EOL:

  • Are you talking about toolforge in general? There will be a deadline in which toolforge will support py2, and there will be a at least few months of prior notice before it breaks. I am expecting the EOL of Debian 10 "buster" to be the final deadline, which will be no later than 2024. Bots hosted on somewhere else are, well, I guess depends on where they are hosted...
  • However, pywikibot will stop support Python 2 soonish. When the EOL of py2 arrives, expect the patches to no longer be tested on py2, and bots may break at any time if ran against pywikibot master (you can still run it against an old release however).
  • As for video2commons, the tool cannot be tested without an actual deployment and I think it would be better to make it 'testable' before doing any porting. Besides, much of its code are so bad and should be better redone than ported. So please hold that for now. I'll see what I can do in the coming months. (Though, you are still free to do the patch for the license verification; I will check if it can work :) )

As for saving other bots, as I said, there will be months of prior notice before toolforge breaks them all. Toolforge admins are unlikely to 'go and fix' each affected bot. And as for the number of affected bots, I'm too lazy to make a super-overpowered-python-2-bot-detector-of-doom :) --Zhuyifei1999 (talk) 06:01, 29 July 2019 (UTC)

Toolforge is a massive VM server farm and a PaaS (Platform as a Service), built atop WMCS (Wikimedia Cloud Services), an IaaS (Infrastructure as a Service). If ClueBot-NG runs on ClueNet, I don't think it runs on Toolforge, considering that ClueNet is not on WMCS.
If you are going to ssh into Toolforge, the IP that is used to connect to Toolforge is technically visible to all those with Toolforge ssh access. The IP may or may not reveal your true identity.
Regarding quota, assuming you mean Toolforge, it depends. There is no storage quota (i.e. if you don't get noticed by admins you are fine), and there is no GPU acceleration at all, due to pretty much all good GPUs and drivers are not FOSS. The RAM quota is per job, not per tool, and depends on the 'job runner' that is used to run the bot / webservice. If the job runner is 'son of grid engine' (most bots, most PHP webservices, and a few non-PHP webservices), then you specify the quota in the submission command. It matches against the VMS size (instead of RSS), and as long as you specify below something like 7GiB the job should be scheduled fine. If the job runner is 'kubernetes' (a few bots, a few PHP webservices, and most non-PHP webservices), then there is a quota, but IIRC it is something like 2 or 4 GiB of RSS per job and cannot be raised. I do not really understand either jobrunner so I cannot confirm that. But you are unlikely to hit it without doing something mad (say, rendering images or videos). As for v2c, the backend is in another cluster on WMCS independent from Toolforge so the quota doesn't apply (yes, I cheated :P) --Zhuyifei1999 (talk) 06:33, 29 July 2019 (UTC)
If you are trying to get the stats of individual servers of Toolforge, here you go: https://tools.wmflabs.org/openstack-browser/project/tools. tools-sge* are 'son of grid engine' instances and tools-worker* are 'kubernetes' instances. --Zhuyifei1999 (talk) 06:42, 29 July 2019 (UTC)
  • Maybe this should be posted at COM:VPP, I hate posting there but don't have any choice either. Do we have any speedy accept criteria on commons, there is already a backlog of 12K video requiring License review. If anyone complains about fake licenses on Vimeo/YouTube, ask them about Flickr. BTW YouTube is far better than Wikimedia Commons/Flickr in the takedown of copyvio videos. Best, Eatcha (talk) 06:16, 26 July 2019 (UTC)
    I am unaware of a speedy accept criteria --Zhuyifei1999 (talk) 17:46, 26 July 2019 (UTC)

21:42, 29 July 2019 (UTC)

SignBot seems stuck

Please check it.   — Jeff G. please ping or talk to me 09:06, 23 July 2019 (UTC)

90% sure deadlock at https://github.com/wikimedia/pywikibot/blob/8e53d6b5d94b926329896cea442c97699084e18d/pywikibot/site.py#L1074, investigating... --Zhuyifei1999 (talk) 00:18, 24 July 2019 (UTC)
Ah. Your typical race condition.... o.O --Zhuyifei1999 (talk) 00:27, 24 July 2019 (UTC)
https://gerrit.wikimedia.org/r/#/c/pywikibot/core/+/525179/ --Zhuyifei1999 (talk) 00:41, 24 July 2019 (UTC)
Could you kick it and watch the edits before the patch is approved? Please ping me.   — Jeff G. please ping or talk to me 02:50, 26 July 2019 (UTC)
@Jeff G.: I'll do that if the patch don't get approved this weekend. I would prefer running it with the patch than without --Zhuyifei1999 (talk) 03:48, 26 July 2019 (UTC)
I restarted it without the patch. Sigh. --Zhuyifei1999 (talk) 03:28, 30 July 2019 (UTC)
It seems not to be contributing.   — Jeff G. please ping or talk to me 14:39, 1 August 2019 (UTC)
Trying again --Zhuyifei1999 (talk) 16:27, 1 August 2019 (UTC)

13:24, 5 August 2019 (UTC)

Hurtigruten minute by minute

Hurtigruten minute by minute
Hi Zhuyifei! Thank you very much for all your hard works on transcoding the whole 554 GB of the Hurtigruten minute by minute. Enjoy the "minute for 24hour" version of the show. Cheers! Meisam (talk) 21:29, 7 August 2019 (UTC)

Signbot on the English Wikipedia

Hi! Over on the English Wikipedia, it seems like our signing bot has been down for a week or so. Would it be possible to configure the Commons SignBot to run over there too, maybe as a temporary measure? Alternatively one of the ENWP bot operators could do it, but I thought you would be the best person to ask. Thank you! Enterprisey (talk) 09:35, 5 August 2019 (UTC)

@Enterprisey: SignBot hasn't run here for 5 days. :(   — Jeff G. please ping or talk to me 15:04, 12 August 2019 (UTC)
Monkey-patched, restarted --Zhuyifei1999 (talk) 15:27, 12 August 2019 (UTC)

18:18, 12 August 2019 (UTC)

15:20, 19 August 2019 (UTC)

Wiki Loves Monuments 2019


Hallo Zhuyifei1999,

bald ist es soweit: Vom 1. bis zum 30. September 2019 findet zum neunten Mal der internationale Wettbewerb Wiki Loves Monuments statt. Dabei können Bau- und Kulturdenkmale fotografiert und die Fotos hochgeladen werden. Du hast an einem der vergangenen Fotowettbewerbe teilgenommen. Deshalb laden wir dich ein, dieses Jahr wieder mitzumachen. Wir freuen uns auf deine Fotos!

Es sind viele spannende Motive überall in Deutschland zu fotografieren. Neben beeindruckenden märchenhaften Schlössern, Burgen und Kirchen können auch andere Kulturdenkmale wie Brücken, Industrieruinen, Bauernhöfe oder Parks fotografiert werden, um sie unter anderem in der Wikipedia zu dokumentieren. In den letzten Jahren sind zahlreiche neue Denkmallisten entstanden, die sich über Fotos freuen. Für die Suche nach Motiven gibt es bei Wikipedia zahlreiche Listen und Karten. Als Einstieg hilft diese Übersichtsseite. Weitere Informationen erhältst du auf der Mitmach-Seite.

Du bist interessiert, am Wettbewerb mitzuwirken, dir fehlt aber die richtige Technik? Dann wirf doch mal einen Blick in den Technikpool und das Technikleihportal von Wikimedia Deutschland! Dort findest du Kameras, Objektive und Zubehör verschiedenster Art. Sollte noch Technik fehlen, die aber in Zukunft unbedingt benötigt wird, dann freut sich Wikimedia Deutschland über dein Feedback zum Technikpool.

Außerdem laden wir Dich ein, ab Mitte September 2019 an der Vorjury teilzunehmen. Diese bewertet die hochgeladenen Bilder und ermittelt so gemeinsam mit der Jury, die im Oktober tagt, die Sieger von Wiki Loves Monuments 2019 in Deutschland. Das Vorjurytool ist hier bald freigeschaltet. Du benötigst dafür nur deinen Benutzernamen und das Passwort.

Für Fragen steht das Organisationsteam gerne auf der Support-Seite zur Verfügung.

Viel Spaß und Erfolg bei größten Fotowettbewerb wünscht dir im Namen des Organisationsteams --Z thomas 14:28, 28 August 2019 (UTC)

Timeless Newsletter • Issue 5

Newsletter • August 2019

Welcome to the fifth and final issue of the Timeless newsletter!

Progress was made. True story.

I am happy to announce that after about a year, this delightful project in which absolutely nothing has gone according to plan is coming to a close. Or at least, the grant-funded portion is. Which means we will now be resuming our regular schedule of random whims-based development, you probably won't notice any difference whatsoever unless you use MonoBook, and there's a report.

What's new from the past two months:

A lot less than we'd hoped, frankly. We:

We also wound up with:

  • Patches resulting in RelatedArticles working in MonoBook, and FlaggedRevisions showing up in Minerva, unless someone actually managed to turn that off as well. (Blame T181242.) It's possible we went a little overboard with the whole 'let's close all the tasks!' sprint.
  • An unfortunate repeated discovery that themes (the Night/Winter variants I keep insisting will happen at some point) are still pretty far off on the horizon. Um.
  • Possibly a quite a few more bugs coming your way. This month, and especially the past week, have been a bit of a mess, development-wise. While hopefully none of the worse issues make it to production, please keep the reports coming for whatever you do find and we'll get it fixed as soon as we can. Y'all've been amazing about this, and it's really appreciated.

And I guess that's that. I'm really bad at reporting, and this is a report. For the purposes of the grant, this was a requirement, but do you want me to keep trying to send these out?

-— Isarra 18:08, 31 August 2019 (UTC)

Wiki Loves Monuments in the United States – Back for 2019!

This user participated in Wiki Loves Monuments 2019.

Want to show your participation in Wiki Loves Monuments 2019? Add {{User Wiki Loves Monuments 2019}} to your userpage!

Hello! Last year you contributed to Wiki Loves Monuments 2018 in the United States. Thanks to people like you it was a great success, with over 1,900 people contributing over 10,000 photos of cultural and historic sites from all over the country. Hundreds of these photos now help illustrate Wikipedia articles, improving our open knowledge about United States history, culture, and heritage. If you haven't seen the winners yet, be sure to check them out here.

I'm pleased to say that we're back this year with Wiki Loves Monuments 2019 in the United States, and I'd like to welcome you to participate once again in the event. Check out our updated event page for more information. Just like last year, you'll be able to upload your photos of any registered historical site in the United States through the end of September (even if the photos were taken before this month).

If you've traveled and taken photos of monuments in non-US countries, you can see if those countries are also participating here.

Once again, thank you for participating in Wiki Loves Monuments 2018, and we hope to see you again in this year's event! If you'd like to respond to this message directly, please do so on on my talk page. ~Kevin Payravi (talk) 05:16, 1 September 2019 (UTC)

SignBot autosigning posts that already contain a signature

SignBot has been repeatedly autosigning in the middle of templates I post on others' user talk pages. I'm already signing the posts, and I don't want to have to spam edit histories with "!nosign!" to work around this. My signature is the default one, so SignBot should be able to recognize it. SignBot also does not appear to respect the {{NoAutosign}} on my talk page, and I don't want to add it to my userpage because I don't want a local userpage on Commons. What am I doing wrong?

Thanks for looking into this! --Alex Cohn (talk) 17:11, 2 September 2019 (UTC)

It doesn't think lines starting with | is part of a comment. Hmm. --Zhuyifei1999 (talk) 17:30, 2 September 2019 (UTC)
Should be fixed. --Zhuyifei1999 (talk) 17:43, 2 September 2019 (UTC)
Thank you! Alex Cohn (talk) 17:44, 2 September 2019 (UTC)