Commons:Bots/Requests/Fbot

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search

Fbot (talk · contribs)

Operator: FASTILY (TALK)

Bot's tasks for which permission is being sought: This account shall be used to make test uploads as a part of a new bot mw-API library I've been developing. Any files uploaded by this bot will be deleted within hours of being uploaded, once the results have been verified.

Automatic or manually assisted: Automatic supervised

Edit type (e.g. Continuous, daily, one time run): Sporadically

Maximum edit rate (eg edits per minute): 6/min | 50mb/min (whichever limit is reached first)

Bot flag requested: (Y/N): Yes, so as not to swamp recent changes with temporary test edits.

Programming language(s): Java

FASTILY (TALK) 11:00, 10 June 2012 (UTC)[reply]

Discussion

  • Please create bot page and make a test run. --EugeneZelenko (talk) 14:30, 10 June 2012 (UTC)[reply]
  • I recall something similar (of yours?) effectively caused denial of service timeouts to uploads at times during the weekend of June 2nd (I was affected, and I imagine others were too). A script was uploading about 5 videos (broken 50Mb ogg files) per minute, which caused the upload server to hang / reject other uploads. (They were subsequently deleted.) I had been hoping it was an isolated one off issue, but if you plan to continue these tests, it needs to be under much tighter constraints. In particular the maximum edit rate should not be "tested" to see how much the code/server can handle, but instead should be limited to avoid causing problems for others. --99of9 (talk) 01:56, 18 June 2012 (UTC)[reply]
  • Ah, I've found them: here. And it's way higher volume than I was imagining (e.g. sometimes 30 50Mb files per minute for an entire hour = 90Gb!). Since the bot request was not even submittted in advance of these many thousands of huge useless uploads, and the bot user pages still do not comply with the requirements for bots (not even stating that they are automated accounts), I am blocking both accounts until we see a full explanation of the requirements and constraints. --99of9 (talk) 02:12, 18 June 2012 (UTC)[reply]
  • DOS? That's utter nonsense. Perhaps you ought to be certain of your terminology before slinging such impenitent accusations. Just so we're clear:
  • It's true I'm not an expert with that terminology, and I'm not implying that it was deliberate or long term, but "saturating the target machine with external communications requests, such that it cannot respond to [other] legitimate traffic" is roughly what I meant. --99of9 (talk) 07:57, 18 June 2012 (UTC)[reply]
  • My bots possess built-in replag detection. If replag should exceed five seconds, execution is halted until replag falls under five seconds again. This number, is stricter than what is mandated by most mediawiki-API bot frameworks.
  • I'm not sure what kind of lag caused the problems, but I do know that I saw repeated problems (a handful of times pywikipediabot had to retry multiple times uploading the file: 1, 2, 4, 8 ... minutes apart) that only occurred during your runs, and you were the only other significant uploader at the time (and I had good connectivity with the rest of the internet, including other WMF servers). For my information, is your replag detection performed during the middle of an upload, or in between uploads? How often is it performed? --99of9 (talk) 07:57, 18 June 2012 (UTC)[reply]
  • Between uploads obviously; it seems that none of us on that bot project can be arsed to implement chunked uploads. Regardless, this concern is now moot with the addition of two lines of code. Also, please do not accuse me, on the basis of pure speculation, of tying up server resources, unless you have the server logs to prove it. I take the upmost precaution when it comes to consuming my fair share of resources and I find it highly offensive for you to be accusing me of doing otherwise. -FASTILY (TALK) 23:52, 18 June 2012 (UTC)[reply]
  • Utmost precaution that does not even include throttling the rate? Hmmm. If you want to pursue this line of argument, please list the precautions in detail, and please also state what rate of 50Mb ogg file uploads per minute that you believe the commons upload server can handle. --99of9 (talk) 00:41, 19 June 2012 (UTC)[reply]
  • Correct, upmost precaution. Since dynamically calculating replag seems to have failed, the limit is now hardcoded at 6 edits/min or 50Mb a minute, whichever limit we hit sooner. If I didn't care, I honestly doubt I'd be making the changes you're requesting here. -FASTILY (TALK) 00:59, 19 June 2012 (UTC)[reply]
  • The problem is not the replication lag of the database (which contains wikitext, comments, relations(categories,...), timestamps) but the overall media handling of MW(WMF) which seems to be very slow, I think. -- RE rillke questions? 12:42, 18 June 2012 (UTC)[reply]
  • This bot framework is mostly complete, is multithreaded, and is constrained by one's network speed (do note that I edit off a Tier 2 network). An upload rate of thirty per minute for a few minutes is a total fluke, and could only have occurred when WMF network traffic was extremely low. Frankly, I only wish I could upload files that fast on a normal basis. At any rate, this can easily be remedied by building a counter into the code.
  • Yes, that's the remedy I'm asking for. Let us know when it is implemented. I think your performance is occasionally pushing the WMF server to or beyond its limit, and I don't want a single user to be able to do that.--99of9 (talk) 07:57, 18 June 2012 (UTC)[reply]
  • What limits have you put on it? Please list these under Maximum edit rate in your request. In this case I think both an uploads/minute and megabytes/minute limit is necessary. --99of9 (talk) 00:41, 19 June 2012 (UTC)[reply]
  • I hope you're not intentionally trying to make matters difficult, especially since we're on the same side here. My goal here is to create an upload framework with a planned GUI wrapper that our contributors will be able to benefit from. -FASTILY (TALK) 00:59, 19 June 2012 (UTC)[reply]
  • No I'm not. I'm trying to make sure what is written down is correct, so that other users will understand, and be able to determine if there is a problem. --99of9 (talk) 01:10, 19 June 2012 (UTC)[reply]
  • I now  Support further testing of the new framework with this bot account, but will leave the request open for a while so that others can examine the type of edits the bot would likely make. Other opinions? --99of9 (talk) 01:36, 19 June 2012 (UTC)[reply]

This request was withdrawn by Fastily. --99of9 (talk) 03:29, 10 October 2012 (UTC)[reply]

Retrieved from "https://commons.wikimedia.org/w/index.php?title=Commons:Bots/Requests/Fbot&oldid=80517476"