Commons talk:Restricted uploads

Related mailing list posts:

• http://lists.wikimedia.org/pipermail/wikitech-l/2010-October/050038.html
• http://lists.wikimedia.org/pipermail/wikitech-l/2010-November/050389.html

Please announce this

Why isn't this proposal being announced on Commons talk:File types, one of the main traditional discussion areas for issues of this kind? -- AnonMoos (talk) 06:58, 28 November 2010 (UTC)

Done.--Eloquence (talk) 19:55, 29 November 2010 (UTC)

Support but why free files only?

I support this idea - I think it'd be immediately valuable to me personally for uploading DNG raw versions of photos that I have uploaded in the past - but why the restriction to only free and unencumbered file types? While I certainly understand that that's part of our mission, I've sometimes had to deal with e.g. PNG files created based on Adobe Illustrator or other proprietary sources. While SVG is more appropriate for this kind of thing, converting to SVG requires substantial effort that the original author is frequently not willing to invest. Another case is proprietary video formats, which the uploader doesn't understand how to convert to OVG, or proprietary camera raw formats, where the uploader doesn't know how to convert to DNG. I'd rather delegate these conversion tasks to users with more skills or time than lose the media. Dcoetzee (talk) 06:11, 4 December 2010 (UTC)

My thinking exactly. I can't see why we should restrict uploading to free file types. For example I don't think it's possible to convert files made using Autodesk 3DS Max or Maya (industry standards) into Blender formats. Same for Adobe Photoshop and GIMP. On the other hand, free software are able to read popular unfree types most of the time, so usability is not encumbered. And finally, restriction might put down some willing uploaders who don't have the time or will to do conversion (which is usually a headache). -- Orionist ★ talk 19:51, 9 December 2010 (UTC)

By being able to upload every file format the support for free software is reduced since people can stick with their proprietary software while being able to upload the orig file here. Even worse: if someone likes to use and modify the orig source file he needs to buy the proprietary software. These are my thoughts about this effort here. Good idea in general but I am not feeling good about this. Cheers --Saibo (Δ) 21:19, 4 February 2011 (UTC)

MediaWiki don't generate thumbnails for proprietary formats, so anyway a file should be converted to a free format to use it anywhere in Wikipedia. And Commons as a media archive has another important role -- to preserve originals in their original form. All proprietary formats eventually will be free, sometimes very soon (say, H.261 or MP3), but if an original was lossy converted to a free format for upload to Commons, information loss can never be recovered. Trycatch (talk) 05:55, 5 February 2011 (UTC)

Thanks for your response Trycatch. That was clear to me, however, if you want to modify a file like File:Fundraising_2010_-_colorful_fibers_banner.png you can nearly (except tweaking the colours all-over) only do it with the src file. If the src file is a photoshop file I have to buy photoshop to be able to modify. Well, of course: if we would not have the photoshop file and only the png output you could not modify at all. But I think by being able only to upload free formats I guess there is at least a tiny move towards free formats by file creators to enable others to modify.

Do you understand my concerns? Cheers --Saibo (Δ) 14:15, 5 February 2011 (UTC)

That's where you're mistaken: Free software support many - if not most - proprietary formats, while the opposite isn't necessarily true. So if you want to modify native Photoshop format (PSD) you can do so with GIMP, similarly you can modify Illustrator native Ai format with Inkscape. The whole point of allowing the upload of original files is to have them in their original, fully editable form. Because conversion, even to similar file types (for example Ai->SVG) is almost always lossy, and the result is very unpredictable. All that is exacerbated, and many more problems arise when the formats become more complex, like 3D and video files. Not to mention the time and effort (and willingness) required. And finally, proprietary software that's become industry standards are widespread. At worst, you can download a free 30 days trial version if you need to. Regards, -- Orionist ★ talk 15:25, 5 February 2011 (UTC)

I never tried to open a psd file with GIMP, but I guess this conversion is not perfect and having to proprietary orig software would be better to modify it.

Just to get this clear: I am not against this effort here but we should try to support free software. Cheers --Saibo (Δ) 17:06, 5 February 2011 (UTC)

I support the idea of only allowing free and open formats to be uploaded, but with a catch: Whenever an open and free format has a text equivalent (such as SVG which has .svg [plain text] and .svgz [zipped text]), allow only the text equivalent. So the ODF inside zip containers wouldn't be allowed, but Flat ODF (and even Flat OOXML) would be. Still, it's better to upload those files only as "source files" for the uploaded files' description pages, i.e., a SVG picture generated from a .fodg, a PDF generated from a .fodt or a .fodp, and so on.

I also propose a recommendation for not embedding any pictures/movies/sounds inside Flat ODF / Flat OOXML files, only linking them when inserting (ODF with SVG would be the sole exception, as SVG pictures can be copy-pasted inside a document as drawings instead of embedded/linked pictures -- LibreOffice Writer and Microsoft Word even have a drawings toolbar for creating them without an external drawing program). This way, 1) the pictures/movies/sounds could be uploaded to Wikimedia Commons first and then linked; and 2) the text files wouldn't waste much space because of the encoding needed to insert binary files inside text files.

Joaopaulo1511 (talk) 04:56, 6 August 2018 (UTC)

how would this work?

As a programmer, it seems odd to refer to graphics files as "source". However, the intent is clear and I can't think of a better term. I also don't understand why these files are any less secure than, say, JPEGs, but I'll assume for the moment that they are.

I'm interested in uploading some Blender files referred to in Wikibooks:Blender 3D: Noob to Pro. However, I'm not convinced that this proposal will help me. If I'm not a trusted user, how do I go about finding one? How do I convince him/her that my file is safe and should be uploaded to Commons? How should I convey the file to him/her? If (as I suspect) the only practical way to upload the file is to become a trusted user myself, how would I do that? --Stepheng3 (talk) 19:12, 9 December 2010 (UTC)

Suppressing the thumbnail

Suppressing the thumbnail generation would eliminate most of the problems as, if I understand correctly, the risk is when the server generates the thumbnail. A generic thumnail could be used automaticaly, based on the file MIME type like it is done on most Operating Systems. Once the upload is done, the uploader could add a thumbnail him/herself as a regular JPG or PNG file. Lionel Allorge (talk) 21:28, 22 December 2010 (UTC)

That is a concern, but my understanding is that there is also a concern that people will download files from Wikimedia Commons and open them, possibly spreading malware (e.g. if some of our restricted uploads were executable binaries, or contained executable script, this wouldn't be too hard to imagine). Of course this is why these uploads would be subject to some kind of review. Dcoetzee (talk) 02:49, 23 December 2010 (UTC)

Moving forward

I'm currently adding more to the proposed design and clarified alternatives.

Has a bug already been created for this in Mediawiki's bug database? If so it would not only make the devs aware of the issue, but also make it easier for volunteers like us to implement the feature and submit a patch. Also I think we're only getting limited feedback about the idea right now and it'd be good to advertise it more widely, before committing effort to implementing it. Dcoetzee (talk) 08:01, 23 January 2011 (UTC)

must use virus scanning software

"The uploader must use virus scanning software to scan the content for viruses or other malware."

Virus scanning software does not detect everything and most non-M$ OS users - like me - do not have virus scanning software. A more clever approach is to run a server-side scanner for uploaded files - although as mentioned it will not detect all viruses.

→ remove this restriction. Cheers --Saibo (Δ) 02:57, 24 January 2011 (UTC)

That would be clever, but is unlikely to be implemented due to cost (virus scanners require regular updates, have high CPU overhead, etc). And yes, it would not detect all problems but it's intended just as a quick and easy check. For people on alternate OSs they can just use a web-based virus scanner like virscan.org, where you upload a file and it runs a gamut of scanners on it. In fact I'll just add it to the description because it suits the needs of something like this pretty spectacularly well. Dcoetzee (talk) 05:53, 24 January 2011 (UTC)

en:Clam AntiVirus is afaik opensource/free and is often used on server side. Maybe we could run it on Toolserver? To check all new uploaded "restricted files"? However, since we have no virus scanning tool ready to use it would mean much programming effort which is not worth it in my point of view.

Online Scanners often have a size restriction of 20 MB which is will be too low for source files in many cases. en:VirusTotal.com is the one I use most often.

Maybe we should advise users that they should use a up-to-date program if they like to use/open the restricted files uploaded here since they could potentially exploit vulnerabilities in the programs. I guess such security vulnerabilities happen more often with the programs "own" file formats as with jpg/png/... See e.g. here: 1, 2. Cheers --Saibo (Δ) 14:12, 24 January 2011 (UTC)

Strong Support

Registering me Strong Support for this idea. I hope it goes live soon! Basket of Puppies (talk) 18:58, 4 February 2011 (UTC)

Alternatives and their respective drawbacks

Alternative 1: Allow uploads from anyone

If this alternative is selected, as a way to avoid Wikimedia Commons being used as a free hosting service I propose a file limit cap: Initially, 1MB per month for files other than the classic PNG/SVG/PDF, which would grow 1MB per month the user account is old (a three year/36 months old account would have 36MB monthly as a limit), until the monthly cap is the same as the maximum file upload (which I think is 100MB). Joaopaulo1511 (talk) 05:49, 6 August 2018 (UTC)

Alternative 2: Do nothing / continue to slowly add support for new file formats

If this alternative is selected, as a way to speed certain things a little, it should be allowed to upload plain text files, so XML files such as Flat ODF (.fodt/.fodg/.fodp/.fods) and LilyPond (.ly) could be uploaded as .txt files and the user oriented to change rename them to their original extensions. Joaopaulo1511 (talk) 05:49, 6 August 2018 (UTC)

Alternative 3: Only implement an improved ZIP file handler (or alternative archive format)

I don't see any need to allow Zip files. SVG, ODF and OOXML have their so called flat formats which are plain text XML files (.svg instead of .svgz, .sodt/.sods/.sodp/.sodg instead of .odt/.ods/.odp/.odg). LilyPond sometimes uses several .ly files to produce a single .pdf file, and this could be a valid reason to pack all .ly files inside a single .zip, but even so the various .ly files can be converted as a single .ly file (by copy-pasting the referenced .ly files in the place of the \include tags).

Joaopaulo1511 (talk) 05:49, 6 August 2018 (UTC)