Commons:Bots/Requests/Smallbot (10)

Smallbot (talk · contribs)

Operator: Smallman12q (talk · contributions · Statistics · Recent activity · block log · User rights log · uploads · Global account information) (assign permissions)

Bot's tasks for which permission is being sought: To upload ~500k files from the US National Archives and Records Administration based on a database dump provided in partnership with the Digital Public Library of America.

Example

JSON source

{ "id": "nara--1693425", "key": "nara--1693425", "value": { "rev": "1-a8e9cee50a8cea9d1724c12a0d0d69e5" }, "doc": { "_id": "nara--1693425", "_rev": "1-a8e9cee50a8cea9d1724c12a0d0d69e5", "hasView": [{ "url": "http://media.nara.gov/Public_Vaults/14755_2006_001_a.jpg", "format": "image/jpeg" }, { "url": "http://media.nara.gov/nwl/berryman/H-009_7-1-1928_Yes_We_Have_No_Ambitions_print.pdf", "format": "application/pdf" } ], "sourceResource": { "date": { "begin": "1928-07-01", "end": "1928-07-01", "displayDate": "07/01/1928" }, "description": "This cartoon plays off a line from a popular 1923 song (\\" Yes, We Have No Bananas!\\") to characterize car maker Henry Ford\'s Presidential ambitions--or lack thereof. Ford blames his busy schedule for his hesitation to jump into the \\" Presidential contest pool, \\" while eager supporters encourage him to \\" come on in !\\" Berryman was correct in his prediction: Ford chose not to pursue the Presidency.", "title": "Yes, We Have No Ambitions Today!", "rights": "Restrictions: Unrestricted; Use status: Unrestricted", "collection": { "@id": "http://dp.la/api/collections/15e82b12ef89a63d03737461e2440df8", "id": "15e82b12ef89a63d03737461e2440df8", "title": "Records of the U.S. Senate, 1789 - 2011" }, "stateLocatedIn": { "name": "DC" }, "creator": "U.S. Senate. Office of Senate Curator.\\t(? -)", "isPartOf": "Series: Berryman Political Cartoon Collection, 1896 - 1949", "type": "image" }, "object": "http://media.nara.gov/Public_Vaults/14755_2006_001_t.jpg", "ingestDate": "2013-04-11T21:27:25.187803", "originalRecord": { "access-restriction": { "specific-access-restrictions": null, "restriction-status": "Unrestricted" }, "contributors": { "contributor": { "contributor-display": "Berryman, Clifford Kennedy, 1869-1949", "contributor-record-type": "PER", "contributor-type": "Artist", "standard": "Y", "num": "1", "contributor-id": "3119843" } }, "hierarchy": { "hierarchy-item": [{ "hierarchy-item-inclusive-dates": "1896 - 1949", "hierarchy-item-id": "306080", "hierarchy-item-lod": "Series", "hierarchy-item-title": "Berryman Political Cartoon Collection, 1896 - 1949" }, { "hierarchy-item-id": "375", "hierarchy-item-lod": "Record Group", "hierarchy-item-title": "Records of the U.S. Senate, 1789 - 2011", "hierarchy-item-record-group-number": "46" } ] }, "production-dates": { "production-date": "07/01/1928" }, "created-timestamp": "1/20/2013 4:36:49", "arc-id": "1693425", "use-restriction": { "specific-use-restrictions": null, "use-status": "Unrestricted" }, "title": "Yes, We Have No Ambitions Today!", "title-only": "Yes, We Have No Ambitions Today!", "general-records-types": { "general-records-type": { "num": "1", "general-records-type-desc": "Photographs and other Graphic Materials", "general-records-type-id": "4237050" } }, "scope-content-note": "This cartoon plays off a line from a popular 1923 song (\\" Yes, We Have No Bananas!\\") to characterize car maker Henry Ford\'s Presidential ambitions--or lack thereof. Ford blames his busy schedule for his hesitation to jump into the \\" Presidential contest pool, \\" while eager supporters encourage him to \\" come on in !\\" Berryman was correct in his prediction: Ford chose not to pursue the Presidency. ", "parent": { "parent-title": "Berryman Political Cartoon Collection, compiled 1896 - 1949", "parent-lod": "Series", "parent-id": "306080" }, "edited-timestamp": "[g_x128_110, g_x32_443, g_x64_221, g_x2_7090, g_x16_886, g_x8_1772, g_x4_3545, b_x1_14180]", "objects": { "object": [{ "thumbnail-url": "http://media.nara.gov/Public_Vaults/14755_2006_001_t.jpg", "object-sequence-number": "1", "file-size": "579687", "mime-type": "image/jpeg", "num": "1", "file-url": "http://media.nara.gov/Public_Vaults/14755_2006_001_a.jpg" }, { "description": "Download PDF", "object-sequence-number": "2", "file-size": "209895", "mime-type": "application/pdf", "num": "2", "file-url": "http://media.nara.gov/nwl/berryman/H-009_7-1-1928_Yes_We_Have_No_Ambitions_print.pdf" } ] }, "title-date": "07/01/1928", "subject-references": { "subject-reference": { "subject-type": "SRT", "display-name": "cartoons (humorous images)", "num": "1", "subject-id": "4170951", "standard": "Y" } }, "level-of-desc": { "level-id": "NAVI", "lod-display": "Item" }, "physical-occurrences": { "physical-occurrence": { "media-occurrences": { "media-occurrence": { "num": "1", "media-type": "Paper" } }, "reference-units": { "reference-unit": { "city": "Washington", "fax": "202-357-5911", "name": "Center for Legislative Archives", "ref-id": "36", "address2": "Room 8E, 7th and Pennsylvania Avenue NW", "summary": "true", "phone": "202-357-5350", "state": "DC", "num": "1", "postcode": "20408", "address1": "National Archives Building", "mailcode": "LL", "email": "legislative.archives@nara.gov" } }, "copy-status": "Preservation-Reproduction-Reference" } }, "creators": { "creator": { "creator-id": "1107050", "standard": "Y", "num": "1", "creator-record-type": "ORG", "creator-type": "Most Recent", "creator-display": "U.S. Senate. Office of Senate Curator.\\t(? - )", "summary": "true" } }, "variant-control-numbers": { "variant-control-number": { "mlr": "false", "variant-number": "NWL-46-BERRYMAN-H009", "num": "1", "variant-type": "NAIL Control Number", "variant-number-desc": "NWL-46-BERRYMAN-H009" } }, "arc-id-desc": "1693425", "indexable-dates": { "date-range": "[b_x16_120, b_x8_237, g_x64_30, b_x4_486, b_x16_119, g_x128_14, g_x32_59, g_x128_15, g_x16_118, g_x8_243, g_x32_60, g_x64_29, b_x2_974, b_x8_242, g_x16_121, g_x4_487]" }, "parent-control-group": { "parent-control-title": "Records of the U.S. Senate, 1789 - 2011", "parent-control-lod": "Record Group", "parent-control-id": "46" }, "_id": "1693425" }, "isShownAt": "http://research.archives.gov/description/1693425", "provider": { "@id": "http://dp.la/api/contributor/nara", "name": "National Archives and Records Administration" }, "@context": { "begin": { "@id": "dpla:dateRangeStart", "@type": "xsd:date" }, "@vocab": "http://purl.org/dc/terms/", "hasView": "edm:hasView", "name": "xsd:string", "object": "edm:object", "dpla": "http://dp.la/terms/", "collection": "dpla:aggregation", "edm": "http://www.europeana.eu/schemas/edm/", "end": { "@id": "dpla:end", "@type": "xsd:date" }, "state": "dpla:state", "aggregatedDigitalResource": "dpla:aggregatedDigitalResource", "coordinates": "dpla:coordinates", "isShownAt": "edm:isShownAt", "stateLocatedIn": "dpla:stateLocatedIn", "sourceResource": "edm:sourceResource", "dataProvider": "edm:dataProvider", "originalRecord": "dpla:originalRecord", "provider": "edm:provider", "LCSH": "http://id.loc.gov/authorities/subjects" }, "ingestType": "item", "dataProvider": "Center for Legislative Archives", "@id": "http://dp.la/api/items/02b4f072d067494f67b08d6a4100f143", "id": "02b4f072d067494f67b08d6a4100f143" } }

File

Yes, We Have No Ambitions Today! - Nara - 1693425.jpg

Author	Berryman, Clifford Kennedy, 1869-1949
Description	English: This cartoon plays off a line from a popular 1923 song ("Yes, We Have No Bananas!") to characterize car maker Henry Ford's Presidential ambitions--or lack thereof. Ford blames his busy schedule for his hesitation to jump into the "Presidential contest pool," while eager supporters encourage him to "come on in!" Berryman was correct in his prediction: Ford chose not to pursue the Presidency.
Date	1 July 1928 date QS:P571,+1928-07-01T00:00:00Z/11
Collection	National Archives and Records Administration    Native name National Archives and Records Administration Parent institution Federal Government of the United States  Location Washington, D.C. (headquarters), and many regional facilities and presidential libraries nationwide in the USA Coordinates 38° 53′ 34″ N, 77° 01′ 23″ W   Established 19 June 1934  Website www.archives.gov Authority file : Q518155 VIAF: 132254586 ISNI: 0000000121603036 ULAN: 500263780 LCCN: n84176101 NLA: 35718998 WorldCat institution QS:P195,Q518155
Record ID InfoField	This media is available in the holdings of the National Archives and Records Administration, cataloged under the National Archives Identifier (NAID) 1693425. This tag does not indicate the copyright status of the attached work. A normal copyright tag is still required. See Commons:Licensing. Record group: Record Group 46: Records of the U.S. Senate, 1789 - 2011 (National Archives Identifier: 37) Series: Berryman Political Cartoon Collection, 1896 - 1949 (National Archives Identifier: 306080) NWL-46-BERRYMAN-H009
Source	U.S. National Archives and Records Administration
Permission (Reusing this file)	This work is in the public domain in the United States because it is a work prepared by an officer or employee of the United States Government as part of that person’s official duties under the terms of Title 17, Chapter 1, Section 105 of the US Code. Note: This only applies to original works of the Federal Government and not to the work of any individual U.S. state, territory, commonwealth, county, municipality, or any other subdivision. This template also does not apply to postage stamp designs published by the United States Postal Service since 1978. (See § 313.6(C)(1) of Compendium of U.S. Copyright Office Practices). It also does not apply to certain US coins; see The US Mint Terms of Use. This file has been identified as being free of known restrictions under copyright law, including all related and neighboring rights. https://creativecommons.org/publicdomain/mark/1.0/PDMCreative Commons Public Domain Mark 1.0falsefalse
Other versions	Please do not overwrite this file: any restoration work should be uploaded with a new name and linked in this page's "other versions=" parameter, so that this file represents the exact file found in the NARA catalog record to which it links. The metadata on this page was imported directly from NARA's catalog record; additional descriptive text may be added by Wikimedians to the template below with the "description=" parameter, but please do not modify the other fields. (Note: Editors who post this notice are strongly encouraged to add details explaining how it applies to this file.)

Automatic or manually assisted: Automatic

Edit type (e.g. Continuous, daily, one time run): One time

Maximum edit rate (e.g. edits per minute): 10-15, as fast as it uploads

Bot flag requested: (Y/N): No

Programming language(s): Python 3.2

Will use metadata from DPLA bulk download for NARA. The metadata is in json, and is converted formatted to the template by the bot.

Smallman12q (talk) 23:41, 8 May 2013 (UTC)

Discussion

For reference, a previous NARA batch upload was approved at Commons:Bots/Requests/US National Archives bot.Smallman12q (talk) 23:41, 8 May 2013 (UTC)

• Yeah sure, looks good to me. --Dschwen (talk) 21:21, 9 May 2013 (UTC)

Usual suggestion: please use language template for Author/Source/Record ID fields. --EugeneZelenko (talk) 13:44, 11 May 2013 (UTC)

• Please can you put a deeplink in the "source" field, as that is where most editors will look. I tried to get the original of this example, but apparently "The Online Public Access (OPA) system will be down for maintenance from May 10 to May 25.", so we may not be able to thoroughly test this for a couple of weeks. --99of9 (talk) 13:01, 14 May 2013 (UTC)

Yes, I recently heard some details about that as well. I'll try to keep updated on the status. Bdcousineau (talk) 14:50, 16 May 2013 (UTC)

• What kind of label is: "NWL-46-BERRYMAN-H009"? It might help to add the name of this kind of identifier. --99of9 (talk) 13:02, 14 May 2013 (UTC)

That is an old catalog number used by NARA. It is no longer in use, but since it is in the current template used by NARA on Commons, it has been included. It most likely refers to the "NAIL" database, which was the in use prior to ARC/OPA, the current database. For a sample, see File:Football team on the field, Haskell Institute, Lawrence, Kansas, 1914 - NARA - 519149.jpg. Better removed? Bdcousineau (talk) 14:50, 16 May 2013 (UTC)

I'd suggest leaving it in there, but having the template do nothing with it (i.e. not display it). That way we can easily reintroduce it if someone thinks it is useful later. --99of9 (talk) 15:36, 16 May 2013 (UTC)

• 500k files! Wow, this is huge, congratulations and good luck! --99of9 (talk) 13:09, 14 May 2013 (UTC)

• Great start! Since this is a large set, and since the metadata will not be perfect (never is for a transfer of this size): are you thinking of staging this? Say, a few hundred to start, then 1k, then 10k, with pauses to see what sort of cleanup is needed? --SJ+ 22:40, 15 May 2013 (UTC)

 On hold-As stated at Online Public Access, access to records is suspended from the 10th to the 25th. (2 weeks is a loong roll out). Once access is restored, will do an initial batch upload of 100, 1000, then auto after that. Will also make source available once upload starts.Smallman12q (talk) 00:06, 24 May 2013 (UTC)

• Online Public Access online again. Bdcousineau (talk) 11:06, 29 May 2013 (UTC)

Can someone explain the process here? What does this have to do with DPLA (which does not host NARA images)? If you are just planning on copying the mostly low-resolution images from the catalog, I think we should slow down and concentrate on acquiring more of the high-resolution TIFF files like we did for the first mass upload. Also, with a separate mass upload based on a different set of source files, how are you planning to prevent uploading tens of thousands of duplicates? Dominic (talk) 16:37, 4 June 2013 (UTC)

@Smallman12q, @99of9, @Dominic: What's the state of this? Is there anything we're waiting for here? odder (talk) 16:17, 16 December 2013 (UTC)

If you ask me, this proposal wasn't very well-formed from the beginning. We already have a full-time staff member inside NARA (myself) who is working on preparing this sort of an upload, and I am working on doing it so we get the high resolution, use the full metadata from their own catalog, not DPLA, and so that it is consistent with the tens of thousands of other uploads already done. I think it is telling that my questions were never answered. Dominic (talk) 16:15, 21 December 2013 (UTC)

Ok, unless User:Smallman12q speaks up soon, I propose that we decline this request given that User:Dominic has something superior in the works. --99of9 (talk) 03:30, 9 January 2014 (UTC)

---

Declined per above. Also, noting Smallman12q's retirement, I'd like to thank him for all his efforts in bot writing. --99of9 (talk) 03:23, 13 January 2014 (UTC)