Commons:Batch uploading/US National Archives 2

US National Archives

I am hoping to begin a bulk upload of media from the US National Archives in the next few weeks. This will be a very different approach from the first upload, which was based on uploading files from an offline drive and scraping HTML for the metadata. This time around, NARA has an API for our online catalog, and so I am building a bot, using mwclient, to upload using the live metadata and files from the API. Some details:

Dataset

The dataset includes all PD materials at https://catalog.archives.gov (API: https://catalog.archives.gov/api/v1). I plan to begin with a series of ~100,000 WWI-era photos. Technically, there are over 15 million files (and counting) in this dataset.

File names

The script is currently configured to name files with the formula: For single-page items:

• "File:[TITLE] - NARA - [NAID].ext"

  Where "[TITLE]" is the catalog record's title field, and "[NAID]" is the National Archives Identifier. If this is over the character limit, "[TITLE]" is automatically truncated, with "(...)" appended.

For multi-page items (since the above formula would give all files belonging to one catalog record the same title):

• "File:[TITLE] - NARA - [NAID] (page X).ext"

Metadata

We are developing a custom metadata mapping, since NARA does not adhere to a metadata standard. You can see the metadata template we use here: {{NARA-image-full}}. Some notes:

While all the records in this catalog come from NARA or partner institutions, there are many different facility locations, and some NARA facilities have their own institutions templates already (e.g. US presidential libraries). Therefore, I am creating institution templates to go along with all NARA locations, and the script will insert the correct institution template based on a mapping.

NARA's authority file is not yet mapped to Wikidata, however that is definitely something that would be useful in the future. For now, we will upload files with NARA's creator and author names and their NAIDs and links back to the catalog authority record. However, including the NAIDs in a Commons template field means that in the future, Wikidata could be used to make creator templates appear instead. Any help with this would be appreciated.

Licenses

Because NARA records are nearly all (>99%) derived from the records of US federal agencies, these uploads will use {{PD-USGov}} or its subtemplates. Most NARA records are in one of about 600 record groups based on their creating agency, so I am using a mapping of NARA record groups to Commons PD-USGov templates so that the bot can apply the more specific agency templates in most cases. Help filling out this mapping would be appreciated.

Nearly all holdings of the US National Archives are in the public domain as a work of the federal government (or, otherwise, due to age). This is marked in the "use restriction" field in the catalog, with a value of "Unrestricted" indicating public domain determination by the archivists. Therefore, the script will be configured to skip over any records in which the use restriction is anything other than "unrestricted" (even "possibly" ones, which could ultimately be PD, but need a human determination).

Categories

All uploads will be automatically categorized by the metadata template into Category:Media contributed by the National Archives and Records Administration and a category for the series they belong to (such as Category:US National Archives series: DOCUMERICA: The Environmental Protection Agency's Program to Photographically Document Subjects of Environmental Concern, compiled 1972 - 1977). Eventually, the script will be designed to create the series category if a file is uploaded for a series which does not yet have one.

When it comes to topical categories, past NARA uploads utilized the {{Uncategorized}} tag to encourage the community to add topical tags. However, since this creates work for the community, I am planning this time around to run uploads a small batch (hundreds to a few thousand) at a time, so I can upload them with one or more topical categories that apply to all records in the batch, rather than uncategorized.

Code

You can find the upload bot's code at https://github.com/usnationalarchives/wikimedia-upload. This project is being developed in public on NARA's official GitHub account. I would welcome collaboration (pull requests or otherwise) there. In addition, the Commons community is welcome to file issue reports on that repo.

Examples

The most recent test uploads can be viewed in Category:US National Archives series: American Unofficial Collection of World War I Photographs. I am still polishing the upload script, but these examples essentially represent what should be expected from the bot once it gets started.

Opinions

The bot account is technically already flagged from the last bulk upload a couple of years ago, however I would like to submit the current plan to community review before restarting uploads. If there are any opinions on the bot's design or the format of uploads or other issues, I am happy to hear them. We'd also like to know whether to limit what is uploaded in any way—as in, would Commons actually be interested in 15 million files, or might some of these, like the millions of census cards, not be of interest. Also, if anyone is interested in helping out with the coding or other tasks, please feel free to let me know. This is a big undertaking. Thanks! Dominic (talk) 17:25, 31 May 2017 (UTC)

Assigned to	Progress	Bot name	Category
User:Dominic	Coding	User:US National Archives bot	Category:Media contributed by the National Archives and Records Administration