Commons:Spacemedia

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search

Spacemedia is a tool written by Don-vip that continuously harvests media libraries of various space agencies in order to find free media not yet uploaded to Wikimedia Commons.

The tool runs on Toolforge (Kubernetes/jdk17 cluster) and uses OptimusPrimeBot account to upload automatically new discovered media.

The discussion to authorize this tool takes place here.

As of 2024-05-05 the tool has already imported more than 200.000 files.

Links[edit]

Useful discussions / projects[edit]

Monitored repositories and status[edit]

Repositories providing free media with compatible licences
Agency / repository Licence(s) Remark Status
US Air Force / Space Force
https://www.flickr.com/photos/airforcespacecommand Cc-by-2.0, Cc-zero Should be PD-USGov-Military-Air Force ✓ Done
https://www.flickr.com/people/129133022@N07/ PD Should be PD-USGov-Military-Air Force ✓ Done
https://www.dvidshub.net/search/?q=&filter%5Btype%5D=image&filter%5Bbranch%5D=Space+Force&view=grid&sort=publishdate PD-USGov-Military-Space Force ✓ Done
https://www.spaceforce.mil/Multimedia/Photos/ PD-USGov-Military-Space Force ✓ Done
DLR (Deutsches Zentrum für Luft- und Raumfahrt, German Aerospace Center)
https://www.flickr.com/photos/dlr_de Cc-by-2.0, Cc-by-sa-2.0 ✓ Done
https://www.dlr.de/EN/organisation-dlr/news/all-news.html Cc-by-sa-3.0 Many news images in cc-by-sa, need to check individually though  Not done
ESA (European Space Agency)
http://www.esa.int/spaceinimages/Images Cc-by-sa-3.0-IGO ✓ Done
https://www.flickr.com/photos/europeanspaceagency/ Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark ✓ Done
https://www.flickr.com/photos/esa_events Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark ✓ Done
https://www.youtube.com/channel/UClB8L8TJEQfZ41Ii0gJRTSQ Cc-by-sa-3.0-IGO  Not done
EU (European Union)
https://emergency.copernicus.eu/mapping/list-of-activations-rapid - https://emergency.copernicus.eu/mapping/list-of-activations-risk-and-recovery Attribution-Copernicus required to inform about the source of information and/or data by using provided reference  Not done
https://www.copernicus.eu/en/media/image-day Attribution-Copernicus ✓ Done
https://erccportal.jrc.ec.europa.eu/ECHO-Products/Maps#/maps/latest PD-European-Commission Category:ECHO Daily Map of 2023  Half done
Observatories, Space telescopes, International Astronomical Union
https://www.eso.org/public/images/ ESO (CC-by-4.0)  Half done
https://www.iau.org/public/images/ CC-by-4.0  Half done
https://www.spacetelescope.org/images/ ESA-Hubble  Half done
https://hubblesite.org/resource-gallery/images PD-Hubble  Half done
https://webbtelescope.org/ PD-Webb ✓ Done
https://esawebb.org/ ESA-Webb ✓ Done
https://gallery.lsst.org Cc-by-4.0 Webdam (old)  Not done
https://rubin.canto.com/v/gallery/ Cc-by-4.0 Canto (new)  Not done
https://www.almaobservatory.org/en/copyright-notice/ Cc-by-4.0  Not done
https://noirlab.edu/public/images/ NOIRLab ✓ Done
https://nso.edu/about/image-use-policy/ Cc-By 4.0  Not done
INPE (Instituto Nacional de Pesquisas Espaciais) - Brazil
http://www.dpi.inpe.br/galeria/ INPE, Cc-By-Sa 4.0  Half done
https://www.flickr.com/photos/observacao-da-terra/ INPE, Cc-By-Sa 2.0 ✓ Done
ISRO (Indian Space Research Organisation)
http://pib.gov.in/PhotoCategories.aspx?MenuId=8 GODL-India (through PIB) https://gitlab.com/Gazoth/pib-upload/issues/1  Not done
KARI (Korean Aerospace Research Institute)
https://www.kari.re.kr/kor/kariimg/list.do?img_gbn=PHO KOGL ✓ Done
NASA (National Aeronautics and Space Administration) - live
https://images.nasa.gov/ PD-USGov-NASA  Half done
https://www.nasa.gov/image-article/ PD-USGov-NASA  Half done
https://www.nasa.gov/gallery/ PD-USGov-NASA  Half done
https://photojournal.jpl.nasa.gov/ PD-USGov-NASA Use Template:NASA Photojournal  Half done
https://earthobservatory.nasa.gov/images PD-USGov-NASA Use Template:ISS Crew Earth Observations  Not done
https://eol.jsc.nasa.gov/Collections/ PD-USGov-NASA To coordinate with User_talk:Askeuhd/Archive_2#Usefulness_of_ISS_image_dumps

Use Template:ISS Crew Earth Observations

 Not done
https://www.nasa.gov/multimedia/imagegallery/iotd.html PD-USGov-NASA To do. https://kaijento.github.io/2017/05/06/web-scraping-nasa-image-of-the-day/  Not done
https://solarsystem.nasa.gov/resources/ PD-USGov-NASA  Not done
https://www.sti.nasa.gov/harvesting-data-from-ntrs/ PD-USGov-NASA To do. Run requests between 8PM-8AM U.S. ET. No more than one request every 3 seconds.  Not done
https://svs.gsfc.nasa.gov/ PD-USGov-NASA  Half done
https://www.uahirise.org/ PD-NASA-HiRISE https://www.uahirise.org/media/usage.php  Not done
https://visibleearth.nasa.gov/ PD-USGov-NASA  Not done
https://asterweb.jpl.nasa.gov/ PD-USGov-NASA ✓ Done
https://modis.gsfc.nasa.gov/gallery/showall.php PD-USGov-NASA  Half done
https://sdo.gsfc.nasa.gov/ PD-USGov-NASA ✓ Done
https://epic.gsfc.nasa.gov/galleries PD-USGov-NASA  Not done
https://ails.arc.nasa.gov/ PD-USGov-NASA  Not done
https://chandra.si.edu/photo/ PD-USGov-NASA  Not done
https://www.dfrc.nasa.gov/Gallery/Photo/index.html PD-USGov-NASA  Not done
https://www.flickr.com/photos/atmospheric-infrared-sounder Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/earthrightnow Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/gsfc Cc-by-2.0, Cc-zero Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/morpheuslander Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa2explore Cc-by-2.0, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa_appel Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa_goddard Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa_ice Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasablueshift Cc-by-2.0, Cc-by-sa-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasacommons Flickr-no known copyright restrictions Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasaearthobservatory Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasafo CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasahubble Cc-by-2.0, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasakennedy Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasarobonaut Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasawebbtelescope Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/uahirise-mars Cc-zero, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://video.ibm.com/channel/nasa-tv-wallops TBC Should be PD-USGov-NASA  Not done
https://nasa-external-ocomm.app.box.com/s/onrtmdvofqluv5ei5kfu5u1pf8v4xqtl/folder/109925769802 PD-USGov-NASA ✓ Done
https://www.missionjuno.swri.edu/junocam/processing?source=public&ob_from=&ob_to=&perpage=100 Cc-By Be careful, the licence must be checked for each image, some are cc-by-nc  Not done
https://lroc.sese.asu.edu/posts PD-USGov-NASA ✓ Done
https://www.shadowcam.asu.edu/images PD-USGov-NASA ✓ Done
https://tothemoon.ser.asu.edu/ PD-USGov-NASA  Not done
NASA (National Aeronautics and Space Administration) - dead
http://grin.hq.nasa.gov PD-USGov-NASA Category:Great Images in NASA - Commons:Batch uploading/Great Images in NASA ✓ Done
https://www.ssc.nasa.gov/sirs/scripts/xmlWelcome.pl PD-USGov-NASA Stennis Image Retrieval System (SIRS) - shut down around 2020  Not done
http://lisar.larc.nasa.gov/UTILS/info.cgi? PD-USGov-NASA Langley Image Scanning, Archival and Retrieval  Not done
https://mix.msfc.nasa.gov/ PD-USGov-NASA Marshall image eXchange  Not done
NOAA (National Oceanic and Atmospheric Administration)
https://www.flickr.com/photos/noaasatellites/ CC-PD-Mark Should be PD-USGov-NOAA or PD-USGov-NASA  Half done
https://photolib.noaa.gov/Collections/NOAA-In-Space PD-USGov-NOAA  Not done
https://www.ngdc.noaa.gov/stp/satellite/goes-r.html PD-USGov-NOAA Curated SUVI images and animations processed at the National Centers for Environmental Information (NCEI)  Not done
http://sxi.ngdc.noaa.gov/ PD-USGov-NOAA Older SXI image data from GOES 12-15  Not done
https://www.ncei.noaa.gov/data/goes-r-series-l2-operational-space-weather-products/ PD-USGov-NOAA Original operational versions of SUVI images are also available through NCEI  Not done
USGS (United States Geological Survey)
https://lima.usgs.gov/access.php PD-USGov-USGS  Not done
https://astrogeology.usgs.gov/site/annex PD-USGov-USGS + PD-USGov-NASA  Not done
Individuals
https://www.flickr.com/photos/charlesandmaggie/ Cc-by-2.0 ✓ Done
https://www.flickr.com/photos/geckzilla Cc-by-2.0 ✓ Done
https://www.flickr.com/photos/harrystrangerphotographyhttps://www.flickr.com/photos/194849271@N04 PD ✓ Done
https://www.flickr.com/photos/192271236@N03 Cc-by-2.0 ✓ Done
https://www.flickr.com/people/kevinmgill Cc-by-2.0 ✓ Done
https://www.flickr.com/photos/pierre_markuse Cc-by-2.0 ✓ Done
Universities
https://www.cfa.harvard.edu/resources/image-gallery PD https://www.cfa.harvard.edu/news/image-use-policy  Not done
https://www.cassis.unibe.ch/ Cc-By-Sa 3.0 IGO  Not done
https://cimss.ssec.wisc.edu/satellite-blog/ PD https://cimss.ssec.wisc.edu/satellite-blog/about  Not done
Private companies
https://www.youtube.com/@arianespace Videos no longer free :( ✓ Done
https://www.planet.com/gallery/ Formerly Cc-by-4.0. No longer :(  Not done
https://www.flickr.com/photos/sentinelhub Cc-By 2.0 ✓ Done
https://www.flickr.com/photos/spacex Formerly Cc-zero-SpaceX. No longer :( ✓ Done
https://registry.opendata.aws/umbra-open-data/ Cc-By 4.0 https://umbra.space/open-license ✓ Done
https://registry.opendata.aws/capella_opendata/ Cc-By 4.0  Half done

Technical description[edit]

The tool is developed using Maven, Spring Boot, Spring Data JPA, Spring Cache, Jsoup, Dozer, Hibernate Search, Flickr4Java and Apache Commons libraries.

The tool connects to the Commons database replica to search for existing medias (by their base36 SHA-1) in these three tables:

  • image: current version of images
  • oldimage: old versions of images
  • filearchive: deleted images (when another version with a different SHA-1 exists)

The result of the analysis is persisted in a dedicated user database.

Most of the configuration is in application.properties. Sensitive information are not stored in the git repository and must be provided at runtime, on the command line.

Launch tool on local machine[edit]

java -Dcommons.datasource.username=$user -Dcommons.datasource.password=$password -Dflickr.api.key=$flickr_api_key -Dflickr.secret=$flickr_secret -jar spacemedia.jar --spring.profiles.active=dev

Launch tool on Toolforge[edit]

java -Xmx4G -Ddomain.datasource.url=jdbc:mariadb://tools.db.svc.eqiad.wmflabs:3306/${user}__spacemedia -Ddomain.datasource.username=$user -Ddomain.datasource.password=$password -Dcommons.datasource.username=$user -Dcommons.datasource.password=$password -Dflickr.api.key=$flickr_api_key -Dflickr.secret=$flickr_secret -jar spacemedia.jar --spring.profiles.active=toolforge

Resource usage[edit]

The production database takes around 300Mb.

The tool can be quite memory hungry when checking if ESA files are valid (to avoid uploading corrupted files to Wikimedia Commons), especially for large TIFF files.

Future[edit]