Commons:SPARQL query service/Upcoming General Availability release

The WMF Search Platform team are happy to announce (16 December 2021‎) that Wikimedia Commons Query Service (WCQS) will soon be in production for general availability (GA), with a release date planned for 1 Feb, 2022. This SPARQL endpoint for the Structured Data on Commons (SDoC) dataset has been available as a beta service since July 2020, and we are excited to be able to now release it as a production-level service.

Moving WCQS from beta to production comprises some key updates to enhance stability and functionality of the service to avoid situations like the recent service outage – more details are provided below. The biggest change to user behavior will be the requirement for user authentication to use all endpoints. While we recognize that this may make it more difficult for some users to use WCQS, it allows us to address problematic queries that can cause service disruptions for all users, as frequently happens on Wikidata Query Service (WDQS).

WCQS is built upon the same principles and architecture as WDQS (with authentication being the only major exception). In general, this means that:

• The Flink-based Streaming Updater is responsible for updating WCQS with the newest Structured Data on Commons changes.
• Blazegraph is the query engine for WCQS.
• Blazegraph allows for federation, most importantly with WDQS.
• Same (or similar) limits for WDQS apply to timeouts for WCQS: one client is allowed 60 seconds of processing time each 60 seconds; one client is allowed 30 error queries/minute; we will treat generic user agents the same way, etc.

The WCQS beta service was served by Blazegraph and allowed federation as well, but we only used a single instance and updates happened in a weekly cadence. Now, we have a complete, 3 instances per data center cluster with real-time updates.

Similar to our WDQS Service Level Objective (SLO), our SLO for WCQS is for 99% of updates to SDoC to be handled within 10 minutes (this dashboard tracks our SLO performance). Keep in mind that, similarly to WDQS, while the new Streaming Updater resolved some of our issues with latency, Blazegraph’s instability can still affect update lag.

As mentioned above, the biggest difference between WDQS and WCQS is that WCQS requires authentication. While the beta service already had the authentication enabled for all endpoints, we had a bug there that turned it off for the /sparql endpoint. While we did not fix this bug in beta, authentication will be functional and maintained in production. We are aware that this authentication makes it more difficult to use apps and bots, and we plan to address this in the future by integrating WCQS with Wikimedia’s API gateway.

While authentication can make using WCQS more difficult than WDQS, we believe this will result in a better overall experience, as the current un-authenticated WDQS situation leads to frequent Blazegraph problems that are difficult to identify and resolve. Implementing user authentication helps us avoid these same issues on WCQS, and gives us more precise tools for pinpointing and banning user queries causing issues that can lead to service outages. In the future, the API Gateway will give us additional tooling that will help us improve the ease of using WCQS.

While there is always more work to do, having WCQS in production for general availability is a big step forward for Structured Data on Commons, and we are happy to be able to provide a more generally reliable and functional service for working with this dataset. We want to provide as much support through this transition as we can, especially in light of the change in authentication requirements, so please reach out to us if you have relevant issues with adapting to this change by reporting a problem, or leaving a comment on this page's talk page. We will also release a follow-up reminder before we deploy to production.

Thank you for all your patience and support in getting this service into production!