Module talk:TNTExpandByCountries

Luxembourg

See Commons talk:Freedom of panorama/Europe#Luxembourg. Commons:Freedom of panorama/Europe invokes TNTExpandByCountries|main|Europe to render a list of FoP sections, which in turn uses Module:Countries to get the country names, which gets the list of names from Module:Countries/Europe. Somewhere in there Luxembourg gets turned into Lëtzebuerg. But I cannot see Lëtzebuerg anywhere in the modules. I am mystified.

@Dalba, 4nn1l2, and Johnuniq: Any ideas on where the problem is? Aymatth2 (talk) 11:35, 20 September 2020 (UTC)

The problem is not limited to Luxembourg or the FOP page. The following countries have problems in all of the pages (e.g., Commons:Currency/Europe):

• République Démocratique du Congo
• République du Congo
• Ambazonia
• Canary Islands
• Madeira (Why the flag?)
• State of Palestine
• Deutschland
• Lëtzebuerg
• Vaticano
• Gagauzia
• UN Buffer Zone in Cyprus

4nn1l2 (talk) 01:14, 21 September 2020 (UTC)

The problem is bigger than it first appears. For example, see Commons:Freedom_of_panorama/Europe#Czech_Republic where it says "No information available", while there is a long section at Commons:Copyright_rules_by_territory/Czech_Republic#Freedom_of_panorama which should have been transcluded. The European FOP page tries to get the information from Commons:Copyright rules by territory/Česko, a redirect created on 9 March 2019. I wonder how long this problem has been around. 4nn1l2 (talk) 02:56, 21 September 2020 (UTC)

The reason Luxembourg becomes Lëtzebuerg is that the former is a redirect to the latter. Module:Countries determines that Luxembourg is a redirect and uses the target (Lëtzebuerg) as the linked title. I wrote the module but it's a couple of years since I thought about it and Verdy p has made a large number of changes since then. Verdy p might be in a better position to assist. I can't see a flag for Madeira—where's that? Johnuniq (talk) 04:56, 21 September 2020 (UTC)

The flag is at Commons:Freedom_of_panorama/Africa#_Madeira. It's because of Template:Madeira. The flag issue is not related to this problem. 4nn1l2 (talk) 05:28, 21 September 2020 (UTC)

Iran is also a redirect to ایران. ‌But it does not suffer from this problem. 4nn1l2 (talk) 05:33, 21 September 2020 (UTC)

The reason Luxembourg and Iran are processed differently is that Luxembourg has an alternate name ('Grand Duchy of Luxembourg') whereas Iran does not. I forget why Module:Countries works that way, but it does—if the country code (LU) has an alternate name, the module does what I said about the redirect. If it doesn't (IR), it just uses the given name. The LU and IR codes are defined in Module:Countries/Europe

LU = { 'Luxembourg', 'Grand Duchy of Luxembourg', qid = 'Q32', }

and Module:Countries/Asia

IR = { 'Iran', qid = 'Q794', }

The module is immensely complex because it has to deal with a lot of variables. For example, at Category:Luxembourg, the link is "Luxembourg" (not "Lëtzebuerg") because that category is not a redirect. Similarly, Category:Iran shows "Iran". I would need to do a great deal of investigation to work out why having an alternate name causes the redirect to be checked. It might have been a performance issue (a desire to not bother with all the other checks that are also performed), or perhaps there was some reason that I have forgotten. My guess is that all the alternate names that might be used are supposed to be listed in the country module, so if there aren't any, the module assumes there is no need to do a check since the only possible name is the one given. That is, perhaps the redirect for Iran should be added to Module:Countries/Asia. Johnuniq (talk) 07:31, 21 September 2020 (UTC)

TNTExpandByCountries procedure

I've poked around in an attempt to determine the underlying problem in the section above. It appears that Commons:Freedom of panorama/Europe uses

{{#invoke:TNTExpandByCountries|main
|Europe
|lang={{PAGELANGUAGE}}
|template=FOP Country
}}

to call Module:TNTExpandByCountries. That module calls Module:Countries to get the wikitext that would be generated by {{Countries of Europe}}. The wikitext is then split into what it hopes are country names, and those names are used to call other templates to generate what is seen at Commons:Freedom of panorama/Europe. That seems unwise to me because it hopes that Module:Countries will generate titles that match those used in Template:CRT list2. Why not simply parse the wikitext in that template? Johnuniq (talk) 10:37, 21 September 2020 (UTC)

I think one reason would be that Commons is a multilingual project, and localization (i18n) is a burden. Translating {{CRT list2}} (simple wikitext) is a difficult job because you need to translate the name of all countries manually and then sort them all. But translating Commons:Freedom of panorama/Asia (created by the module) is a straightforward job and you don't even need to be concerned about sorting. {{CRT list}} used to benefit from Module:Countries, but Aymatth2 changed its procedure[1]. Let me ping Dalba who created the module. 4nn1l2 (talk) 11:15, 21 September 2020 (UTC)

Some comments

• Sorting is a major problem, and one of the reasons {{CRT list2}} does not use Module:Countries. It is translated into over 20 languages, most of which are not sorted by Module:Countries. The solution was to hardcode the country list for each region into templates like {{CRT format Africa index}}, which take a lang parameter. User:Aymatth2/Sandbox shows what it renders with lang=de. The translator can simply click on "Sort by Country" to see the list in German language sequence, then copy-paste into the translation window.
• Pages like Commons:Freedom of panorama/Europe or Commons:Stamps/Public domain/Africa which use this module were developed largely for transitional purposes. The overhaul of Commons:Copyright rules by territory involved consolidating all information about copyright for a given country into a page for that country. Pages like Commons:Freedom of panorama/Europe generate similar results to the old topic-oriented lists of rules by country. I suspect that few people use them, so this problem may have been around for some time, but I would have spotted it if it existed before April 2019.
• According to Commons:Language policy, gallery names should generally be in the local language, but categories should be in English and policies and guidelines should have master versions in English with translations to other languages. We are not consistent. Democratic Republic of the Congo correctly redirects to République Démocratique du Congo and China redirects to 中国, but Côte d'Ivoire incorrectly redirects to Ivory Coast.
• The redirect rule does not seem to work consistently. Austria redirects to Österreich and Germany redirects to Deutschland, but Commons:Freedom of panorama/Europe is o.k. with Austria, not with Germany.
• I am not a Lua programmer, but wonder if it would help to have greater modularity in the Module:Countries structure, so this module could pick the logic it needed, and skip special logic for redirects etc. Otherwise, perhaps the logic could be namespace-specific. In galleries, apply the redirect rule, but in categories and commons, do not.

Aymatth2 (talk) 15:58, 21 September 2020 (UTC)

Thanks all for the info. I have started investigating how Module:TNTExpandByCountries works and will see what can be done with Module:Countries to resolve at least some of the problems mentioned. I don't understand all the points above and will probably have some questions. I will start by refactoring Module:TNTExpandByCountries but off-wiki things mean I won't do anything for a day or two. It's unlikely I would do anything significant in under a week. Question: what is the magic that makes Commons:Freedom_of_panorama/Europe/mk work? Is there something that makes a subpage special if the subpage name is a language code? Adding ?uselang=mk to the URL at Commons:Freedom of panorama/Europe changes some interface labels as expected but of course the text does not change. So (briefly) where does the text for the /mk subpage come from? What I'm really wondering is whether it uses the ?uselang=mk version of Template:Countries of Europe. Johnuniq (talk) 10:03, 22 September 2020 (UTC)

See the page information[2] where it says Page content language mk - Macedonian. This can't be changed by anyone (even admins), as it is a subpage that uses the Translate extension. However, admins can change the content language of normal pages (those that do not use the Translate extension). You can get the language of a page by the magic word {{PAGELANGUAGE}}. 4nn1l2 (talk) 10:44, 22 September 2020 (UTC)

This is a bit complicated. mw:Help:Extension:Translate/Page translation administration gives an overview of translation and Commons:Project page translation gives more information. Briefly,

• If you edit Commons:Freedom_of_panorama/Europe you will see some text enclosed in <translate> ... </translate>. That is the text that must be manually translated for each language version of this page.
• If you view Commons:Freedom_of_panorama/Europe and click on "Translate this page" above the "Other languages:" bar, you will be prompted to select a language. If you choose Macedonian you will get this page, which shows translatable English to the left and editable Macedonian text to the right. In this case all that can be translated is the page title and lead paragraph. The translations are stored in two small files used by the translation system when it displays the page in Macedonian. I forget where they are.
• Everything other than the translatable text is generated by code in Commons:Freedom of panorama/Europe: <languages/> renders the list of language versions at the top, supported by the translation system. {{FOP region index}}, {{Horizontal TOC}}, Module:TNTExpandByCountries and {{CRTtransfoot}} do the rest. They all try to find a translated version using {{PAGELANGUAGE}}, and fall back on English.
• The details for each country are formatted by {{FOP Country}}, which is called once for each country in Europe by Module:TNTExpandByCountries, passed the country name as first parameter, with the calls in Macedonian alphabetic sequence. If page language is mk and the country name is Austria, {{FOP Country}} will look for Commons:Copyright rules by territory/Austria/mk#слобода на панорама, or failing that Commons:Copyright rules by territory/Austria/en#Freedom of panorama. It transcludes the first it finds, or else renders the Macedonian translation of "No information available".
• The problem occurs when it is not passed the English form of a country name but is passed Lëtzebuerg, Deutschland, Vaticano, Gagauzia, or a country for which there is no COM:CRT entry like UN Buffer Zone in Cyprus.

Clear as mud? Aymatth2 (talk) 13:10, 22 September 2020 (UTC)

Thanks, that's helpful. You'll see that I refactored Module:TNTExpandByCountries and in the next day or two will investigate in more detail. Johnuniq (talk) 00:08, 23 September 2020 (UTC)

Perhaps the cleanest approach would be to have Module:Countries take an optional parameter to suppress the redirect logic, which Module:TNTExpandByCountries would pass to it. Aymatth2 (talk) 11:56, 24 September 2020 (UTC)

Yes, although I'm still not clear why the list of titles known to Module:Countries would always be what TNTExpandByCountries wants. At any rate, I'm getting closer to looking at the issue although probably not until next week. Johnuniq (talk) 00:18, 25 September 2020 (UTC)

Good point.

• We have a COM:Copyright rules by territory/country page for every country, former country or territory for which a user might expect to find copyright rules defined. That includes "normal" countries like France and Germany, but also includes Yugoslavia, Hong Kong, Guadeloupe, Antarctica etc.
• Module:Countries meets the need well at present, although there does not seem to be a clearly defined rule for what gets included in it. Maybe the informal rule is much what we need for COM:CRT, that "a user might expect Wikimedia to have a page describing this country or terriitory".
• I notice that UN Buffer Zone in Cyprus was added recently, and does not have a COM:Copyright rules by territory/ entry. But it is real enough, users might ask about the copyright rules for that zone, so we probably should give some guidance.
• {{CRT list2}} does not use Module:Countries, because it needed support for more languages, with the ability to easily make sorted lists of names in new languages. So possibly TNTExpandByCountries should also go its own way with separate lists.

However, my instinct is to keep using Module:Countries, and tweak it when problems like this come up. Aymatth2 (talk) 12:20, 25 September 2020 (UTC)

Investigation

Please see Module talk:Sandbox/Johnuniq which lists data extracted by my sandbox module. On the assumption that I work out how to use the first name for a country (and not follow redirects), the tables show what problems would remain. I'm thinking it would be straightforward to add exception data to the countries modules when needed but first I want to work out what exceptions are needed. Johnuniq (talk) 05:02, 26 September 2020 (UTC)

• My notes on the exceptions in the table below. I am concerned with Africa, Americas, Asia, Europe and Oceania, but have annotated the others. The "Wikidata en name" is the name of the country/territory used in the English Wikipedia, and presumably has been discussed and decided there, so can be taken as the "official" English name.
  • Ambazonia is marginal. It seems an unlikely term, but now has a stub COM:CRT entry referring to Commons:Copyright rules by territory/Cameroon. I have started an entry.
  • Canary Islands and Madeira now both have stub COM:CRT entries pointing to Commons:Copyright rules by territory/Portugal. Readers may not know they have the same laws as the home country. I can do that have done that.
  • United Nations Buffer Zone in Cyprus is an oddball. It could have a stub COM:CRT entry referring to Commons:Copyright rules by territory/Cyprus and Commons:Copyright rules by territory/Northern Cyprus. I can do that have done that: Commons:Copyright rules by territory/United Nations Buffer Zone in Cyprus. I used the full name as in the English wikipedia, and will move the category to match.
  • Gagauzia does not belong in Module:Europe. It is a region of Moldova.

Aymatth2 (talk) 13:41, 26 September 2020 (UTC)

Module	code	name1	Wikidata en name	Error	COM:CRT name
Africa	CI	Ivory Coast	Ivory Coast	#REDIRECTTemplate:Côte d'Ivoire	Ivory Coast
Africa	CMXAM	Ambazonia	Ambazonia	Template:Ambazonia	Ambazonia
Africa	ESCN	Canary Islands	Canary Islands	Template:Canary Islands	Canary Islands
Africa	PTMA	Madeira	Madeira	Category:Flag icon templates of the regions of Portugal	Madeira
Americas	BL	Saint Barthélemy	Saint Barthélemy	#redirect Template:Saint-Barthélemy	Saint Barthélemy
Asia	CN	China	People's Republic of China	{{Label|Q29520...}}	China
Europe	CYXUN	UN Buffer Zone in Cyprus	United Nations Buffer Zone in Cyprus	Template:UN Buffer Zone in Cyprus	United Nations Buffer Zone in Cyprus
Europe	MDGA	Gagauzia	Gagauzia	Template:Gagauzia	None. Country region

Following are not relevant to TNTExpandByCountries

Module	code	name1	Wikidata en name	Error	COM:CRT name
Caribbean	MF	Saint-Martin (France)	Saint-Martin	Template:Saint-Martin (France)	Saint Martin (France)
Caribbean	NLCAR	Caribbean Netherlands	Caribbean Netherlands	Template:Caribbean Netherlands	None. Territory
Caribbean	SX	Sint Maarten, Dutch Caribbean	Sint Maarten	Template:Sint Maarten, Dutch Caribbean	Sint Maarten
European Union	AX	Åland	Åland	Template:Åland	None. Region
European Union	BQ	Caribbean Netherlands	Caribbean Netherlands	Template:Caribbean Netherlands	None. Territory
European Union	CYXUN	UN Buffer Zone in Cyprus	United Nations Buffer Zone in Cyprus	Template:UN Buffer Zone in Cyprus	None. Oddball
European Union	DEXBU	Büsingen am Hochrhein	Büsingen am Hochrhein	Template:Büsingen am Hochrhein	None. Municipality
European Union	DEXHE	Heligoland	Heligoland	Template:Heligoland	None. Region
European Union	EL69	Mount Athos	Mount Athos	Template:Mount Athos	None. Oddball
European Union	ESCN	Canary Islands	Canary Islands	Template:Canary Islands	None. Territory
European Union	ITXCA	Campione d'Italia	Campione d'Italia	Template:Campione d'Italia	None. Commune
European Union	ITXLI	Livigno	Livigno	Template:Livigno	None. Region
European Union	MF	Saint-Martin (France)	Saint-Martin	Template:Saint-Martin (France)	Saint Martin (France)
European Union	PN	Pitcairn Islands	Pitcairn Islands	Template:Pitcairn Islands	None. Territory
European Union	PT20	Azores	Azores	Azores	None. Territory
European Union	PT30	Madeira	Madeira	Madeira	None. Territory
European Union	SX	Sint Maarten, Dutch Caribbean	Sint Maarten	Template:Sint Maarten, Dutch Caribbean	Sint Maarten
United Kingdom	GG	Bailiwick of Guernsey	Bailiwick of Guernsey	Template:Bailiwick of Guernsey‎	Guernsey
United Kingdom	PN	Pitcairn Islands	Pitcairn Islands	Template:Pitcairn Islands	None. Territory

Miscellaneous problems that need to be fixed:

Module	Module name	COM:CRT name	Fix
Africa	République Démocratique du Congo	Democratic Republic of the Congo	Do not redirect
Africa	République du Congo	Republic of the Congo	Do not redirect
Asia	State of Palestine	Commons:Copyright rules by territory/State of Palestine	(See below: fixed)
Asia	Republic of Artsakh	Commons:Copyright rules by territory/Republic of Artsakh	(See below: fixed)
Asia	Macau	Commons:Copyright rules by territory/Macau	(See below: fixed)
Europe	Deutschland	Germany	Do not redirect
Europe	Lëtzebuerg	Luxembourg	Do not redirect
Europe	Vaticano	Vatican state	Do not redirect

Aymatth2 (talk) 13:41, 26 September 2020 (UTC)

1) Canary Islands are in Spain, not Portugal. 2) Can't we treat Palestine like any other territory/country? Let's not complicate the code unnecessarily. 4nn1l2 (talk) 15:26, 26 September 2020 (UTC)

Agreed. 1) Spain it is. 2) Category:Palestinian territories was moved to Category:State of Palestine in June 2020. I moved Commons:Copyright rules by territory/Palestinian territories to Commons:Copyright rules by territory/State of Palestine to match it.

@4nn1l2: Could you please delete the redirects Commons:Copyright rules by territory/Republic of Artsakh and Commons:Copyright rules by territory/Macau? They are stopping me from moving these pages. Thanks, Aymatth2 (talk) 17:15, 26 September 2020 (UTC)

Thanks. Deleted. Regarding Palestine, I actually meant the following snippet from this module

if country == ':Category:State of Palestine' then
		return ''

Let's see what Johnuniq thinks. 4nn1l2 (talk) 17:30, 26 September 2020 (UTC)

I don't know the purpose of the special code for Palestine. I understand there is a lot of confusion/argument about a suitable name but I don't know the problem here. Let me know if there is something I might do regarding any outstanding issues. Johnuniq (talk) 09:35, 27 September 2020 (UTC)

I went ahead and boldly removed that part of code. Asian pages still look good at the moment. 4nn1l2 (talk) 12:47, 27 September 2020 (UTC)

Progress

I have made some changes that need to be assessed. See the history at Module:TNTExpandByCountries + Module:Countries + Module:Countries/Europe. The before/after effect on Commons:Freedom of panorama/Europe was:

• Template:Deutschland • fixed
• Template:Lëtzebuerg • fixed
• Template:Vaticano • fixed (but see below)

Under "Limited recognition", these are omitted:

• Gagauzia • good
• United Nations Buffer Zone in Cyprus • do not know how this happened

Problem: Vatican City shows "No information available".

• Template:Vatican City is the correct page but
• Commons:Copyright rules by territory/Vatican City is a redirect to Commons:Copyright rules by territory/Vatican state
• Something goes wrong.

Johnuniq (talk) 03:59, 27 September 2020 (UTC)

I moved the Vatican page, and it is fixed now. As far as I can see, everything is in order now. Thanks 4nn1l2 (talk) 04:52, 27 September 2020 (UTC)

The results in the COM:FOP/region pages all look good to me now apart from

• Madeira, which shows ok in the list but for some obscure reason does not render the contents of the Freedom of panorama section. I do not think this has anything to do with Module:Contries.
• United Nations Buffer Zone in Cyprus, which does not show up. Possibly this is because the name in the list is still UN Buffer Zone in Cyprus. It should be renamed to match the other pages/category/Wikidata.

I am going to continue with clean-up. Basically I would like to get rid of redirects and replace them with gallery pages so for each country or territory we consistently have a gallery, category, CRT page and template. This exercise has been very useful. Aymatth2 (talk) 11:47, 27 September 2020 (UTC)

Both fixed. Commons:Copyright rules by territory/Madeira had not been marked for translation. I marked it, and it showed up. I added "United Nations Buffer Zone in Cyprus" to Module:Countries/Europe [3]. Maybe we should get rid of "UN Buffer Zone in Cyprus" altogether. 4nn1l2 (talk) 12:41, 27 September 2020 (UTC)