[5.2] Language fallback for special languages like 'de-CH', 'de-AT' for extensions providing the fallback language

[MacJoom]

Summary of Changes - thanks go to Allon @laoneo and Corinne @webuniverse for discussion and code

• Add an additional tag to langmetadata.xml (admin & frontend)
  e.g. de-DE for a fallback to de-DE
  so extensions will use this fallback language
  (seperate PR will be made for the langmetadata.xml files, after this PR passes...)

• process the tag in Language.php (load)

Testing Instructions

• A Joomla installation with the default language 'de-CH' or 'de-AT' or an another language which is not the most used language which is usually provided by extensions
• Manually add the <fallbackTag>de-DE</fallbackTag> to 'language/de-CH/langmetadata.xml' and 'administrator/language/de-CH/langmetadata.xml' in the <metadata> section
  use your language files descriptions and whatever fallback language you want to use
  the section looks like this afterwards:

<metadata>
                <name>German (Switzerland)</name>
		<nativeName>Deutsch (Schweiz)</nativeName>
		<tag>de-CH</tag>
		<fallbackTag>de-DE</fallbackTag>
		<rtl>0</rtl>
.
.
.

• Have an extension installed that does not provide your special language but that does provide the fallback language
  e.g. have de-CH or de-AT installed as default language and install a component which provides de-DE
  As extension you can use for example DPCalendar as it comes automatically shipped with the de-DE language.

Actual result BEFORE applying this Pull Request

Language falls back to English, best checked in the backend of the component

Expected result AFTER applying this Pull Request

Fallback language is used

Link to documentations

Please select:

• Documentation link for docs.joomla.org:

• No documentation changes for docs.joomla.org needed

• Pull Request link for manual.joomla.org:

• No documentation changes for manual.joomla.org needed

[MacJoom]

Sorry i wanted this PR for 4.3-dev - can anyone change this?

[coolcat-creations]

Means if a particullar string is not available in the main language it falls back to the fallback language? Or the whole file only?

[MacJoom]

Means if a particullar string is not available in the main language it falls back to the fallback language? Or the whole file only?

It tries to use an extensions language file as a possible fallback - if the extension provides a language file, it is loaded. this does not work on the string level

[brianteeman]

Would it not be better to enhance the current behaviour.

As it is today if the de-AT language file for the component does not exist then it falls back to en-GB

1. load de-AT
2. if not exist then load en-GB

I propose that instead the workflow is as follows

1. load de-AT
2. if not exist then load any language file beginning with de
3. if not exist then load en-GB

This way the "fallback" language does not have to be specified. It just follows a logical path

fr-CA --> fr-* --> en-GB

[MacJoom]

Would it not be better to enhance the current behaviour.

As it is today if the de-AT language file for the component does not exist then it falls back to en-GB

1. load de-AT
2. if not exist then load en-GB

I propose that instead the workflow is as follows

1. load de-AT
2. if not exist then load any language file beginning with de
3. if not exist then load en-GB

This way the "fallback" language does not have to be specified. It just follows a logical path

fr-CA --> fr-* --> en-GB

Thats a good point - we did indeed discuss an automatic solution, but i realized that this code was called very often and checking for an available language file may impact performance. On this level (not the string level) there is no fallback to en-GB (which wrongly would be en-EN) - Anyway i would like to change as little as possible - and there are not many countries needing this ;-)

[brianteeman]

Anyway i would like to change as little as possible - and there are not many countries needing this ;-)

exactly why I believe my proposal is the most efficient. (It's also the method used by other applications)

[laoneo]

Another question. When a de-DE file and de-CH file exists are always both of them loaded?

And please update the test instructions so other can test it as well. As extension you can use for example DPCalendar as it comes automatically shipped with the de-DE language.

[MacJoom]

Another question. When a de-DE file and de-CH file exists are always both of them loaded?
Only the first available language is loaded, first language to test is the default language de-CH, then the fallback language.
(See the rather ugly early return at line 717 - existing code)

[laoneo]

I have tested this item ✅ successfully on 64e89d9

Tested it with DPCalendar as it comes with de-DE shipped. After installing the de-CH language I set the fallback language to de-DE in the metadata file of the site language and DPCalendar opened successfully in German. Before the patch it was English.

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/39152.}

[HLeithner]

I think it's the wrong direction. We should load a base language and override it with the localised Version.

[MacJoom]

I think it's the wrong direction. We should load a base language and override it with the localised Version.

You mean we should prepare (put in the array of files to be tried) automatically de-DE (derived from the first 2 characters in the language string) as well as the localised version (prior to the base version) without having the fallback tag?
We could do that - it changes the behaviour of actual installations however

[laoneo]

I think it's the wrong direction. We should load a base language and override it with the localised Version.

Please add more information how you would do it. It is hard to understand what exactly you want to have differently. For the use case mentioned in this pr, this is a solid solution. Of course you can also do it like Brian mentioned but I like this approach more, so it is completely under control of the translation teams.

[ChrisHoefliger]

I have tested this item ✅ successfully on e9bd3a3

I have tested this successfully on localhost with JCE both on site and in the backend.

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/39152.}

[laoneo]

RTC

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/39152.}

[laoneo]

Where does it need to be documented? Is there some documentation for translation managers?

[obuisard]

The language packs need to be created with the <fallbackTag/> tag. The language pack component should generate that tag, unless I am mistaken. It's not much that this PR needs to be documented, but language packs should now be able to contain the tag without needing a manual modification of the xml file. In other terms, these changes are going to be 'forgotten' if language files are not able to include <fallbackTag/>. Functionality will be there but not used. So there needs to be a 'follow up', maybe a PR for https://github.com/joomla-extensions/pkg_languagepack? Sorry, I am not expert on language package creation...

[laoneo]

Only the official core languages can use that tag es they define the fallback. It is not must that they have it. The extension dev doesn't need to add that tag at all to their language.

[zero-24]

The language pack component should generate that tag, unless I am mistaken.

There is no language pack component that would generate some tag, which as it is is the wrong solution in the first place to me too.

In other terms, these changes are going to be 'forgotten' if language files are not able to include .

The bigger issue is as soon as a better solutions without that tag, or a modified solution that also works for the core will be implemented that is going to be a big mess for b/c so it has to be fully documented on docs where the other language XML docs are placed and clearly mention what is intended, what are the ordering etc. Its going to be more of a problem too as core itself can not and will not use that tag in this setup for now. I still think we should think about a solution that can be applied to the core too.

Functionality will be there but not used. So there needs to be a 'follow up', maybe a PR for https://github.com/joomla-extensions/pkg_languagepack?

That component does not have anything todo with the generation and therefor a PR there does not help.

As long as this PR stays at it is it will always be a manuall task to be executed as the existing tooling would not output that tag, only none crowdin languages would have it easier as they could just workaround that and include the setting in a manuall step. Crowdin TTs need to manually alter the download comming from crowdin.

All of that should be documented on a docs page explaining what steps have to be done and what should be the consideration of using that tag and also that needs to be clear to the TT what is the result. The example used above with de-DE and de-AT is stright forward also as its comming from the same team. Not sure whether thats true for all other languages as well.

This is also going to be a B/C issue as it will massivly changing the behavior on a simple update. Right now the fallback in that case will be en-GB now in some places it will be de-AT or de-DE etc. To the regular user it is not understandable or even configuratable in the new setup other than hacking said component or revert this PR here. The other way arround would be a hacked TT package which is also bad.

I see the intention here is good but I would recomend to look deeper into this issue first and not just go with the first solution that comes into the mind and find a solution that can be applied on string basis similiar on how the fallback with en-GB is happening, and not redject comments with "it works as designed". Yes it works as designed but maybe we need a different design for the core setup.

Maybe its worth allowing that to be a setting, defauling to en-GB on user or even TT level but let that final decision of the user. That would allow the user to configure that setting and also allow us to perseve B/C for upgrades. Ontop on that I would say this should be string based too but said setting would allow full controll to the user and siteowner to do that fallback what makes sense to thier site. Maybe some kind of "language pack suggested" information can be provided but not forced to all people using that language pack i would say.

[obuisard]

I agree that for the core, we should place the decision in the hands of the user.
As a third-party developer, I see it as an advantage if a language is missing in my extension and that the user can get a 'close alternative' that I can provide.
I can see valid points here but, in the end, let's make sure the user is the one benefiting from this.

[zero-24]

I agree that for the core, we should place the decision in the hands of the user.

This here is a change to the core which has b/c issues and issues resulting into major changes to the users they dont have a way to control.

As a third-party developer, I see it as an advantage if a language is missing in my extension and that the user can get a 'close alternative' that I can provide.

Again I fully agree about the problem statment, is something I was thinking about looking into myself too a few times but never came to look into the code. But this solution here only makes matters worse by not allowing the users to decide whats happening. We have to think about the core to not just a workaround for extensions.

I can see valid points here but, in the end, let's make sure the user is the one benefiting from this.

I do too but I dont think that we should do this by forcing a major changes over them they have not control over to disable and changing an important part of joomla just half way in a minor update.

On the extension side of things there is no need to relay on core changes when you want to ship the de-DE strings for de-AT go and copy the de-DE file to de-AT, problem solved and your users are happy. But when we do a change in the core including chnages to translation packs we should think about all consequences and find a solution that has been checked and verified to be sutibel for the core. Translations are a big part where we in Joomla are good its not something that we should take easy and just apply major changes without any options aviable to the users.

[laoneo]

@zero-24 feel free to make an alternative pr.

On the extension side of things there is no need to relay on core changes when you want to ship the de-DE strings for de-AT go and copy the de-DE file to de-AT, problem solved and your users are happy

If you would maintain an extension which thousands of strings, then you would see that this is not an easy task. Especially when you use translation plattforms like crowdin or transifex.

I really see no point in making this another option which nobody understands.

[zero-24]

If you would maintain an extension which thousands of strings, then you would see that this is not an easy task. Especially when you use translation plattforms like crowdin or transifex.

A one to one copy of a translation file? You are using a build script right? What holds you back from adding one cp line? The tools mentiond should make it even easier and you know as good as me that the count does not matter much here as all the translations does not need to be managed by you anymore. You could add to your translation sync script the required cp statments and after that its also fully automated, cp de-DE to de-AT upload the strings, download the strings done. I do agree there are some management required when you actually want to manage different languages but this PR does not solve that anyway its a 1:1 copy/fallback for the complete file to de-DE too.

feel free to make an alternative pr.

Yes as mentiond in my comment its on my list already but i have not yet had the time to do a PR for that.

[Elfangor93]

In general I like to have a fallback language in Joomla, especially for extensions. Like every Swiss Joomla beginner a had hard times as I started Joomla to find out what to do in order for extensions to show up in German. Of course I had installed de-CH back then like every Swiss newby will probably do. So thank you very much @MacJoom for this solution.

I see this discussion heading towards a question of design principle...

• Is Joomla designed to be customized/controlled as much as possible by the user. Then I would suggest to add some kind of setting in the Joomla global configuration where the user can decide a fallback language by its own.
  (This is good for advanced users who want the freedom of customize everything to their taste.)

• Or is Joomla designed to be very easy in usage. Then I like this approach since every translation team decides for its userbase.
  (This principle is good for users who want the system to run with as few clicks as possible.)

Do we have some general design priciples from Joomla which can be used to make the decision? Something like "We want to have a CMS which is highly customizable." or "We want to have a CMS which is as easy to set up and use as possible."?

[zero-24]

Or is Joomla designed to be very easy in usage. Then I like this approach since every translation team decides for its userbase.
(This principle is good for users who want the system to run with as few clicks as possible.)

That solution is not only that, its breaking B/C on a update just because we can, that current code does not support core translations but only extensions and is adding a setting which is going to cause more complex coding to keep the than old behavior once a better solution has been build. ;)

When an extension developer like to implement the changes proposed here go for it just copy the de-DE file to de-AT (and similiar languages). But when something is implemented into the core we should think about all places where this has an impact.

I does not have to be a setting by the user, its just that I think we should not force it on updates and I personally think it should be configurable. But if anything we should not break b/c as well as not merge something thats only half way done and not transparent to the user.

[laoneo]

@zero-24 what breaks here bc?

[zero-24]

two ways. Sites that have setup an extension thats expected to show en-GB strings for de-AT or where de-AT overrides for en-GB have been created to mitigate the issue. As well as this is changin stuff thats inconsistend for core and some extensions too without any indication to the user that and why its changed.

[HLeithner]

This pull request has been automatically rebased to 5.1-dev.

[Elfangor93]

So how are we going on here?
Its out of discussion that a fallback possibility is needed, but we havent found yet the best way to do it right?

[HLeithner]

This pull request has been automatically rebased to 5.2-dev.