Fallback mapping only for output

[apascual]

I noticed that the fallback mapping is only used for the output and not for the input. This creates some troubles like exporting a set of files that cannot be inserted back because incorrect language codes are resolved.

It also looks like so far the only purpose of the fallback mechanism is to support the Android regional codes that everybody use for simplified and traditional Chinese.

And then my question is, shouldn't the language code fix for Chinese be inside the Android formatter instead?

[sebastianludwig]

It's been a while since I worked on this part but IIRC the fallback is applied to all formatters, so also Apple would use zh-Hant before falling back to en, right?

This creates some troubles like exporting a set of files that cannot be inserted back because incorrect language codes are resolved.

What's your workflow? When/how/why do you re-import platform specific formats? Would specifying --lang work for you? What languages do you maintain in the Twine file and what are your desired output languages?

[apascual]

Hi @sebastianludwig,

It's been a while since I worked on this part but IIRC the fallback is applied to all formatters, so also Apple would use zh-Hant before falling back to en, right?

Correct, but in practice, with the existing default configuration it feels like it is there just to fix the problem with Android not supporting zh-Hans / zh-Hant.

What's your workflow? When/how/why do you re-import platform specific formats? Would specifying --lang work for you? What languages do you maintain in the Twine file and what are your desired output languages?

Our configuration is simple and I think quite typical. One (twine) file with multiple languages including zh-Hans and zh-Hant.

Four different actions that we run:

1. Export iOS strings: ✅ It does not need to use the fallback mechanism since the language codes in the twine file and the project are a perfect match.

2. Export Android strings: ✅ It uses the fallback mechanism to match zh-Hans -> zh-CN and zh-Hant -> zh-TW.

3. Export strings for translation: ⁉️ All the untranslated strings are exported at the same time using generate-all-localization-files into a new folder. There is no fallback mechanism in place here and using Android format produces folders called values-zh-rHans and values-zh-rHant which are incorrect in Android.

4. Consume translations: ❌ Importing translations with the folders mentioned above will not work as the Android formatter parser expects two characters for the regional code part instead of four. Manually renaming those folders into values-zh-rCN and values-zh-rTW will not work either because it will insert those strings as new languages (zh-CN/zh-TW) instead of match the existing ones (zh-Hans/zh-Hant).

And that is why I think that the problem of Android not supporting the standard codes for simplified/traditional Chinese should be addressed inside the Android formatter instead of using the generic fallback mechanism, that is also only one way (exporting).

And yes, manually incorporating each language file with the --lang parameter would probably work, but then we cannot use consume-all-localization-files to directly insert what we get back from translators.

[scelis]

Yes, I believe the current Android formatter does not support the newer BCP 47 support introduced in Android 7.0. It would be nice to add support for that and then remove the older fallback mechanism. Does this sound correct to you?

[apascual]

Yes, I believe the current Android formatter does not support the newer BCP 47 support introduced in Android 7.0. It would be nice to add support for that and then remove the older fallback mechanism. Does this sound correct to you?

@scelis Yes, that would work. If there would be support for BCP 47, then Android could use the modern language codes and no need to use fallbacks.

The abstract formatter (and maybe other parts of Twine) would also need to support BCP 47:

LANGUAGE_CODE_WITH_OPTIONAL_REGION_CODE = "[a-z]{2}(?:-[A-Za-z]{2})?"

[pastk]

3. Export strings for translation: All the untranslated strings are exported at the same time using
   generate-all-localization-files into a new folder. There is no fallback mechanism in place here and
   using Android format produces folders called values-zh-rHans and values-zh-rHant which are incorrect in Android.

I second that.
We have to use a custom patch to rename zh-Hans to zh and zh-Hant to zh-rTW.

BCP-47 support sounds good, but wouldn't that mean that android formatter will be Android 7.0+ only? What about earlier versions? I think there should be a fallback mechanism anyway.