Move zh-tw and zh-cn locales to zh-hant and zh-hans, respectively

[diox]

Recent version of django moved zh-tw and zh-cn locales to zh-hant and zh-hans. There is some code for backwards-compatibility, but that will go away and we'll need to handle that.

I had started doing that in mozilla/addons-server#5811 but it requires more thought so I reverted that commit and moved the discussion here. One of the issues is that if we change the locales like this, we need to add redirects from the old locales and migrate data in translated fields.

This blocks #4292 (See #4515)

[eviljeff]

tbh, this change (in django) makes me uneasy. zh-CN is a locale code - Chinese understood in mainland China, whereas zh-hans is a language+script code - Chinese written in Simplified Chinese (not just understood in mainland China). It's converting one thing to something different - although it's true that zh-CN has typically been misused to mean Simplified Chinese anyway.

[diox]

See https://code.djangoproject.com/ticket/18419 for the discussion in Django. It looks like zh-CN was indeed considered to be Simplified Chinese and that's why they changed it.

In any case, I'm not sure we'll be able to avoid the change, unless we keep a copy / symlink the old locales files for the strings translated in django itself and ignore the warning.

[yookoala]

In practice, zh-CN and zh-SG are subsets of zh-Hans; zh-TW and zh-HK are subset of zh-Hant. So zh-CN if not resolved directly, should be resolved as zh-Hans.

The difference within the subsets are usually vocabularies and translation costumes (much like the difference between en_US and en_UK). But the scripts used are common within the sets.

[diox]

Re-reading the django code I don't think it's a blocker for us... yet. When django removes the backwards-compatibility code, we'll be in trouble, but at the moment it will still work - and produce a deprecation warning.

[yookoala]

According to the discussion in the ticket, there should be a fallback for zh-tw, zh-hk to zh-Hant (and zh-cn, zh-sg to zh-Hans). This would be the ideal behaviour given the situation.

The pull request django/django#1868 seems to have implemented the idea and was merged years ago. But the behaviour of https://addons.mozilla.org/ does not follow the default Django. So is there something in the middle that went wrong?

[diox]

We don't follow the default because Mozilla products don't - the source of truth for our languages is currently https://product-details.mozilla.org/1.0/languages.json and it doesn't support zh-hant/zh-hans.

Right now, everything works - because of the fallback code. I created this issue because in more recent versions of Django, the fallback code emits a deprecation warning indicating that it might go away in the future. However, as I said in my previous comment, this is not necessarily a blocker for upgrading to Django 1.11, we can live with the deprecation warning, as long as the fallback code is still there.

[muffinresearch]

This may also be impacted by https://github.com/mozilla-services/cloudops-deployment/issues/1404

[eviljeff]

blocked by mozilla-services/cloudops-deployment#1404 pretty much

[eviljeff]

not blocked by mozilla-services/cloudops-deployment#1404 any more. Looks like the fallback was removed in django1.9 @diox 🐼

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If you think this bug should stay open, please comment on the issue with further details. Thank you for your contributions.