Proposal - UN M.49 region support to IETF language (#43)

[ratoaq2]

This is a proposal implementation for the region support in IETF language tag.

subtags occur in the following order:

1. A single primary language subtag based on a two-letter language code from ISO 639-1 (2002) or a three-letter code from ISO 639-2 (1998), ISO 639-3 (2007) or ISO 639-5 (2008), or registered through the BCP 47 process and composed of five to eight letters;
2. Up to three optional extended language subtags composed of three letters each, separated by hyphens; (There is currently no extended language subtag registered in the Language Subtag Registry without an equivalent and preferred primary language subtag. This component of language tags is preserved for backwards compatibility and to allow for future parts of ISO 639.)
3. An optional script subtag, based on a four-letter script code from ISO 15924 (usually written in Title Case);
4. An optional region subtag based on a two-letter country code from ISO 3166-1 alpha-2 (usually written in upper case), or a three-digit code from UN M.49 for geographical regions;
5. Optional variant subtags, separated by hyphens, each composed of five to eight letters, or of four characters starting with a digit; (Variant subtags are registered with IANA and not associated with any external standard.)
6. Optional extension subtags, separated by hyphens, each composed of a single character, with the exception of the letter x, and a hyphen followed by one or more subtags of two to eight characters each, separated by hyphens;
7. An optional private-use subtag, composed of the letter x and a hyphen followed by subtags of one to eight characters each, separated by hyphens.

This proposed change adds a region attribute to the Language class which makes this a non-backwards compatible change. It would be possible to reuse the country attribute to pass a country alpha2 code or a UN M.49 region code, but that would be a misuse of that attribute. Open to discuss alternatives.

The UN M.49 is a 3-letter numerical code. Since it is numerical and it should be 3-letter, the code should always be str, e.g.: '001', '419', etc

The data use was the official CSV downloaded from United Nations Statistics Division

A Region class was created to be as simple as the Script class. Although it has an English name, no converters for Region is created (similar to Script where there's no converter as well)

Some regions have country alpha2 code because they represent the country itself. So a country converter was added to be able to create countries from an UN M.49 code.

Not all countries have a UN M.49 code, like TW

When creating a language, the region attribute is defaulted to the country region, if any. Or the country attribute is defaulted if creating a language using the region code and the region represents a country.

The behaviour can be described with the sample snippet:

    assert Language.fromietf("es-419") == Language("spa", region="419")
    assert Language.fromietf("pt-076") == Language("por", "BR")
    assert Language.fromietf("pt-076") == Language.fromietf("pt-BR")
    assert Language.fromietf("pt-BR").region == "076"
    assert Language.fromietf('zh-TW').region is None

And one last point that I noticed that's not really related to this change:
In wikipedia (as quoted above), the script sub-tag comes before the country/region sub-tag:

3. An optional script subtag
4. An optional region subtag based on a two-letter country code or a three-digit code from UN M.49

Also mentioned here:
https://www.w3.org/International/questions/qa-choosing-language-tags
with some examples: zh-Latn-CN-pinyin

So, fra-FR-Latn in the tests should be fra-Latn-FR

That itself could result already in a non-backwards compatible fix in babelfish.

[ratoaq2]

my bad here... to be fixed later

[getzze]

I'm thinking that country could accept 3 digit numbers, so we don't have to create a region variable and it would simplify this proposal.