add twitter-cldr for bidirectional text support

[mikemorris]

This looks like it's currently inflating the size of the browserify'd package by over 1MB. I'm not sure how much of this could be avoidable by extracting the bidi logic from https://github.com/twitter/twitter-cldr-js (ugh, CoffeeScript) - right now this just pulls in the entire en.min.js localization file with datetime logic and such too. I THINK the bidi logic is templated and identical across all locales?

BUT, at least preliminarily, It Works™

Before

After

Refs mapbox/DEPRECATED-mapbox-gl#4

[mourner]

don't rely on node_modules path, you can't just do require('twitter_cldr/min/en.min.js')

[mourner]

In theory, if we write a simple bidi algorithm implementation, we could cut this down to just a few tens of kilobytes. All we need is the codepoint mapping to see if a particular character is RTL or not:

• https://github.com/mathiasbynens/unicode-8.0.0/blob/master/bidi/AL/code-points.js (Arabic RTL)
• https://github.com/mathiasbynens/unicode-8.0.0/blob/master/bidi/R/code-points.js (Hebrew RTL)

Total 14kb.

[mikemorris]

Extracted the bidi logic from twitter-cldr-js in 0ed31d5 and dropped the uncompressed size down to ~25kb. 5d1d891 is supposedly faster, but balloons bidi_classes.json up from ~8kb to over 40kb - should we rewrite this to be smaller, or is there a compression tool in the browserify pipeline that will rewrite the verbose keys?

[mourner]

5d1d891 is supposedly faster, but balloons bidi_classes.json up from ~8kb to over 40kb

I think the structure rewrite can be easily done in JS (once), not statically.

[camertron]

@mourner The codepoint mapping is where most of the size of the bidi feature comes from. When I initially wrote the feature I struggled to compress the mapping into a reasonable size. I'm sure more could be done in this regard (prefix tree, perhaps?).

[mikemorris]

Alright, finally got a chance to look at this again. Most of the weight appears to come from the "bidiClass", "min" and "max" key labels. I'm going to try writing a script to generate data from the Unicode source tables in a format like the following:

{
    "BN": [[0,8],[14,27]],
    "B": [[10,10],[13,13],[28,30]]
    "S": [[9,9],[11,11],[31,31]],
    "WS": [[12,12],[32,32]]
}

There are a few variations I want to benchmark for performance and size, one with single-entry arrays for the single-codepoint ranges, and another splitting out a big array for each bidiClass for the single-codepoint entries and a second array for the ranges. I think Variant B (below) might be the best option.

Variant A

{
    "BN": [[0,8],[14,27]],
    "B": [[10],[13],[28,30]]
    "S": [[9],[11],[31]],
    "WS": [[12],[32]]
}

for (var bidiClass in Object.keys(data)) {
    var group = data[bidiClass];
    for (var range in group) {
        if (range.length === 1 && codepoint === range[0]) {
            return bidiClass;
        } else {
            if (codepoint >= range[0] && codepoint <= range[1]) return bidiClass;
        }
    }
}

Variant B

{
    "BN": [[],[[0,8],[14,27]]],
    "B": [[10,13],[[28,30]]]
    "S": [[9,11,31],[]],
    "WS": [[12,32],[]]
}

for (var bidiClass in Object.keys(data)) {
    var group = data[bidiClass];
    var points = group[0];
    var ranges = group[1];

    if (points.indexOf(codepoint) !== -1) {
        return bidiClass;
    } else {
        for (var range in ranges) {
            if (codepoint >= range[0] && codepoint <= range[1]) return bidiClass;
        }
    }
}

@camertron I'm not actually sure what a prefix tree is, would love a little more detail if you think it's a worthwhile approach.

[mikemorris]

@camertron Would appreciate if you could point me to the source data for https://github.com/twitter/twitter-cldr-js/blob/master/lib/twitter_cldr/js/mustache/implementation/shared/prerender/bidi_classes.json in http://unicode.org/Public/8.0.0/ucd/

[camertron]

@mikemorris Here's the code that generates them.

[lucaswoj]

Have you considered formats other than JSON for storing this data? In particular, the StructArray class might enable a more compact representation.

[lucaswoj]

I am closing this PR because it has been inactive for a long time. The branch isn't going anywhere so please keep working on this feature and re-open the PR when it is ready!

Let me know if you have any questions.