Repository navigation
Update default n-gram length for Text Transform to match default text recipe #2870
Description
@justinormont and the text team tuned default n-gram lengths for the default text recipe in the internal repo
These defaults are:
Word -- bigrams (w/ unigrams)
Character -- trigrams (w/o unigrams and bigrams)
One chart from his findings:
The line w/ the light blue call-out represents current ML.NET defaults (Unigram + Trichar)
The line w/ the light green call-out is the requested change (Bigram + Trichar)
The line w/ the pink call-out shows the Trigram+Trichar is better in terms of accuracy, but with a time hit, and accuracy has a cross over at NumIterations > 8 for Averaged Perceptron learner.
Activity
[-]Update default n-gram length in text transforms to match TLC[/-][+]Update default n-gram length in Text Transform to match TLC[/+][-]Update default n-gram length in Text Transform to match TLC[/-][+]Update default n-gram length for Text Transform to match TLC[/+][-]Update default n-gram length for Text Transform to match TLC[/-][+]Update default n-gram length for Text Transform to match default text recipe[/+]rogancarr commentedon Mar 6, 2019
Related to #2802
zeahmed commentedon Mar 25, 2019
@justinormont and @shauheen, do you want this to go in V1.0?
justinormont commentedon Mar 25, 2019
That's up to @shauheen. I'd say yes, as there's strong upsides of accuracy. You'll notice the large jump in accuracy (y-axis) when we move from the blue to green lines in the above graph.
The power of defaults should never be underestimated.
Related: #2305
najeeb-kazmi commentedon Jan 31, 2020
Tracking in #4749