[browser][non-icu] `HybridGlobalization` SortKey by ilonatommy · Pull Request #84621 · dotnet/runtime

ilonatommy · 2023-04-11T11:30:14Z

Implements a chunk of web-api based globalization. Is a part of HybridGlobalization feature and contributes to #79989.

Old, icu-based private API: GlobalizationNative_GetSortKey

New, non-icu private API: none, Web API does not expose SortKey. We throw PlatformNotSupportedException.

Affected public API (see: tests in CompareInfoTests.cs):

CompareInfo.GetSortKey
CompareInfo.GetSortKeyLength
CompareInfo.GetHashCode

All changes in behavior are listed in docs\design\features\hybrid-globalization.md

This PR also refactors CompareInfo tests - it moves common properties/fields to CompareInfoTestsBase.

cc @SamMonoRT

ghost · 2023-04-11T11:30:23Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Implements a chunk of web-api based globalization. Is a part of HybridGlobalization feature and contributes to #79989.

Old, icu-based private API: GlobalizationNative_GetSortKey

New, non-icu private API: none, Web API does not expose SortKey. We throw PlatformNotSupportedException.

Affected public API (see: tests in CompareInfoTests.cs):

CompareInfo.GetSortKey
CompareInfo.GetSortKeyLength
CompareInfo.GetHashCode

All changes in behavior are listed in docs\design\features\hybrid-globalization.md

This PR also refactors CompareInfo tests - it moves common properties/fields to CompareInfoTestsBase.

cc @SamMonoRT

Author:	ilonatommy
Assignees:	ilonatommy
Labels:	`arch-wasm`, `area-System.Globalization`
Milestone:	-

ghost · 2023-04-11T11:30:35Z

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

Implements a chunk of web-api based globalization. Is a part of HybridGlobalization feature and contributes to #79989.

Old, icu-based private API: GlobalizationNative_GetSortKey

New, non-icu private API: none, Web API does not expose SortKey. We throw PlatformNotSupportedException.

Affected public API (see: tests in CompareInfoTests.cs):

CompareInfo.GetSortKey
CompareInfo.GetSortKeyLength
CompareInfo.GetHashCode

All changes in behavior are listed in docs\design\features\hybrid-globalization.md

This PR also refactors CompareInfo tests - it moves common properties/fields to CompareInfoTestsBase.

cc @SamMonoRT

Author:	ilonatommy
Assignees:	ilonatommy
Labels:	`arch-wasm`, `area-System.Globalization`
Milestone:	-

pavelsavara · 2023-04-12T08:12:23Z

What are the size wins ?
What are the use-cases which would be affected by this change ?

ilonatommy · 2023-04-12T08:35:24Z

What are the size wins ? What are the use-cases which would be affected by this change ?

Size win:
lack of SortKey is a side effect of removing collations data from ICU. Overall size decrease on full WASM ICU file (icudt.dat) ~40% (329770 bytes -> 202330 bytes on brotli compressed).

Use-cases (only of changes from this PR, not the whole collation):

CompareInfo.GetSortKey
CompareInfo.GetSortKeyLength
CompareInfo.GetHashCode

pavelsavara · 2023-04-12T08:50:04Z

(329770 bytes -> 202330 bytes on brotli compressed).

Thanks

Use-cases (only of changes from this PR, not the whole collation):
* CompareInfo.GetSortKey
* CompareInfo.GetSortKeyLength
* CompareInfo.GetHashCode

I mean what are the use-cases where those APIs are useful ?

This is what I tried https://github.com/search?q=CompareInfo.GetSortKey+lang%3AC%23+&type=code&p=3
I see mostly database oriented libraries.

Would it be better if we provide bitwise/invariant sortkey instead of PNSE ? Just an idea.

ilonatommy · 2023-04-12T09:11:45Z

I mean what are the use-cases where those APIs are useful ?

This is what I tried https://github.com/search?q=CompareInfo.GetSortKey+lang%3AC%23+&type=code&p=3 I see mostly database oriented libraries.

Sorry, I did not get it. It seems you are right, mostly DBs.

Would it be better if we provide bitwise/invariant sortkey instead of PNSE ? Just an idea.

It is not a bad idea but I think if we make it a rule for all locales, it could be confusing. User that calls e.g. new CultureInfo("de-DE").CompareInfo.GetSortKey() expects locale-sensitive sort key.
If they need invariant SortKey, they can switch on InvariantGlobalization and when they need locale-sensitive, functional SortKey, they should not use HybridGlobalization.
I can think about a use-case when your solution would be perfectly useful:

the user needs to cut down on runtime size (HybridGlobalization on)
they use sort key for locale-insensitive case, e.g.: CultureInfo.InvariantCulture.CompareInfo.GetSortKey()
but do not want to switch on InvariantGlobalization becuse they need locale-sensitive operations somewhere else.

I will stash this idea in the tracking issue in case there is a demand on the above case.

pavelsavara

LGMT

What is our plan for Android/iOS where maybe Hybrid could support these methods ?

pavelsavara · 2023-04-17T11:31:32Z

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.cs

        public int LCID => CultureInfo.GetCultureInfo(Name).LCID;
+
+#if TARGET_BROWSER
+        private static string GetPNSEText(string funcName) =>  $"{funcName} is not supported when HybridGlobalization=true. Disable it to load larger ICU bundle, then use this option.";


Should this be in resource file and localized ?

Good idea, I can see all exception text in PrivateCoreLib are stored like this, so you are right.

ilonatommy · 2023-04-17T11:50:19Z

What is our plan for Android/iOS where maybe Hybrid could support these methods ?

Android might have an invariant equivalent, it's dedicated for LDAP sorting: https://docs.oracle.com/javase/7/docs/api/javax/naming/ldap/SortKey.html. I do not see any culture-related SortKey.
For iOS I cannot find an equivalent, @mkhamoyan do you know about any?
Even if Hybrid SortKey will be supported on both of them, then PNSE will be left only for WASM.

Throw on SortKey.

715234e

ilonatommy added arch-wasm WebAssembly architecture area-System.Globalization labels Apr 11, 2023

ilonatommy requested review from kg, mkhamoyan and tarekgh April 11, 2023 11:30

ilonatommy self-assigned this Apr 11, 2023

ilonatommy added 6 commits April 11, 2023 13:35

Fix NLS tests.

dd19372

Do not add these files for WASI.

ea4a93e

Removed WASI from targets.

7dcbe65

Run tests with HG only on browser, rename.

020883a

Missing WASI removal.

c65d687

Merge branch 'main' into hg-sortkey

f46f9ac

ilonatommy mentioned this pull request Apr 12, 2023

[wasm][globalization][icu] Tracking issue for HybridGlobalization: Web API + ICU #79989

Closed

23 tasks

ilonatommy requested a review from pavelsavara April 17, 2023 06:17

General docs fix.

0056864

build-analysis bot mentioned this pull request Apr 17, 2023

Tracking issue for CI build timeouts #76454

Closed

pavelsavara approved these changes Apr 17, 2023

View reviewed changes

Storing exception text as resource string.

8eccd2e

runfoapp bot mentioned this pull request Apr 17, 2023

Infra improvements for Helix #68176

Closed

ilonatommy merged commit 907cc1c into dotnet:main Apr 18, 2023

ghost locked as resolved and limited conversation to collaborators May 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[browser][non-icu] `HybridGlobalization` SortKey#84621

[browser][non-icu] `HybridGlobalization` SortKey#84621
ilonatommy merged 9 commits intodotnet:mainfrom
ilonatommy:hg-sortkey

ilonatommy commented Apr 11, 2023

Uh oh!

ghost commented Apr 11, 2023

Uh oh!

ghost commented Apr 11, 2023

Uh oh!

pavelsavara commented Apr 12, 2023

Uh oh!

ilonatommy commented Apr 12, 2023

Uh oh!

pavelsavara commented Apr 12, 2023

Uh oh!

ilonatommy commented Apr 12, 2023

Uh oh!

pavelsavara left a comment

Uh oh!

pavelsavara Apr 17, 2023

Uh oh!

ilonatommy Apr 17, 2023

Uh oh!

ilonatommy commented Apr 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ilonatommy commented Apr 11, 2023

Uh oh!

ghost commented Apr 11, 2023

Uh oh!

ghost commented Apr 11, 2023

Uh oh!

pavelsavara commented Apr 12, 2023

Uh oh!

ilonatommy commented Apr 12, 2023

Uh oh!

pavelsavara commented Apr 12, 2023

Uh oh!

ilonatommy commented Apr 12, 2023

Uh oh!

pavelsavara left a comment

Choose a reason for hiding this comment

Uh oh!

pavelsavara Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

ilonatommy Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

ilonatommy commented Apr 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants