Cassandrich@dalias@hachyderm.io

I'm pleased to announce a big step in the #musl locale overhaul project: collation is working!

This is the result of a long project to design data structures and logic that would admit honoring the full Unicode Collation Algorithm rules, including equivalence under normalization, and keep the size of collation weight tables for the entirety of Unicode under control.

Full status announcement on the mailing list: openwall.com/lists/musl/2026/0

www.openwall.com/lists/musl/2026/09/03/4
Cassandrich@dalias@hachyderm.io

Replying to @⁨dysfun@social.treehouse.systems⁩

@dysfun For the full root collation data without fancy radical-stroke order for CJK (i.e. using legacy implicit weights for CJK), it comes in at just under 200 kB for something like 60k mappings, averaging around 3.3 bytes per mapping. That's including all of the data structure overhead to find the weights.

For the complete radical-stroke data, it's about 560 kB.

This is the mmappable locale data file for a locale, not part of libc .rodata or anything.

Cassandrich@dalias@hachyderm.io

Replying to @⁨dalias@hachyderm.io⁩

@dysfun In libc .rodata, there's 14k of decomposition and canonical combining class table, used for putting the input in form NFD to apply collation weight rules. This is something I'm not aware of any other libc having, much less in such compact form.

(AFAICT most just brush over not being able to sort odd noncanonical reorderings of characters as intended.)

en
Cassandrich@dalias@hachyderm.io

Replying to @⁨dalias@hachyderm.io⁩

@dysfun I'm still not sure how system integrators will want to do collation in locale data. My leaning would be to strip the language-specific locales down to just have culturally-correct logic for their own language's script as it's used in their language, so they're only a few tens of kB, and then have a few optional "big" files you can install and use for LC_COLLATE if you want a multilingual collation order.