01:36
<tewuzij>
Well case insensitivity is terrible anyway
01:43
<sffc>
Databases aren't my area of expertise but if you want fuzzy searching you should normalize your text with NFD and Case Fold and build a search db out of that, rather than making a key with a version-sensitive sorting function
01:49
<sffc>
Collator-based matching is really powerful though so maybe there's a valid reason to consider building an index with a Collator-based transform
03:54
<littledan>
I kinda got the idea that collation keys might be better than NFD + case fold for some of these fuzzy comparison/searching cases from https://www.unicode.org/reports/tr10/#Searching but I hear about a lot more real systems using nfd+fold
03:54
<littledan>
So maybe I misinterpreted that section
03:55
<littledan>
(The non-stability point is well noted!)
06:15
<justingrant>

if you want fuzzy searching

AFAIK, many RDBMSs (like MySQL, Postgres, and SQL Server) default to case-insensitive collation for text columns. There may be per-DBMS variation in whether accent-insensitive (equivalent to Unicode tertiary, I suspect) is the default or not. But it's safe to assume that the majority of text data stored in relational databases is case-insensitively collated.

So to an app developer, a DB query like SELECT lastname FROM inventors WHERE lastname = 'De Havilland' returning a result like"de Haviland" isn't "fuzzy" or special. It's just the default way that most non-Oracle relational databases have worked for many years.

normalize your text with NFD and Case Fold and build a search db out of that

Some challenges to this approach:

  • It doubles the storage requirements for any text column, except in the relatively rare case that you don't need to retain the original case of the column. Database performance and cost is generally constrained by data size, so for most large-scale applications it would be a non-starter to store data twice.
  • It requires application code (or code in the DB, like a computed column or trigger) to do the NFD + Case Fold operation when storing new rows. This adds complexity, hurts perf, and introduces failure cases like the two columns getting out of sync. The alternative of relying on the DB itself to take care of collation is easier, faster, and safer. (Assuming collation updates don't corrupt indexes!)

So it seems unlikely that many DB app developers would want to store an extra, normalized copy of text data. Instead, they'd probably demand that their hosting providers freeze the collation library. This is what AWS seems to be doing, FWIW.

A DBMS system relying on the OS's collation library seems dangerously brittle. I'm amazed that Postgres does this. Until today, I assumed that all DBMSs statically linked their collation library, and only revved it like any other backwards-incompatible change: put in a major version release, require rebuilding indexes to pick up the change, fixup or rebuild indexes when upgrading the DBMS, etc.

07:00
<hsivonen>
Got it. So Postgres relies on the OS's glibc for collation instead of statically linking a particular snapshot of a collation library with each release? Or is the OS's glibc fixed to match Postgres's behavior so that client apps will sort exactly like the DBMS does, even when not running any DB-related code?

Regardless, how can this problem actually be solved? If I want to have a case-insensitive, accent-insensitive DB index, and the rules for accent-insensitive comparison change between ICU releases, then the index has to be rebuilt, right?
As I understand it, Postgres calls glibc (unless configured to call ICU4C) for collation and expects the output for given input to stay the same, which is assuming a guarantee that glibc (other than the AWS special version) doesn't provide across glibc updates. I'm not a Postgres admin and don't know how the Postgres developers view this or how Postgres continues to do this despite it appearing extremely ill-advised from a collator implementor perspective. https://postgresql.verite.pro/blog/2018/08/27/glibc-upgrade.html (with enough searching, there should also be a blog post out there about a horror story of a Postgres database backup being unrestorable after a glibc update.)
07:18
<hsivonen>
sffc: Given https://github.com/unicode-org/icu4x/issues/3906#issuecomment-1688682654 , does there exist documentation for what are considered ECMA-402 design errors that i18n libraries should not copy?
09:46
<tewuzij>
Would the db work with strings akin to 'Maθiw'?