| 01:36 | <tewuzij> | Well case insensitivity is terrible anyway |
| 01:43 | <sffc> | Databases aren't my area of expertise but if you want fuzzy searching you should normalize your text with NFD and Case Fold and build a search db out of that, rather than making a key with a version-sensitive sorting function |
| 01:49 | <sffc> | Collator-based matching is really powerful though so maybe there's a valid reason to consider building an index with a Collator-based transform |
| 03:54 | <littledan> | I kinda got the idea that collation keys might be better than NFD + case fold for some of these fuzzy comparison/searching cases from https://www.unicode.org/reports/tr10/#Searching but I hear about a lot more real systems using nfd+fold |
| 03:54 | <littledan> | So maybe I misinterpreted that section |
| 03:55 | <littledan> | (The non-stability point is well noted!) |
| 06:15 | <justingrant> |
AFAIK, many RDBMSs (like MySQL, Postgres, and SQL Server) default to case-insensitive collation for text columns. There may be per-DBMS variation in whether accent-insensitive (equivalent to Unicode tertiary, I suspect) is the default or not. But it's safe to assume that the majority of text data stored in relational databases is case-insensitively collated. So to an app developer, a DB query like
Some challenges to this approach:
So it seems unlikely that many DB app developers would want to store an extra, normalized copy of text data. Instead, they'd probably demand that their hosting providers freeze the collation library. This is what AWS seems to be doing, FWIW. A DBMS system relying on the OS's collation library seems dangerously brittle. I'm amazed that Postgres does this. Until today, I assumed that all DBMSs statically linked their collation library, and only revved it like any other backwards-incompatible change: put in a major version release, require rebuilding indexes to pick up the change, fixup or rebuild indexes when upgrading the DBMS, etc. |
| 07:00 | <hsivonen> | Got it. So Postgres relies on the OS's glibc for collation instead of statically linking a particular snapshot of a collation library with each release? Or is the OS's glibc fixed to match Postgres's behavior so that client apps will sort exactly like the DBMS does, even when not running any DB-related code? |
| 07:18 | <hsivonen> | sffc: Given https://github.com/unicode-org/icu4x/issues/3906#issuecomment-1688682654 , does there exist documentation for what are considered ECMA-402 design errors that i18n libraries should not copy? |
| 09:46 | <tewuzij> | Would the db work with strings akin to 'Maθiw'? |