The Turkish I Problem: Locale Dependent Case Conversion, Database Collations and Catching It in Tests

An application runs happily on English text for years, then on a machine with a Turkish locale "ID".toLowerCase() returns "ıd", a configuration key goes missing, an enum fails to parse and the same email address registers twice. The culprit is the four letter i family in Turkish and Azerbaijani: I pairs with dotless ı, and dotted İ pairs with i. Unicode's default mapping lowercases I to i; the Turkish rule lowercases it to ı. In many runtimes the rule that applies depends on the process locale, and that locale differs between a developer laptop, CI and production. This post separates the two faces of the problem, shows the correct calls in common languages and in the database, and explains how to catch the bug in tests.

There are two problems, and one fix does not cover both

Case conversion is done for two different reasons, and most bugs come from mixing them up.

The first is machine text: protocol tokens, HTTP header names, enum values, file extensions, configuration keys, SQL keywords. These are not human language. Comparisons must be locale independent, and usually they are only defined over ASCII. Applying the Turkish rule here is simply wrong.

The second is human text: a name the user typed, a product title, a word in a search box. Here not applying the Turkish rule is wrong. A search for "İSTANBUL" should match "istanbul", and uppercasing "ılık" should produce "ILIK", not "İLIK".

The rule is short: use a fixed, culture independent conversion for machine text, and pass the culture explicitly for human text. Never rely on whatever locale the operating system has right now. A call that uses the default locale is guaranteed to be wrong in one of the two cases, and which one depends on the machine it runs on.

Default behaviour, language by language

What makes this painful is that the defaults do not agree with each other.

Java is the most dangerous. String.toLowerCase() and toUpperCase() without arguments use the JVM default locale. Under a Turkish JVM, "title".toUpperCase() yields "TİTLE" and Enum.valueOf throws. Write toLowerCase(Locale.ROOT) for machine text and toLowerCase(Locale.forLanguageTag("tr")) for Turkish human text. equalsIgnoreCase compares character by character and ignores the locale, which is fine for identifiers and not enough for Turkish search.

.NET is similar: ToLower() and ToUpper() use the current culture. For machine text use ToLowerInvariant(), or better, skip the conversion and compare with string.Equals(a, b, StringComparison.OrdinalIgnoreCase). Raising the CA1304 (specify CultureInfo) and CA1309 (use ordinal StringComparison) analyzer rules to errors catches this whole class at build time.

JavaScript toLowerCase() is locale independent, so it is safe for machine text and wrong for Turkish. On top of that, "İ".toLowerCase() returns two code points, i followed by a combining dot above (U+0307). A conversion that changes length breaks length limits and offset arithmetic. For Turkish text use toLocaleLowerCase("tr"); calling toLocaleLowerCase() with no argument falls back to the browser or Node process setting again.

Python str.lower() and casefold() are locale independent and show the same U+0307 behaviour. The standard library has no Turkish specific conversion; use an ICU binding, or map I and İ yourself before lowercasing.

Go strings.ToLower applies the Unicode default. For Turkish, the standard library ships strings.ToLowerSpecial(unicode.TurkishCase, s).

So the locale independent runtimes are right for machine text and wrong for human text, while the locale dependent ones are wrong for one of the two depending on their environment. In no language does the default call get both right.

The same problem in the database, but persistent

A bug in application code shows up in one request and is gone. A bug in the database ends up in indexes, unique constraints and stored data.

In PostgreSQL, lower(), upper() and ILIKE follow the column's collation. If the database was created with a Turkish libc locale, lower('ID') returns 'ıd'. If you enforce email uniqueness with a unique index on lower(email), the same expression behaves differently in a database created with a different locale, and restoring a backup into such a database can surface duplicates.

For fields that are really machine text (email, username, slug, external id), the most robust approach is to normalise in the application with a locale independent conversion and store the already lowercased value. The database then only checks byte equality and nothing is ambiguous:

ALTER TABLE users
  ADD CONSTRAINT email_is_normalized CHECK (email = lower(email COLLATE "C"));
CREATE UNIQUE INDEX users_email_key ON users (email);

lower() under COLLATE "C" only touches ASCII letters, ignores the locale, and the constraint behaves the same everywhere.

For searching human text you can define a nondeterministic ICU collation:

CREATE COLLATION tr_ci (provider = icu, locale = 'tr-TR-u-ks-level2', deterministic = false);
CREATE TABLE products (name text COLLATE tr_ci);

ks-level2 ignores case but keeps accents; ks-level1 ignores accents too, so "şeker" equals "seker". For a search box the latter is often more useful. This has costs: LIKE is not supported on nondeterministic collations before PostgreSQL 18, equality is more expensive than byte comparison, and indexes must be built with that collation. Sorting is a separate concern; using the ICU provided "tr-TR-x-icu" collation in ORDER BY puts Ç, Ğ, Ö, Ş and Ü in their proper places instead of after Z.

Pitfalls

The bug only appears in one environment. Most container images ship with C or C.UTF-8. Tests fail on the developer's Turkish laptop while CI stays green. Or the other way round: someone sets LANG=tr_TR.UTF-8 on a server so Turkish characters "display properly", and a library that worked for years breaks. Treating the locale as a display setting is the most common root cause.

Libraries are affected too. Fixing your own code is not enough; a template engine, a JSON mapper or a driver that calls toUpperCase without arguments produces the same bug. Pinning the process default (-Duser.language=en for the JVM, invariant globalization mode for .NET) is worth having as a defensive layer, but it should not be the only one.

Conversion changes length. Lowercased İ becomes two code points, and German ß uppercases to SS. If you check length limits after conversion, or cut strings at precomputed offsets, the result is wrong. Apply limits to the normalised value.

Unicode normalisation is a separate step. İ can be written as one code point or as I plus a combining dot. If you do not convert user input to NFC before comparing, two values that look identical will not be equal.

Comparing by converting is slow and unnecessary. a.toLowerCase() == b.toLowerCase() allocates two new strings and hides which rule it uses. Use the language's case insensitive comparison function and pass the rule explicitly.

Catching it in tests

Code review misses this, because toLowerCase() looks innocent. The reliable way to catch it is to run the test suite at least once under a Turkish locale: a second CI job with LANG=tr_TR.UTF-8, and -Duser.language=tr -Duser.country=TR for the JVM. If the locale is not generated in the image, the setting silently falls back to C and the job proves nothing, so print and assert the effective locale at the start. Add fixed test data containing I, ı, İ and i: an enum name used as an identifier, an email address and a search query.

When you do not need all of this

If your application only deals with ASCII identifiers and never converts or compares human text, enforcing locale independent calls at the language level plus one Turkish locale CI job is enough. ICU collations, a Turkish aware search layer and locale correct sorting only pay off when you have user facing search or listings. If search already lives in a dedicated search engine, configuring a Turkish analyzer there is less painful than wrestling with nondeterministic collations in the database.