Unicode Normalization: Strings That Look the Same but Are Not, Choosing Between NFC and NFKC, and Where to Normalize
Two file names that look identical slip past a unique constraint and search only half works. The difference between NFC, NFD and NFKC, normalizing at the boundary, enforcing it in the database and leaving signed data alone.
A user uploads a file called "Résumé.pdf". Later they upload another file with the same name and the system happily treats it as a new one. Someone searching for "résumé" finds only one of the two. The unique constraint has a hole in it and search is half broken, yet the two names on screen are identical down to the pixel. The cause is usually that Unicode lets you encode the same character as more than one sequence of code points. This post explains where that difference comes from, what each normalization form is for, and where in a system normalization belongs.
Two strings that look the same
The letter "é" can be written as a single code point (U+00E9). It can also be written as a plain "e" (U+0065) followed by a combining acute accent (U+0301). The first is precomposed, the second decomposed. They render the same, but they differ byte for byte, in length and in any hash you compute over them.
Decomposed text shows up more often than people expect. Some operating systems have historically stored file names in decomposed form, and names that leave those systems through archives, sync tools or drag and drop carry that form with them. Certain keyboard layouts and input methods, text copied out of PDFs and old data exports do the same. Linux file systems treat a name as a byte sequence and convert nothing, so both forms can sit side by side in one directory.
Four forms and what each is for
The Unicode standard (UAX #15) defines four normalization forms.
NFC composes characters wherever possible. It is the form the web and most databases expect, and it changes only the representation, not the text. Make it your default for storage.
NFD splits everything into base letters and combining marks. It is rarely a good storage form, but it is a useful intermediate step when you want an accent-insensitive search key: decompose, then drop the combining marks.
NFKC and NFKD add compatibility equivalence on top. The ligature "fi" becomes "fi", fullwidth "A" becomes a plain "A", superscript "²" becomes "2". This is lossy: "x²" and "x2" end up as the same string. So use NFKC for comparison and identifiers, not for storing what the user actually typed.
In short: store content as NFC, and build matching values such as usernames and search keys with NFKC plus case folding.
Where to normalize
The most common mistake is spreading normalization across the read side, at every place a comparison happens. One spot gets missed and the inconsistency is back. The better rule is to normalize at the boundary: text is converted to NFC the moment it enters the system (API request, form, file upload, import job) and only normalized text travels inside.
Every mainstream language has a standard call for this:
import unicodedata
def clean(s: str) -> str:
return unicodedata.normalize("NFC", s)
unicodedata.is_normalized("NFC", s)
const clean = (s) => s.normalize("NFC");
In Java it is java.text.Normalizer.normalize(s, Normalizer.Form.NFC), and in Go norm.NFC.String(s) from golang.org/x/text/unicode/norm.
Boundary normalization is not enough on its own; make the database enforce it too. PostgreSQL 13 and later provide the normalize() function and the IS NFC NORMALIZED predicate in UTF-8 databases. Tying that to a constraint stops a forgotten code path from quietly polluting the data:
ALTER TABLE files
ADD CONSTRAINT files_name_nfc CHECK (name IS NFC NORMALIZED);
UPDATE files SET name = normalize(name, NFC)
WHERE name IS NOT NFC NORMALIZED;
Fix existing rows before adding the constraint, or it will refuse to be created. During that cleanup two rows may collapse into the same NFC value and violate a unique index. Find those collisions with a query first and decide on each by hand. They are exactly the duplicates normalization was supposed to catch.
Identifiers play by different rules
Usernames, organization handles and similar values are a different class. The goal is not to preserve the text but to make sure two people cannot claim names that look the same. Apply NFKC, then case folding. In Python, casefold() rather than lower() is the tool built for this, and it handles letters like German "ß" correctly. Turkish dotted and dotless i is a separate, locale dependent problem and needs its own rule on top of folding.
Do not invent this yourself. The IETF PRECIS framework (RFC 8264) and its username and password profiles (RFC 8265) define which characters are allowed and which transformations run in which order.
Normalization does not solve visual lookalikes. Cyrillic "а" and Latin "a" are different letters and no normalization form merges them. Against impersonation, Unicode's security mechanisms (UTS #39) provide a confusables table and mixed-script detection. For most products, allowing a single script per username is a simple rule that goes a long way.
Passwords need normalization as well: a user typing the same password on two devices can send two different byte sequences and get locked out. NIST's digital identity guidelines recommend applying NFKC or NFKD to Unicode passwords before hashing. Pick one form once and never change it, or existing hashes will stop matching.
Pitfalls
Concatenating NFC strings does not give you NFC. Put an NFC "e" next to a lone combining acute accent and the result is no longer NFC. Code that normalizes pieces and then joins them must normalize the joined result again.
Length limits depend on the form. Decomposed text has more code points and more bytes. If you cap length by code points, the same name passes in one form and fails in the other. Most Linux file systems cap a file name in bytes, so a name that looks short on screen can still exceed the limit. And what the user perceives as a "character" is neither a code point nor a byte but a grapheme cluster. Measure after normalizing and document what you are counting.
Never normalize signed data. Normalize an HMAC signed webhook body, a hashed payload or a token and verification breaks. Normalization is for text; values that travel as bytes and are verified as bytes must stay untouched. That is why a blanket "sanitize every request" middleware is dangerous.
The name you write to disk may not be the name you read back. Some file systems transform names on write or perform lookups in a normalization-insensitive way. A sync tool that keeps NFC names in its database and meets a different form on disk can see "one file deleted, one file created" and produce spurious copies or deletions. In anything that syncs, compare names after normalizing both sides. The fact that Git has a dedicated setting for this (core.precomposeUnicode) tells you how common the problem is.
The search index and the query must go through the same pipeline. Build the index from NFC and leave the query raw, and a user who pastes decomposed text gets no results. If you want accent-insensitive search (the user types "resume" and finds "résumé"), do it in a separate search column with NFD plus mark removal, and leave the original data alone.
Nondeterministic collations are an escape hatch with a price. With PostgreSQL ICU collations you can make equality insensitive to normalization differences. That brings restrictions on pattern matching and some operators, plus a performance cost; read the documentation for your exact version before going that way. In most cases normalizing on write is simpler and faster.
When not to normalize
In systems that must hand back exactly the bytes they were given (backups, object storage, byte-exact file sync), rewriting the original name is data loss. Keep two columns there: the original name as submitted, and a normalized key used for uniqueness and search. Put the constraint on the key, show the user the original.
Also stay away from NFKC where changing content can change meaning, such as source code, configuration files or legal text; even NFC should be applied there only as a deliberate decision.
The working rule: content is stored as NFC, matching keys are built with NFKC and case folding, signed bytes are never touched, and the database enforces the contract with a constraint. Get those four decisions right and most "looks the same but isn't" bugs stop coming back.