TL;DR
Unicode normalization answers a defined text-equivalence question. It does not decide whether two people own the same account, whether names are visually confusable, or which spelling a product should display. Build an explicit identifier policy, preserve stable account IDs, and compute a separate lookup key. Before changing an existing policy, inventory collisions under the proposed transformation and decide how each will be resolved. A successful migration preserves ownership and recovery behavior as well as database uniqueness.
Exhibit A: the duplicate that a person cannot see
Imagine a collaboration service called LedgerRoom. Two customers independently register names that appear to be “café”. One name ends with the single code point U+00E9. The other ends with U+0065 followed by U+0301, a letter followed by a combining acute accent. Their stored sequences differ even though an ordinary rendering can look identical.
The service originally compared the stored bytes. It therefore accepted both registrations. Months later, the search team improves matching and begins treating the two forms as equivalent. Search now finds both people, but the login service still selects one exact sequence. Support receives screenshots that appear to describe a single account while the database contains two.
The following investigation, schema, and migration choices are a constructed design case. LedgerRoom is fictional. The point is to make the consequences of an equality rule inspectable before applying it to real customer records. The relevant Unicode mechanisms are linked separately; the proposed product policy is not a Unicode requirement.
Start the investigation by retrieving account IDs and code-point sequences, rather than repeatedly copying names through interfaces that may transform them. The account ID is the durable identity in this case. The typed name is an identifier chosen under an older policy. A screenshot can establish what someone saw, but it cannot establish which code points were stored or which person owns either record.
Do not begin by updating every stored name to a preferred form. That destroys the easiest evidence of the collision and may create an immediate uniqueness failure. First establish the current relationships among names, credentials, projects, memberships, recovery channels, and externally published links. The text problem sits inside an ownership problem.
Four transformations that answer different questions
Unicode Standard Annex #15 defines NFC and NFD for canonical equivalence, and NFKC and NFKD for compatibility equivalence. NFC composes where the normalization rules permit; NFD decomposes. The compatibility forms also remove certain distinctions, including some presentation variants. These are specified equivalence mechanisms, rather than generic instructions to simplify every string.
In the café example, NFC supplies a common representation for the two canonically equivalent endings. It does not remove the accent. An accent-insensitive search feature is an additional matching policy, with different consequences. A team that describes both operations as normalization can accidentally make login equality much broader than intended.
Case handling is another choice. ICU's case-mapping documentation distinguishes case folding from uppercasing and lowercasing, including locale-sensitive mappings. A case-insensitive identifier policy needs a specified operation; calling a locale-dependent lowercase function somewhere in the stack is not a complete cross-service contract.
Finally, visual similarity is not canonical equality. Unicode Technical Standard #39 describes confusable detection and internal skeletons. It explicitly distinguishes those mechanisms from a display representation. A suspiciously similar name can require review even when its normalized lookup key is different from every existing name.
LedgerRoom therefore has four separate questions to answer: what name to show, what sequence to use for exact account lookup, what alternatives search may suggest, and what resemblance should cause a warning. Combining them into one increasingly aggressive transformation makes the code look smaller while leaving the product behavior harder to explain.
Choose the identifier contract before the database function
LedgerRoom's proposed contract preserves the submitted spelling as display data and creates an NFC lookup key that remains case-sensitive. That is an illustrative conservative choice, not a universal recommendation. A service with established case-insensitive handles would need a different contract and a collision analysis covering its chosen case-folding behavior.
The contract must identify the namespace. A name unique within one organization is different from a globally unique public handle. If the lookup key is unique only within a tenant, every login or invitation lookup must carry a trustworthy tenant context. Removing that context during a migration can turn a valid local duplicate into an ambiguous global identity.
Also define which strings are eligible identifiers. Unicode Standard Annex #31 provides identifier specifications and the vocabulary for documenting profiles. A product should state its allowed repertoire and modifications rather than claiming that any text which normalizes successfully is an acceptable account name.
For this case, the contract distinguishes an account handle from a full personal name. A display name can contain spaces and punctuation that the handle policy disallows. This distinction lets people represent themselves without making every decorative spelling a routable identifier. It also prevents a later display-name edit from unexpectedly changing login behavior.
Write the policy as user-observable examples. Registering the two café sequences should produce the same lookup key. A name with an unaccented final letter remains distinct under this specific policy. A case variant remains distinct because case sensitivity is intentional. Each example states a promise that a reviewer can discuss without reading the implementation.
There are costs to every choice. A narrower equivalence rule preserves more distinctions but may surprise people who expect broad matching. A broader rule simplifies some lookups while collapsing names that previously referred to different accounts. The correct decision depends on the namespace's history, user expectations, and recovery obligations, not on which transformation yields the shortest expression.
Keep three fields and one stable relationship
The proposed account record has a stable account ID, a display handle, a lookup key, and an identifier-policy version. Credentials and permissions refer to the account ID. They do not refer to the current spelling of the handle. A rename changes how the account is found and shown without changing which projects it owns.
The display handle retains the spelling the product has agreed to show. The lookup key is computed under the applicable policy. The version records which contract produced it. This does not mean normalizing ordinary text is inherently unstable; it makes the broader product policy explicit, including eligibility and any additional mappings beyond normalization.
Apply the uniqueness rule to the lookup key within its namespace. Application validation can offer immediate feedback, but the durable uniqueness constraint must resolve concurrent registrations. Two requests can both observe that a name is available before either writes. A migration that adds a preliminary check without the corresponding storage invariant leaves that race unresolved.
Never make a confusable skeleton the account's primary identity. Under this design it is an auxiliary signal used for review or warnings. Its purpose is to help assess resemblance, not to redirect credentials or combine memberships. The UTS #39 documentation also warns that skeleton data has versioning considerations; treat an update as a change to a security signal, not as an ownership transfer.
The stable relationship is from authenticated principal to account ID. A typed handle helps locate a candidate record; it does not itself establish authorization. This sounds basic until a migration introduces aliases, fallback lookups, and automatic corrections. Every convenience path must still end at one unambiguous account before the normal authentication and authorization checks occur.
Inventory collisions before proposing a winner
Build a read-only candidate table containing each existing account ID, its current handle, its proposed key, and its namespace. Group by namespace and proposed key. A group containing multiple account IDs is a collision under the new policy, even if the current database considers every original handle unique.
Do not report only the number of colliding rows. Count collision groups, affected active accounts, published handles, recovery dependencies, and organizational relationships. Ten collisions involving inactive test accounts have a different migration cost from one collision involving two paying organizations whose public links circulate widely.
For each group, preserve evidence about the old lookup behavior. Could each account previously log in with its own exact spelling? Were invitations addressed by handle? Did password reset use a broad search before selecting an account? The last question may reveal an existing defect that the migration needs to contain rather than perpetuate.
A normalized-key collision is not evidence that the records are duplicates. Two independent customers can legitimately have registered different sequences under the original rules. Merging their accounts would combine private data and permissions without any proof of common ownership. The collision table should therefore use neutral labels such as “unresolved,” rather than “duplicate customer.”
LedgerRoom identifies three resolution classes. Accounts controlled by the same verified owner may follow an ordinary account-consolidation process. Independent owners need a rename or namespace decision with support involvement. Unverifiable or disputed ownership remains quarantined from automatic lookup changes until the identity issue is resolved through the service's established process.
Decide how abandoned names are reserved. If an account renames during resolution, immediately allowing another customer to claim its old visible spelling can create misleading historical links. A reservation period or an explicit tombstone may be appropriate. That is a product and support decision with a retention cost, not something the normalization algorithm decides.
Resolve one collision all the way through
Take the two café accounts again. Account A owns a design studio's projects; account B belongs to an unrelated consultant. Both owners can authenticate normally under the old exact-sequence rule. Neither has made a mistake merely by choosing a name the product allowed. The migration team needs a policy for scarce identifiers, not a technical claim that one account is invalid.
LedgerRoom offers both owners an explanation and a controlled rename process. Suppose the studio accepts a new public handle while the consultant retains the shared normalized spelling. The old studio spelling cannot simultaneously be a transparent login alias to the studio and an ordinary normalized lookup for the consultant. That conflict should be explained before either owner agrees to the change.
One resolution is to retire the ambiguous public spelling entirely and give both accounts new handles, while supporting recovery through verified channels. Another is to retain the spelling for one account but require the other to use an explicitly different identifier. A historical exact-sequence alias might exist in a tightly controlled recovery tool, yet it should not be advertised as a normal public handle if ordinary typing cannot reliably distinguish it.
The costs are concrete. Retiring both names creates more customer work. Awarding one name raises a fairness question. Preserving exact-sequence distinctions indefinitely extends support complexity. Writing down these alternatives makes the product decision reviewable and avoids presenting a database constraint as an unavoidable ruling about who deserves a name.
After resolution, test a mention, a project invitation, a saved profile link, and a recovery attempt for each owner. The expected destination must be stated separately for each workflow. If the team cannot explain where an old spelling goes, the collision is still unresolved even if the new unique index can be created successfully.
Make storage comparison match the promised contract
A computed lookup key is only useful if the storage layer compares it according to the intended rule. A database index or query may apply a collation that treats additional characters as equivalent. The application might preserve case in its key while the database compares keys without case distinctions. That combination implements a different policy from the one shown to users.
Treat the comparison mode as part of the schema contract. For LedgerRoom's proposed policy, the persisted normalized keys need a deterministic comparison that preserves the distinctions the contract promises. The precise database configuration depends on the engine and schema; this design does not prescribe a portable collation name or assume that defaults agree across environments.
Include deliberate negative examples in an integration check against the real storage configuration. Two inputs that must remain distinct should both be insertable in the same namespace. Two inputs that must collapse should produce one accepted registration and one defined conflict. Repeat the lookup checks through the same query paths used by login and invitations.
This catches a class of bugs that a unit test of the normalization function cannot see. The function may produce the expected sequence perfectly while a database, search index, or cache applies a broader comparison afterward. Agreement must hold through the full route from typed name to selected account, not merely at the transformation step.
Stage the migration without a split identity system
A safe rollout starts with observation. Compute proposed keys alongside existing records without using them to select accounts. Run the collision inventory, compare results across the services that process handles, and investigate disagreements. At this stage a defect in the candidate transformation changes a report rather than a customer's login destination.
Next, enforce the proposed rule for new registrations while maintaining the current behavior for existing unresolved accounts. This requires a deliberate admission strategy. New names must not claim keys already represented in the candidate inventory, including collision groups awaiting resolution. Otherwise the migration can create more affected users while support is resolving the first set.
Backfill only records whose outcomes are understood. A unique candidate key can be assigned after validation; a collision group needs the agreed resolution. Track these states explicitly. A single boolean named “migrated” cannot distinguish unprocessed data, a valid unique key, a disputed collision, and a completed rename with reserved aliases.
During a rolling deployment, older application instances may still write handles. Either route relevant writes through a version-aware service or use a deployment boundary that prevents obsolete writers from creating inconsistent records. Database uniqueness alone cannot correct a key that an old writer never computed. The rollout plan must account for every writer, including administrative tools and import jobs.
The eventual lookup switch should be one bounded policy change. Avoid an indefinite sequence that tries normalized lookup, then raw lookup, then case-insensitive search, then the first approximate result. Such a ladder can make the same input select different accounts depending on which records happen to exist. If a transitional input is ambiguous, the system needs a defined recovery path instead of a guess.
Rollback also needs precision. Reverting a reader implementation may be possible; undoing communicated renames and aliases is a separate operation. Record which changes are reversible in code and which have become customer-facing commitments. A rollback that restores old lookup code while leaving new names active can produce a combination that never existed in either policy.
Trace the invitation before declaring login fixed
Login is only one place where a handle selects an account. LedgerRoom also has invitations, mentions, public profile routes, audit exports, support searches, and administrative scripts. Each can apply a different matching rule unless the product assigns them a clear role. The migration should trace one concrete workflow through all these surfaces.
Suppose a project administrator pastes a café handle into an invitation dialog. The dialog should show the candidate's stable distinguishing information before sending, especially if the search feature supports broader matches than exact handle lookup. An invitation is an authorization-related action; approximate search should help a person choose, rather than silently choose for them.
The stored invitation should bind to an account ID once the intended account is known. If the product supports invitations to unregistered handles, those invitations require a separate claim policy. A later registration under a transformed spelling should not automatically inherit a pending invitation merely because a newly broadened comparison says the strings match.
Public routes create a similar question. An old profile URL may need an alias after a rename, but an alias is a durable mapping to an account, not a fresh search at request time. Otherwise a future registration could change where the old URL leads. Keep alias ownership and reservation behavior consistent with the collision-resolution decision.
Audit records should preserve enough historical context to explain what happened. Store the stable actor ID and, where appropriate, the handle displayed when the event occurred. Re-rendering every historical event with today's handle may be convenient, but it can erase why an administrator originally selected a particular account. The migration should not turn an identity repair into an unexplained rewrite of history.
Test the boundaries rather than a favorite example
The café pair is a useful starting point, but passing one example proves little about a transformation pipeline. Test that canonically equivalent inputs obtain the same proposed key and that deliberate distinctions remain distinct under the chosen policy. Include empty inputs, rejected characters, maximum lengths, and sequences that become different lengths after the complete transformation.
Use the Unicode Consortium's normalization conformance data when validating a normalization implementation. Application tests then sit above that mechanism: namespace scoping, collision reporting, key persistence, and ownership resolution. Passing a standards test does not prove that the product's account migration preserves the right person-to-project relationships.
Length needs an explicit unit. Bytes, code points, and user-perceived characters are different quantities. A product may enforce an input limit for resource control and a display limit for layout, but those limits should be described accurately. Check limits at the stages where they matter rather than assuming that the original and transformed representations have identical sizes.
Test fragments as well as whole strings if the implementation builds a handle from pieces. Independently normalizing components and concatenating them should not be assumed to produce the desired normalized whole; UAX #15 addresses normalization and concatenation. The proposed application should compute its lookup key after assembling the complete identifier under its documented policy.
Cross-service fixtures matter when a browser, an API, and a background job use different runtimes. Give each fixture an input sequence, expected eligibility result, expected lookup key, and policy version. Compare the stored key with the expected key, not merely with whatever the same local function returns during a test. Otherwise the test can reproduce its own mistake perfectly.
Add ownership assertions to migration tests. Both original café accounts must retain their private projects throughout collision resolution. A failed registration must not attach to the existing account. A renamed account's reserved alias must not become a fresh invitation target for someone else. These are the failures users would experience, even if every resulting string is valid NFC.
Preserve choice where visual resemblance remains
After canonical collisions are resolved, similar-looking names can still exist. That residual problem needs a product response calibrated to the namespace. LedgerRoom may warn an administrator when inviting an account whose handle resembles a colleague's, show organization and profile context, or reserve certain high-impact service names. These measures have different false-positive costs.
A universal ban on every resembling string would create its own usability problems. People using different writing systems can encounter rejections they cannot predict from the visible name. Review the warning experience with examples from the product's actual languages, and provide a reason that explains the relevant ambiguity without exposing private account details.
The display layer should help people distinguish accounts without requiring them to read code points during ordinary use. Stable profile context, an organization label, and an explicit confirmation screen can be more useful than a technical badge saying “Unicode checked.” A successful normalization policy should not promise that visual impersonation has become impossible.
Support tools need a deeper inspection mode. Showing escaped sequences, account IDs, aliases, policy versions, and collision history can let a trained operator diagnose a report. Keep that diagnostic view separate from the ordinary customer interface, whose job is to support a clear decision rather than teach Unicode internals.
LedgerRoom can call the migration complete when every identifier has a defined policy outcome, ambiguous groups have a controlled resolution path, all lookup writers agree, and credentials still lead to the intended account. The final asset is an explainable equality contract. The normalized string is one component of that contract, while ownership, recovery, and the meaning of a public name remain explicit product responsibilities.