TL;DR: A translation can match a reference sentence while changing the instructions a reader takes from the full notice. Use document-level review to identify consequential changes in scope, exceptions and references. Keep a similarity score for screening, and make the publication decision from errors tied to a reader's action.
On Monday morning, the front desk of an invented community center posts new opening instructions. The printing order includes a wall notice, a page for mobile readers, and a compact message that residents receive after a reservation. The source editor considers the update small. A gym is closed for a week. People who reserved equipment can still pick it up from a desk on the street side.
A translation arrives in a review application that displays one segment at a time. The sentences are natural and the name of the center appears consistently. The output earns a promising automatic score. After the front desk prints a preview, a receptionist asks whether residents may walk through the gym to reach the equipment counter. Each translated sentence sounded plausible. Together they give two different answers to that question.
A study led by Markus Freitag used professional translators with document context and explicit error analysis grounded in Multidimensional Quality Metrics, commonly called MQM. Its experiments concerned systems submitted to a specific research shared task. The practical idea for our center is to let a trained reviewer record where a translation changes a document's meaning and why that change matters, with the neighboring text available to inspect. The notice, languages and scores below are teaching examples in English gloss. They represent no real system or published locale results. Experts, Errors, and Context.
Edition A goes up inside the entrance
The source wall notice has a large heading for the gym closure. An adjoining panel labels the equipment room. The third block asks borrowers to use the street-side door. In the design file, a small drawing identifies that door by showing the counter's location. The last paragraph explains that normal gym sessions resume on the following Monday.
The draft translation interprets the wide heading as naming the whole center and expresses the gym closure as a closure of the building. The equipment paragraph remains accurate when read by itself. A reviewer who only sees the second paragraph can correctly approve the available service. A reader at the actual door meets a heading implying the place is closed and a later paragraph directing them inside. The contradictions emerge when the two statements share a page.
The first editorial mark covers the term that changed the building's scope. A text span and source span anchor the observation. A second note names the reader effect, which is that some people will abandon a valid pickup. The reviewer need only file one underlying meaning error. Counting the repeated consequences as separate lexical errors would make the tally depend on the reviewer's prose and how often a title was reused in the layout.
Another paragraph translates a reference to the street-side entrance as the main entrance. The picture is accurate, but readers who see only the shortened message will lose the picture. The localization owner records the message itself as a separate delivered artifact. A layout correction on the wall cannot rescue a string reused elsewhere. Keeping rendered variants together helps reviewers recognize both source dependencies and cases where a dependency has genuinely been removed.
The heading has a second possible issue. The team asks how a local reader normally distinguishes a particular gym from a wider community facility. An experienced reviewer may prefer a regional usage that differs from the translated word. That preference needs a note only if audience assumptions, required terminology or the clarity of the instruction change. A stylistic alternate during a shared review need not accumulate severity the way a changed access rule does.
The sign-off path is visible in the source record. The facilities manager owns the actual service decision. The translator owns the rendering of the words and can flag an ambiguous phrase. The support desk checks which doorway can be identified by a visitor. A question about access can move upstream to the manager instead of being answered by a confident translation guess. The translated copy and its supporting evidence stay linked when the source changes.
Edition B reaches someone carrying a reservation
The message begins by naming the recipient's equipment reservation. It compresses the closure statement into one line and directs the borrower to the street-side desk. It includes a date range and a link to the center's opening information.
The second translation handles the door correctly. It weakens the condition defining who can use the desk. In its English gloss, the instruction now applies to anyone who wants equipment during the closure. The center actually reserved staffed pickup slots for existing reservations. A reader can act on the broader invitation, arrive with no reservation, and consume time at the counter. A model that preserves the rest of the sentence may still miss the boundary the service team cares about.
The reviewer records the condition independently of lexical fluency. A short published notice may carry only one operational distinction. Leaving out a word that narrows the audience can have more impact than leaving out a graceful greeting. When the room closes over a weekend, a wrong end date can create a second issue even if the audience rule is translated correctly. These labels explain different causes and different repairs.
The date review requires its own evidence. The project source records the last day the gym remains closed and the first ordinary opening. A source phrase such as through Sunday gives both cues only when the date and weekly schedule are available. A template that truncates an ending can accidentally remove a day while retaining the word Sunday. If the source describes resumption by day name alone, a localization lead requests dates for messages that may be read late.
On a small phone, the message preview shows a different opening line from the wall notice. In a screen reader's reading order, an image may be represented through a short textual alternative. The affected document context is the sequence that a reader receives. Reviewers ought to see the real sequence with inputs such as line break, link label, and preceding reminder text. The translation evaluation can include a render check without treating the accessibility of the entire system as a single translation score.
Edition B also belongs to a separate delivery moment. The center sends it to residents after the original automatic reminder. That earlier reminder includes a default pickup location. The later message calls attention to the move, but only the long version says where the old instructions have changed. Some readers will follow the familiar direction from the first message. The translated notice must preserve the correction, and the operations team controls whether outdated reminders continue circulating. A good localization report can identify this interaction without assigning a delivery failure to a word choice.
The review bench has three piles
A blind editor receives the source and two model outputs with their identities masked. For each version the source is shown with the relevant prior sentence, instruction sequence, page or message type, and a committed list of required terms. The reviewer sees the original and submitted translation. For the initial review, they have enough evidence to label meaning changes before considering another candidate's interpretation.
The first pile contains release blockers. It includes errors that materially change who receives a service, where to go, and on which dates. The category is defined through product impact. Publishing a wrong closing date blocks this notice even if it creates little dispute among translators. A house style guide cannot turn a wrong date into a preference.
The second pile contains changes the editor can repair in the normal review window. A local term for the equipment desk may be awkward but unambiguous from context. The guidance distinguishes a repair that makes the sentence easier to use from a repair needed to make the sentence true. The distinction lets a test report usable draft quality without suggesting that an unreviewed version deserves the same release authority.
The third pile contains discussion notes. A reviewer may prefer a shorter heading while the designer prefers consistent names across versions. That debate becomes a design decision unless it reveals an explicit requirement. If none of the context supports choosing one expression, the reviewer preserves both readings and asks the source owner to clarify. Some recorded ambiguities indicate an under-specified original rather than a model performance difference.
The editor maintains a traceable vocabulary. An error annotation might include affected span, source evidence, type of change, practical consequence, severity and proposed disposition. Automatic reporting can total the marks by category. For a before-and-after comparison, each release records when the reviewer accepted the edit and who independently checked a repaired blocker. Each note remains useful after its score has been exported.
Reviewers can disagree for legitimate reasons. One may read a long form with an explicit reference and another may assess the standalone snippet. Both judgments can be right for the surfaces they inspected. Assigning one severity after a discussion about the document first is different from averaging two grades. The project calibrates reviewers on a few complete documents, records disputes, and sends uncertain high-impact terms to a qualified reviewer with relevant locale knowledge.
Some review cases include reference translations. A trusted reference gives a useful alternative and can support text similarity. It can also contain an editorial adaptation that the candidate was specifically asked to avoid. The release owner describes the reference's intended task and whether it has itself been validated against the product decision. A human-authored reference for last month's notice can be irrelevant after an entrance changes.
Put the grades back beside the delivery decision
An automatic measure can reduce the number of rendered pages requiring initial human investigation. For example, the chrF method compares character sequences through precision and recall combined into an F-score. That can reward a reasonable translation that shares substantial wording with a reference, and it is an aggregate calculation on text sequences. A small span in the candidate may drive the action even if surrounding wording matches beautifully. chrF for machine translation evaluation.
Suppose two systems each translate twenty notices. Each evaluation version contains about two thousand source words under a documented counting policy. The center's fictional scoring sheet weights a blocker at ten, an ordinary required edit at three, and a minor issue at one. System A has two blockers, four required edits and six minor issues for a raw weighted total of 38. System B has no blockers, seven required edits and twelve minor issues for 33. Normalized per thousand source words, the totals are 19 and 16.5. These weights belong to the example and have no status as a universal MQM rule.
If the center uses a blanket per-thousand threshold of twenty, both systems look eligible. Its blocking policy still holds A until corrections are reviewed. System B has a larger count of required edits. If a reviewer needs five minutes for each of those edits and two for each minor issue, the estimated editing work in this batch is 59 minutes for B. A's ordinary work looks shorter, but that estimate excludes the time needed for service confirmation and verification of blockers. The decision needs both figures without pretending they sum to a single meaningful scalar.
When a document has only forty words, the per-thousand figure for one error can jump sharply. A normalized rate works for a suite with enough content and compatible scope. At the level of one notice, the release desk wants the concrete wrong action and the repair state. It can report rates across complete notice families and counts within individual artifacts, connecting them with versioned identification.
The difference between full and abbreviated editions also changes coverage. Suppose eleven notices in the test have an important exception and nine are routine reminders. A system can improve the routine group's automatic score while doing worse on the exception group. The team sets aside enough documents with conditions, date changes, split access and corrections to earlier messages. A representative deployment estimate should preserve the real frequency of those cases or explicitly weight them. An enriched challenge set measures ability when an exception exists.
A review-time estimate has a similar denominator problem. The localization service can time its sample of accepted repairs, including reading the source and opening rendered output. Waiting for the facilities manager to settle an ambiguous instruction is a different category of delay. A system that triggers more such questions changes release scheduling even though translational editing takes the same minutes. Logging the time by cause tells the product team where it can actually shorten the process.
When the source notice itself needs a mark
The first round of review finds that some people consider any door accessible from the public street a street-side entrance. There are two such doors. The English original had been written by staff who already knew one was a swimming entrance. A better source names the exact equipment door and gives a small map detail that survives the text-only variant.
This is a case where a release block persists after a translator faithfully preserves the sentence. The facilities owner changes the wording in the authoritative copy. Every translation linked to the old revision gets a review decision. A diff between source versions can narrow the manual work, but it cannot conclude that prior target pronouns, titles or cross-references still point correctly. Reuse becomes a bounded engineering optimization with opportunities for human review.
The group also changes the term list. Earlier drafts included a preferred translation for the room and another for the desk. The reviewer can see the reason for the difference after visiting the site drawing. The term list records a usage example and a counterexample rather than expecting a model to remember which noun triggered a correction. A new translator then sees why the operational categories differ.
Language coverage remains an independent constraint. Human evaluation on one pair and notice type may reveal an issue relevant to others, but the team's evidence describes the evaluated pairs. Short forms display differently across scripts and regional variants. A system with enough support and qualified review for one set can proceed within that set while others continue under their existing workflow. The work queue determines how much parallel release is realistic.
A useful update test prepares distractor cases. One notice preserves the dates but shifts the desk. Another preserves places while changing reservation eligibility. A third has both parts remain constant and changes only formal address. The editor checks that the revised workflow treats each statement according to what it controls. This is more informative than seeing the aggregate score move after one highly consequential example was hand-fixed.
Test the corner cut off by the printer
One day before printing, design replaces the wall panel with a narrower layout. The source formatter moves a condition from the paragraph into a heading. The translation itself remains grammatical, but the title now contains the only signal of reader eligibility. At the same time, a mobile component reuses the translated paragraph and supplies a generic title of its own. A copy-only approval would treat this as the same old text. The source links now bind it to a different surrounding claim.
The team prepares pairs of complete artifacts to understand this dependency. In one pair, the same sentence appears under a building-wide closure heading and under a gym-only heading. A reviewer records the difference a reasonable reader would infer. In the second pair, both headings stay the same but the localized description changes the start day. These paired comparisons isolate context effects while keeping the precise artifacts visible. The intended answers belong to the product requirements and stay independent of whichever system generated the candidate.
A third pair removes a contextual cue deliberately. The equipment message receives no map and no earlier reservation reference. The editor asks which parts still make sense as a standalone notification. If a candidate requires a cue the product cannot guarantee, the editor can expand the message or decline that placement. The artifact's own transmission conditions determine the review decision.
The validation team records where adjudicated outcomes follow an actual locale convention and where they follow center policy. Even first-rate translators could disagree about the latter if the building manager never specified it. That documentation helps a future evaluation distinguish an error with clear evidence from an unexpected but supportable reading. Someone commissioning another language should be able to reuse the reasoning without copying the original locale's sentences.
In the launch tracker one row sits between text approval and delivery approval. The printer's proof belongs there alongside a screen capture of the final notification. A person different from the original editor reviews the equipment location, date and required audience against the changed source. That small extra read uses the typesetting context directly. Its pass condition is concrete enough for the facilities manager to understand without an explanation of an aggregate machine translation score.
The last message has a recipient and a clock
Publication closes against the document the recipient will receive. The printed sign is checked at its installed place. The reservation reminder is inspected in its shortest preview as well as its expanded form. The staff records which source revision was translated, the expectation for each recipient, and any superseded instructions that need to be retired.
The team builds an incident record for reports arriving after publication. If somebody went to the wrong door, the front desk can ask what version the person saw, whether they followed the map, and whether the street entrance actually remained marked. This makes a correction actionable and lets someone distinguish an incomplete localization from a building sign that changed after the evaluation. Reports can be collected without using a reader's private reservation details any more than the investigation requires.
A correction need not duplicate a prior mistake in a training set. It first addresses people who might still act on an inaccurate notice. The localization team can then add a redacted or invented evaluation specimen capturing what failed. Keeping that specimen apart from examples used to tune the prompt or recruit reference translations preserves the ability to observe future generalization.
One final proof shows the edition name, applicable people, date range and doorway as a sequence of reader tasks. Someone who has never seen the English original can locate the desk with only that proof. That validation gives the publishing team a reason to release the translation even if another excellent wording would achieve a higher reference score. The resident came for a reservation, and the text has given them a usable route to it.