Two People Choose the Same Answers: What a Forced-Choice Personality Score Can Tell You

TL;DR: Choosing one personality statement over another reveals a comparison, not automatically an absolute level of either attribute. Simple fixed-sum scoring can make profiles useful for describing priorities within a person while making comparisons between people misleading. Modern item response models can estimate more comparable scores under specified designs and assumptions, so forced choice does not always mean ipsative scoring. Inspect the response format, scoring method, and evidence for the proposed interpretation separately before treating a colorful profile as a basis for comparing people.

Open the report before believing the ranking

A questionnaire asks you to choose which of several appealing descriptions sounds most like you. Its report then places four attributes in a descending order. One bar is tall, another short. It is tempting to conclude that the first is a strength, the last a weakness, and a colleague's taller bar evidence that they possess more of the attribute than you do.

Those conclusions may describe three different comparisons. The first compares attributes within your profile. The second compares your behavior with some external standard. The third compares you with another person. A report can support one without supporting the others. The visual appearance of a common scale does not establish which comparison the scoring procedure permits.

This matters for readers familiar with MBTI and other personality language because the response interface can look deceptively simple. Two alternatives appear, you select one, and a label emerges. But a family resemblance between questionnaires does not make their scoring identical. The examples here do not reproduce or evaluate the proprietary scoring of an official MBTI instrument. They isolate a general measurement problem that readers should ask about any comparative questionnaire.

The distinction has a research history. Brown and Maydeu-Olivares' 2011 paper explains problems with classical scoring of multidimensional forced-choice responses and develops a Thurstonian item response approach. Its central implication for a report reader is bounded: the format alone does not settle what the scores mean. The item design and the model connecting choices to attributes matter too.

We can make that problem concrete without assuming that an invented quiz measures personality. The numerical profiles and scoring rules below are original teaching devices. They are not a validated test, a reconstruction of a commercial instrument, or a recommendation to classify employees with homemade questions.

Dossier A: equal rankings can conceal different levels

Imagine three descriptions: “I enjoy initiating conversation,” “I enjoy organizing the details,” and “I enjoy exploring unfamiliar ideas.” Call them conversation, organization, and exploration for convenience. These are ordinary descriptions, not formally defined psychological scales. A respondent must rank them from most to least descriptive.

Suppose two people have the following illustrative underlying attraction to the descriptions. The numbers are invented latent utilities on an arbitrary scale; they are not observed ratings or trait scores:

The ranking is identical. Yet in the invented setup Ari has a higher utility for every description. A rank-only response does not distinguish those two rows. Adding the same amount to every utility preserves their order, so the observed ranking remains unchanged. That is an arithmetic feature of this example, not evidence that Ari actually has more of every personality attribute.

Now consider a third row: conversation 9, organization 4, exploration 1. Its ranking is also identical, although the gaps differ. The ranking does not show whether the top two descriptions were almost tied or far apart. A person completing the questionnaire may experience those cases differently, but a strict rank response records the same ordering.

This does not make ranking useless. It tells us which description won against the alternatives presented. The mistake is to silently replace that relative statement with “Ari is highly conversational” or “Bea lacks exploration.” Exploration can be last in a group of attractive descriptions, and conversation can be first in a group that fits someone poorly. The response needs its comparison set to remain intelligible.

The alternative you were not shown changes the question

Suppose Bea now chooses between conversation and a fourth description that fits even less well. Conversation wins again. Several wins could make the profile appear decisive while all the alternatives remain weak descriptions of Bea. A tally records repeated preference within the offered comparisons; it does not by itself establish a population reference point.

The opposite can happen when an attribute is repeatedly paired with especially attractive alternatives. It loses more often, even though the person identifies with it strongly. Item selection is therefore part of measurement, not a neutral container around the “real” response. Ask what the attribute competed against before interpreting how often it won.

An instrument developer may address such issues through calibrated items, carefully designed blocks, and a suitable model. A reader does not need to solve that model to ask whether it exists. They do need to avoid treating a raw win count as self explaining merely because the final report uses precise numbers.

Dossier B: the fixed sum manufactures tradeoffs

Consider a different toy questionnaire with twelve choices. Each choice awards one point to one of three broad categories. Every respondent therefore receives twelve points in total. This is a simple example of an ipsative structure: the scores describe a constrained allocation within the person.

Three profiles might look like this:

Casey's six points in P exceed Drew's four. It does not follow, under this rule alone, that Casey exhibits more of the behavior represented by P in daily life. Casey may simply have selected P more often relative to the competing descriptions. The profile provides an allocation of choices, and interpreting that allocation as an absolute behavioral quantity adds an unsupported step.

The constant total also creates mathematical dependence. If Casey's P score rises from six to seven while the number of choices remains twelve, at least one other category must lose a point. The change does not establish that Casey's actual level of another behavior declined. The scoring system requires a reduction somewhere whether or not the corresponding real world attributes are opposed.

This becomes especially consequential when someone computes correlations across a large collection of such profiles. The category scores cannot vary freely because their sum is constrained. Apparent opposition among them can partly reflect the scoring rule. One should not read the resulting pattern straightforwardly as a discovery that certain human qualities compete in the same way.

Convert the scores to percentages and the constraint remains

Dividing each score by twelve yields percentages. Casey receives 50 percent, roughly 33 percent, and roughly 17 percent. The report now resembles a composition chart, which may be an honest representation of the allocation. It still does not mean that half of Casey's personality consists of category P, or that Casey performs P behaviors half the time.

Changing the display to a hundred-point scale does not add information. Nor does replacing the categories with appealing names. A ten-point lead on a rescaled report can look substantial while remaining a consequence of a few choices within constrained blocks. To interpret it, return to the response and scoring rules rather than the visual prominence of the difference.

A helpful caption would say that these are relative selections within this questionnaire. A misleading caption would imply independent quantities without explaining the constraint. The same arithmetic can support a modest reflection exercise or an unjustified comparison, depending on how the result is described and used.

A falling score can accompany an unchanged preference

Suppose the toy questionnaire is revised to include more attractive alternatives to P. Casey chooses P on fewer occasions, even though Casey's feelings about the original P descriptions have not changed. A lower P tally across versions may reflect the changed comparisons rather than personal development.

This thought experiment exposes why version differences matter. If a report claims that someone became less organized over a year, ask whether the items, alternatives, instructions, and scoring remained comparable. A visible difference between two profiles is the starting observation. Explaining that difference as a change in the person requires additional evidence.

Dossier C: count the comparisons, then ask whether they connect

A third toy design contains four descriptions, A, B, C, and D. It asks only two questions: choose between A and B, then choose between C and D. A respondent selects A and C. What have we learned? A defeated B in one comparison, and C defeated D in another. We have not directly learned whether the respondent would choose A over C, or B over D.

The two comparisons form separate islands. Many overall rankings are compatible with them. A could be the most attractive description, with C only slightly preferable to D. C could be the most attractive, with A only slightly preferable to B. A report that draws one complete ranking from these two choices must be adding a scoring convention or model beyond the directly observed relations.

That additional machinery is not necessarily illegitimate. Measurement routinely depends on models. The point is to identify it, because a result produced with its help should not be presented as if every relationship were directly observed. A user evaluating the questionnaire needs to know how the unobserved comparisons are inferred and what supports the inference.

Now add a third question comparing B with C. Suppose the respondent chooses B. Together the choices support the ordering A above B above C above D within this toy setup, provided we treat the comparisons as consistent expressions of one ordering. The new question connects the islands. Its informational role differs from simply asking the original two questions again.

This gives a concrete reason to inspect questionnaire design rather than just its length. Twenty repeated comparisons within disconnected sets do not automatically provide the same information as comparisons that connect the relevant attributes. In an actual personality instrument, item content and model assumptions make the problem more complicated than this four-letter example. The design still matters; a large item count alone cannot settle it.

A complete ranking does not provide independent evidence for every pair

Suppose a respondent ranks three descriptions A, B, and C in that order. We can express the ranking as A above B, A above C, and B above C. It is tempting to count these as three separate observations that independently corroborate the profile. But once A above B and B above C are established within a consistent ranking, A above C is already implied.

The three pair statements are a representation of one ranking task. Treating them as though three unrelated judgments were collected would misdescribe the data. This is an elementary logical point in the invented example, not a recipe for scoring real questionnaires. A suitable statistical model must handle the information structure its response task creates.

A report reader can translate this into a practical question: does the technical explanation distinguish the number of choices from the information those choices contain? “Hundreds of comparisons” might refer to relationships expanded from fewer ranking blocks. That count may be perfectly accurate, yet it should not be mistaken for hundreds of independent questions a person answered.

The same caution applies to repeated administrations. Identical answers can reflect a stable ordering within the presented comparisons while leaving absolute levels unresolved. Repetition may strengthen confidence in what was actually observed; it does not automatically supply a different kind of information. Stability of the response and breadth of the supported interpretation are separate matters.

What modern scoring changes, and what it does not

It would be a mistake to conclude that every forced-choice questionnaire produces only fixed-sum scores. Response format and score construction are separate design decisions. Brown and Maydeu-Olivares' 2011 research uses item response modeling to relate comparative responses to latent attributes, with questionnaire design conditions affecting how well person scores can be recovered. That is a specific modeling result, not a license to assume that any forced-choice test supports unrestricted comparisons.

Anna Brown's 2016 framework further separates the format of the comparisons, the measurement model relating items to attributes, and the decision model for choices. This is a useful antidote to arguments based solely on how a question looks. A pair of statements can be part of materially different measurement systems.

For a nontechnical reader, the practical distinction is between a raw allocation and an estimate obtained from a model. An estimate uses information about the items and assumptions about how responses arise. It should arrive with an explanation of its scale, the population or calibration context, and its uncertainty. The sophistication of the model does not make those details optional.

The toy utility table cannot establish the limitations of every modern instrument. Its purpose was to show what a single ranking, by itself, does not identify. A designed instrument can gather additional comparisons and use calibrated relationships that go beyond that isolated observation. The question for an actual assessment is whether its particular design and evidence justify the claim being made, not whether a simple example can describe everything its model does.

Ask the vendor to connect three separate statements

A provider may say that a questionnaire uses forced-choice items, that it estimates normative scores, and that its report helps a particular decision. Those are three claims, each needing a connection to the next. The first describes what respondents do. The second describes scoring and interpretation. The third concerns a use that may demand evidence beyond the score's construction.

A precise inquiry is: “Are the displayed scores raw within-person allocations or model-based estimates intended to support comparisons between people? What technical documentation supports that interpretation for this version?” This asks about a defined inference rather than demanding that a provider prove the instrument is universally valid.

If the answer is merely “the algorithm uses AI” or “our questions prevent gaming,” the scoring question remains unanswered. A product can have a modern interface and still leave the measurement logic obscure. Conversely, technical documentation may be substantial even when the report itself is simple. Judge the connection between the evidence and the claim rather than the age or polish of the presentation.

A comparison format does not eliminate strategic responding

Forced choice can make some response strategies less straightforward. When several alternatives are attractive, selecting every socially desirable description is no longer available within that block. That design motivation should not be inflated into a claim that respondents cannot manage impressions or infer which answer an evaluator prefers.

Consider an obviously desirable statement paired with a visibly unattractive one. The choice may reveal what someone believes is expected rather than a subtle preference. Replacing it with two desirable descriptions changes the task, but desirability can differ across jobs, cultures, and contexts. Someone applying for a sales role might infer that conversation is more valued than detailed organization, even if both statements sound positive.

These are possible response mechanisms, not accusations about a particular respondent. A report reader should avoid treating a forced-choice format as a lie detector. The relevant evidence would concern how the actual instrument behaves under the conditions in which it is used. A claim about resistance to one response bias does not establish immunity to every source of distortion.

Instructions also shape what respondents compare. “Most like you generally” differs from “most like you at work” or “most important for success.” A person may answer each honestly and produce different choices because the questions concern different things. A report should preserve that context rather than merge preference, frequency, value, and competence into a single personality claim.

What happens when neither description fits?

A person may select the less inaccurate statement because a choice is required. That selection is valid as a record of the response, but its interpretation depends on how the instrument handles such comparisons. A reflection conversation can ask what felt mismatched. A technical scoring system needs its own defensible treatment; a facilitator should not improvise a new scoring rule after seeing a surprising profile.

Similarly, a close choice can be experienced as arbitrary. The respondent may have selected an option that fit only slightly better. Do not assume that a categorical output reflects a categorical internal experience. Ask whether the report communicates uncertainty or whether it encourages the reader to convert a narrow comparative preference into a dramatic identity statement.

For casual self reflection, the difficulty of choosing can be informative in its own right. It can reveal that context changes the answer or that both descriptions fit. That conversational value is different from a claim that the numerical score measures those nuances accurately. The conversation should not borrow scientific authority from a scoring system it has set aside.

Read an actual report through an inference table

Instead of beginning with the label, write down the statement you want to make from the report. Then identify the evidence required for that statement. The following table separates common interpretations without assuming that all instruments fail or support them.

The table helps locate an unsupported leap. A score can be interpretable without predicting job performance. A test can support some comparisons while providing little basis for a particular development recommendation. Each inference should carry only the authority its evidence supplies.

The International Test Commission's Guidelines on Test Use call for evidence supporting intended purposes and the inferences drawn from scores. That principle is more demanding than asking whether a test has any research behind it. A study of one use does not automatically support another, and a familiar brand does not remove the need to identify the inference.

For a team exercise, this may mean replacing a ranking of people with a discussion of specific choices. Ask when the lower-ranked description fits, what alternatives were difficult to choose between, and what examples would challenge the report's interpretation. Those questions keep the exercise connected to experience while avoiding the fiction that a short bar establishes an inability.

Close the dossier with a sentence that the data can bear

Imagine that Casey brings the twelve-point profile to a coaching conversation. A careful opening is: “In this exercise, I selected the P descriptions more often than the alternatives. I want to examine when that preference appears in my work.” This makes a bounded claim and identifies a question worth exploring.

An overreach would be: “I have more P than Drew, so I should own that responsibility.” The proposed responsibility may require knowledge, availability, practice, and judgment that the questionnaire did not measure. Even if the score supported a between-person trait comparison, the allocation decision would need its own reasoning.

A useful follow up asks for contrasting examples. Casey might describe enjoying the start of a discussion but avoiding follow through, while Drew might dislike initiating yet handle difficult conversations well. The apparent disagreement with the profile need not be resolved by declaring either person mistaken. It may show that the category combines behaviors the decision should separate.

For the individual reader, the strongest habit is to keep the comparison visible. Compared with which alternatives, scored by which method, for which proposed inference? These questions do not require rejecting personality assessment. They make it possible to use a modest result modestly and recognize when a stronger claim needs stronger evidence.

Sources

• Brown and Maydeu-Olivares, 2011: Item Response Modeling of Forced-Choice Questionnaires — original paper and author's accepted manuscript in the University of Kent repository.
• Brown, 2016: Item Response Models for Forced-Choice Questionnaires—A Common Framework — distinctions among response format, measurement model, and choice model.
• International Test Commission: Guidelines on Test Use — evidence for intended score interpretations and uses. All numerical dossiers and illustrative questionnaire items above are original, unvalidated teaching examples.