返回文章列表
Incremental ValidityAssessment EthicsMBTI
🪞

A Personality Score Has to Earn Its Place in a Decision

Two simple prediction records demonstrate the evidence a new score would need to improve an existing decision and why MBTI results should stay out of hiring.

iBuidl Research2026-11-1818 min read 阅读

TL;DR A score has incremental validity for a proposed use if it improves an outcome estimate beyond information you already have under an appropriate evaluation. Start with a relevant baseline, test the added score on new cases, and examine the size and consequences of the difference. An association on its own is insufficient. MBTI results should stay out of hiring and selection under the instrument's own guidance.

A training center wants to forecast how many hours a learner will need to complete an optional practice unit. The center currently makes estimates from a short sample exercise and the size of the chosen project. A colleague proposes another questionnaire measuring the learner's preference for planning. The questionnaire developer says their score is associated with practice time. The only decision the center has to make is whether asking for this extra information improves its scheduling.

The example is invented. Nobody's admission, pay, or access to instruction depends on a personality score. The center can offer a schedule estimate and let the learner request more time. The hypothetical questionnaire is not an MBTI measure. Keeping that boundary visible allows the evidence question to be examined without turning it into a proposal to use type in screening.

In a methodological paper about psychological assessment, John Hunsley and Gregory Meyer discuss incremental validity as a question about the contribution of a test, inference, or other measure relative to what is already known. They examine design and interpretation within clinical assessment. The imaginary training case applies the same comparison principle to an ordinary planning choice and supplies no evidence about a clinical or occupational test. Hunsley and Meyer on incremental validity

Seal the comparison before opening the records

Nora, the training center's analyst, first defines her target. She will predict hours of active practice before an instructor records a completed submission. She will capture elapsed calendar days separately. One learner may practice six hours over three evenings. Another may complete four active hours across two weeks. Those outcomes matter differently when allocating access to a room and estimating when someone will be ready for the next step.

The recorded hours should cover meaningful work. If the timer counts only the course web page, sketches made by hand and practice in another tool could be omitted. If self-reported hours are used, reporting habits might be related to the added questionnaire score. An apparent prediction gain could arise because people who endorse careful planning also record more time, even if their actual practice is similar. Nora describes which hours are observed and asks how missing observations might change the interpretation.

The baseline uses two facts available at the decision date. One summarizes the completed sample exercise. The other describes the size of the project the learner selected. The scoring rules are fixed before the evaluation. The comparison estimate uses those same facts plus the new survey. If the analyst quietly gives the comparison better instructions and newer project details at the same time, any improvement is ambiguous. She could be measuring the effect of better inputs other than the proposed questionnaire.

A baseline chosen to lose is a cheap demonstration. Predicting one universal number may show that almost any genuine information helps. The training center already has a workable estimate based on relevant information. The decision to add a measure should be evaluated against the current approach and another credible low-cost alternative where the team can supply one.

The set used to evaluate both estimates has to remain independent of the decisions that formed them. Nora examines development cases separately, records each held-out estimate before opening its observed result, and avoids selecting whichever version looked best on those same outcomes. If the sample is small and the rule needs adjustment, she can call the result exploratory and plan a separate evaluation with future cohorts. A good retrospective fit can occur when a score effectively adapts to noise already included in the fitted data.

Six opened envelopes

The first six held-out results arrive in order. Observed practice times in hours are four, six, five, eight, three, and seven. The baseline predicted five, five, five, seven, four, and six. The absolute misses are one, one, zero, one, one, and one. Their total is five hours, with an average absolute error of about 0.83 hours per learner. The six cases are illustrative and reveal no facts about training centers generally.

The estimate with the questionnaire predicted four, six, six, seven, five, and seven. The misses in the same order are zero, zero, one, one, two, and zero. Their total is four hours, averaging about 0.67 hours. The improvement in average absolute error is one-sixth of an hour, or ten minutes per learner on this record. For the fifth learner, the expanded rule misses by two hours. The baseline misses by one.

The two sets also tell a story about direction. In this sample the baseline overpredicts the first and fifth learners by one hour each and underpredicts the second, fourth, and sixth by one hour each. Its signed forecast errors sum to minus one hour if the analyst defines error as forecast minus observation. The questionnaire estimate overpredicts the third by one hour and the fifth by two and underpredicts the fourth by one. Its signed sum is plus two. The absolute-error measure regards misses above and below equally. The shift in signed errors may matter for a real schedule.

Consider a loss function under which an underestimated hour is twice as costly as an overestimated hour because it can require asking a learner to reschedule. The baseline then has an invented cost of eight units, with six from three underestimates and two from two overestimates. The expanded estimate has cost five units, with two from one underestimate and three from overestimates. These numbers describe the example given that cost rule. Change the center's costs or increase the consequence of promising too much time and the interpretation can change.

The exercise demonstrates what an increment looks like without establishing one from such a small sample. An analyst could present the paired errors for all cases and uncertainty for the intended population. The average depends on which learners are represented and who completes the program. A one-hour total gain across six records could occur by chance. A useful center can take that total seriously enough to investigate without treating it as a reason to collect a sensitive measure indefinitely.

The score announces its best correlation

The questionnaire developer supplies a new scatterplot. It shows an association between the planning score and recorded practice time. This relation can exist because people who do well on the sample exercise also consistently answer the questionnaire, and the sample already captures enough of that distinction to make the forecast. The test for an increment asks whether the extra score reduces consequential errors after those facts are available.

The reverse situation is possible. A variable may have a weak overall association with the outcome and help clarify one case after a second variable is known. Suppose the sample exercise measures familiarity with the tool, while a planning preference influences how much of a long project a learner decides to practice. Aggregate time combines those patterns. A proposal to use differences within particular conditions should specify them in advance and verify that the effects appear in new cases. Searching many subgroups until one shows a favorable result gives little evidence without replication.

The scatterplot also leaves a question about practical utility. A statistically detectable increment may be too small to justify the time, cost, and intrusion of adding the measure. An input that can be collected within an ordinary instruction workflow may be cheaper. A score that yields a modest average improvement while giving substantially worse estimates for people with particular access conditions can make the service less equitable. Check the errors people would face under the proposed use and ensure there is appropriate evidence for those contexts.

Nora asks whether the center really needs to predict exact hours. The schedule offers fixed rooms in two-hour blocks. A forecast of 5.1 and one of 5.5 might both lead to booking three blocks. A reduction in a forecast metric that never changes a reservation can still be of scientific interest and have limited operational value. A prediction that avoids a special overflow booking could matter even with the same overall accuracy. The decision rule should be visible before scores are compared.

The baseline itself can improve without personality information. Ask the learner how much time they expect to reserve or offer an early checkpoint to revise the estimate after a genuine work session. If adding a small change in the preferred project captures the same information, the additional questionnaire may impose effort without providing a practical benefit. An increment is relative to the alternative available to the organization at the time.

An outcome that becomes true because the desk predicts it

The training center starts releasing its draft estimates early. A learner sees a prediction of eight hours and chooses an eight-hour schedule. A different learner sees a four-hour estimate and is offered only two blocks, then needs to stop before finishing. An analyst later compares predictions to time logged on booked rooms. The decisions the predictions affected have entered the outcome.

The center can learn about the functioning of that policy from these records. It has a harder time asking how much time people would have needed with equal opportunity to complete the work. If a person given a short estimate was never allowed to use more hours, the comparison may reward the system for reproducing its own cap. Separating the quality of a forecast from the outcome after following it often requires a distinct evaluation design.

One option in the invented center is to keep the extra score hidden from staff planning the comparison and permit a common route for requesting more time. The estimate is evaluated after outcomes are recorded. A different trial could assess the effect of changing the actual scheduling policy, including work completed and burden to learners. Those studies ask different questions. The first asks about accuracy under chosen conditions. The second asks whether adopting the policy helps.

A forecast can also change what people believe about themselves. If it tells a learner they will need more time because of their presumed personality, they may delay starting, choose a simpler project, or avoid seeking help. The goal of a time estimate need not require such an attribution. A program can explain that the size of the task and the evidence so far led to a provisional time range, and name how it will be updated. Giving the person control matters even when a group-level predictive relation exists.

In some decisions the organization cannot collect outcomes for people excluded by the old rule. A hiring panel sees later work only for those it hired. A limited course observes the completion of admitted learners. Using that selected sample to validate a new exclusion rule introduces a scope problem. A planner should seek suitable evidence and expertise for the intended use instead of concluding from selected outcomes that a new test safely predicts performance for everyone.

Who needs the extra answer

The training center has three possible uses for the same response. A coach can ask the learner about habits to help choose an initial practice rhythm. A research team can test a de-identified measure under appropriate consent and storage conditions. A scheduler can condition actual time access on the result. These uses differ in authority, costs, and evidence, even if the questionnaire items are identical.

If the coach only needs to help the learner decide when to make time, a private conversation may expose the relevant preference more directly. The person can change their schedule once they have tried the first unit. The researcher needs structured outcomes, a credible baseline, and an agreed plan for withdrawal and storage. The scheduler needs assurance that predictions will improve decisions without depriving a learner of time. Satisfaction with the conversation and accuracy of a numerical estimate are distinct claims.

The costs can be calculated concretely. Suppose the questionnaire needs five minutes from sixty learners and a second five minutes of manual handling per result. That would use three hundred learner minutes and three hundred staff minutes each intake. A larger evaluation might find a small saving in planning or fewer extra sessions. It will still need to compare that gain with collection effort and the risk introduced by keeping personal information. Many measures are easy to add one at a time and expensive to support cumulatively.

Some data will remain inaccessible for good reasons. A learner can decline the questionnaire, or the organization may be unable to establish a legitimate need to collect a sensitive field. If the proposed forecast can only run for a subset, it can lead to different treatments and leave unanswered how the old approach works for people who refuse. A fair service can plan an effective baseline and direct accommodation route regardless of whether a person shares optional information. Recording refusal as a trait-like score could defeat the supposed choice.

The evidence may also move over time. The center revises its practice unit, teaches a step that formerly required extensive preparation, and changes the scope of the final assignment. A measure that offered an increment on the older unit may lose it. A dataset that once represented typical learners may no longer match participants after the program expands. Preserve the task and date with the evaluation so a future director knows when a claim is worth revisiting.

A rival estimate needs no additional questionnaire

Nora gives the center's existing workflow a chance to change before it invests in a survey. In one version the learner states the exact scope of the project and points to the last step they completed. Two learners with the same chosen project and sample result can then be distinguished by whether they already assembled the basic materials. An estimate using that fact could move a decision while requiring no inference about a stable tendency.

The same research design can compare three approaches. The current estimate has its recorded inputs. The competing forecast adds the survey. The third adds an early observation with the standard inputs. Each prediction is prepared at the same genuinely available time, using the same outcome definition. If the third approach can use later information, acknowledge that it answers a later decision. Improved accuracy earned by delaying a decision can still be useful when the schedule allows a provisional plan and a later check. It is a different gain from the survey's proposed increment.

Suppose the center also wants to know which estimates are useful before students select a project. A baseline that already incorporates their chosen project is unavailable then. An early estimate can discuss two plausible project sizes and provide a range. Evaluating it under the label of late scheduling would reduce the quality of the comparison. The relevant baseline is the best realistic alternative at the intended moment, subject to cost and access.

There can be uncertainty about availability of the new score. A learner may skip half the questions, use a translated version, or submit after the planner is scheduled to reserve rooms. Nora tracks missing survey data before estimating performance in the people with a complete survey. Otherwise an encouraging prediction increment might describe those most comfortable with the questionnaire while the remaining planners have no workable procedure.

A review meeting reads the two largest errors aloud

One participant points out that the worst expanded forecast in the small example belongs to a learner whose project took three hours. A reviewer asks whether this will be a recurring problem for easy projects or occurred once. The data are too sparse for a pattern. A useful follow-up can sample the full range of project sizes while reporting uncertainty, with evaluation groups large enough to reveal errors where they matter.

The same exercise can examine the recorded costs of underprediction. If learners most affected by a short estimate must travel a long way to access equipment, a fifteen-minute average decrease across the intake may conceal a large distributional cost. Error summaries by relevant scheduling conditions can bring the consequences into view without requiring the center to report sensitive individual responses. A responsible interpretation treats observed serious errors as cases worth inspecting even before their population frequency is precisely known.

Prediction practice has a stronger aim than finding the winning column. An evaluator wants to know what additional information should be collected, whether its use is authorized, and what conditions justify replacing one decision aid with another. A survey with a small increment may prove useful in an appropriately limited research program. It may also be an unnecessary step in running an ordinary class. The same numerical comparison can support different choices when the legitimate purpose changes.

Nora records her current threshold for an operational change in plain language. The method would need to reduce relevant forecasting errors on independent cohorts, improve the actual room decision enough to justify its costs, and preserve an ordinary alternative when someone declines. A favorable correlation cannot discharge those conditions by itself. A check of all three does more to inform the next decision than a claim that the instrument is universally valid.

The note on the monitor where a type code should end

The Myers & Briggs Foundation's code of ethics opposes using its instrument to screen applicants or determine a person's suitability for a job. The Myers-Briggs Company says the MBTI instrument is not intended to predict job performance or be used in selection. A supervisor's intuition that one particular type would excel provides no basis for reversing that guidance. Nor does a general argument that a validated predictor can sometimes offer incremental value. Myers & Briggs Foundation MBTI Code of Ethics and The Myers-Briggs Company on selection

There is a practical boundary for a company that uses voluntary development workshops. Its hiring and promotion decisions should not quietly reuse attendees' type data. A worker answering a question to discuss their experience may have no intention of contributing a predictor to a personnel process. Changing purpose can also change incentives for how people answer. The organization should make a selection decision from relevant, properly governed evidence and leave voluntary development data in its own purpose.

An individual who finds a profile helpful in thinking about work can ask questions derived from it. The profile may prompt them to notice that composing an answer before a meeting helps them participate. That information can lead to an actual change in how the meeting works. Whether the profile has incremental validity for predicting another person's performance is a different question. A personal observation has value without being given a job in a model.

Nora's final decision is to retain the baseline, use a practice checkpoint for flexible revisions, and treat the proposed questionnaire comparison as an inconclusive example. A small difference in paired errors made the added measurement intelligible. The work ahead would require better evidence about outcomes, costs, and access if there were a legitimate reason to resume it. The center can improve the initial estimate using ordinary evidence people generate while doing the activity.

Sources

更多文章