You can spend a quarter arguing about a rating scale. The hour that decides whether your ratings mean anything is the calibration session. 

This is the second of four pieces expanding on The Four Questions Every Performance Management Program Must Answer. Question two is “how do we rate it?”, and most programs answer it by redesigning the form. 

Scale design gets weeks. Four levels or five, what to call the middle, whether to force a directional call. Calibration gets an hour on a Friday, or gets skipped. That is backwards. 

The scale is a small lever 

There is no research establishing an optimal number of rating levels. Landy and Farr’s review in Psychological Bulletin put rating format at 4 to 8 percent of the variance in ratings. The form is not where your outcomes come from. 

Three things about the scale do matter, and none of them is the number of levels. 

Define each level in observable terms, so that two managers reading “exceeds” picture the same behavior. 

Say what your middle level actually means. “Meets Expectations” describes someone doing the job they were hired to do, and most of your workforce should land there. If your managers treat it as a C, the label is doing that, and no amount of training will fix it. 

Decide whether you want an even number of levels. An even scale removes the safe middle and forces a directional call. It is about the only scale decision worth a long meeting. 

Then pick a number and move on. 

Why calibration earns the time 

Calibration is where individual manager judgment becomes an organizational rating. Five reasons it is worth the meeting. 

A rating starts meaning the same thing everywhere. Without it, a 4 from a generous manager and a 4 from a demanding one carry identical weight in every decision downstream. 

Pay decisions get a defensible basis. Everything in question three assumes the rating is comparable across managers. If it is not, a merit matrix just distributes the inconsistency with more decimal places. 

Managers learn the standard by hearing it argued. A manager who sits through two hours of peers defending ratings against evidence calibrates themselves for next year. No training module does that. 

Leadership meets people it would not otherwise hear about. Strong performers two levels down become visible in the room, which feeds succession work later. 

It works. In the best study available, calibration committees adjusted about one rating in four, and post-calibration ratings correlated more closely with actual performance measures than the pre-calibration ones did. 

Calibration starts at goal setting, not in December 

Here is the failure most programs never diagnose. The session turns into an argument about whether one team’s goals were harder than another’s, nobody can settle it, and the ratings stand as written. 

That is not a calibration problem. It is a goal-setting problem arriving nine months late. Four things set in the first quarter decide whether calibration can work at all. 

One goal-setting window and one template. If half the organization sets goals in January and half in April, you are comparing different time periods at year end. 

Peer review of goals up front. Have managers at the same level read each other’s team goals before they are locked. Ten minutes per manager in February prevents the December argument entirely. This is calibration, just earlier and far cheaper. 

Comparable stretch. Ask the same question of every manager’s goal set: what would have to go right for all of these to be met? If one team’s answer is “a solid year” and another’s is “everything breaks our way,” the ratings will never be comparable. 

Evidence captured as it happens. A manager reconstructing a year from memory in December produces recency bias and little else. A short quarterly note per person, in whatever system you already have, is what makes the calibration room work. 

How to run the session 

Who is in the room. Managers at the same level, discussing a comparable population, with one level up facilitating or observing. HR runs the process and does not own the ratings. Keep it small enough that everyone speaks. 

Combine small teams. A manager with three direct reports cannot produce a distribution. Group them with peers so the population is large enough to compare. 

Send the distribution in advance. Managers should see the aggregate before they walk in. It turns the conversation from defending a number into explaining a pattern. 

What each manager brings. A proposed rating for every person, plus two or three pieces of specific evidence, especially for anyone at the top or the bottom. “She is a rock star” is not evidence. 

Start with the outliers. Work the highest and lowest proposed ratings first, while people are fresh and before the clock forces shortcuts. Alphabetical order guarantees you spend your energy in the middle, where least is at stake. 

Ground rules, said out loud at the start. Evidence over impression. This year, not a career. No discussion of pay, at all. The manager owns the rating; the group’s job is to test it. 

Define what “challenged” means. The goal is not consensus. It is that no rating leaves the room unexamined. 

Write down what changed and why, in both directions. This is the record that makes the system defensible later, and it is the step most often skipped. 

Agree what managers say next. A rating that moved in committee still has to be explained to a person. Decide in the room who says what. 

Things to watch 

Challenge the low ratings as hard as the high ones. In that same study, downward adjustments outnumbered upward ones four to one, and the committees made centrality bias worse, not better. A room that only demands evidence when someone proposes a high rating is a machine for pushing everyone to the middle. 

Watch the manager who says nothing. Research on calibration committees found that supervisors facing less scrutiny came away with fewer adjustments and higher final ratings for their people, even controlling for actual performance. Quiet is not agreement. Spread the challenge evenly. 

Keep pay out of the room and off the calendar. If the raise is on the table, the rating drifts to justify it. 

Look at the outcome by group before anything is final. The EEOC’s guidance treats evaluations of performance used for employment decisions as selection procedures, which puts your rating distribution by race, sex and ethnicity under the four-fifths adverse impact screen. Run it while you can still do something about it. 

Give new managers a dry run. A first-time manager in a live session will defer to whoever speaks loudest. 

Where this lands 

Spend less time on the form than you want to. Define the levels, name the middle honestly, pick a number. 

Then put the effort where it changes outcomes: comparable goals in the first quarter, evidence collected through the year, and a session where the low end is defended as carefully as the high end. 

Question three, how the rating links to pay, assumes the rating means the same thing from one manager to the next. That is next. 

Coming up in this series 

  • Building a merit matrix that actually differentiates 
  • Performance vs. potential: the 9-box and succession planning 

Let’s talk 

Rethinking your performance management program, or starting from scratch? Wilson Group has helped over 400 organizations design employee total compensation, executive rewards, and sales incentive programs that fit their stage, scale, and mission. Schedule a conversation → 

Chelsea Martens is an HR leader and consultant with 15+ years of experience in Total Rewards, HR technology and operations, pay equity, and organizational design. She designs compensation and incentive programs, job architecture, and technology-enabled HR processes that improve performance, efficiency, and retention; she holds CCP, CECP, and CBP credentials and a B.S. in Managerial Economics.