By Kennedy Backer, Director of The Rosenberg Survey
Benchmarking reports can look authoritative. They include tables, charts, percentages, and comparisons that appear precise. But polished presentation isn’t the same as reliable data.
Trustworthy benchmarking depends on what happens before the report is published: how questions are defined, submissions are reviewed, inconsistencies are resolved, and the limits of the data are communicated.
Data quality isn’t created at the end of the process. It should be built into the survey from the beginning.
Clear definitions come first
Two firms may use the same term and mean different things by it. This can happen with revenue, owners, staff, compensation, billable hours, and many other measures. One firm may include something another excludes. Another may calculate a measure using a different time period or method.
If those differences aren’t addressed, the final comparison may look exact while combining numbers that aren’t truly comparable.
That’s why a good benchmarking survey should make clear what each question is asking, what should be included, and how the response will be used. Clear definitions don’t eliminate every judgment call, but they give participants a common starting point.
Review is part of data collection
Submitting a survey isn’t the end of the data collection process.
At Rosenberg, I review every submission. I look at how the responses fit together, whether the answers appear consistent, and whether a significant change from a prior year has a reasonable explanation.
Sometimes the issue is simple: a number may have been entered in the wrong field or reported in dollars instead of thousands. In other cases, a change may reflect an acquisition, a new partner, a change in firm structure, or another event that needs context.
The goal isn’t to make every firm’s results look typical. It’s to understand whether the submission accurately reflects the firm.
A large dataset isn’t useful if the individual submissions haven’t been examined carefully.
Follow-up is a sign of rigor
When something appears inconsistent, we follow up. That may mean asking a participant to confirm a number, explain a change, or clarify how an answer was calculated.
Sometimes the original response is correct. Sometimes the conversation reveals that a question was interpreted differently than intended. Either outcome improves the process.
Participants occasionally worry that a follow-up means they completed the survey incorrectly. Usually, it means the review process is working as it should.
A benchmarking survey shouldn’t accept every response at face value simply because the form is complete. Uncertainty should be resolved before the data is analyzed and published.
Outliers require judgment
An unusual result isn’t automatically an error. An acquisition, leadership transition, new service line, or deliberate investment may produce a result that falls outside the usual range.
Removing every outlier would erase meaningful differences. Accepting every outlier without review could allow mistakes to distort the results.
The question isn’t simply, “Is this number unusual?” It’s, “Does this number make sense given what we know about the firm and the rest of its submission?”
Good data review protects legitimate differences while identifying responses that need correction or explanation.
More data isn’t always better data
The number of participants matters in benchmarking. A broader pool can provide a stronger view of the profession and create more useful comparison groups. But volume alone doesn’t guarantee quality.
A smaller group of carefully reviewed submissions can be more useful than a larger set built on inconsistent definitions or unresolved errors. Participation and rigor should grow together.
The goal isn’t to combine as many firms as possible into one average. It’s to organize the data in ways that support relevant comparisons while protecting the integrity of the dataset.
Trust includes knowing the limits
No benchmarking survey can account for every difference among firms.
The data can show patterns, ranges, and relationships. It can help leaders identify where their firm appears similar to or different from peers. It can’t explain every cause or make a decision on the firm’s behalf.
Trustworthy benchmarking should be clear about those limits. That means avoiding conclusions the data can’t support and distinguishing between what the numbers show and what leaders still need to investigate.
Confidence in the data shouldn’t require pretending that the data is complete.
A process leaders can rely on
The quality of a benchmarking report depends on more than the final calculations. It depends on the care taken at every stage: defining the questions, reviewing responses, resolving inconsistencies, preserving legitimate differences, and communicating the results responsibly.
That work is less visible than the finished report, but it’s what makes the report useful.
Firm leaders should be able to trust that the comparison in front of them was built thoughtfully. Not because every number is perfect, but because the process was designed to make the data as accurate, consistent, and meaningful as possible.
