Every quarter I sit with a team that just spent six weeks scoring vendors. They built a spreadsheet with weighted criteria. They invited stakeholders to a demo day. They collected scores. And six months later they are unhappy with the pick. The spreadsheet said the vendor was the best option. The spreadsheet was wrong.
The problem is not the tool. The problem is how teams design the scoring exercise. Most scoring matrices look rigorous but actually measure confidence, not performance. I have watched a dozen teams score the same vendor by the same criteria and get results that differed by 40 points. That variance does not come from the vendor. It comes from how people interpret the rows. A small team can fix this without hiring a procurement department. You can borrow the methods that enterprise buyers use, stripped down to what actually works. I have used a lightweight scoring framework from apluses us with client teams, and it solves the hardest part: getting people to score the same thing in the same way.
The weighted score trap
Weighted scoring feels scientific. You assign 30 percent to price, 25 percent to features, 20 percent to support. Then you multiply. The math gives you confidence. But the weights are usually made up on the spot. I asked one team where their 30 percent came from. Nobody remembered. The CFO guessed it during a meeting. That number drove a purchase decision worth eight months of payroll.
I found the same pattern across five companies last year. Every one of them had a weight column that nobody could justify. The weights were emotional placeholders disguised as numbers. When I asked people to explain a single score on their matrix, most could not describe what a 4 meant versus a 3. They just knew a 4 felt better.
Score inflation is a team sport
When stakeholders score a vendor demo, they do not score the product. They score the presenter. A confident salesperson gets high marks for “ease of use” and “roadmap clarity.” A quiet engineer gets low marks for the same features. I compared scores for two vendors at one company. The vendor with the worse uptime record scored 15 points higher because the demo was smoother.
Most vendor scores describe the sales call, not the software.
That gap does not close with more training. It closes when you change what people observe. Demos are theater. The people who act in them are professionals at making things look easy. Your team is scoring a performance, and the performance is rehearsed.
What actually correlates with vendor success
I started tracking which criteria predicted good outcomes for clients who bought software. The usual suspects, feature count and UI polish, predicted almost nothing. The criteria that predicted success were boring ones.
- How the vendor handles a support ticket during your trial, not during the demo
- Whether the vendor publishes uptime and performance data publicly
- How quickly the vendor responds to a clarifying question in writing
- The average tenure of their support staff, which they will tell you if you ask
These four items explained most of the difference between happy and unhappy clients in my files. Feature count explained almost none. That is counterintuitive, but it is what the data from a dozen projects shows.
Running a cleaner comparison
You do not need a bigger matrix. You need fewer rows and sharper definitions. Write each criterion as a question with a concrete answer, not a vague statement. Replace “good mobile experience” with “can an agent complete an order on a phone in under two minutes.” That single change cut scoring variance in half for one team I worked with.
- Define every criterion with a scenario, not an adjective
- Have two people score the same demo independently and compare notes before averaging
- Score the vendor’s written documentation, not just the slide deck
- Run one real task from your own workflow during the trial and score that separately
The second item is the one most teams skip. Independent scoring feels redundant. But when I made two people score the same demo alone, their scores differed by 20 points on average. After they talked, they settled on a number neither had originally chosen. That conversation is where the real evaluation happens.
The reference call you are skipping
Most teams skip reference calls because they feel awkward. The references are chosen by the vendor. The calls are short. But a reference call, even a managed one, tells you more than any demo. Ask the reference about the last time the software failed. The answer reveals how the vendor behaves under pressure. One client learned that way that a shortlisted vendor took three weeks to fix a billing bug. That changed the decision.
Keep the matrix, change the inputs
The scoring matrix is not broken. The inputs are. Weighted criteria still work when you define them sharply, score independently, and add reference calls to the list. My advice to every small team is the same: spend less time polishing the spreadsheet and more time defining what “good” looks like in your own workflow. The scoring will take care of itself. And the pick you make will survive contact with real work.
