Choosing AI sales roleplay software
What to test before you buy: realistic buyers, strict scoring and progress you can see.
The SalesOS team
8 min read
The first session with any AI roleplay tool is impressive. The buyer pushes back on price, asks how you compare with the incumbent, and at the end you get a score and a tidy paragraph of feedback. Almost every product on the market can produce that moment. What separates them is what has happened by week six.
We built roleplay into SalesOS, and building it forced us to decide what good looks like. This is how we would tell the tools apart. Some of it argues against products like ours, which seems fair.
Start with the buyer, not the score
A roleplay is only as useful as the person on the other side of it. If the simulated buyer is agreeable, reps learn that a confident tone closes deals. That lesson costs money the first time they meet a real procurement lead.
Good buyer personas differ in two ways that matter. The first is temperament. An analytical buyer wants numbers and gets suspicious when you dodge them; a driver wants you to get to the point; an amiable buyer will be pleasant right up to the moment they stop replying. The second is the kind of objection they reach for: technical risk, price, timing or authority (“I like it, but this isn’t my call”). Those combinations produce genuinely different conversations, and a rep who only ever practises against one of them is rehearsing for a single meeting.
There is a simple test for this. In the trial, try to win the call by agreeing with everything the buyer says and asking for the meeting. A well built buyer should not give it to you. If the AI signs because you were polite, everything downstream of it is suspect.
Scoring has to be strict
Generous scoring is the most common failure, and it is easy to see why vendors drift into it. Reps like seeing 88. Managers like dashboards full of green. Nobody complains, and nobody improves.
A useful score does three things. It is broken into parts, so a rep can see that discovery was strong and the close was weak instead of reading one blended number. It points at the transcript, quoting the moment the rep talked over an objection or missed a buying signal. And it is calibrated low. Most competent reps should land somewhere in the sixties and seventies, with the eighties reserved for calls you would play to the whole team.
The parts we score, and how much each counts, look like this: rapport (15%), discovery (25%), objection handling (25%), value articulation (20%) and next steps (15%). You might weigh them differently. The point is that discovery and objections carry the most weight, because that is where real deals are won and lost, and a tool that weighs friendliness the same as discovery is training the wrong habit.
To test calibration, run the same mediocre call twice in a trial. The two scores should be close to each other and neither should be flattering. If one says 61 and the other says 83, the number is noise.
Progress is the product
One good session changes nothing. What you are paying for is a rep who handles pricing objections better in December than they did in October, and you should be able to see that without exporting anything.
For each rep, the tool should show you at least:
- how many sessions they did this week, because practice that stopped in week two is the usual story;
- their average score over completed sessions;
- their weakest dimension across all sessions, not just the most recent one;
- a trend that compares recent sessions with earlier ones.
Be a little suspicious of how the trend is calculated. A trend built from two sessions is mostly luck. Ours compares the average of the last three completed sessions with the three before them once a rep has six, and falls back to first versus latest only until then. Ask any vendor how theirs works. If they cannot tell you, assume it flatters.
The question is not whether the rep scored well today. It is whether the same weakness is still the weakest thing a month from now.
Keep a manager in the loop
AI feedback is fast and tireless, and it is no substitute for a manager who knows the territory. The best setups use the AI to do the volume and the manager to do the judgement: the rep practises three times, the manager reads one transcript, adds a note, and decides what the next drill should be.
So check that a manager can open a finished session, mark it reviewed and leave notes the rep will see. Check too that a manager can see their whole team at once without clicking into every rep.
A word about leaderboards
Leaderboards are good for one thing: getting people to practise. Ranking reps by score is riskier. Reps are competitive and quickly work out which personas are easy. If you use a leaderboard, rank by sessions completed and treat scores as private between rep and manager.
Practise for real meetings
Generic practice fades. Practice for the renewal call on Thursday, with a buyer who has the same title and the same worries as the person you are about to meet, tends to stick. The rep has a reason to take it seriously and the feedback is immediately useful.
That is the strongest argument for roleplay that lives next to your pipeline rather than in a separate training product. When the scenario can be described in terms of a real account, and when the persona speaks the language of your industry, practice stops being an exercise and becomes preparation.
Questions for the demo
- Can I build a buyer from one of our lost deals?
- What happens if I agree with everything and ask for the meeting?
- How is the overall score made up, and can I see the weighting?
- Does the feedback quote the transcript?
- How is a rep’s improvement trend calculated, and from how many sessions?
- Can a manager review a session and leave notes the rep sees?
- What happens when the AI model is unavailable? Is the feedback labelled, or passed off as the real thing?
The last question sounds like a technicality. It is not. A tool that quietly returns template feedback when its model fails is teaching your reps from a script while telling them it listened. When ours cannot reach its model, the scorecard says it is a rough estimate and the score is kept out of averages and the leaderboard.
Where SalesOS fits
Roleplay is part of Sales Coaching in SalesOS, on the Professional and Enterprise plans. Reps choose from built in buyers with different temperaments and objection styles, or the team adds its own. Each session ends with the five part scorecard above, highlights and areas to work on taken from the transcript, and a longer note on what to practise next. Managers can review sessions, and every rep has their totals, weekly sessions, average, trend and strongest and weakest areas in one place. If your workspace uses an industry pack, the buyers speak that industry’s language.
Practice is a text conversation today. If you need voice roleplay above everything else, or you only want roleplay and already have a CRM you are happy with, a standalone tool may suit you better. Where we fit is the team that wants practice to sit beside the pipeline, the forecast and the rest of the sales operating system, so the rep preparing for Thursday never has to leave it.