Five things qualitative testing taught us this year

You can build a digital product that says all the right things and still fail the person using it. You won’t spot this in a design review, because everyone involved already knows what every term means. It only shows up when you sit with someone who doesn’t, and watch in real time, where the meaning starts to fall away.

This year we’ve run qualitative and quantitative testing across financial planning, clinical prescribing, vocational learning and more. Five findings kept recurring. None of them a failure on the user’s part, but each one being an important design signal.

1. People report understanding more than they actually do

This should concern anyone building regulated financial products, and we’ve seen it across several pieces of comprehension testing this year. Ask someone whether they understood a document and they’ll almost always say yes, confidently. Ask them specific questions about it and the picture changes. They recall the headline figures, because those are the boldest things on the page. The detail around those figures, the part that depends on the reader having actually read and understood it, gets skimmed far more often than their confidence suggests.

This is where the two methods need each other. Quantitative testing shows you how wide the gap is between how well people think they’ve understood and how well they actually have. Qualitative sessions show you where that gap opens up, and why. Without the numbers you have anecdotes. Without the sessions you have a score and no idea what to fix. The design response is to stop treating self-reported understanding as a sign-off measure, and to give the detail that matters the same visual weight as the headline figure.

2. What people say they’ll do isn’t what they do under pressure

On testing for a clinical decision-support tool used by GPs, clinicians told us they saw real value in explaining why they’d rejected a piece of prescribing guidance. That feedback helps the system, and their local health board, improve. In principle, they were fully on board. But within a busy clinic environment that can be a challenge, and this feedback provided vital insight into our design process. Where did it take us? Well, the answer isn’t to nag harder. It’s to make the fast path and helpful path the same: prefill their most recent answer, so doing the right thing costs one click instead of a written explanation.

3. A number without context isn’t information

The same testing surfaced a simpler pattern. When a message told a clinician that switching a prescription would save money, the figure alone didn’t tell them whether to care. The same saving carried very different weight depending on whether it was per prescription, per month or per year. Clinicians said so directly: the number could be worth acting on or not worth the click, and nothing in the message told them which. A figure only becomes information when the reader can tell what it applies to, and whether it applies to them. The fix is simple: always state the unit and the time frame.

4. Accurate language can still be misread as a risk

On a pensions proposition we built for a major UK pension provider, one feature let people request a ‘guaranteed quote’. The phrase tested well; people found it reassuring. But on reflection, several weren’t sure what was being guaranteed. One asked, in effect, whether requesting the quote meant committing to retire, and whether the process carried an obligation they hadn’t meant to take on.

Nothing about the wording was wrong. It was accurate, compliant and the right term for what the feature did. The gap was between the certainty the word projected and what readers assumed it meant for their own choices. Precise language doesn’t automatically produce a precise understanding of what it does and doesn’t commit you to, and that gap matters more the higher the stakes. The answer isn’t to change the term. It’s to say plainly, right next to it, what it doesn’t commit you to.

Shorthand caused a similar problem. Abbreviations like DB, DC and TFC tripped up people who already understood the concepts, because the shorthand assumed a fluency they had no reason to have. The fix is almost too obvious to state: write the term out in full every time, not just the first time. But it’s the kind of fix you only see once you’ve watched someone hesitate.

5. One ambiguous term on a safety-critical button is a real risk, not a copy nitpick

Back in the clinical tool, a button labelled ‘Acknowledge’ sat on a patient safety message. It tested ambiguously. Clinicians weren’t sure whether acknowledging meant they agreed with the message, had simply seen it, or were dismissing it. On a safety-critical interface that ambiguity isn’t cosmetic. It sits on top of a decision about patient care, hidden inside a label that looked entirely reasonable on paper. The fix is a label that says exactly what the clinician is confirming.

So, what does this all mean?

None of these findings would have surfaced in a design critique, a stakeholder review or a heuristic evaluation, because the people in those rooms already have the fluency the real user lacks. ‘Acknowledge’ reads as a perfectly reasonable label if you already know what acknowledging a clinical alert is meant to do.

Qualitative testing puts the product in front of someone without that fluency and asks them to think out loud, so the gap between what the product assumes and what the person brings to it becomes visible before launch, not after. Quantitative testing tells you how widespread that gap is, so you know which fixes matter most and can make the case for them with confidence. One shows you the problem. The other tells you how scale.

This ties into the argument we’ve been making across FinTech and HealthTech all year. Trust can’t be bolted on at the end, and it can’t be assumed just because an output might be technically accurate. It has to be tested with the people who will rely on it, in real conditions and in their own words. Otherwise you’re only guessing whether the trust you designed for actually resonates.