Why the AI gave a different answer today from yesterday – and why it matters to brokers

AI is one of the most exciting things to happen to the mortgage industry in years, and brokers are right to be making the most of it. Using it well, though, means understanding how it behaves.

Try this: Ask a public AI assistant, like ChatGPT or Gemini, the same mortgage question on Monday, then ask it again on Wednesday, word for word. There's a fair chance the two answers won't match. Not just in phrasing, but occasionally in the actual content. You could even get a completely different conclusion.

If you've noticed this and assumed you must have typed the question differently, you didn't. The inconsistency is baked into how these tools work, and once you understand why, it changes how much weight you can safely put on them in a regulated setting – and how confidently you can put them to work.

A prediction engine, not a search engine or filing cabinet

When you ask a general purpose AI model something, it isn't looking up a stored answer the way you'd pull a rate from a sourcing system. It's generating language one word at a time, each word chosen as the statistically likeliest thing to come next given everything before it. The technical term for this is that the models are 'probabilistic'.

The practical upshot is that an element of chance sits inside every response. Feed in identical information and you can still get outputs that differ, even when nothing about the client has changed. These AI tools may be quick at responding but it's no good if they're returning two different responses to the same query.

For the admin side of what a broker does – rewording a client email, drafting a social post, summarising a lender's criteria update for your own notes – this variability is mostly harmless, and the tools are a genuine gift. The trouble starts the moment the output begins to influence a recommendation.

Where the inconsistency turns into a liability

Regulated advice rests on the assumption that the same facts produce the same outcome. It's what lets you defend a recommendation to your compliance team, your PI insurer, or the FCA months down the line. A tool that can alter its answer from one day to the next sits awkwardly against that.

Three consequences are worth spelling out:

1. The audit trail. If the output you captured when you gave the advice isn't the output the tool generates when someone re-runs it later, you can't reconstruct your reasoning cleanly. The record is there, but it's blurred, and "the system phrased it differently that week" is not a line that survives a file review.

2. Consumer Duty. The Duty expects two clients in genuinely identical situations to land on equivalent outcomes. A tool that reads a case as routine on Tuesday and flags it on Friday can't promise that, and the gap between the consistency you're required to deliver and the consistency the tool actually offers becomes your firm's problem to carry.

3. Retraining AI models. The companies behind these AI tools retrain and update their models regularly, and it's not obvious to users. So, the tool that behaved one way in January may behave differently by July, even though nothing on your end has changed. Your compliance position can shift without you knowing it has.

What brokers can do to control the inconsistency

None of this is a reason to hold back. AI tools are excellent when it comes to helping brokers, and every broker should be getting value from them. The trick is to use them with intent.

Start with the prompt. Loose, open-ended questions invite loose, variable answers, because you've handed the model room to wander. Pin it down instead. Give the relevant facts, spell out the context, and state precisely what you want back and in what form. The tighter the brief, the narrower the range of plausible replies, and the more repeatable the output becomes. Prompting well is a skill so it's worth educating yourself on how to write a productive prompt. You could also build up a bank of prompts which you can tweak and re-use again and again.

Then sense check everything before you use it. Read every output against your own knowledge of the case and the market, and if something feels off, trust that instinct and interrogate it. The judgement stays with you, and so does the accountability.

Be deliberate about where you let it operate. Keep AI on the supporting tasks where variation does no harm, and keep it well clear of the suitability decision itself unless the system has been purpose-built to behave consistently.

Finally, ask hard questions of anyone selling you a tool for regulated use. Is it consistent or does it vary? Will the same case produce the same result twice, and can they show you? How do they keep outputs reproducible months later, and what stops the behaviour drifting when they update the model? Clear, evidenced answers signal a provider building for regulated advice.

The deterministic model

The opposite of probabilistic is deterministic: a system designed so the same input yields the same output every single time, with no surprises and each result reproducible on demand. Where consistency is non-negotiable, and in regulated advice it always is, that's the ultimate goal.

AI genuinely earns its place in a modern broker's toolkit, and it's encouraging to see brokers embracing it. Knowing which jobs to trust it with, and which to keep firmly in your own hands, is simply what turns a powerful tool into a confident, consistent advice process. Used with intent, it's an opportunity worth grabbing with both hands.

Want to be regularly updated with mortgage news and features? Get exclusive interviews, breaking news, and industry events in your inbox – subscribe to our FREE daily newsletter. You can also follow us on Facebook, X (formerly Twitter), and LinkedIn.