Source Insurance director on AI accountability, the German court ruling that could reshape the industry, and why "human in the loop" is no longer optional
Chris Lynch (pictured top) has a test. Call it the work experience rule. If you would hand a task to someone who walked in on their first day – capable, eager, but entirely unproven – AI can probably manage it. If you would not, neither should it.
Very few businesses are applying anything like it. And Lynch, IT director at Source Insurance, believes the consequences are only just starting to arrive.
"Up until this point, all the AI providers have kind of gotten away with being able to say this tech makes mistakes," he told Mortgage Introducer. "It's there at the bottom of ChatGPT, it's there at the bottom of Claude. Every time you search Google and get an AI summary, there's a little disclaimer saying this might not actually be right. That isn't new. We've accepted a much lower level of quality from software than from anything else ever made and sold."
What does the German court ruling mean for UK businesses?
Something shifted in May. The Regional Court of Munich ruled Google is directly liable for false claims made by its AI-powered search summaries, finding that the summaries are the company's own content, not a list of links to third-party sources, and the liability protections search engines have long enjoyed do not apply. The ruling is being appealed. But the direction of travel is clear enough.
Lynch had been making the same argument for months. "Google's position has always been, ‘We're just telling you what we can find on the internet. We're not the arbiters of truth’. But now they've taken all that information and put it through a mincing machine and given you an answer they've created. That's a fundamentally different thing."
The Air Canada case sits in the same territory. The airline was held liable for incorrect bereavement fare information given by its chatbot, after arguing the chatbot was a separate legal entity and not its responsibility. McKinsey's Global Survey on AI, published in November, found 51% of organisations using AI had already experienced at least one negative consequence, with 30% attributing it specifically to inaccuracy. "We are responsible for what our AIs are doing in the same way we're responsible for what our employees are doing," Lynch said. "Turning around and saying, 'I just got that from ChatGPT' is no longer going to fly."
What makes this harder is you cannot opt out of the problem by opting out of AI. Lynch works in insurance. It is now trivial to generate a convincing photograph of damage that never happened, or a fake receipt for a high-value item. AI-assisted complaint letters, fed into ChatGPT, returned professionally formatted with citations to case law, are already circulating. Sometimes the cases cited do not exist. "You receive an email saying, 'In the case of Frobisher versus Duckworth, 1976...' and now I've got to go and look that up. We're already seeing judges throw out cases because all the legal precedent cited was just made up. What kind of pressure does that create on a legal system?"
World models and word models
Lynch came to Source Insurance from e-commerce, where he specialised in complex, regulated products. Before that, health service IT. He talks about large language models – the AI systems underpinning ChatGPT and Claude, trained to predict plausible text rather than retrieve verified facts – with a fluency that is unusual in his peer group, and a scepticism that is perhaps even rarer.
He is drawn to a distinction made by Yann LeCun, the Turing Award laureate who left Meta in late 2025 to build what he calls "world models" – AI designed to understand physical reality rather than pattern-match on text. The gap between the two, Lynch believes, explains most of the problems businesses are walking into.
"If you spoke both English and Chinese and I asked you a question, you would synthesise that information," he said. "You hold concepts in a world model, not a word model. LLMs don't. They don't know that this token in English and that token in Chinese are the same logical equivalent, so unless you really explain things in great detail and put in all the guardrails, you can't just rely on the AI to do it."
The word "hallucination" bothers him. The model is not imagining things, it has followed a chain of data to a wrong conclusion, with no mechanism to know it has done so. "It's either hit a piece of information that is just wrong – and it doesn't know that – or it's made a link between two tokens that is invalid. It's seen those words near each other a few times and thinks there's something there."
All of which shapes everything Source Insurance is doing with its AI referral chatbot, which lets customers complete a general insurance quote via a WhatsApp-style interface without tying up advisers. Getting the system to collect data and generate a quote took no time at all. "That was day one," Lynch said. Everything since has been about safety. "Can we make this robust? Can we get this to the point where it's not going to miss something that's really important for the customer? In a lot of instances, the solutions are not more AI. They are more traditional, deterministic, symbolic programming, where we're taking the output from the AI and checking it." He compares it to baking with a child. You measure out the salt. You do not hand them the box.
Building something brokers can trust
The feedback from advisers testing the system has been less about what it does than how it sounds. Warm without being unsettling. Helpful without feeling fake. That is harder than the engineering.
Lynch reaches for the uncanny valley – the unease people feel when something appears almost human but falls just short – to describe what they are trying to avoid. "Be a good, friendly, useful, personable machine, but be a machine. Don't try to be anything else. And if someone mentions they've got a thatched roof and the system needs to escalate, you don't want the customer going, 'But I thought I was talking to a person'."
Behind the conversational layer, conventional code runs continuously, watching for signs of distress, confusion, or repeated questions. Transcripts are kept throughout. "In the early days, we will have people who will check those transcripts," Lynch said. "We will have to check its work. Of course we will, and we will build our confidence in that way. That's just your normal continuous improvement. That's no different to running a normal customer service team. I think this is maybe where some people get lost with AI. They think that by having AI, you can magically not have customer services or not have QA. Actually, you probably need more of it."
Why AI will not replace the adviser
Klarna cut large numbers of customer service staff, then started hiring them back. Ford let go of experienced engineers. Some have been difficult to rehire. Lynch has watched both from a distance. He is not sure the industry has drawn the right conclusions.
"I view it as an augmentative technology," he said. "It's not like the toaster, where we all stopped holding bread on a metal spike over our coal fires. Think about email. We thought that was going to make everything better, and now we all sit with inboxes that are out of control."
Mortgage brokers face something specific. Clients are arriving at meetings having already asked ChatGPT whether to fix or track, what the latest base rate decision means, whether they can afford what they think they can afford. Some of what they have been told will be wrong, or incomplete, or subtly misleading. The adviser has to unpick it, with authority and without dismissing the client.
"Your broker needs to be able to say, ‘I know it's told you that, but it makes mistakes, and this is the why to that and why I’m going to tell you this’, so it’s a huge challenge," Lynch said. "It used to be a mate down the pub told me this. Now the mate in the pub is in everyone's pocket, 24 hours a day."
He is not pessimistic, but he is direct about where the risk sits. "Go and ask AI about something you know nothing about and you'll think it's fantastic. Go and ask it about something you're a real expert in, and you'll go, ‘That's not actually right’. Where's the nuance? Where's the value? That would be the difference between reading a random blog and going to an adviser and saying, ‘This is my problem’."
Want to be regularly updated with mortgage news and features? Get exclusive interviews, breaking news, and industry events in your inbox – subscribe to our FREE daily newsletter. You can also follow us on Facebook, X (formerly Twitter), and LinkedIn.
