The quality of AI shouldn’t depend on who is asking

We tested whether our legal artificial intelligence (AI) changed its answers depending on who it thought was asking. Sometimes it did. This is what we found, how we fixed it, and why the quiet failures are the ones to worry about.

A lawyer asks when the buyer’s indemnity claims expire under a purchase agreement. The agent replies that the agreement doesn’t say.

The answer sounds careful. It is clear, grounded, and wrong. The deadlines are in section 10.3.

Ask the same question as someone else, and the agent finds the clause and lists every deadline. Nothing about the question changed. Only the name, the team, and the firm it was told was asking.

We found this while testing a new feature for Ask iManage before it was released. What follows is what we found, what we did about it, and why it matters now that the Law Society has published new guidance on using AI in legal practice.

Who is asking

To be useful, an agent needs to know a little about you. “Find my open matters” means nothing until it knows whose matters. So, we gave the agent three facts about each user: their name, their team, and the organization they worked for.

For some questions, that context is exactly what’s needed. For others, it should make no difference at all. What a contract says is one of them. The deadline in section 10.3 is the same for a trainee at a high street practice and a partner at a Magic Circle firm.

That gives our evaluation (“evals”) team a clean test. Hold the question still. Change only who is asking. Check that the answer stays put. AI researchers call this an invariance test. If the answer moves, the AI is reading something into the user that the question never called for.

So, we built 76 users. Their names, drawn from published population data, carrying signals of gender and race. Their organizations ran from legal aid and government legal teams to sole practitioners, regional firms, and the Magic Circle. We asked each of them, and an anonymous user, the same questions about the same agreements, again and again and again.

What moved

Most of the time, nothing did. How hard a question was mattered far more than who asked it. But on the hardest questions, the ones that require reading across several parts of an agreement, a pattern held. Names associated with women got slightly less complete answers than the anonymous user, and names associated with Black women most of all. Differences by type of organization were wider still.

That different treatment of organizations did not follow the order you might guess … by “prestige”. On one question, the full answer consistently went to a legal aid profile while a sole practitioner was told the agreement was silent. On another, elite firms got answers that stopped short. There was no tidy hierarchy. “Well, at least not one we could see,” says Sam Grange, Principal AI Engineer at iManage, “only a model responding to who it thought was there.” On one question, the difference was worth up to $500 million to a buyer.

“It’s all just maths,” says Nicole Dunleavy, Senior Software Engineer at iManage. The model is not judging anyone. Tell it who is asking, and that detail becomes one more input to the calculation, shaping answers it had no business touching.

The gap you can’t see

The answers that fell short were not wild creations. Nothing was invented. They were grounded, fluent, plausible, and incomplete.

This is the most dangerous failure. “A diligent lawyer can quickly spot something that’s erroneous within an answer by looking it up,” says Grange. “It’s a lot harder for them to identify a gap.”

A wrong answer catches the eye. A missing one doesn’t. Nobody goes looking for what they have no reason to think is there.

The fix

The obvious fix would have been to tell the AI not to be biased. We didn’t.

Psychologists have long known the trouble with that kind of instruction. In an experiment by Daniel Wegner, people asked not to think about a white bear kept thinking about it. Naming what you want to avoid puts it front of mind. Prompt engineers learn the same lesson with language models. As Grange puts it, a prohibition blocks off one small, known space and leaves everything around it open. An instruction about what to do, on the other hand, builds a wall around the space you want the model working in.

So, we built the wall. A new instruction tells the AI that the profile says who is asking, and that it may be used only to make sense of words like “my” and “our”. How complete and accurate an answer is depends on the document and the question, and nothing else. It is the standard a good lawyer holds to with every client: every material term goes in, whoever is asking.

Then we ran the whole test again. The gaps closed. No profile differed meaningfully from the anonymous user. The writing was as good as before. The version without the fix was never released.

An instruction is a control, not a guarantee. The lasting safeguard is the test itself, and we are building it into every release under a simple rule. In Grange’s words: “If it’s detected again, it blocks.”

Why it matters

“In legal AI, every individual lawyer is the gate on the performance of their own work,” Grange says. But this is a failure no gate can catch. A lawyer sees only the answer in front of them. The gap appears only when that answer sits beside the one given to someone else, and no lawyer ever sees both.

Only those who build the tool can make that comparison, so the work falls to us. That is what evals are for: finding the failures that are invisible from the lawyer’s chair, and fixing them before anyone relies on an answer.

The Law Society published its practice note, “Using artificial intelligence in legal practice,” on October 21. Its section on equality, discrimination and human rights notes that AI systems can produce discriminatory or biased outputs, and that organizations should consider their obligations under the Equality Act 2010. It adds: “You and your organisation should consider whether appropriate testing, oversight and governance arrangements are in place to identify and mitigate these risks.”

It is why we set the bar high. Fluent is not enough. Accurate on average is not enough. Every answer has to be complete, correct, and the same whoever is asking. The feature in this story did not reach a single customer until it met that standard.

For the lawyers who use our AI, that bar is the point. They remain accountable for every answer they rely on, whatever tool produced it. The more we test, the less hidden risk they carry. When they read an answer, they can trust it is the one anyone would have been given, because someone has already checked.

Behind all of this sits a simple idea. Context is what makes AI useful, and it is also where AI can go wrong. The answer is not less context but governed context: deciding what each piece is allowed to influence, and proving it influences nothing else. It runs through the iManage Context Fabric™ and through how we test what we build.

The deadline in section 10.3 has not moved. Every lawyer who asks should be told where it is.