A Courtroom Blunder That Exposed the Dangers of Blind AI Reliance

In August 2025, one of Australia’s most respected defence lawyers found himself in an awkward and instructive moment that has resonated far beyond the courtroom in Melbourne. Rishi Nathwani KC, a senior barrister with the prestigious title of King’s Counsel, stood before the Supreme Court of Victoria and publicly apologised to Justice James  Elliott after AI had led him and his team seriously astray in a high‑stakes murder case.

Nathwani’s written submissions included quotes from speeches and case judgments that were presented as if they were real legal authority. They were not. These quotes, and the “precedents” cited, were manufactured by a generative AI tool the defence team had used to help draft their legal arguments. Court associates could not find the cited cases anywhere in the official records, and when pressed for copies, the legal team had to concede that the citations did not exist and the quotes were fictitious.

The AI‑generated errors forced a 24‑hour delay in what was expected to be a swift final resolution of the teenager’s murder charge, a case in which Justice Elliott later ruled the accused was not guilty due to mental impairment. The interruption was more than an inconvenience; it reminded everyone involved that the justice system depends on the reliability of legal research and the professionalism of those presenting it. “The ability of the court to rely upon the accuracy of submissions made by counsel is fundamental to the due administration of justice,” Justice Elliott remarked sternly.

For many observers, this episode prompted a mix of surprise, disbelief and concern. On the one hand, AI is now commonly used as a research assistant, lawyers and scholars alike are experimenting with tools that can summarise case law or draft argumentative language. On the other hand, what happened here was not a minor typo or a misremembered date; it was entirely fabricated legal content paraded as established jurisprudence. The embarrassment was palpable, not just for Nathwani but for the profession at large, because it hit at the heart of what legal practice is meant to uphold: trust, accuracy and accountability.

The incident also highlighted something even deeper: AI’s outputs can sound convincing without being true. When a generative model produces text, it doesn’t “know” whether a case exists or a quote is real, it predicts plausible language based on patterns in data. Left unchecked, those plausible outputs can be utterly false, yet still look professionally drafted. In this case, the defence team admitted they’d checked some initial citations and then assumed the rest were correct, an error in judgement that cost time, credibility and judicial patience.

So what lessons arise from this courtroom misstep? First and foremost, AI should never replace rigorous human verification. Lawyers have professional duties, to their clients, to the court, and to the rule of law, that demand independent checking of every source, whether it came from a database, a textbook or an AI assistant. These duties cannot be abdicated to an algorithm, no matter how polished its language may appear.

Second, this incident has contributed to a broader conversation about regulation and best practices for AI in professional fields. Courts in Australia have already issued guidelines restricting how AI can be used in drafting evidence or affidavits unless its output is fully verified. Other jurisdictions are watching closely, reinforcing the idea that AI use must be paired with ethics, not convenience.

Finally, this story serves as a cautionary tale for anyone tempted to treat AI as an infallible knowledge engine. Whether it’s law, medicine, journalism, or public policy, the human mind, trained, sceptical and accountable, must remain in charge. Generative AI can help with drafting and brainstorming, but it cannot replace the deep expertise and responsibility of a trained professional.

In the end, the legal system weathered this incident, and the case was resolved. But for practitioners, scholars, and even everyday users, the message is clear: check your sources, question your tools, and never trust AI without verification. The law and society depends on it.