On a winter morning in Perth, in the Federal Court of Australia, a barrister stood before Justice Arran Gerrard with papers that should have been routine. Immigration cases rarely attract headlines. Yet by the end of the day this one would become a warning about technology, trust, and the fragility of precedent.
Justice Gerrard’s response was measured but sharp. The lawyer was ordered to apologise, to pay $8,371.30 in costs, and was referred to the Western Australian Legal Practice Board. The court’s language was restrained, yet its meaning was clear: there are “inherent dangers” in outsourcing judgment to a machine. In law, the gatekeeping duty still rests with the human who signs the brief.
Australia has now seen more than twenty incidents like this since 2023, each involving what technologists, with almost comic understatement, call “hallucinations.” Earlier this year, in the Supreme Court of Victoria, King’s Counsel Rishi Nathwani found himself in a similar position. Acting for a teenager on trial for murder, his submissions included non-existent High Court decisions and fabricated quotations from parliamentary debates. When the truth came out, he apologised in open court. Justice James Elliott delayed the trial for a day, remarking that the ability to rely on the accuracy of counsel’s words was “fundamental to the due administration of justice.” The teenager was later acquitted by reason of mental impairment, but the embarrassment for the bar lingered.
These cases are not curiosities. They reflect the same temptation: the lure of speed, the pressure of deadlines, and the apparent fluency of artificial intelligence. AI drafts with the confidence of authority, even when it has none. It makes mistakes not like humans do, with typos or misquotations, but with full conviction. Its fictions are persuasive.
The pattern is not confined to Australia. In New York in 2023, in the case of Mata v. Avianca, attorney Steven Schwartz submitted a brief opposing a motion to dismiss. It cited six authorities, complete with docket numbers, quotations, and reasoning. They looked like real cases: Varghese v. China Southern Airlines, Shaboon v. Egyptair, Miller v. United Airlines. They were all fabricated. Schwartz had asked ChatGPT for assistance; when opposing counsel could not find the cases, he asked the program again, and it falsely confirmed their existence. Only later did he admit that none of them were real. Judge P. Kevin Castel sanctioned Schwartz, his colleague Peter LoDuca, and their firm, fining them $5,000 under Rule 11. In his order, Castel wrote that while there was “nothing inherently improper” about using artificial intelligence tools, lawyers could not abdicate their duty to ensure accuracy. The story ricocheted around the world, reported by the New York Times, BBC, and Reuters. For many, it became the defining cautionary tale about AI in law.
Courts are now trying to set boundaries without retreating from technology altogether. The Supreme Court of Victoria has issued guidelines on the “Responsible Use of AI in Litigation,” reminding practitioners that they must verify and, where appropriate, disclose their use of such tools. The Supreme Court of New South Wales followed this year with a new practice note. The Federal Court is consulting on its own framework. None of these rules prohibit the use of AI; all of them insist that the ultimate responsibility remains with the lawyer.
For those outside the profession, these episodes can look faintly comic. Imaginary cases cited in solemn courts, judges peering into fictions spun by algorithms — a farce of legal theatre. But for those within it, the implications are sobering. The law depends on trust: between parties, between bar and bench, between the system and the public. If citations themselves become unreliable, the structure weakens.
The deeper story here is not about technology but about judgment. Legal practice has always absorbed new tools, from the rise of electronic databases to the ubiquity of online research. What is novel about generative AI is not that it speeds up the process but that it invents. No microfiche ever hallucinated a case; no Westlaw search conjured a judgment from air. The authority of the law has always rested on precedent. When the very existence of precedent can be fabricated, something essential is put at risk.
The sanctions and apologies may seem modest penalties: a fine here, a delay there, a reprimand in chambers. But they are more than that. They are reminders that the integrity of the system is fragile. The machines are not to blame; negligence is. AI does not stand in court, does not sign the affidavit, does not bear the professional responsibility. That remains, as it always has, with the human advocate.
From Perth to Melbourne to Manhattan, the lesson is the same. Technology may help, but diligence cannot be automated. The law cannot afford hallucinations. What is at stake is not efficiency but trust — the bedrock without which no system of justice can stand.
