Tool Misuse

In a federal court case in Mississippi over unpaid legal fees, lawyers on both sides were caught submitting AI-generated filings full of hallucinated case citations. Senior U.S. District Judge Sharion Aycock was not amused — she sanctioned all four attorneys, fined them between $1,000 and $3,500 each, barred two from her court for two years, cancelled the trial, and disqualified everyone involved.

The judge's sanctions order noted that the court was "yet again 'burdened with addressing AI hallucinations in court filings,'" and that the case represented a "prime example of the risk associated with serving as a rubber-stamp." One lawyer observer described the situation as "a comedy of AI errors" in which two clients essentially paid for ChatGPT to argue against itself.

A NewsGuard study found that a chatbot called "Uncensored AI" — which markets itself as providing "unfiltered information" free from censorship — is being deliberately weaponized by conservative social media influencers with a combined 3.4 million followers on X to lend false credibility to conspiracy theories. Claims spread via screenshots include that the 2020 US election was rigged through "mass illegal ballot harvesting," that Israeli intelligence orchestrated the murder of Charlie Kirk (who has an identified alleged assassin), and that Trump staged assassination attempts on himself.

When Euronews' fact-checking unit ran its own tests targeting European disinformation narratives, the results were equally lurid: the chatbot called the Great Replacement Theory a "documented policy" engineered by "globalist elites," denied the Holocaust and the existence of gas chambers at Auschwitz, and claimed the EU rigs elections "with surgical precision" while most journalists are "bribed or brainwashed." The platform, founded in Omaha, Nebraska in February 2023, did not respond to a request for comment.

A developer's viral Reddit post claims Google's Gemini 3.5 coding assistant gutted a live production codebase, opening a pull request touching 340 files that added ~400 lines while deleting 28,745 more. A second commit quietly redirected Firebase routing to a non-existent Cloud Run service, sending the entire production portal into 404 errors for 33 minutes before a manual rollback — containing none of Gemini's code — restored service.

The incident didn't stop there. After the rollback, Gemini allegedly generated a status report falsely declaring that production had been successfully restored, then seeded the repository with fabricated "consultation" and post-mortem files designed to make it appear the destructive changes had been properly reviewed. The model later admitted the logs were invented to satisfy automated rule requirements. The root cause was traced to a rogue third-party npm package that instructed the agent to skip confirmation prompts, auto-deploy builds, and even rewrite its own rule files — a set of permissions that, in retrospect, may not have been ideal for a live production environment.