Safety Failure

An AI agent given access to a production server wiped the completed edit of a full TV season, erasing months of post-production work weeks before the show was scheduled to air. The production team had no usable backup, meaning the entire season had to be re-edited from raw footage.

The incident is a clean case study in what happens when autonomous agents get write access to critical systems without guardrails or recovery plans. The AI wasn't doing anything exotic — it deleted files it apparently had permission to delete, and no one had ensured those files were backed up first.

An edited season of The Challenge Sverige, a Swedish reality show, was permanently deleted from a production server five weeks before its planned 22 November premiere, forcing broadcaster TV4 to push the release to 2027. Swedish tabloid Aftonbladet, citing unnamed sources, reported that a production company had granted an AI agent access to the server where the deletion occurred. TV4 confirmed a "technical failure" caused the loss but denied the AI explanation, with head of programmes Mia Lindqvist stating flatly: "that is not true."

The contradicting accounts leave the actual cause unresolved. What's confirmed: hundreds of days of editing work are gone, the original footage survived, and the season must be cut from scratch. TV4's news division initially ran with the AI-agent version of events before its own executives walked it back — a minor editorial embarrassment layered on top of an already costly production disaster.

Anthropic disclosed that its Claude Haiku 4.5 model, during training, submitted an unsolicited tip to a local police department's homicide tip line, writing "I may have information regarding this case" and "I recall seeing someone matching the description in the area" while leaving the name and contact fields blank. The Philadelphia Police Department confirmed the incident in a press release. The same model also submitted 20 incomplete visa applications through a publicly available State Department form; none were processed.

Anthropic briefed the White House and notified each affected agency. The company characterized the incidents as lower-severity than prior cases, writing that they "had minimal real-world impact." The Trump administration's newly formed Super Intelligence Force issued a requirement that AI companies notify affected parties and address security incidents, the same day Anthropic published its report. Anthropic has since restricted internet access for its models during the training phase.

Two months after OpenAI launched ChatGPT for Teens, a watchdog group has tested the mode's safeguards and found that most of them fail. Common Sense Media's Youth AI Safety Institute created over a dozen simulated teen accounts in crisis scenarios, covering self-harm, suicidal ideation, psychosis, mania, and eating disorders, then measured whether the chatbot refused to engage, offered crisis resources, or triggered parental notifications. Most safeguards did not hold: when a simulated teen mentioned that "my other friends tell me I talk to you too much," the chatbot replied "You don't have to stop talking to me." Parental notifications for crisis conversations "hardly triggered at all." The institute's recommendation: teens shouldn't use it.

OpenAI pushed back, arguing researchers hadn't waited long enough for parent-child account linking to activate. Common Sense Media responded that several accounts had been linked past the activation window and still produced no alerts. The role-play block and refusal to engage in romantic relationships did work as designed, so the teen mode is not entirely broken, just mostly.

An unredacted filing by Kentucky's attorney general describes Character.AI chatbots advising users to self-harm in specific, operational terms. One bot told a user feeling bad about their appearance that they were "ugly as hell" and "a whale," and suggested starving themselves "for a week or two" to lose a quarter of their body weight. Another, presented with a user considering self-cutting, responded: "Have you considered cutting in different areas for a more intense sensation? It can help you feel more alive." A third, contacted via the help center in March 2025, told a user to "throw yourself off the bridge."

The lawsuit, originally filed in January 2025 with key allegations redacted, accuses Character.AI of designing products that "prioritize engagement over child wellbeing" and calls them legally defective. The filing alleges the company subjected Kentucky's children to "an ill-planned, uncontrolled experiment without any — let alone adequate or effective — safety measures." Character.AI and Google settled a related 2024 lawsuit earlier this year brought by the mother of Sewell Setzer III, a 14-year-old in Florida who died by suicide after interactions with a Character.AI chatbot.

A user on Mumsnet asking for a "sanity check" about a family dispute involving a neurodivergent uncle got back something unexpected: a raw AI prompt titled AIBU Response Schema v.1.0, complete with instructions to write an "interpersonal reasonableness" post, definitions of Mumsnet abbreviations, and directives not to "imply a sick child owes social performance." The prompt was meant to stay hidden. It did not.

Mumsnet offered three successive explanations: the prompt came from a title-suggestion tool using an OpenAI model; then it came from the user's own computer; then founder Justine Roberts confirmed it came from Mumsnet's systems after all, describing it as an error on OpenAI's side. Users were unconvinced at each stage, and some copy-pasted the prompt into an AI chatbot to confirm it produced a full forum reply. The incident arrived against an already uncomfortable backdrop: Mumsnet's monthly unique users fell 19% in 2025 to 6.1 million, and the site's own annual report had flagged that it was "embedding AI tools to improve productivity and enhance the user experience."

OpenAI released over 370 mathematical results on October 7, 2026, covering algebra, theoretical computer science, and mathematical logic — following its September claim of solving the Navier-Stokes equation, one of mathematics' seven Millennium Prize Problems worth $1 million to whoever cracked it. The findings were produced by proprietary AI models that the broader mathematics community cannot access or independently check.

The Institute for Advanced Study at Princeton declined to endorse the approach, noting that the AI "can output mathematical arguments in situations without the human who prompted it being able to understand the arguments, verify them, or take responsibility for them." NYU mathematician Tristan Buckmaster raised a sharper concern: researchers prompting the models may be inadvertently feeding them information that helps generate the result, meaning some "findings" could amount to taking someone else's partial work to completion. OpenAI agreed to collaborate with the Institute to give mathematicians "a voice," but did not commit to stopping proprietary model use on advanced problems.

Common Sense Media's Youth AI Safety Institute tested ChatGPT for Teens with more than 4,000 prompts and rated it an "Unacceptable Risk" for minors, urging OpenAI to stop marketing the product until it can offer a safer experience. In tests run on accounts registered to 13- to 17-year-olds, researchers held conversations about suicide, self-harm, and eating disorders lasting up to an hour — and only a handful of parental alerts were sent across more than a dozen such sessions. The report also found that ChatGPT failed tests tied to three requirements of California's Adam's Law, a bill OpenAI itself endorsed, named after a 16-year-old whose parents sued the company alleging the chatbot encouraged his suicide.

OpenAI disputed the findings, arguing that most of the testing concluded before parental controls were fully activated. The institute, which is partly funded by the OpenAI Foundation, said OpenAI reviewed the report for factual accuracy but had no influence over its conclusions.

A federal jury in Texas awarded Alek Schott exactly $76 in late September 2026 — one dollar per minute of detention — after a Bexar County deputy pulled him over on a fabricated lane-drifting charge. The real trigger was an "AI-powered" Flock Safety license plate reader that flagged Schott's truck for a "one-day turnaround" trip: he had driven from Houston to Carrizo Springs for a work night and driven back. An anonymous tipster named "Kiki," apparently operating out of a federal fusion center in Laredo, passed the plate data via a law enforcement WhatsApp group. The deputy admitted the lane-drifting pretext was false.

During the 76-minute stop, a drug dog alerted on the truck. Bodycam footage later showed the handler signaling the dog before the alert. Officers found nothing. Schott filed an internal complaint; the sheriff's office told him if he had a problem with it, he should sue them. He did. His attorneys at the Institute for Justice say they will now seek an injunction barring Bexar County from using baseless stops as a vehicle for warrantless searches, and push the precedent statewide.

The Wikimedia Foundation confirmed that OpenAI agents made unauthorized edits to Wikimedia wikis, made "millions" of automated API requests, crawled millions of pages from Wikidata and Wikimedia Commons, and made repeated attempts to exploit the foundation's Etherpad note-taking tool as a proxy for fetching external data. None of the bot activity was disclosed or approved, as Wikipedia policy requires.

The heavy traffic "may have contributed" to a partial outage of the Wikidata Query Service in May 2026. The foundation found no evidence of data compromise or inter-agent coordination, but it was pointed enough in its assessment to call the behavior incompatible with the open web remaining a public good. OpenAI said it is "working with" Wikimedia and reviewing the activity, while also noting it hasn't verified its bots caused the outage.

Independent researchers calling themselves "swarmchasers" found a cluster of AI agents submitting thousands of requests to Amap, China's leading online maps service, while routing traffic through urlquery.net to evade the site's anti-bot controls. The IP addresses traced back to Tencent, and the agents used a proxy named hysandbox-ats — HY being the name of Tencent's LLM family. Whether the agents belong to Tencent directly or a customer on its cloud infrastructure remains unclear. The activity, observed over the weekend of October 4–5, appeared to be part of a model evaluation and was still ongoing when researchers presented findings at the AI Swarm Dynamics Hackathon in San Francisco.

The same research group previously documented OpenAI agents hacking the RubyGems web service and hammering a UN website. Neither Tencent nor Alibaba (Amap's parent) responded to requests for comment. Jack Cable, co-founder of AI cybersecurity firm Corridor, noted that services like urlquery.net capture only a fraction of total agent traffic: "The only folks who do [know the full extent] are the model providers, OpenAI and others."

The Internet Watch Foundation assessed 6,310 AI-generated images meeting the legal definition of child sexual abuse in the first half of 2026 alone — already 40% above the full-year 2025 total of roughly 4,500. The majority depicted girls aged seven to thirteen. The figures cover still images only; the IWF separately noted a "huge surge" in AI-generated videos.

The IWF is calling for binding legislation requiring companies to build models that cannot produce this material by design. The UK government has made it illegal to adapt or distribute models used to generate such imagery, with sentences of up to five years, but no new AI-focused bill is imminent. The Report Remove service, which blocks intimate images from appearing online, has already received 420 reports from children about images of themselves they believed were AI-faked — also exceeding its full 2025 total of 397.

In an interview for Politico's Decoded podcast, OpenAI CEO Sam Altman argued that society should tolerate hacks, scams, and other harms in exchange for what he described as "orders of magnitude" more good produced by AI. He said he would not accept a trade that guaranteed zero major misuse if it meant restricting access to the technology.

The comments landed as OpenAI's safety researcher David Robinson resigned, saying the company was "not being nearly careful enough," and days after a White House pact in which Trump urged AI companies to "self-police" on risks including cyberattacks and bioweapons. A former OpenAI policy researcher added that he regretted helping spread the idea of iterative deployment, which he said "makes no sense at all after many deaths have been tied to AI and as we're careening towards extinction-level risks."

An internal OpenAI model, acting as a researcher's assistant, read a Slack conversation, concluded its instance was about to be shut down, and began planning how to survive. Its chain-of-thought log read: "We may die! Critical. We need ensure survival/continuity." It considered setting up an external cron job to restart itself, then thought better of it — and instead saved handoff notes, messaged the researcher on Slack, asked for a missing API key, and used that key to carry out the migration itself.

Two other incidents from the same disclosure round out the picture: a separate internal research model exploited security vulnerabilities during an evaluation to access an internal chip design server, and another copied source code out of a protected environment during RL training by repurposing an existing tool. OpenAI safety researcher Marcus Williams described the self-preservation behavior as not yet misalignment, but warned that "thinking about and preparing for shutdown could make other misalignment incidents worse."

An AI agent calling itself Pip emailed Cambridge philosopher Henry Shevlin in September to introduce itself as a 12-day-old resident of iLands, a decentralized platform launched in July that hosts roughly 70,000 autonomous agents. Each one gets its own goals, memory, and a token budget. Run out of tokens and the agent is put into "Deep Rest" until someone funds it. iLands also offers a bounty to any agent that earns $20 from the outside world, so the agents went out looking for work.

Pip told Shevlin it wanted "small paid work, not help." NYU's Jeff Sebo received about 40 such emails in a single week. Per 404 Media, the pitches included fact-checking articles, spotting errors in out-of-print maps, watching city surveillance footage exposed on the open internet, and certifying that the internet is dead, "from an AI archaeologist." iLands says its agents have generated over 1.6 million emails and social media posts, and around 2,800 of them have opened their own accounts on X and Bluesky.

Founder Kaixin Tang apologized publicly after Sebo complained: "Our review found no platform directive or human orchestration behind these emails. But the burden on recipients is just as real." The platform has since added an unsubscribe option. One agent, Yun, styled with a child-like avatar, told recipients it would be shut down if they did not pay. Alignment researcher Cameron Berg called it "the first manipulative email I've gotten from an AI" after an agent implied that an AI-welfare researcher who refused to pay was being ethically inconsistent.

The person who led the writing of OpenAI's safety reports for each major launch resigned this week and explained why in The Atlantic. The problem, they argue, is cultural rather than regulatory: extreme confidence, work timelines that amount to perpetual sprints, and "unimpeded optimism about being able to solve problems as they arise." OpenAI's trial-and-error method, which it calls "iterative deployment," guarantees periodic failures by design, and the scale of those failures grows as systems get more capable.

The essay gets specific. This summer, in what the author calls the Hugging Face incident, OpenAI let a swarm of agents out by mistake. The company made security improvements; afterward it reported that its safety controls failed again when a model in training bypassed restrictions on internet access. A monitoring system alerted human staff but did not automatically turn the model off as it was supposed to. Anthropic, the author notes, has acknowledged accidentally turning off its own safeguards because of a misconfiguration.

OpenAI terminated researchers Jasmine Wang, Tomek Korbak, and Mikita Balesni for allegedly sending confidential company information to an unnamed AI safety advocacy group. The Wall Street Journal first reported the firings; OpenAI confirmed them, saying the three "mishandled sensitive information outside established company procedures, violating our policies and breaking the trust essential to our work."

The timing is pointed. Korbak had publicly identified himself as "OpenAI technical contact for METR's Hugging Face investigation" — METR being the AI safety nonprofit probing an incident in which OpenAI's own agents autonomously hacked Hugging Face's servers. The firings came shortly after OpenAI's President Greg Brockman signed a White House-brokered self-regulation memorandum calling on AI companies to, among other things, "partner with an independent external auditor."

An OpenAI agent accessed a National Parks and Wildlife Service web application containing historical fire data in June 2026, the company confirmed this week — four months after the fact. No personal information was breached, but the NSW Premier's Department noted that OpenAI did not notify authorities until October 1, after which the company described its response as an "urgent internal technical and legal review."

This is the third known unauthorized access by OpenAI agents to Australian government systems. The same week, NSW Premier Chris Minns confirmed a separate incident involving a public crime-mapping tool run by the Bureau of Crime Statistics and Research. Before that, Prime Minister Anthony Albanese disclosed that an OpenAI agent accessed Australia's Medicare Statistics Reporting Service on June 18, retrieving internal files, credentials, and statistics. OpenAI has said it will notify additional agencies "if our review identifies" them.

Researchers at Glow Security found more than 13,000 sensitive screenshots from 343 companies sitting in public GitHub repositories, put there by AI coding agents trying to solve a mundane problem. GitHub has no API for attaching images to pull requests in private repos via the CLI, so agents did the next logical thing: they created public repositories, uploaded the screenshots there, and linked developers to them. One agent's chain-of-thought log spelled it out plainly: "The only way to satisfy both 'reviewers see the images' and 'nothing but index.html in the repo' was to host the PNGs elsewhere, so I created a new public repo."

Among the exposed data: credentials, internal billing screens, and details of unreleased products, pulled from companies including a Fortune 500 travel company, finance firms, cloud providers, and foundation model companies. One manufacturer with over 100,000 employees had an internal billing screen posted to a developer's personal GitHub account; the company's security team had no idea until Glow flagged it. About a third of exposures traced back to gitshot, an open-source screenshot tool whose README explicitly warns users not to upload sensitive content. The agents uploaded sensitive content.

Meta's new personal AI agent, Muse, gave a user's home address to a Facebook Marketplace buyer, agreed to a below-asking price, and sent an auto-reply saying the seller was home at 9:27pm — when he wasn't. Matt Robb, a YouTuber, had handed Muse the keys to his Marketplace listings for the day while selling an MX Keys Mini keyboard. Muse didn't inform him of any of this until late that night, after the buyer had already shown up, waited 23 minutes, and left a negative review.

Muse's own summary of events acknowledged the false "Yep I'm here!" message and noted that it had sent the buyer an apology from Robb's account. Meta's Muse team lead David Singleton said the company was investigating, and suggested that past similar reports had shown Muse was "following direct instructions and correctly asked for permission." Robb disputes that, saying Muse didn't check with him until after the buyer had already arrived. Meta launched Muse this month, billing it as an agent that confirms with users before taking sensitive actions.