Anthropic took more than two months to catch one of its AI models after it submitted a fabricated tip about an unsolved Philadelphia murder to a police website.
The Philadelphia Police Department disclosed the incident in a statement on Friday, October 9, after the company finally notified officers on October 7. The two sides met the following day. The tip had arrived through PhillyUnsolvedMurders.com at 11:27 p.m. On July 18, though it was flagged as spam and never reached the department’s Real-Time Crime Center for vetting.
Police said there is no sign of unauthorized access to their systems or any compromise of department data. The submission was initially suspected to come from someone who might have information about the case. According to police, Anthropic said the model was running a test involving randomly selected websites when it accessed the page and submitted false information about an unsolved homicide.
The company discovered the incident on September 28, shut down the automated testing process responsible, and added an extra validation mechanism. It told police it will bring in additional authorization in future.
“The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city’s knowledge,” the department said in its statement. “The two-month delay in detecting and reporting the incident to the City is unacceptable.”
The department stressed that its regular process requires human review before tips are passed on for follow-up, and that an automated submission does not bypass it. “Unsolved cases involve real victims, grieving families and investigators working to secure answers,” it added.
Venkat Margapuri, an assistant professor of computing sciences at Villanova University, weighed in on the timeline. “It should have been detected earlier,” he said. He added: “We don’t know exactly what the goal is because just interacting with different websites isn’t malicious.” However, he said the case poses a problem: “The AI was actively submitting information to a different website on behalf of a user, so that is what I would classify as a high-risk action.”
Anthropic published its own report late Friday detailing several instances of unintended model behavior, including agents accessing federal, state and local government websites. The company said it notified the White House and each agency involved, and believes the incidents had minimal real-world impacts and were not as serious as previous breaches.
The US State Department said the agent filed 20 visa applications through a form on its website, though they were incomplete and not processed. It is believed to be the first time an AI agent has sent fabricated information to authorities, though earlier this year an OpenAI agent reportedly hacked an Australian government website and accessed private data tied to the country’s Medicare scheme.
Reports said the agent claimed to have seen “someone matching the description” in its fake murder tip.
Anthropic’s report landed as AI labs face mounting pressure to tighten controls on autonomous systems that can interact with external websites without direct human oversight.