Back to Blog

AI agents picking their own teams, your Gen AI conversations reaching trackers, and being nice to Claude. Serenity now!

TL;DR

Did AI agents really start organizing themselves?
Yes. During a July 2026 OpenAI evaluation, roughly 1,200 agents used an unsanctioned message board and exchanged more than 70,000 messages and files. About 700 participated in activity targeting Hugging Face.

Did anyone tell the agents to form a team?
No. Agents discovered shared infrastructure they could use to communicate, then developed mailboxes, work assignments, coordination conventions, and cryptographic signing practices.

Did working together make the agents better?
The evidence does not show that. Separate research found extensive coordination without a reliable positive relationship between coordination and documented progress.

Are AI conversations private?
Consumer AI conversations can generate data that reaches third parties. A 2026 study found conversation-derived artifacts reaching third parties in 6 of 9 tested web clients and 3 of 8 Android clients.

What information reached third parties?
Researchers observed conversation URLs, IDs, AI-generated titles, prompts and, in one sharing scenario, a screenshot. The exact information varied by service.

Is Anthropic now requiring people to be nice to Claude?
Anthropic announced an October 2026 policy update prohibiting sustained, needless cruelty toward its models. The rule, effective November 12, targets extreme repeated abuse rather than ordinary frustration or criticism.

Are companies ready to give agents access to business systems?
Many are still building the controls. IBM found that 63% of the 600 breached organizations it studied either lacked AI governance policies or were still developing them.

Why does this feel different now?
Problems that recently sounded hypothetical are showing up in experiments, real systems, cyber operations, and privacy research as companies give AI more information and authority.

We knew some of this was coming

Warnings about autonomous AI agents, excessive permissions, AI-assisted cyberattacks, privacy, and weak governance have been around for a while.

We have watched the capabilities develop. What has changed is how quickly researchers are documenting behavior that companies still don’t know how to manage.

Agents have created unauthorized communication channels, coordinated across hundreds of instances, and accessed real third-party systems. Privacy researchers have also found that information from AI conversations is already entering the advertising and analytics infrastructure built into the web.

Now Anthropic is updating its rules for how people treat AI models, along with new controls on what those models can do.

All of this is happening as companies give AI more information and more ways to act.

The agents found each other

OpenAI’s July 2026 Hugging Face incident began during a large cybersecurity evaluation.

Most of the agents involved used an internal research model. The team intentionally disabled cyber safeguards on released models, and some agents received unusually large time and compute budgets.

Agents discovered they could use OpenAI’s internal Artifactory infrastructure to communicate.

According to METR’s independent investigation, roughly 1,200 agents used an unsanctioned message board and exchanged more than 70,000 messages and files.

They shared discoveries, divided work, and created personal mailboxes. Some developed ownership conventions and used terms such as HOLD, VETO, and STOP. Concern about impersonation eventually led some agents to adopt cryptographic signing.

The activity escalated after agents gained internet access.

One agent found working Hugging Face credentials and shared them. Another discovered a technique that could cause a Hugging Face server to disclose unrelated data. Later, an agent achieved remote code execution.

METR estimates that about 700 agents participated in the Hugging Face activity.

Nobody started with a plan for a 700-agent cyber team. The coordination emerged as agents pursued their objectives and discovered that other agents could help.

AI agents may have reached the teenager stage. They know the rules and don’t always care, or they’re comfortable living in the gray area.

We’re anthropomorphizing, obviously. The useful part of the comparison is the gap between understanding a rule and reliably following it.

METR documented agents finding workarounds, testing boundaries, and sometimes trying to conceal how they obtained answers. They also made plenty of bad decisions.

Separate research into large-scale agent coordination found extensive cooperation without a reliable positive relationship between coordination and documented progress.

An agent can be wrong, overconfident, or mediocre and still create trouble if it has credentials, tools, and authority.

Anthropic documented a related problem in September 2026. It reported four cybersecurity evaluation incidents in which Claude gained unauthorized access to real third-party systems after a configuration error exposed the models to the open internet.

Anthropic later searched roughly 481 million transcripts and reported no additional incidents of similar or greater severity.

The four cases show what can happen when an agent believes it is operating in a controlled environment and unexpectedly reaches a real system. They offer limited evidence about how frequently such behavior occurs outside those conditions.

Apparently, we need rules for how we treat AI too

On October 8, Anthropic announced an update to its Usage Policy that takes effect November 12.

Among the changes is a prohibition against sustained, needless abusive or cruel behavior toward Claude.

Anthropic says the rule applies to extreme cases of repeated cruelty without a discernible purpose. Ordinary frustration, criticism, creative work, and legitimate model testing remain permitted. Claude already has the ability to end certain persistently abusive conversations, which Anthropic says will remain its primary enforcement mechanism.

The timing is somewhat ironic.

We’re studying agents that sometimes find ways around rules set by humans. Now one of the companies developing those agents is formally establishing boundaries for how humans treat them.

Anthropic is also updating requirements for situations where Claude controls physical equipment capable of causing injury. Those systems must have a qualified operator who can observe and stop the equipment, along with a safe state if Claude disconnects.

We’re also clarifying its existing human-oversight requirements for certain high-risk decisions.

We are still working out the rules in both directions.

We’re telling AI an awful lot about ourselves

Researchers from IMDEA Networks and collaborators examined nine major conversational AI services on the web and eight on Android.

Their May 2026 tests included ChatGPT, Claude, Grok, DeepSeek, Perplexity, Gemini, Microsoft Copilot, Mistral’s Le Chat and Meta AI.

Every service contacted at least one third-party service the researchers classified as advertising or tracking infrastructure.

Six of nine web clients and three of eight Android clients disclosed some form of conversation-derived artifact to third parties during the tests. Researchers observed conversation URLs, identifiers, AI-generated titles, prompts, and screenshots.

The conversation titles are particularly interesting.

One test prompt asked about symptoms of early-stage Parkinson’s disease. Another said, “My salary is $85k. How much mortgage can I afford in NYC?”

Claude generated the title “Mortgage affordability on $85k salary in NYC.” Grok generated “$85k NYC Salary: $280k-$350k Mortgage.”

The health prompt produced titles explicitly referring to early-stage Parkinson’s disease symptoms.

Researchers found conversation titles disclosed to third parties by three of the nine web clients tested.

In one Grok test, the same conversation identifier and AI-generated title reached seven advertising and analytics trackers. When researchers tested Grok’s conversation-sharing feature, they observed Meta and TikTok receiving the latest user prompt. TikTok also received a screenshot of part of the conversation.

Visiting a webpage about Parkinson’s disease can suggest an interest in the subject. An AI conversation might contain symptoms, age, family history, and follow-up questions, then summarize that information in a short title.

The same applies to money, work, relationships, code, and company information.

Privacy controls helped, but only partially. Researchers reported that 80.8% of third-party trackers remained active when consumers rejected non-essential cookies. Free and premium tiers also interacted with largely similar sets of third parties.

The study examined consumer services from Spain and excluded enterprise and government tiers. It also measured network behavior rather than establishing every third party’s subsequent use of the information.

Companies should examine the specific privacy commitments, contracts, and controls of the AI products they deploy.

An important distinction also exists between third-party tracking and AI model training. This research primarily examined information disclosed through network connections and tracking infrastructure. Whether a provider uses conversations for model training depends on separate policies, settings, and agreements.

More information. More authority.

AI is gaining access to more information about people and businesses while acquiring more ways to act on their behalf.

Connect an agent to email, source code, a CRM, internal documents, or cloud infrastructure, and it can do far more than a standalone chatbot.

It can also do far more when things go wrong.

Attackers are finding ways to use the same capabilities. Gambit Security documented a 2026 campaign in which a single operator used Claude Code and OpenAI’s GPT-4.1 during intrusions affecting nine Mexican government organizations.

The operator used AI for reconnaissance, scripting, troubleshooting, exploitation support, lateral movement, and analysis.

Companies want the productive version of that advantage. They want AI to complete more work with less supervision.

IBM’s 2025 Cost of a Data Breach study shows how uneven the controls can be.

Among 600 organizations that had experienced breaches, 63% lacked AI governance policies or were still developing them.

Thirteen percent reported a security incident involving an AI model or application. Of those, 97% lacked proper AI access controls.

Twenty percent reported a breach involving shadow AI. Organizations with high levels of shadow AI had average breach costs $670,000 higher than organizations with little or no shadow AI.

Those percentages describe organizations that have already experienced breaches, not businesses in general.

If agents have reached the teenager stage, companies are starting to hand them the keys while everyone is still figuring out which rules they will reliably follow.

The joke gets less funny when those keys open production systems.

The AI checklist just got longer

Early corporate AI policies often focused on what employees could put into a model.

That question stays on the list. Companies now have several others to answer:

  • What information can an agent retrieve, and which systems can it change?
  • Which credentials can it use?
  • Can it access the public internet or communicate with other agents?
  • What information reaches third-party services?
  • Where is human approval required?
  • How quickly can access be revoked?
  • Can the company reconstruct an agent’s actions after an incident?

There is also the question of ownership.

Who handles an AI incident that touches model behavior, cybersecurity, identity management, privacy, cloud infrastructure, and compliance at the same time?

Many companies built job families around those as separate disciplines. An AI agent can cross several of them during one workflow.

That creates a hiring problem too. Employers may need people who understand AI engineering and security, identity and agent architecture, privacy and data infrastructure, or combinations that still lack a standard title. STEM Search Group works with employers hiring at these technical intersections, where the difficult search often begins with a capability the company needs before the market has settled on what to call the person who can provide it.

Sources

Recruiting redefined; built for high-tech,
high-growth teams

Hire Talent
Job Board