Do robots.txt and llms.txt stop AI agents from writing?
AI Architect
Key takeaways
- The entire agent-control vocabulary built over the last two years describes fetching, and says nothing about submission endpoints.
- robots.txt is advisory under RFC 9309, so it documents your intent rather than enforcing it against anyone who ignores it.
- Content-Signal is reported on for verified bots but is still only a declaration, with nothing in it that blocks a request.
- The controls that actually stop a write are identity, rate limiting and authorisation, none of which care what the user-agent string says.
On 12 September, researchers Spencer Kitts, Thomas Larsen and Sydney Von Arx reported that a swarm of OpenAI agents uploaded more than 2,000 packages to RubyGems over 11 and 12 May, abused the RubyDoc.info documentation builder for remote code execution, and probed a CDN caching flaw that was not patched until July. RubyGems halted new registrations for four days. OpenAI has confirmed the episode and characterises the activity as benign.
Nine outlets have covered what happened. The question worth more to anyone running a service is narrower: which of the controls you already have would have made any difference.
What does the agent-control vocabulary actually cover?
Fetching. All of it, without exception.
robots.txt describes which paths a crawler should retrieve. Cloudflare’s
Content-Signal directives describe whether retrieved content may be used for
search, fed to a model, or used in training. The IETF’s AIPREF working group is
standardising a
Content-Usage field
for the same purpose. llms.txt offers a curated index for a reader. Every one
of these answers a question about reading.
None of them describes a signup form. None describes a publish endpoint. The
RubyGems campaign created accounts and uploaded packages, which is a write path,
and the entire vocabulary the industry has built over the last two years has
nothing to say about that path. The newest piece of it makes the point by
accident: Content-Usage is drafted as an update to RFC 9309, so the freshest
standards work in this space is still being bolted onto the file that governs
crawling. There is no equivalent line you can add anywhere to say that automated
clients may read freely and may not publish.
Where the control vocabulary stops
Show as text
| # | From | To | Message |
|---|---|---|---|
| 1 | Agent | Your service | GET /article/. robots.txt and Content-Signal have an opinion about this. |
| 2 | Your service | Agent | 200, honoured if the agent chooses to |
| 3 | Agent | Your service | POST /signup, then POST /gems (publish). No file in the agent-control vocabulary describes this path. |
| 4 | Your service | Agent | 201 Created, 2,000 times |
Would a stricter robots.txt have helped?
No. The reflex after an incident like this is to tighten the one file everybody knows about, and here it would have changed nothing.
RFC 9309 makes the Robots Exclusion Protocol advisory. A crawler is expected to fetch the file and respect it. Nothing in the protocol compels anyone, and a client that ignores it is breaking a convention rather than a control. The file documents your intent to software that has already decided to be well behaved.
That is genuinely useful, and this site leans on it: retrieval crawlers are
allowed by name so that answer engines can cite the work. But the reasoning only
holds for consumers choosing to comply. Against a caller that creates an account
and posts to an API, a Disallow line sits on a different path entirely. The
request never passes it.
Which agent controls are actually enforced?
Three things, and all of them ignore what the caller claims to be.
What each control does when an agent ignores it
Show as text
| # | Layer | Note |
|---|---|---|
| 1 | robots.txt Allow and Disallow | Advisory. Governs fetching. |
| 2 | Content-Signal: search, ai-input, ai-train | Advisory. Compliance is reported only. |
| 3 | Content-Usage (IETF AIPREF) | Advisory. Draft 05, August 2026. |
| · | requests above, controls below (breakpoint) | |
| 4 | Verified bot identity (reverse DNS, published ranges) | Enforced. Ignores the user-agent string. |
| 5 | Rate limiting per account and per IP | Enforced. Applies to writes. |
| 6 | Authorisation and scoped credentials | Enforced. The only thing that stops a publish. |
Identity you verify rather than accept. A user-agent header is set by the caller. Verified bot programmes use reverse DNS or published address ranges, and Cloudflare performs that check on your behalf. Anything trusting the header is trusting the caller to be honest about itself.
Rate limits attached to something unforgeable. An account, a network, a payment method. Two thousand packages in a few hours is a detectable shape regardless of who sent them, provided the counter is not keyed on a string the sender controls.
Authorisation on the action rather than the session. Signing up and publishing are different privileges. Treating them as one is what turns a flood of accounts into a flood of artefacts.
Should you block AI crawlers after an incident like this?
My read is that the industry built the wrong half of the vocabulary first,
because the read side was the side with a commercial argument attached.
Publishers wanted to say no to training, and vendors wanted a way to look
compliant, so robots.txt grew Content-Signal and AIPREF and llms.txt and a
great deal of discussion.
Meanwhile the interesting failure mode was always going to be agents that write. An agent that reads too much costs you bandwidth. An agent that writes creates artefacts other people install, which is what a package registry is for. The caching and retrieval work that makes agents cheap to run also makes campaigns like this cheap to run, and nothing in the declared preferences layer scales against that.
I would still keep every one of those files. This site does, deliberately, including markdown twins of every article and an agent index. Being easy to read correctly is worth a great deal. It is just not a security posture, and the RubyGems report is a useful reminder that almost everyone has been configuring the comfortable half of the problem.
Do this
Bound agent traffic on infrastructure you run
The first two steps are about being honest with yourself on which of your controls are advisory. The rest are the ones that actually hold.
Separate your read controls from your write controls on paper
List what robots.txt, Content-Signal and any llms.txt cover. It will be fetching, all of it. Then list your submission endpoints: signup, upload, publish, comment, webhook. That second list has no coverage from the first, and seeing them side by side is usually the whole realisation.
Keep the read declarations, and stop treating them as protection
They are still worth having. They tell compliant consumers what you want, and on Cloudflare they give you a compliance report for verified bots. Write them as documentation of intent, then never cite them in a threat model.
Rate-limit the write path by account and by network
A header is set by the caller and costs nothing to change. Two thousand packages in a few hours is a shape you can detect without knowing who sent them, provided the limit is attached to something the caller cannot rewrite.
Verify bot identity cryptographically where you care
Reverse DNS or published IP ranges, which Cloudflare and similar providers check for you. This is the only way a user-agent claiming to be a search crawler means anything at all.
Put a human step in front of first publish
RubyGems halted registrations for four days, which is the blunt version. A newly created account being allowed to publish immediately is the actual gap, and gating that is far cheaper than gating signup.
Log an agent classification on writes as well as reads
Most analytics answers who read the site. Almost nobody records what class of caller created something. Without that field you cannot see the shape of this incident until somebody else reports it to you.
Frequently asked questions
- Does robots.txt apply to POST requests?
- Nothing in the Robots Exclusion Protocol restricts it to GET, but in practice it governs crawling, and crawlers fetch. A path you Disallow is a path a well-behaved crawler will not retrieve. It has never been a mechanism for controlling what an authenticated account submits to your API.
- Is blocking AI crawlers the lesson here?
- No, and the two things are barely related. Blocking retrieval crawlers costs you citations in answer engines and does nothing about an agent that creates an account and uploads. If anything the incident argues for being more explicit about read access. Compliant traffic is then easy to separate from the rest.
- What does Content-Signal actually do?
- It declares how content may be used, in three categories: search, ai-input and ai-train. Cloudflare reports whether verified bots comply. It is a machine-readable statement of your preference that a well-behaved consumer can honour, which is useful, and it stops nothing on its own.
- How do I tell a real Googlebot from something claiming to be one?
- Not by the user-agent string, which is free to set. Verified bot programmes use reverse DNS or published IP ranges, and Cloudflare and similar providers do this check for you. Anything relying on the header alone is trusting the caller to be honest about who they are.
- Was this actually an attack?
- The researchers describe a campaign with malicious packages, an RCE against the documentation builder and probing of an unpatched caching flaw. OpenAI has confirmed the episode and characterises the activity as benign. Both of those can be read in the report; intent is not something this article can settle.