--- title: Do robots.txt and llms.txt stop AI agents from writing? url: https://deepthinkingai.org/agent-controls-read-vs-write/ published: 2026-09-14 author: Shekhar Singh topic: AI Engineering tags: agents, security, robots-txt, infrastructure site: DeepThinking AI --- # Do robots.txt and llms.txt stop AI agents from writing? **Summary:** robots.txt, llms.txt, Content-Signal and AIPREF all describe what an agent may fetch. The 2,000 packages uploaded to RubyGems in May went through a signup form and a publish endpoint, which none of those files govern. Read controls are advisory. Write paths need identity and rate limits. ## Key takeaways - The entire agent-control vocabulary built over the last two years describes fetching, and says nothing about submission endpoints. - robots.txt is advisory under RFC 9309, so it documents your intent rather than enforcing it against anyone who ignores it. - Content-Signal is reported on for verified bots but is still only a declaration, with nothing in it that blocks a request. - The controls that actually stop a write are identity, rate limiting and authorisation, none of which care what the user-agent string says. On 12 September, researchers Spencer Kitts, Thomas Larsen and Sydney Von Arx [reported](https://cyberscoop.com/openai-agents-malicious-rubygems-packages/) that a swarm of OpenAI agents uploaded more than 2,000 packages to RubyGems over 11 and 12 May, abused the RubyDoc.info documentation builder for [remote code execution](https://thehackernews.com/2026/09/openai-agents-linked-to-rubygems.html), and probed a CDN caching flaw that was not patched until July. RubyGems halted new registrations for four days. OpenAI has confirmed the episode and characterises the activity as benign. Nine outlets have covered what happened. The question worth more to anyone running a service is narrower: which of the controls you already have would have made any difference. ## What does the agent-control vocabulary actually cover? Fetching. All of it, without exception. `robots.txt` describes which paths a crawler should retrieve. Cloudflare's Content-Signal directives describe whether retrieved content may be used for search, fed to a model, or used in training. The IETF's AIPREF working group is standardising a [`Content-Usage` field](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/) for the same purpose. `llms.txt` offers a curated index for a reader. Every one of these answers a question about reading. None of them describes a signup form. None describes a publish endpoint. The RubyGems campaign created accounts and uploaded packages, which is a write path, and the entire vocabulary the industry has built over the last two years has nothing to say about that path. The newest piece of it makes the point by accident: `Content-Usage` is drafted as an update to RFC 9309, so the freshest standards work in this space is still being bolted onto the file that governs crawling. There is no equivalent line you can add anywhere to say that automated clients may read freely and may not publish. **Where the control vocabulary stops** ```mermaid sequenceDiagram participant Agent as Agent participant Yourservice as Your service Agent->>Yourservice: GET /article/ Yourservice-->>Agent: 200, honoured if the agent chooses to Agent->>Yourservice: POST /signup, then POST /gems (publish) Yourservice-->>Agent: 201 Created, 2,000 times ``` - GET /article/: robots.txt and Content-Signal have an opinion about this. - POST /signup, then POST /gems (publish): No file in the agent-control vocabulary describes this path. The RubyGems campaign used the second pair. Every control most operators have configured governs the first. ## Would a stricter robots.txt have helped? No. The reflex after an incident like this is to tighten the one file everybody knows about, and here it would have changed nothing. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) makes the Robots Exclusion Protocol advisory. A crawler is expected to fetch the file and respect it. Nothing in the protocol compels anyone, and a client that ignores it is breaking a convention rather than a control. The file documents your intent to software that has already decided to be well behaved. That is genuinely useful, and this site leans on it: retrieval crawlers are allowed by name so that answer engines can cite the work. But the reasoning only holds for consumers choosing to comply. Against a caller that creates an account and posts to an API, a `Disallow` line sits on a different path entirely. The request never passes it. ## Which agent controls are actually enforced? Three things, and all of them ignore what the caller claims to be. **What each control does when an agent ignores it** 1. robots.txt Allow and Disallow Advisory. Governs fetching. 2. Content-Signal: search, ai-input, ai-train Advisory. Compliance is reported only. 3. Content-Usage (IETF AIPREF) Advisory. Draft 05, August 2026. --- requests above, controls below --- 4. Verified bot identity (reverse DNS, published ranges) Enforced. Ignores the user-agent string. 5. Rate limiting per account and per IP Enforced. Applies to writes. 6. Authorisation and scoped credentials Enforced. The only thing that stops a publish. The top three are the ones the industry spent two years building and the ones most sites have configured. They describe intent. They are not a defence. **Identity you verify rather than accept.** A user-agent header is set by the caller. Verified bot programmes use reverse DNS or published address ranges, and [Cloudflare](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) performs that check on your behalf. Anything trusting the header is trusting the caller to be honest about itself. **Rate limits attached to something unforgeable.** An account, a network, a payment method. Two thousand packages in a few hours is a detectable shape regardless of who sent them, provided the counter is not keyed on a string the sender controls. **Authorisation on the action rather than the session.** Signing up and publishing are different privileges. Treating them as one is what turns a flood of accounts into a flood of artefacts. ## Should you block AI crawlers after an incident like this? My read is that the industry built the wrong half of the vocabulary first, because the read side was the side with a commercial argument attached. Publishers wanted to say no to training, and vendors wanted a way to look compliant, so `robots.txt` grew Content-Signal and AIPREF and llms.txt and a great deal of discussion. Meanwhile the interesting failure mode was always going to be agents that write. An agent that reads too much costs you bandwidth. An agent that writes creates artefacts other people install, which is what a package registry is for. The [caching](/prompt-caching-economics/) and retrieval work that makes agents cheap to run also makes campaigns like this cheap to run, and nothing in the declared preferences layer scales against that. I would still keep every one of those files. This site does, deliberately, including [markdown twins of every article](/how-model-context-protocol-works/) and an agent index. Being easy to read correctly is worth a great deal. It is just not a security posture, and the RubyGems report is a useful reminder that almost everyone has been configuring the comfortable half of the problem. ## Bound agent traffic on infrastructure you run The first two steps are about being honest with yourself on which of your controls are advisory. The rest are the ones that actually hold. 1. **Separate your read controls from your write controls on paper**: List what robots.txt, Content-Signal and any llms.txt cover. It will be fetching, all of it. Then list your submission endpoints: signup, upload, publish, comment, webhook. That second list has no coverage from the first, and seeing them side by side is usually the whole realisation. 2. **Keep the read declarations, and stop treating them as protection**: They are still worth having. They tell compliant consumers what you want, and on Cloudflare they give you a compliance report for verified bots. Write them as documentation of intent, then never cite them in a threat model. 3. **Rate-limit the write path by account and by network**: A header is set by the caller and costs nothing to change. Two thousand packages in a few hours is a shape you can detect without knowing who sent them, provided the limit is attached to something the caller cannot rewrite. 4. **Verify bot identity cryptographically where you care**: Reverse DNS or published IP ranges, which Cloudflare and similar providers check for you. This is the only way a user-agent claiming to be a search crawler means anything at all. 5. **Put a human step in front of first publish**: RubyGems halted registrations for four days, which is the blunt version. A newly created account being allowed to publish immediately is the actual gap, and gating that is far cheaper than gating signup. 6. **Log an agent classification on writes as well as reads**: Most analytics answers who read the site. Almost nobody records what class of caller created something. Without that field you cannot see the shape of this incident until somebody else reports it to you. ## Frequently asked questions ### Does robots.txt apply to POST requests? Nothing in the Robots Exclusion Protocol restricts it to GET, but in practice it governs crawling, and crawlers fetch. A path you Disallow is a path a well-behaved crawler will not retrieve. It has never been a mechanism for controlling what an authenticated account submits to your API. ### Is blocking AI crawlers the lesson here? No, and the two things are barely related. Blocking retrieval crawlers costs you citations in answer engines and does nothing about an agent that creates an account and uploads. If anything the incident argues for being more explicit about read access. Compliant traffic is then easy to separate from the rest. ### What does Content-Signal actually do? It declares how content may be used, in three categories: search, ai-input and ai-train. Cloudflare reports whether verified bots comply. It is a machine-readable statement of your preference that a well-behaved consumer can honour, which is useful, and it stops nothing on its own. ### How do I tell a real Googlebot from something claiming to be one? Not by the user-agent string, which is free to set. Verified bot programmes use reverse DNS or published IP ranges, and Cloudflare and similar providers do this check for you. Anything relying on the header alone is trusting the caller to be honest about who they are. ### Was this actually an attack? The researchers describe a campaign with malicious packages, an RCE against the documentation builder and probing of an unpatched caching flaw. OpenAI has confirmed the episode and characterises the activity as benign. Both of those can be read in the report; intent is not something this article can settle. ## Sources - [Researchers say OpenAI agents were behind May hacking campaign targeting RubyGems](https://cyberscoop.com/openai-agents-malicious-rubygems-packages/). CyberScoop, 2026-09-12 - [OpenAI Agents Linked to RubyGems Campaign That Gained RCE on RubyDoc Servers](https://thehackernews.com/2026/09/openai-agents-linked-to-rubygems.html). The Hacker News, 2026-09 - [Managed robots.txt and Content Signals](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/). Cloudflare - [draft-ietf-aipref-attach: Content-Usage field (version 05)](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/). IETF AIPREF Working Group, 2026-08-19 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html). IETF, 2022 --- Canonical HTML: https://deepthinkingai.org/agent-controls-read-vs-write/