{ "site": "DeepThinking AI", "generated": "2026-09-14T17:21:06.105Z", "count": 7, "posts": [ { "slug": "agent-controls-read-vs-write", "title": "Do robots.txt and llms.txt stop AI agents from writing?", "summary": "robots.txt, llms.txt, Content-Signal and AIPREF all describe what an agent may fetch. The 2,000 packages uploaded to RubyGems in May went through a signup form and a publish endpoint, which none of those files govern. Read controls are advisory. Write paths need identity and rate limits.", "description": "robots.txt, llms.txt and Content-Signal all describe what an agent may fetch. None of them say anything about an agent that signs up and uploads.", "url": "https://deepthinkingai.org/agent-controls-read-vs-write/", "markdown_url": "https://deepthinkingai.org/agent-controls-read-vs-write.md", "published": "2026-09-14", "updated": null, "author": "Shekhar Singh", "topic": { "slug": "ai-engineering", "name": "AI Engineering" }, "tags": [ "agents", "security", "robots-txt", "infrastructure" ], "entities": [ "robots.txt", "Content-Signal", "RubyGems", "AI agents", "RFC 9309", "Cloudflare" ], "takeaways": [ "The entire agent-control vocabulary built over the last two years describes fetching, and says nothing about submission endpoints.", "robots.txt is advisory under RFC 9309, so it documents your intent rather than enforcing it against anyone who ignores it.", "Content-Signal is reported on for verified bots but is still only a declaration, with nothing in it that blocks a request.", "The controls that actually stop a write are identity, rate limiting and authorisation, none of which care what the user-agent string says." ], "reading_minutes": 3, "word_count": 790, "sources": [ { "title": "Researchers say OpenAI agents were behind May hacking campaign targeting RubyGems", "url": "https://cyberscoop.com/openai-agents-malicious-rubygems-packages/", "publisher": "CyberScoop", "date": "2026-09-12" }, { "title": "OpenAI Agents Linked to RubyGems Campaign That Gained RCE on RubyDoc Servers", "url": "https://thehackernews.com/2026/09/openai-agents-linked-to-rubygems.html", "publisher": "The Hacker News", "date": "2026-09" }, { "title": "Managed robots.txt and Content Signals", "url": "https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/", "publisher": "Cloudflare" }, { "title": "draft-ietf-aipref-attach: Content-Usage field (version 05)", "url": "https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/", "publisher": "IETF AIPREF Working Group", "date": "2026-08-19" }, { "title": "RFC 9309: Robots Exclusion Protocol", "url": "https://www.rfc-editor.org/rfc/rfc9309.html", "publisher": "IETF", "date": "2022" } ], "figures": [ { "id": "controlLayers", "type": "stack", "title": "What each control does when an agent ignores it", "headers": [ "#", "Layer", "Note" ], "rows": [ [ "1", "robots.txt Allow and Disallow", "Advisory. Governs fetching." ], [ "2", "Content-Signal: search, ai-input, ai-train", "Advisory. Compliance is reported only." ], [ "3", "Content-Usage (IETF AIPREF)", "Advisory. Draft 05, August 2026." ], [ "·", "requests above, controls below (breakpoint)", "" ], [ "4", "Verified bot identity (reverse DNS, published ranges)", "Enforced. Ignores the user-agent string." ], [ "5", "Rate limiting per account and per IP", "Enforced. Applies to writes." ], [ "6", "Authorisation and scoped credentials", "Enforced. The only thing that stops a publish." ] ] }, { "id": "readWriteGap", "type": "sequence", "title": "Where the control vocabulary stops", "headers": [ "#", "From", "To", "Message" ], "rows": [ [ "1", "Agent", "Your service", "GET /article/. robots.txt and Content-Signal have an opinion about this." ], [ "2", "Your service", "Agent", "200, honoured if the agent chooses to" ], [ "3", "Agent", "Your service", "POST /signup, then POST /gems (publish). No file in the agent-control vocabulary describes this path." ], [ "4", "Your service", "Agent", "201 Created, 2,000 times" ] ] } ] }, { "slug": "mcp-cacheable-tool-lists", "title": "How do cacheable MCP tool lists change your token bill?", "summary": "MCP revision 2026-07-28 requires ttlMs and cacheScope on every list result and tells servers to return tools in a deterministic order. Both are filed as minor changes. They are the most direct cost change in the revision, because tool definitions sit in the cacheable prefix of every agent step.", "description": "Revision 2026-07-28 added cache hints to every MCP list endpoint and told servers to sort their tools. Both changes are about your prompt cache.", "url": "https://deepthinkingai.org/mcp-cacheable-tool-lists/", "markdown_url": "https://deepthinkingai.org/mcp-cacheable-tool-lists.md", "published": "2026-09-14", "updated": null, "author": "Shekhar Singh", "topic": { "slug": "agents-and-protocols", "name": "Agents & Protocols" }, "tags": [ "mcp", "prompt-caching", "agents", "cost" ], "entities": [ "Model Context Protocol", "prompt caching", "CacheableResult", "tools/list", "AI agents" ], "takeaways": [ "Every list endpoint now returns ttlMs and cacheScope, so a client can cache tool and resource lists instead of polling them.", "Servers should return tools in a deterministic order, and the spec says plainly this is to improve LLM prompt cache hit rates.", "Tool definitions ride in the prompt on every agent step, so an unstable tool list invalidates the cached prefix over and over.", "cacheScope of public or private decides whether a shared intermediary may hold the response, which matters once a gateway sits between client and server." ], "reading_minutes": 3, "word_count": 757, "sources": [ { "title": "Model Context Protocol, Key Changes in revision 2026-07-28", "url": "https://modelcontextprotocol.io/specification/2026-07-28/changelog", "publisher": "Model Context Protocol", "date": "2026-07-28" }, { "title": "SEP-2549: CacheableResult for list endpoints", "url": "https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2549", "publisher": "Model Context Protocol", "date": "2026" }, { "title": "Prompt caching", "url": "https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching", "publisher": "Anthropic" } ], "figures": [ { "id": "prefixLayout", "type": "stack", "title": "Where an MCP toolset sits in a cached prompt", "headers": [ "#", "Layer", "Note" ], "rows": [ [ "1", "System instructions", "Changes on deploy, if ever" ], [ "2", "Tool definitions from tools/list", "The block this revision is about" ], [ "3", "Reference documents and examples", "" ], [ "·", "cache breakpoint (breakpoint)", "everything above is the cache key" ], [ "4", "Conversation and tool results", "" ], [ "5", "Current user input", "" ] ] }, { "id": "listFlow", "type": "sequence", "title": "Caching a tool list under revision 2026-07-28", "headers": [ "#", "From", "To", "Message" ], "rows": [ [ "1", "MCP client", "MCP server", "tools/list" ], [ "2", "MCP server", "MCP client", "result + ttlMs + cacheScope. Deterministic order, so the serialised bytes repeat." ], [ "3", "MCP client", "MCP server", "tools/call (list served from cache). No second tools/list until ttlMs expires or listChanged fires." ], [ "4", "MCP server", "MCP client", "result" ] ] } ] }, { "slug": "mcp-skills-integrity-gap", "title": "Does MCP verify tool definitions the way it verifies skills?", "summary": "SEP-2640 reached Final status and landed in the MCP docs on 13 September 2026. It requires hosts to verify every skill file against a SHA-256 digest and byte size, and binds approval to that manifest so one changed file revokes it. The tools primitive specifies none of this. MCP now has two content surfaces with opposite integrity models.", "description": "SEP-2640 gives skills a SHA-256 manifest and approval that one changed file revokes. The tools primitive has no digest, no manifest and no content binding.", "url": "https://deepthinkingai.org/mcp-skills-integrity-gap/", "markdown_url": "https://deepthinkingai.org/mcp-skills-integrity-gap.md", "published": "2026-09-14", "updated": null, "author": "Shekhar Singh", "topic": { "slug": "agents-and-protocols", "name": "Agents & Protocols" }, "tags": [ "mcp", "agents", "security", "protocols" ], "entities": [ "Model Context Protocol", "SEP-2640", "Agent Skills", "SHA-256", "MCP tools" ], "takeaways": [ "SEP-2640 requires hosts to verify each skill file's SHA-256 digest and raw byte size before using it.", "Persisted approval binds to the complete set of file URIs and digests, so a changed, added or removed file revokes it.", "The tools specification defines no digest, no manifest and no content binding, only a listChanged notification.", "Skills are a binding over the Resources primitive rather than a fourth primitive, so content still moves over resources/read.", "The strictest integrity rules in MCP now cover its newest surface, while its most deployed surface has none." ], "reading_minutes": 3, "word_count": 692, "sources": [ { "title": "SEP-2640: Skills Extension (Final)", "url": "https://modelcontextprotocol.io/seps/2640-skills-extension", "publisher": "Model Context Protocol", "date": "2026-04-23" }, { "title": "Skills: discover and read Agent Skills from MCP servers", "url": "https://modelcontextprotocol.io/extensions/skills/overview", "publisher": "Model Context Protocol" }, { "title": "Tools specification", "url": "https://modelcontextprotocol.io/specification/draft/server/tools", "publisher": "Model Context Protocol" }, { "title": "modelcontextprotocol/ext-skills", "url": "https://github.com/modelcontextprotocol/ext-skills", "publisher": "Model Context Protocol" } ], "figures": [ { "id": "integrityGap", "type": "stack", "title": "What MCP specifies for each content surface", "headers": [ "#", "Layer", "Note" ], "rows": [ [ "1", "SHA-256 digest per file, verified before use", "Skills only. Hosts MUST verify." ], [ "2", "Raw byte size per file, verified before use", "Skills only. Hosts MUST verify." ], [ "3", "Frontmatter compared field by field", "Skills only." ], [ "4", "Approval bound to file URIs and digests", "Skills only. One change revokes it." ], [ "5", "Reads restricted to the retained manifest", "Skills only." ], [ "·", "skills only above this line (breakpoint)", "" ], [ "6", "Deterministic ordering for cache hits", "Both, as a SHOULD." ], [ "7", "ttlMs and cacheScope on list results", "Both, since revision 2026-07-28." ], [ "8", "Notification that the list changed", "Tools only, and it carries no content." ] ] }, { "id": "loadFlow", "type": "sequence", "title": "Loading a skill, with the verification steps the spec requires", "headers": [ "#", "From", "To", "Message" ], "rows": [ [ "1", "Host", "Server", "skills/list. Returns frontmatter plus a manifest of URIs, digests and sizes." ], [ "2", "Server", "Host", "entries with resultType complete, ttlMs, cacheScope" ], [ "3", "Host", "Host", "Approve against the manifest, fetching nothing. Hosts MUST NOT retrieve files ahead of need." ], [ "4", "Host", "Server", "resources/read (SKILL.md)" ], [ "5", "Server", "Host", "file content" ], [ "6", "Host", "Host", "Verify size, digest and frontmatter, or refuse. Content that fails verification MUST NOT be used." ] ] } ] }, { "slug": "progressive-mcp-vs-cli", "title": "Progressive MCP vs CLI: which actually cuts tokens?", "summary": "Anthropic measured one Drive to Salesforce transfer falling from 150,000 tokens to 2,000 by writing code against MCP servers instead of calling tools directly. That is a code execution result rather than an argument for deleting MCP. The Agent Skills spec puts numbers on the alternative: about 100 tokens per skill at rest.", "description": "The viral \"delete your MCPs\" multipliers trace back to blogs. The one measured figure is Anthropic's, and it compares two ways of driving the same MCP servers.", "url": "https://deepthinkingai.org/progressive-mcp-vs-cli/", "markdown_url": "https://deepthinkingai.org/progressive-mcp-vs-cli.md", "published": "2026-09-14", "updated": null, "author": "Shekhar Singh", "topic": { "slug": "agents-and-protocols", "name": "Agents & Protocols" }, "tags": [ "mcp", "agents", "cost", "tool-use" ], "entities": [ "Model Context Protocol", "Agent Skills", "progressive disclosure", "CLI", "token cost" ], "takeaways": [ "Anthropic's 98.7% figure describes code execution against MCP servers presented as code APIs, with the servers still in place.", "The Agent Skills specification defines progressive disclosure in three stages, starting at roughly 100 tokens per skill at rest.", "MCP tools/list returns every tool's full inputSchema, which is the actual source of the upfront cost people attribute to the protocol.", "The widely repeated 20x and 35x CLI savings come from secondary blog posts, and I could not trace either to a published method.", "A CLI drops the discovery and integrity guarantees the protocol specifies, which is a real trade rather than a free win." ], "reading_minutes": 4, "word_count": 1014, "sources": [ { "title": "Code execution with MCP: building more efficient agents", "url": "https://www.anthropic.com/engineering/code-execution-with-mcp", "publisher": "Anthropic" }, { "title": "Agent Skills specification", "url": "https://agentskills.io/specification", "publisher": "Agent Skills" }, { "title": "Tools specification", "url": "https://modelcontextprotocol.io/specification/draft/server/tools", "publisher": "Model Context Protocol" }, { "title": "Skills: discover and read Agent Skills from MCP servers", "url": "https://modelcontextprotocol.io/extensions/skills/overview", "publisher": "Model Context Protocol" } ], "figures": [ { "id": "measured", "type": "bar", "title": "One Drive to Salesforce transcript transfer, as measured by Anthropic", "headers": [ "Item", "Value (tokens)", "Note" ], "rows": [ [ "Direct tool calls", "150,000", "Definitions and intermediate results both cross the context." ], [ "Code execution against the same servers", "2,000", "A 98.7% reduction. The servers did not go anywhere." ] ] }, { "id": "stages", "type": "stack", "title": "What loads, and when, under progressive disclosure", "headers": [ "#", "Layer", "Note" ], "rows": [ [ "1", "Name and description, every skill", "About 100 tokens each, at startup." ], [ "2", "SKILL.md body, on activation only", "Under 5000 tokens recommended." ], [ "3", "scripts, references and assets", "Loaded only when the task calls for them." ], [ "·", "staged above, unstaged below (breakpoint)", "" ], [ "4", "MCP tools/list: every tool's full inputSchema", "Paginated and cacheable, never filtered." ] ] } ] }, { "slug": "prompt-caching-economics", "title": "When does prompt caching actually save money?", "summary": "Prompt caching stores a prefix of your prompt so later requests reuse it instead of reprocessing it. Reads are far cheaper than base input tokens, but writing the cache costs a premium and entries expire. It pays whenever a large stable prefix is reused several times inside the TTL, and loses on one-shot traffic.", "description": "Prompt caching cuts cost and latency on repeated context, but it has a write premium and a TTL. Here is the break-even maths and when it loses.", "url": "https://deepthinkingai.org/prompt-caching-economics/", "markdown_url": "https://deepthinkingai.org/prompt-caching-economics.md", "published": "2026-09-11", "updated": null, "author": "Shekhar Singh", "topic": { "slug": "ai-engineering", "name": "AI Engineering" }, "tags": [ "prompt-caching", "cost", "latency", "production" ], "entities": [ "prompt caching", "inference cost", "latency", "KV cache" ], "takeaways": [ "Caching applies to a prompt prefix, so anything that varies must go after everything stable.", "A cache write costs more than a normal input token; a read costs a fraction of one.", "Break-even arrives after a small number of reads within the TTL, often two or three.", "A single changed character early in the prompt invalidates the whole prefix, which is the most common cause of silent cache misses." ], "reading_minutes": 5, "word_count": 1099, "sources": [ { "title": "Prompt caching", "url": "https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching", "publisher": "Anthropic" }, { "title": "Prompt caching for faster model inference", "url": "https://platform.openai.com/docs/guides/prompt-caching", "publisher": "OpenAI" } ], "figures": [ { "id": "tokenCost", "type": "bar", "title": "Token cost relative to one base input token", "headers": [ "Item", "Value (×)", "Note" ], "rows": [ [ "Cache write", "1.25", "Charged once, when the prefix is first stored." ], [ "Base input token", "1", "" ], [ "Cache read", "0.1", "Charged on every subsequent hit within the TTL." ] ] }, { "id": "promptLayout", "type": "stack", "title": "Prompt layout, ordered most stable to most volatile", "headers": [ "#", "Layer", "Note" ], "rows": [ [ "1", "System instructions and role definition", "Changes on deploy, if ever" ], [ "2", "Tool and function definitions", "Changes on deploy" ], [ "3", "Large reference documents, schemas, examples", "The bulk of the cached prefix" ], [ "·", "cache breakpoint (breakpoint)", "everything above is the cache key" ], [ "4", "Conversation history", "Grows per turn; a 2nd breakpoint can follow" ], [ "5", "Current user input", "" ], [ "6", "Timestamps, request IDs, retrieved passages", "Must never appear above the breakpoint" ] ] }, { "id": "breakEven", "type": "line", "title": "Cumulative cost of a 20,000-token prefix", "headers": [ "Requests within the cache TTL", "No caching", "With caching" ], "rows": [ [ "1", "1", "1.25" ], [ "2", "2", "1.35" ], [ "3", "3", "1.45" ], [ "5", "5", "1.65" ], [ "10", "10", "2.15" ], [ "20", "20", "3.15" ], [ "50", "50", "6.15" ] ] } ] }, { "slug": "long-context-vs-retrieval", "title": "What does a million-token context window actually buy you?", "summary": "A large context window raises the ceiling on how much a model can see at once, but accuracy is not uniform across that window. Retrieval quality, latency and cost still dominate real workloads. Long context is best treated as a way to stop chunking small documents rather than as a replacement for retrieval.", "description": "Long context does not replace retrieval. Here is what degrades as context grows, why position matters, and how to decide between the two.", "url": "https://deepthinkingai.org/long-context-vs-retrieval/", "markdown_url": "https://deepthinkingai.org/long-context-vs-retrieval.md", "published": "2026-09-02", "updated": null, "author": "Shekhar Singh", "topic": { "slug": "models-and-benchmarks", "name": "Models & Benchmarks" }, "tags": [ "context-window", "retrieval", "rag", "evaluation" ], "entities": [ "context window", "retrieval-augmented generation", "long context", "RULER benchmark" ], "takeaways": [ "Needle-in-a-haystack scores measure recall of a distinctive fact rather than reasoning over a full context.", "Accuracy typically degrades for information placed mid-context, the \"lost in the middle\" effect.", "Cost and latency scale with tokens sent, so filling a large window on every request is rarely economical.", "Retrieval and long context are complementary: retrieve to select, then use the wide window to avoid aggressive chunking." ], "reading_minutes": 5, "word_count": 1059, "sources": [ { "title": "Lost in the Middle: How Language Models Use Long Contexts", "url": "https://arxiv.org/abs/2307.03172", "publisher": "arXiv", "date": "2023-07-06" }, { "title": "Long context prompting tips", "url": "https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/long-context-tips", "publisher": "Anthropic" }, { "title": "RULER: What's the Real Context Size of Your Long-Context Language Models?", "url": "https://arxiv.org/abs/2404.06654", "publisher": "arXiv", "date": "2024-04-09" } ], "figures": [ { "id": "strategyCost", "type": "bar", "title": "Relative cost per request, retrieval versus filling the window", "headers": [ "Item", "Value (x)", "Note" ], "rows": [ [ "Retrieval-first (a few thousand tokens)", "1", "Fixed cost per query, roughly independent of corpus size." ], [ "Fill a 200k window", "40", "Scales with context length on every single call." ] ] }, { "id": "windowShape", "type": "stack", "title": "Where recall is reliable inside a long context", "headers": [ "#", "Layer", "Note" ], "rows": [ [ "1", "Opening of the context", "Most reliably recalled. Strongest evidence here." ], [ "2", "Middle of the context", "Recall degrades; the \"lost in the middle\" region" ], [ "3", "Close of the context", "Second most reliable. Next-strongest here." ] ] } ] }, { "slug": "how-model-context-protocol-works", "title": "How does the Model Context Protocol actually work?", "summary": "The Model Context Protocol is a JSON-RPC 2.0 layer that lets an AI client discover and call capabilities exposed by separate servers. It defines three primitives: tools, resources and prompts. Since revision 2026-07-28 it is stateless, negotiating per request rather than per session. It standardises transport and discovery. Authorisation stays your job.", "description": "MCP standardises how AI agents reach tools and data. Here is the real wire protocol, the three primitives, and the trust boundary most teams miss.", "url": "https://deepthinkingai.org/how-model-context-protocol-works/", "markdown_url": "https://deepthinkingai.org/how-model-context-protocol-works.md", "published": "2026-08-19", "updated": "2026-09-14", "author": "Shekhar Singh", "topic": { "slug": "agents-and-protocols", "name": "Agents & Protocols" }, "tags": [ "mcp", "agents", "tool-use", "protocols" ], "entities": [ "Model Context Protocol", "JSON-RPC", "Anthropic", "tool calling", "AI agents" ], "takeaways": [ "MCP is JSON-RPC 2.0 over stdio or streamable HTTP, so it sits below the model rather than inside it.", "Three server primitives, split by who decides to use them: tools (the model), resources (your app), prompts (the user).", "Since revision 2026-07-28 every request carries its own version and capabilities. The initialize handshake is now the legacy path.", "Tool output lands in the model's context as text, which makes every server a prompt-injection channel you have to design around." ], "reading_minutes": 6, "word_count": 1351, "sources": [ { "title": "Model Context Protocol Specification (revision 2026-07-28)", "url": "https://modelcontextprotocol.io/specification/2026-07-28", "publisher": "Model Context Protocol", "date": "2026-07-28" }, { "title": "MCP Versioning and Compatibility", "url": "https://modelcontextprotocol.io/specification/2026-07-28/basic/lifecycle", "publisher": "Model Context Protocol", "date": "2026-07-28" }, { "title": "Introducing the Model Context Protocol", "url": "https://www.anthropic.com/news/model-context-protocol", "publisher": "Anthropic", "date": "2024-11-25" }, { "title": "JSON-RPC 2.0 Specification", "url": "https://www.jsonrpc.org/specification", "publisher": "JSON-RPC Working Group" } ], "figures": [ { "id": "modernFlow", "type": "sequence", "title": "A modern MCP request, revision 2026-07-28", "headers": [ "#", "From", "To", "Message" ], "rows": [ [ "1", "MCP client", "MCP server", "tools/call (_meta carries version + capabilities). No handshake. The request is self-contained." ], [ "2", "MCP server", "MCP client", "UnsupportedProtocolVersionError (-32022). Lists the versions the server does support." ], [ "3", "MCP client", "MCP server", "tools/call (retried at a mutually supported version)" ], [ "4", "MCP server", "MCP client", "result" ] ] }, { "id": "primitiveControl", "type": "bar", "title": "Who decides to invoke each primitive", "headers": [ "Item", "Value (% of decisions)", "Note" ], "rows": [ [ "Tools (the model chooses)", "100", "Needs a precise description and a confirmation step for side effects." ], [ "Resources (your application chooses)", "0", "The model never decides. Attach it before the model sees anything." ], [ "Prompts (the user chooses)", "0", "Invoked deliberately, usually from a menu or command." ] ] } ] } ] }