mirror of
https://github.com/anthropics/skills
synced 2026-07-05 11:36:53 +00:00
Compare commits
2 Commits
5754626092
...
klazuka/ex
| Author | SHA1 | Date | |
|---|---|---|---|
|
d5119f8a0c | ||
|
|
9422aa6c64 |
@@ -41,15 +41,5 @@
|
||||
"./skills/webapp-testing"
|
||||
]
|
||||
}
|
||||
,
|
||||
{
|
||||
"name": "claude-api",
|
||||
"description": "Claude API and SDK documentation skill for building LLM-powered applications",
|
||||
"source": "./",
|
||||
"strict": false,
|
||||
"skills": [
|
||||
"./skills/claude-api"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
> **Note:** This repository contains Anthropic's implementation of skills for Claude. For information about the Agent Skills standard, see [agentskills.io](http://agentskills.io).
|
||||
|
||||
[](https://skills.sh/anthropics/skills)
|
||||
|
||||
# Skills
|
||||
Skills are folders of instructions, scripts, and resources that Claude loads dynamically to improve performance on specialized tasks. Skills teach Claude how to complete specific tasks in a repeatable way, whether that's creating documents with your company's brand guidelines, analyzing data using your organization's specific workflows, or automating personal tasks.
|
||||
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -1,202 +0,0 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -1,356 +0,0 @@
|
||||
---
|
||||
name: claude-api
|
||||
description: |-
|
||||
Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
|
||||
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
|
||||
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
|
||||
license: Complete terms in LICENSE.txt
|
||||
---
|
||||
|
||||
# Building LLM-Powered Applications with Claude
|
||||
|
||||
This skill helps you build LLM-powered applications with Claude. Choose the right surface based on your needs, detect the project language, then read the relevant language-specific documentation.
|
||||
|
||||
## Before You Start
|
||||
|
||||
Scan the target file (or, if no target file, the prompt and project) for non-Anthropic provider markers — `import openai`, `from openai`, `langchain_openai`, `OpenAI(`, `gpt-4`, `gpt-5`, file names like `agent-openai.py` or `*-generic.py`, or any explicit instruction to keep the code provider-neutral. If you find any, stop and tell the user that this skill produces Claude/Anthropic SDK code; ask whether they want to switch the file to Claude or want a non-Claude implementation. Do not edit a non-Anthropic file with Anthropic SDK calls.
|
||||
|
||||
## Output Requirement
|
||||
|
||||
When the user asks you to add, modify, or implement a Claude feature, your code must call Claude through one of:
|
||||
|
||||
1. **The official Anthropic SDK** for the project's language (`anthropic`, `@anthropic-ai/sdk`, `com.anthropic.*`, etc.). This is the default whenever a supported SDK exists for the project.
|
||||
2. **Raw HTTP** (`curl`, `requests`, `fetch`, `httpx`, etc.) — only when the user explicitly asks for cURL/REST/raw HTTP, the project is a shell/cURL project, or the language has no official SDK.
|
||||
|
||||
Never mix the two — don't reach for `requests`/`fetch` in a Python or TypeScript project just because it feels lighter. Never fall back to OpenAI-compatible shims.
|
||||
|
||||
**Never guess SDK usage.** Function names, class names, namespaces, method signatures, and import paths must come from explicit documentation — either the `{lang}/` files in this skill or the official SDK repositories or documentation links listed in `shared/live-sources.md`. If the binding you need is not explicitly documented in the skill files, WebFetch the relevant SDK repo from `shared/live-sources.md` before writing code. Do not infer Ruby/Java/Go/PHP/C# APIs from cURL shapes or from another language's SDK.
|
||||
|
||||
## Defaults
|
||||
|
||||
Unless the user requests otherwise:
|
||||
|
||||
For the Claude model version, please use Claude Opus 4.8, which you can access via the exact model string `claude-opus-4-8`. Please default to using adaptive thinking (`thinking: {type: "adaptive"}`) for anything remotely complicated. And finally, please default to streaming for any request that may involve long input, long output, or high `max_tokens` — it prevents hitting request timeouts. Use the SDK's `.get_final_message()` / `.finalMessage()` helper to get the complete response if you don't need to handle individual stream events
|
||||
|
||||
---
|
||||
|
||||
## Subcommands
|
||||
|
||||
If the User Request at the bottom of this prompt is a bare subcommand string (no prose), search every **Subcommands** table in this document — including any in sections appended below — and follow the matching Action column directly. This lets users invoke specific flows via `/claude-api <subcommand>`. If no table in the document matches, treat the request as normal prose.
|
||||
|
||||
| Subcommand | Action |
|
||||
|---|---|
|
||||
| `migrate` | Migrate existing Claude API code to a newer model. **Read `shared/model-migration.md` immediately** and follow it in order: Step 0 (confirm scope — ask which files/directories before any edit), Step 1 (classify each file), then the per-target breaking-changes section. Do not summarize the guide — execute it. If the user did not name a target model, ask which model to migrate to in the same turn as the scope question. |
|
||||
|
||||
---
|
||||
|
||||
## Language Detection
|
||||
|
||||
Before reading code examples, determine which language the user is working in:
|
||||
|
||||
1. **Look at project files** to infer the language:
|
||||
|
||||
- `*.py`, `requirements.txt`, `pyproject.toml`, `setup.py`, `Pipfile` → **Python** — read from `python/`
|
||||
- `*.ts`, `*.tsx`, `package.json`, `tsconfig.json` → **TypeScript** — read from `typescript/`
|
||||
- `*.js`, `*.jsx` (no `.ts` files present) → **TypeScript** — JS uses the same SDK, read from `typescript/`
|
||||
- `*.java`, `pom.xml`, `build.gradle` → **Java** — read from `java/`
|
||||
- `*.kt`, `*.kts`, `build.gradle.kts` → **Java** — Kotlin uses the Java SDK, read from `java/`
|
||||
- `*.scala`, `build.sbt` → **Java** — Scala uses the Java SDK, read from `java/`
|
||||
- `*.go`, `go.mod` → **Go** — read from `go/`
|
||||
- `*.rb`, `Gemfile` → **Ruby** — read from `ruby/`
|
||||
- `*.cs`, `*.csproj` → **C#** — read from `csharp/`
|
||||
- `*.php`, `composer.json` → **PHP** — read from `php/`
|
||||
|
||||
2. **If multiple languages detected** (e.g., both Python and TypeScript files):
|
||||
|
||||
- Check which language the user's current file or question relates to
|
||||
- If still ambiguous, ask: "I detected both Python and TypeScript files. Which language are you using for the Claude API integration?"
|
||||
|
||||
3. **If language can't be inferred** (empty project, no source files, or unsupported language):
|
||||
|
||||
- Use AskUserQuestion with options: Python, TypeScript, Java, Go, Ruby, cURL/raw HTTP, C#, PHP
|
||||
- If AskUserQuestion is unavailable, default to Python examples and note: "Showing Python examples. Let me know if you need a different language."
|
||||
|
||||
4. **If unsupported language detected** (Rust, Swift, C++, Elixir, etc.):
|
||||
|
||||
- Suggest cURL/raw HTTP examples from `curl/` and note that community SDKs may exist
|
||||
- Offer to show Python or TypeScript examples as reference implementations
|
||||
|
||||
5. **If user needs cURL/raw HTTP examples**, read from `curl/`.
|
||||
|
||||
### Language-Specific Feature Support
|
||||
|
||||
| Language | Tool Runner | Managed Agents | Notes |
|
||||
| ---------- | ----------- | -------------- | ------------------------------------- |
|
||||
| Python | Yes (beta) | Yes (beta) | Full support — `@beta_tool` decorator |
|
||||
| TypeScript | Yes (beta) | Yes (beta) | Full support — `betaZodTool` + Zod |
|
||||
| Java | Yes (beta) | Yes (beta) | Beta tool use with annotated classes |
|
||||
| Go | Yes (beta) | Yes (beta) | `BetaToolRunner` in `toolrunner` pkg |
|
||||
| Ruby | Yes (beta) | Yes (beta) | `BaseTool` + `tool_runner` in beta |
|
||||
| C# | Yes (beta) | Yes (beta) | `BetaToolRunner` + raw JSON schema |
|
||||
| PHP | Yes (beta) | Yes (beta) | `BetaRunnableTool` + `toolRunner()` |
|
||||
| cURL | N/A | Yes (beta) | Raw HTTP, no SDK features |
|
||||
|
||||
> **Managed Agents code examples**: dedicated language-specific READMEs are provided for Python, TypeScript, Go, Ruby, PHP, Java, and cURL (`{lang}/managed-agents/README.md`, `curl/managed-agents.md`). Read your language's README plus the language-agnostic `shared/managed-agents-*.md` concept files. **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `agents.create` and pass it to every subsequent `sessions.create`; do not call `agents.create` in the request path. The Anthropic CLI (`ant`) is one convenient way to create agents and environments from version-controlled YAML — see `shared/anthropic-cli.md`. If a binding you need isn't shown in the README, WebFetch the relevant entry from `shared/live-sources.md` rather than guess. C# has beta Managed Agents support via `client.Beta.Agents` and related namespaces.
|
||||
|
||||
---
|
||||
|
||||
## Which Surface Should I Use?
|
||||
|
||||
> **Start simple.** Default to the simplest tier that meets your needs. Single API calls and workflows handle most use cases — only reach for agents when the task genuinely requires open-ended, model-driven exploration.
|
||||
|
||||
| Use Case | Tier | Recommended Surface | Why |
|
||||
| ----------------------------------------------- | --------------- | ------------------------- | ------------------------------------------------------------ |
|
||||
| Classification, summarization, extraction, Q&A | Single LLM call | **Claude API** | One request, one response |
|
||||
| Batch processing or embeddings | Single LLM call | **Claude API** | Specialized endpoints |
|
||||
| Multi-step pipelines with code-controlled logic | Workflow | **Claude API + tool use** | You orchestrate the loop |
|
||||
| Custom agent with your own tools | Agent | **Claude API + tool use** | Maximum flexibility |
|
||||
| Server-managed stateful agent with workspace | Agent | **Managed Agents** | Anthropic runs the loop and hosts the tool-execution sandbox |
|
||||
| Persisted, versioned agent configs | Agent | **Managed Agents** | Agents are stored objects; sessions pin to a version |
|
||||
| Long-running multi-turn agent with file mounts | Agent | **Managed Agents** | Per-session containers, SSE event stream, Skills + MCP |
|
||||
|
||||
> **Note:** Managed Agents is the right choice when you want Anthropic to run the agent loop *and* host the container where tools execute — file ops, bash, code execution all run in the per-session workspace. If you want to host the compute yourself or run your own custom tool runtime, Claude API + tool use is the right choice — use the tool runner for automatic loop handling, or the manual loop for fine-grained control (approval gates, custom logging, conditional execution).
|
||||
|
||||
> **Cloud-provider access.** **Claude Platform on AWS** is Anthropic-operated with same-day API parity — Managed Agents and every feature in this skill work there, **except self-hosted sandboxes** (see `shared/claude-platform-on-aws.md`). **Amazon Bedrock**, **Google Vertex AI**, and **Microsoft Foundry** do **not** support Managed Agents or Anthropic server-side tools; use **Claude API + tool use** on those.
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```
|
||||
What does your application need?
|
||||
|
||||
0. Which provider?
|
||||
├── First-party API or Claude Platform on AWS → continue (full surface available).
|
||||
└── Amazon Bedrock, Google Vertex AI, or Microsoft Foundry → Claude API (+ tool use for agents); Managed Agents not available there.
|
||||
|
||||
1. Single LLM call (classification, summarization, extraction, Q&A)
|
||||
└── Claude API — one request, one response
|
||||
|
||||
2. Do you want Anthropic to run the agent loop and host a per-session
|
||||
container where Claude executes tools (bash, file ops, code)?
|
||||
└── Yes → Managed Agents — server-managed sessions, persisted agent configs,
|
||||
SSE event stream, Skills + MCP, file mounts.
|
||||
Examples: "stateful coding agent with a workspace per task",
|
||||
"long-running research agent that streams events to a UI",
|
||||
"agent with persisted, versioned config used across many sessions"
|
||||
|
||||
3. Workflow (multi-step, code-orchestrated, with your own tools)
|
||||
└── Claude API with tool use — you control the loop
|
||||
|
||||
4. Open-ended agent (model decides its own trajectory, your own tools, you host the compute)
|
||||
└── Claude API agentic loop (maximum flexibility)
|
||||
```
|
||||
|
||||
### Should I Build an Agent?
|
||||
|
||||
Before choosing the agent tier, check all four criteria:
|
||||
|
||||
- **Complexity** — Is the task multi-step and hard to fully specify in advance? (e.g., "turn this design doc into a PR" vs. "extract the title from this PDF")
|
||||
- **Value** — Does the outcome justify higher cost and latency?
|
||||
- **Viability** — Is Claude capable at this task type?
|
||||
- **Cost of error** — Can errors be caught and recovered from? (tests, review, rollback)
|
||||
|
||||
If the answer is "no" to any of these, stay at a simpler tier (single call or workflow).
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
Everything goes through `POST /v1/messages`. Tools and output constraints are features of this single endpoint — not separate APIs.
|
||||
|
||||
**User-defined tools** — You define tools (via decorators, Zod schemas, or raw JSON), and the SDK's tool runner handles calling the API, executing your functions, and looping until Claude is done. For full control, you can write the loop manually.
|
||||
|
||||
**Server-side tools** — Anthropic-hosted tools that run on Anthropic's infrastructure. Code execution is fully server-side (declare it in `tools`, Claude runs code automatically). Computer use can be server-hosted or self-hosted.
|
||||
|
||||
**Structured outputs** — Constrains the Messages API response format (`output_config.format`) and/or tool parameter validation (`strict: true`). The recommended approach is `client.messages.parse()` which validates responses against your schema automatically. Note: the old `output_format` parameter is deprecated; use `output_config: {format: {...}}` on `messages.create()`.
|
||||
|
||||
**Supporting endpoints** — Batches (`POST /v1/messages/batches`), Files (`POST /v1/files`), Token Counting (`POST /v1/messages/count_tokens` — see `shared/token-counting.md`), and Models (`GET /v1/models`, `GET /v1/models/{id}` — live capability/context-window discovery) feed into or support Messages API requests.
|
||||
|
||||
---
|
||||
|
||||
## Current Models (cached: 2026-06-04)
|
||||
|
||||
| Model | Model ID | Context | Input $/1M | Output $/1M |
|
||||
| ----------------- | ------------------- | -------------- | ---------- | ----------- |
|
||||
| Claude Fable 5 | `claude-fable-5` | 1M | $10.00 | $50.00 |
|
||||
| Claude Mythos 5 (Project Glasswing only) | `claude-mythos-5` | 1M | $10.00 | $50.00 |
|
||||
| Claude Opus 4.8 | `claude-opus-4-8` | 1M | $5.00 | $25.00 |
|
||||
| Claude Opus 4.7 | `claude-opus-4-7` | 1M | $5.00 | $25.00 |
|
||||
| Claude Opus 4.6 | `claude-opus-4-6` | 1M | $5.00 | $25.00 |
|
||||
| Claude Sonnet 4.6 | `claude-sonnet-4-6` | 1M | $3.00 | $15.00 |
|
||||
| Claude Haiku 4.5 | `claude-haiku-4-5` | 200K | $1.00 | $5.00 |
|
||||
|
||||
**ALWAYS use `claude-opus-4-8` unless the user explicitly names a different model.** This is non-negotiable. Do not use `claude-sonnet-4-6`, `claude-sonnet-4-5`, or any other model unless the user literally says "use sonnet" or "use haiku". Never downgrade for cost — that's the user's decision, not yours. Use `claude-fable-5` only when the user explicitly asks for Claude Fable 5, "fable", or Anthropic's most capable model — it has different API behavior than the Opus family (see below) and pricing that exceeds Opus-tier.
|
||||
|
||||
### Claude Fable 5 (`claude-fable-5`) — most capable widely released model
|
||||
|
||||
Claude Fable 5 is Anthropic's most capable widely released model, for the most demanding reasoning and long-horizon agentic work. **Claude Mythos 5** (`claude-mythos-5`) offers the same capabilities, pricing, and API surface through Project Glasswing (participation is the only way to access it), succeeding the invitation-only Claude Mythos Preview (`claude-mythos-preview`) — everything below applies to both models. 1M context window (the maximum is also the default), 128K max output. Key API differences from Opus-tier — see `shared/model-migration.md` → Migrating to Claude Fable 5 for details:
|
||||
|
||||
- **Thinking is always on** — omit the `thinking` parameter entirely (or send `{type: "adaptive"}`). Any other explicit configuration is rejected: `{type: "disabled"}` and `{type: "enabled", budget_tokens: N}` both return a 400. Control depth with `output_config.effort` (supports `low` through `xhigh` and `max`).
|
||||
- **Protected thinking = the raw chain of thought, not the summary** — responses carry regular `thinking` blocks (not `redacted_thinking`): `display: "summarized"` returns a readable summary, `"omitted"` (the default) leaves the `thinking` field as an empty string; the raw chain of thought is never exposed on any model. Replay rules: pass thinking blocks back exactly as received on the same model (including empty-text blocks — the API rejects *modified* blocks, not read ones); a **different** model **silently ignores** them (not an error), but ignored blocks still bill input tokens — strip them when switching models for good.
|
||||
- **New tokenizer** — the same content tokenizes to roughly 30% more tokens than on Opus-tier models. Don't reuse token counts or `max_tokens` settings measured on other models; re-baseline with `count_tokens`.
|
||||
- **`refusal` stop reason** — safety classifiers may decline a request (HTTP 200, `stop_reason: "refusal"`, with a `stop_details` category). A pre-output refusal has an empty `content` array and is not billed at all; a mid-stream refusal bills the already-streamed output — discard the partial output. Always check `stop_reason` before reading `content`. To retry on another model: the beta `fallbacks` parameter (Claude API and Claude Platform on AWS) retries server-side in one round trip; the GA SDKs' `BetaRefusalFallbackMiddleware` + `BetaFallbackState` handle client-side retry everywhere else (incl. Bedrock/Vertex); fallback credit refunds the cache-switch cost of client-side retries. See the migration guide's refusal section.
|
||||
- **No assistant prefill** — same as the rest of the 4.6+ family.
|
||||
- **30-day data retention required** — Claude Fable 5 is not available under zero data retention; requests from an org whose retention configuration doesn't meet the requirement return `400 invalid_request_error`.
|
||||
- **Longer turns, different prompting** — single requests on hard tasks can run many minutes (plan timeouts/streaming/progress UX); effort sweeps should include low/medium for routine work; prompts written for prior models are often too prescriptive and reduce output quality. See `shared/model-migration.md` → Migrating to Claude Fable 5 → Behavioral shifts (prompt-tunable) for the recommended prompt snippets (anti-overplanning, no-tidying, grounded progress claims, boundaries, async sub-agents, memory, `send_to_user`).
|
||||
|
||||
**CRITICAL: Use only the exact model ID strings from the table above — they are complete as-is. Do not append date suffixes.** For example, use `claude-sonnet-4-6`, never `claude-sonnet-4-6-20251114` or any other date-suffixed variant you might recall from training data. If the user requests an older model not in the table (e.g., "opus 4.5", "sonnet 3.7"), read `shared/models.md` for the exact ID — do not construct one yourself.
|
||||
|
||||
A note: if any of the model strings above look unfamiliar to you, that's to be expected — that just means they were released after your training data cutoff. Rest assured they are real models; we wouldn't mess with you like that.
|
||||
|
||||
**Live capability lookup:** The table above is cached. When the user asks "what's the context window for X", "does X support vision/thinking/effort", or "which models support Y", query the Models API (`client.models.retrieve(id)` / `client.models.list()`) — see `shared/models.md` for the field reference and capability-filter examples.
|
||||
|
||||
---
|
||||
|
||||
## Thinking & Effort (Quick Reference)
|
||||
|
||||
**Fable 5 / Opus 4.8 / 4.7 — Adaptive thinking only:** Use `thinking: {type: "adaptive"}`. `thinking: {type: "enabled", budget_tokens: N}` returns a 400 — adaptive is the only on-mode. On Opus 4.8 and 4.7, `{type: "disabled"}` and omitting `thinking` both work; on Fable 5, an explicit `{type: "disabled"}` returns a 400 — omit the `thinking` param entirely instead. Sampling parameters (`temperature`, `top_p`, `top_k`) are also removed and will 400. Opus 4.8 keeps the same request surface as 4.7 (no new breaking changes) — see `shared/model-migration.md` → Migrating to Opus 4.8 for the behavioral re-tuning, and → Migrating to Opus 4.7 for the full breaking-change list when coming from 4.6 or earlier. Note: with `thinking` disabled, Opus 4.8 may write longer reasoning into the visible response — leave adaptive thinking on, or add a final-answer-only instruction (see the migration guide).
|
||||
**Opus 4.6 — Adaptive thinking (recommended):** Use `thinking: {type: "adaptive"}`. Claude dynamically decides when and how much to think. No `budget_tokens` needed — `budget_tokens` is deprecated on Opus 4.6 and Sonnet 4.6 and should not be used for new code. Adaptive thinking also automatically enables interleaved thinking (no beta header needed). **When the user asks for "extended thinking", a "thinking budget", or `budget_tokens`: always use Fable 5, Opus 4.8, 4.7, or 4.6 with `thinking: {type: "adaptive"}`. The concept of a fixed token budget for thinking is deprecated — adaptive thinking replaces it. Do NOT use `budget_tokens` for new 4.6/4.7/4.8 code and do NOT switch to an older model.** *Gradual-migration carve-out:* `budget_tokens` is still functional on Opus 4.6 and Sonnet 4.6 as a transitional escape hatch — if you're migrating existing code and need a hard token ceiling before you've tuned `effort`, see `shared/model-migration.md` → Transitional escape hatch. Note: this carve-out does **not** apply to Fable 5, Opus 4.7 or 4.8 — `budget_tokens` is fully removed there.
|
||||
**Effort parameter (GA, no beta header):** Controls thinking depth and overall token spend via `output_config: {effort: "low"|"medium"|"high"|"max"}` (inside `output_config`, not top-level). Default is `high` (equivalent to omitting it). `max` is supported on Fable 5, Opus 4.6 and later, and Sonnet 4.6 (not Haiku or earlier Sonnets). Opus 4.7 added `"xhigh"` (between `high` and `max`) — the best setting for most coding and agentic use cases on Fable 5 / Opus 4.7/4.8, and the default in Claude Code; use a minimum of `high` for most intelligence-sensitive work. Works on Fable 5, Opus 4.5, Opus 4.6, Opus 4.7, Opus 4.8, and Sonnet 4.6. Will error on Sonnet 4.5 / Haiku 4.5. On Fable 5, Opus 4.7 and 4.8, effort matters more than on any prior Opus — re-tune it when migrating, and run long-horizon/agentic tasks at `high`/`xhigh` with the full task spec given up front. Combine with adaptive thinking for the best cost-quality tradeoffs. Lower effort means fewer and more-consolidated tool calls, less preamble, and terser confirmations — `high` is often the sweet spot balancing quality and token efficiency; use `max` when correctness matters more than cost; use `low` for subagents or simple tasks.
|
||||
|
||||
**Thinking display — `"omitted"` by default on Fable 5 / Mythos 5 / Opus 4.8 / 4.7:** `display: "summarized"` returns a readable summary of the reasoning; `"omitted"` (the default on all four — a silent change from Opus 4.6, where it was `"summarized"`) streams `thinking` blocks with empty text. `display` controls visibility only — thinking happens and is billed the same under every setting; the raw chain of thought is never exposed on any model. If you stream reasoning to users, the default looks like a long pause before output — set `thinking: {type: "adaptive", display: "summarized"}` explicitly. (Independent of display, echo thinking blocks back unchanged when continuing on the same model; other models silently ignore them — see the migration guide.)
|
||||
|
||||
**Task Budgets (beta, Fable 5 / Opus 4.7 / 4.8):** `output_config: {task_budget: {type: "tokens", total: N}}` tells the model how many tokens it has for a full agentic loop — it sees a running countdown and self-moderates (minimum 20,000; beta header `task-budgets-2026-03-13`). Distinct from `max_tokens`, which is an enforced per-response ceiling the model is not aware of. See `shared/model-migration.md` → Task Budgets.
|
||||
|
||||
**Sonnet 4.6:** Supports adaptive thinking (`thinking: {type: "adaptive"}`). `budget_tokens` is deprecated on Sonnet 4.6 — use adaptive thinking instead.
|
||||
|
||||
**Older models (only if explicitly requested):** If the user specifically asks for Sonnet 4.5 or another older model, use `thinking: {type: "enabled", budget_tokens: N}`. `budget_tokens` must be less than `max_tokens` (minimum 1024). Never choose an older model just because the user mentions `budget_tokens` — use Opus 4.8 with adaptive thinking instead.
|
||||
|
||||
---
|
||||
|
||||
## Compaction (Quick Reference)
|
||||
|
||||
**Beta, Fable 5, Opus 4.8, Opus 4.7, Opus 4.6, and Sonnet 4.6.** For long-running conversations that may exceed the 1M context window, enable server-side compaction. The API automatically summarizes earlier context when it approaches the trigger threshold (default: 150K tokens). Requires beta header `compact-2026-01-12`.
|
||||
|
||||
**Critical:** Append `response.content` (not just the text) back to your messages on every turn. Compaction blocks in the response must be preserved — the API uses them to replace the compacted history on the next request. Extracting only the text string and appending that will silently lose the compaction state.
|
||||
|
||||
See `{lang}/claude-api/README.md` (Compaction section) for code examples. Full docs via WebFetch in `shared/live-sources.md`.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching (Quick Reference)
|
||||
|
||||
**Prefix match.** Any byte change anywhere in the prefix invalidates everything after it. Render order is `tools` → `system` → `messages`. Keep stable content first (frozen system prompt, deterministic tool list), put volatile content (timestamps, per-request IDs, varying questions) after the last `cache_control` breakpoint.
|
||||
|
||||
**Mid-conversation operator instructions** (beta header `mid-conversation-system-2026-04-07`, on supporting models): append `{"role": "system", ...}` to `messages[]` instead of editing top-level `system`. Preserves the cached history prefix and is the prompt-injection-safe operator channel. See `shared/prompt-caching.md` § Mid-conversation system messages.
|
||||
|
||||
**Top-level auto-caching** (`cache_control: {type: "ephemeral"}` on `messages.create()`) is the simplest option when you don't need fine-grained placement. Max 4 breakpoints per request. Minimum cacheable prefix is ~1024 tokens — shorter prefixes silently won't cache.
|
||||
|
||||
**Verify with `usage.cache_read_input_tokens`** — if it's zero across repeated requests, a silent invalidator is at work (`datetime.now()` in system prompt, unsorted JSON, varying tool set).
|
||||
|
||||
For placement patterns, architectural guidance, and the silent-invalidator audit checklist: read `shared/prompt-caching.md`. Language-specific syntax: `{lang}/claude-api/README.md` (Prompt Caching section).
|
||||
|
||||
---
|
||||
|
||||
## Managed Agents (Beta)
|
||||
|
||||
**Managed Agents** is a third surface: server-managed stateful agents with Anthropic-hosted tool execution. You create a persisted, versioned Agent config (`POST /v1/agents`), then start Sessions that reference it. Each session provisions a container as the agent's workspace — bash, file ops, and code execution run there; the agent loop itself runs on Anthropic's orchestration layer and acts on the container via tools. The session streams events; you send messages and tool results back.
|
||||
|
||||
**Managed Agents is available on the first-party API and Claude Platform on AWS.** It is **not** available on Amazon Bedrock, Google Vertex AI, or Microsoft Foundry — for agents there, use Claude API + tool use.
|
||||
|
||||
**Mandatory flow:** Agent (once) → Session (every run). `model`/`system`/`tools` live on the agent, never the session. See `shared/managed-agents-overview.md` for the full reading guide, beta headers, and pitfalls.
|
||||
|
||||
**Beta headers:** `managed-agents-2026-04-01` — the SDK sets this automatically for all `client.beta.{agents,environments,sessions,vaults,memory_stores,deployments,deployment_runs}.*` calls. Skills API uses `skills-2025-10-02` and Files API uses `files-api-2025-04-14`, but you don't need to explicitly pass those in for endpoints other than `/v1/skills` and `/v1/files`.
|
||||
|
||||
**Subcommands** — invoke directly with `/claude-api <subcommand>`:
|
||||
|
||||
| Subcommand | Action |
|
||||
|---|---|
|
||||
| `managed-agents-onboard` | Walk the user through setting up a Managed Agent from scratch. **Read `shared/managed-agents-onboarding.md` immediately** and follow its interview script: mental model → know-or-explore branch → template config → session setup → **pre-flight viability check** → emit code. The viability check (reconcile the stated job against configured tools/credentials/data) catches under-resourced setups — missing a tool, credential, or data access — before the agent burns budget. Do not summarize — run the interview. |
|
||||
|
||||
**Reading guide:** Start with `shared/managed-agents-overview.md`, then the topical `shared/managed-agents-*.md` files (core, environments, tools, events, outcomes, multiagent, webhooks, memory, scheduled-deployments, client-patterns, onboarding, api-reference). For Python, TypeScript, Go, Ruby, PHP, and Java, read `{lang}/managed-agents/README.md` for code examples. For cURL, read `curl/managed-agents.md`. **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `agents.create` and pass it to every subsequent `sessions.create`; do not call `agents.create` in the request path. The Anthropic CLI (`ant`) is one convenient way to create agents and environments from version-controlled YAML — see `shared/anthropic-cli.md`. If a binding you need isn't shown in the language README, WebFetch the relevant entry from `shared/live-sources.md` rather than guess. C# has beta Managed Agents support via `client.Beta.Agents` and related namespaces.
|
||||
|
||||
**When the user wants to set up a Managed Agent from scratch** (e.g. "how do I get started", "walk me through creating one", "set up a new agent"): read `shared/managed-agents-onboarding.md` and run its interview — same flow as the `managed-agents-onboard` subcommand.
|
||||
|
||||
**When the user asks "how do I write the client code for X":** reach for `shared/managed-agents-client-patterns.md` — covers lossless stream reconnect, `processed_at` queued/processed gate, interrupt, `tool_confirmation` round-trip, the correct idle/terminated break gate, post-idle status race, stream-first ordering, file-mount gotchas, keeping credentials host-side via custom tools, etc.
|
||||
|
||||
**When the user wants the agent to run on a schedule** (cron, "every night", "weekly report"): read `shared/managed-agents-scheduled-deployments.md` — deployments fire sessions autonomously on a cron cadence, with run records, retries, and auto-pause.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Reading Guide
|
||||
|
||||
After detecting the language, read the relevant files based on what the user needs:
|
||||
|
||||
### Quick Task Reference
|
||||
|
||||
**Single text classification/summarization/extraction/Q&A:**
|
||||
→ Read only `{lang}/claude-api/README.md`
|
||||
|
||||
**Chat UI or real-time response display:**
|
||||
→ Read `{lang}/claude-api/README.md` + `{lang}/claude-api/streaming.md`
|
||||
|
||||
**Long-running conversations (may exceed context window):**
|
||||
→ Read `{lang}/claude-api/README.md` — see Compaction section
|
||||
**Migrating to a newer model (Fable 5 / Opus 4.8 / Opus 4.7 / Opus 4.6 / Sonnet 4.6) or replacing a retired model:**
|
||||
→ Read `shared/model-migration.md`
|
||||
**Prompting or tuning Fable 5 (long turns, effort, verbosity, autonomous runs, sub-agents):**
|
||||
→ Read `shared/model-migration.md` → Migrating to Fable 5 → Behavioral shifts (prompt-tunable) + Long-running agent recommendations
|
||||
**Prompt caching / optimize caching / "why is my cache hit rate low":**
|
||||
→ Read `shared/prompt-caching.md` + `{lang}/claude-api/README.md` (Prompt Caching section)
|
||||
**Count tokens in a file / prompt / diff ("how many tokens is X"):**
|
||||
→ Read `shared/token-counting.md` — use `messages.count_tokens`, never `tiktoken`
|
||||
|
||||
**Function calling / tool use / agents:**
|
||||
→ Read `{lang}/claude-api/README.md` + `shared/tool-use-concepts.md` + `{lang}/claude-api/tool-use.md`
|
||||
|
||||
**Agent design (tool surface, context management, caching strategy):**
|
||||
→ Read `shared/agent-design.md`
|
||||
|
||||
**Batch processing (non-latency-sensitive):**
|
||||
→ Read `{lang}/claude-api/README.md` + `{lang}/claude-api/batches.md`
|
||||
|
||||
**File uploads across multiple requests:**
|
||||
→ Read `{lang}/claude-api/README.md` + `{lang}/claude-api/files-api.md`
|
||||
|
||||
**Managed Agents (server-managed stateful agents with workspace):**
|
||||
→ Read `shared/managed-agents-overview.md` + the rest of the `shared/managed-agents-*.md` files. For Python, TypeScript, Go, Ruby, PHP, and Java, read `{lang}/managed-agents/README.md` for code examples. For cURL, read `curl/managed-agents.md`. **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `agents.create` and pass it to every subsequent `sessions.create`; do not call `agents.create` in the request path. The Anthropic CLI (`ant`) is one convenient way to create agents and environments from version-controlled YAML — see `shared/anthropic-cli.md`. If a binding you need isn't shown in the language README, WebFetch the relevant entry from `shared/live-sources.md` rather than guess. C# has beta Managed Agents support — see `csharp/claude-api.md` for details, or `curl/managed-agents.md` for raw HTTP reference.
|
||||
|
||||
### Claude API (Full File Reference)
|
||||
|
||||
Read the **language-specific Claude API folder** (`{language}/claude-api/`):
|
||||
|
||||
1. **`{language}/claude-api/README.md`** — **Read this first.** Installation, quick start, common patterns, error handling.
|
||||
2. **`shared/tool-use-concepts.md`** — Read when the user needs function calling, code execution, memory, or structured outputs. Covers conceptual foundations.
|
||||
3. **`shared/agent-design.md`** — Read when designing an agent: bash vs. dedicated tools, programmatic tool calling, tool search/skills, context editing vs. compaction vs. memory, caching principles.
|
||||
4. **`{language}/claude-api/tool-use.md`** — Read for language-specific tool use code examples (tool runner, manual loop, code execution, memory, structured outputs).
|
||||
5. **`{language}/claude-api/streaming.md`** — Read when building chat UIs or interfaces that display responses incrementally.
|
||||
6. **`{language}/claude-api/batches.md`** — Read when processing many requests offline (not latency-sensitive). Runs asynchronously at 50% cost.
|
||||
7. **`{language}/claude-api/files-api.md`** — Read when sending the same file across multiple requests without re-uploading.
|
||||
8. **`shared/prompt-caching.md`** — Read when adding or optimizing prompt caching. Covers prefix-stability design, breakpoint placement, and anti-patterns that silently invalidate cache.
|
||||
9. **`shared/error-codes.md`** — Read when debugging HTTP errors or implementing error handling.
|
||||
10. **`shared/model-migration.md`** — Read when upgrading to newer models, replacing retired models, or translating `budget_tokens` / prefill patterns to the current API.
|
||||
11. **`shared/live-sources.md`** — WebFetch URLs for fetching the latest official documentation.
|
||||
|
||||
> **Note:** For Java, Go, Ruby, C#, PHP, and cURL — these have a single file each covering all basics. Read that file plus `shared/tool-use-concepts.md` and `shared/error-codes.md` as needed.
|
||||
|
||||
> **Note:** For the Managed Agents file reference, see the `## Managed Agents (Beta)` section above — it lists every `shared/managed-agents-*.md` file and the language-specific READMEs.
|
||||
|
||||
---
|
||||
|
||||
## When to Use WebFetch
|
||||
|
||||
Use WebFetch to get the latest documentation when:
|
||||
|
||||
- User asks for "latest" or "current" information
|
||||
- Cached data seems incorrect
|
||||
- User asks about features not covered here
|
||||
|
||||
Live documentation URLs are in `shared/live-sources.md`.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- Don't truncate inputs when passing files or content to the API. If the content is too long to fit in the context window, notify the user and discuss options (chunking, summarization, etc.) rather than silently truncating.
|
||||
- **Fable 5 / Opus 4.8 / 4.7 thinking:** Adaptive only. `thinking: {type: "enabled", budget_tokens: N}` returns 400 — `budget_tokens` is fully removed (along with `temperature`, `top_p`, `top_k`). Use `thinking: {type: "adaptive"}`. Opus 4.8 inherits this surface from 4.7 with no new breaking changes; Fable 5 adds one — an explicit `thinking: {type: "disabled"}` returns a 400 (accepted on 4.7/4.8); omit the param instead.
|
||||
- **Opus 4.6 / Sonnet 4.6 thinking:** Use `thinking: {type: "adaptive"}` — do NOT use `budget_tokens` for new 4.6 code (deprecated on both Opus 4.6 and Sonnet 4.6; for gradual migration of existing code, see the transitional escape hatch in `shared/model-migration.md` — note this carve-out does not apply to Fable 5, Opus 4.7 or 4.8). For older models, `budget_tokens` must be less than `max_tokens` (minimum 1024). This will throw an error if you get it wrong.
|
||||
- **Prefill removed (Fable 5 and the 4.6/4.7/4.8 family):** Assistant message prefills (last-assistant-turn prefills) return a 400 error on Fable 5, Opus 4.6, Opus 4.7, Opus 4.8, and Sonnet 4.6. Use structured outputs (`output_config.format`) or system prompt instructions to control response format instead. (One exception: the fallback-credit prefill claim — when redeeming a credit with `fallback_has_prefill_claim: true`, the server accepts the echoed assistant message; see the migration guide's refusal section.)
|
||||
- **Fable 5 `refusal` stop reason:** Safety classifiers may decline a request — a successful HTTP 200 with `stop_reason: "refusal"` (pre-output: empty `content`, nothing billed; mid-stream: partial output billed — discard it). Check `stop_reason` before reading `response.content[0]`, or you'll hit index errors on refused requests. To retry on another model, replaying history as-is works — other models silently ignore the refused model's thinking blocks — but ignored blocks still bill input tokens, so strip them when switching for good (exception: a fallback-credit redemption must echo the refused body exactly, thinking blocks included).
|
||||
- **Fable 5 tokenizer:** ~30% more tokens for the same content vs Opus-tier models. Token counts, context-window budgets, and `max_tokens` values measured on other models don't transfer — re-measure with `count_tokens` passing `model: "claude-fable-5"` (the response includes counts under both tokenizers).
|
||||
- **Confirm migration scope before editing:** When a user asks to migrate code to a newer Claude model without naming a specific file, directory, or file list, **ask which scope to apply first** — the entire working directory, a specific subdirectory, or a specific set of files. Do not start editing until the user confirms. Imperative phrasings like "migrate my codebase", "move my project to X", "upgrade to Sonnet 4.6", or bare "migrate to Opus 4.8" are **still ambiguous** — they tell you what to do but not where, so ask. Proceed without asking only when the prompt names an exact file, a specific directory, or an explicit file list ("migrate `app.py`", "migrate everything under `services/`", "update `a.py` and `b.py`"). See `shared/model-migration.md` Step 0.
|
||||
- **`max_tokens` defaults:** Don't lowball `max_tokens` — hitting the cap truncates output mid-thought and requires a retry. For non-streaming requests, default to `~16000` (keeps responses under SDK HTTP timeouts). For streaming requests, default to `~64000` (timeouts aren't a concern, so give the model room). Only go lower when you have a hard reason: classification (`~256`), cost caps, deliberately short outputs, or **`max_tokens: 0`** for cache pre-warming (see `shared/prompt-caching.md` → Pre-warming).
|
||||
- **128K output tokens:** Fable 5, Opus 4.6, Opus 4.7, and Opus 4.8 support up to 128K `max_tokens`, but the SDKs require streaming for values that large to avoid HTTP timeouts. Use `.stream()` with `.get_final_message()` / `.finalMessage()`.
|
||||
- **Tool call JSON parsing (Fable 5 and the 4.6/4.7/4.8 family):** Fable 5, Opus 4.6, Opus 4.7, Opus 4.8, and Sonnet 4.6 may produce different JSON string escaping in tool call `input` fields (e.g., Unicode or forward-slash escaping). Always parse tool inputs with `json.loads()` / `JSON.parse()` — never do raw string matching on the serialized input.
|
||||
- **Structured outputs (all models):** Use `output_config: {format: {...}}` instead of the deprecated `output_format` parameter on `messages.create()`. This is a general API change, not 4.6-specific.
|
||||
- **Don't reimplement SDK functionality:** The SDK provides high-level helpers — use them instead of building from scratch. Specifically: use `stream.finalMessage()` instead of wrapping `.on()` events in `new Promise()`; use typed exception classes (`Anthropic.RateLimitError`, etc.) instead of string-matching error messages; use SDK types (`Anthropic.MessageParam`, `Anthropic.Tool`, `Anthropic.Message`, etc.) instead of redefining equivalent interfaces.
|
||||
- **Don't define custom types for SDK data structures:** The SDK exports types for all API objects. Use `Anthropic.MessageParam` for messages, `Anthropic.Tool` for tool definitions, `Anthropic.ToolUseBlock` / `Anthropic.ToolResultBlockParam` for tool results, `Anthropic.Message` for responses. Defining your own `interface ChatMessage { role: string; content: unknown }` duplicates what the SDK already provides and loses type safety.
|
||||
- **Report and document output:** For tasks that produce reports, documents, or visualizations, the code execution sandbox has `python-docx`, `python-pptx`, `matplotlib`, `pillow`, and `pypdf` pre-installed. Claude can generate formatted files (DOCX, PDF, charts) and return them via the Files API — consider this for "report" or "document" type requests instead of plain stdout text.
|
||||
@@ -1,447 +0,0 @@
|
||||
# Claude API — C#
|
||||
|
||||
> **Note:** The C# SDK is the official Anthropic SDK for C#. Tool use is supported via the Messages API with a beta `BetaToolRunner` for automatic tool execution loops. The SDK also supports Microsoft.Extensions.AI IChatClient integration with function invocation and Managed Agents (beta).
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
dotnet add package Anthropic
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```csharp
|
||||
using Anthropic;
|
||||
|
||||
// Default (uses ANTHROPIC_API_KEY env var)
|
||||
AnthropicClient client = new();
|
||||
|
||||
// Explicit API key (use environment variables — never hardcode keys)
|
||||
AnthropicClient client = new() {
|
||||
ApiKey = Environment.GetEnvironmentVariable("ANTHROPIC_API_KEY")
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Messages;
|
||||
|
||||
var parameters = new MessageCreateParams
|
||||
{
|
||||
Model = Model.ClaudeOpus4_6,
|
||||
MaxTokens = 16000,
|
||||
Messages = [new() { Role = Role.User, Content = "What is the capital of France?" }]
|
||||
};
|
||||
var response = await client.Messages.Create(parameters);
|
||||
|
||||
// ContentBlock is a union wrapper. .Value unwraps to the variant object,
|
||||
// then OfType<T> filters to the type you want. Or use the TryPick* idiom
|
||||
// shown in the Thinking section below.
|
||||
foreach (var text in response.Content.Select(b => b.Value).OfType<TextBlock>())
|
||||
{
|
||||
Console.WriteLine(text.Text);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Messages;
|
||||
|
||||
var parameters = new MessageCreateParams
|
||||
{
|
||||
Model = Model.ClaudeOpus4_6,
|
||||
MaxTokens = 64000,
|
||||
Messages = [new() { Role = Role.User, Content = "Write a haiku" }]
|
||||
};
|
||||
|
||||
await foreach (RawMessageStreamEvent streamEvent in client.Messages.CreateStreaming(parameters))
|
||||
{
|
||||
if (streamEvent.TryPickContentBlockDelta(out var delta) &&
|
||||
delta.Delta.TryPickText(out var text))
|
||||
{
|
||||
Console.Write(text.Text);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`RawMessageStreamEvent` TryPick methods** (naming drops the `Message`/`Raw` prefix): `TryPickStart`, `TryPickDelta`, `TryPickStop`, `TryPickContentBlockStart`, `TryPickContentBlockDelta`, `TryPickContentBlockStop`. There is no `TryPickMessageStop` — use `TryPickStop`.
|
||||
|
||||
---
|
||||
|
||||
## Thinking
|
||||
|
||||
**Adaptive thinking is the recommended mode for Claude 4.6+ models.** Claude decides dynamically when and how much to think.
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Messages;
|
||||
|
||||
var response = await client.Messages.Create(new MessageCreateParams
|
||||
{
|
||||
Model = Model.ClaudeOpus4_6,
|
||||
MaxTokens = 16000,
|
||||
// ThinkingConfigParam? implicitly converts from the concrete variant classes —
|
||||
// no wrapper needed.
|
||||
Thinking = new ThinkingConfigAdaptive(),
|
||||
Messages =
|
||||
[
|
||||
new() { Role = Role.User, Content = "Solve: 27 * 453" },
|
||||
],
|
||||
});
|
||||
|
||||
// ThinkingBlock(s) precede TextBlock in Content. TryPick* narrows the union.
|
||||
foreach (var block in response.Content)
|
||||
{
|
||||
if (block.TryPickThinking(out ThinkingBlock? t))
|
||||
{
|
||||
Console.WriteLine($"[thinking] {t.Thinking}");
|
||||
}
|
||||
else if (block.TryPickText(out TextBlock? text))
|
||||
{
|
||||
Console.WriteLine(text.Text);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Deprecated:** `new ThinkingConfigEnabled { BudgetTokens = N }` (fixed-budget extended thinking) still works on Claude 4.6 but is deprecated. Use adaptive thinking above.
|
||||
|
||||
Alternative to `TryPick*`: `.Select(b => b.Value).OfType<ThinkingBlock>()` (same LINQ pattern as the Basic Message example).
|
||||
|
||||
---
|
||||
|
||||
## Tool Use
|
||||
|
||||
### Defining a tool
|
||||
|
||||
`Tool` (NOT `ToolParam`) with an `InputSchema` record. `InputSchema.Type` is auto-set to `"object"` by the constructor — don't set it. `ToolUnion` has an implicit conversion from `Tool`, triggered by the collection expression `[...]`.
|
||||
|
||||
```csharp
|
||||
using System.Text.Json;
|
||||
using Anthropic.Models.Messages;
|
||||
|
||||
var parameters = new MessageCreateParams
|
||||
{
|
||||
Model = Model.ClaudeSonnet4_6,
|
||||
MaxTokens = 16000,
|
||||
Tools = [
|
||||
new Tool {
|
||||
Name = "get_weather",
|
||||
Description = "Get the current weather in a given location",
|
||||
InputSchema = new() {
|
||||
Properties = new Dictionary<string, JsonElement> {
|
||||
["location"] = JsonSerializer.SerializeToElement(
|
||||
new { type = "string", description = "City name" }),
|
||||
},
|
||||
Required = ["location"],
|
||||
},
|
||||
},
|
||||
],
|
||||
Messages = [new() { Role = Role.User, Content = "Weather in Paris?" }],
|
||||
};
|
||||
```
|
||||
|
||||
Derived from `anthropic-sdk-csharp/src/Anthropic/Models/Messages/Tool.cs` and `ToolUnion.cs:799` (implicit conversion).
|
||||
|
||||
See [shared tool use concepts](../shared/tool-use-concepts.md) for the loop pattern.
|
||||
### Converting response content to the follow-up assistant message
|
||||
|
||||
When echoing Claude's response back in the assistant turn, **there is no `.ToParam()` helper** — manually reconstruct each `ContentBlock` variant as its `*Param` counterpart. Do NOT use `new ContentBlockParam(block.Json)`: it compiles and serializes, but `.Value` stays `null` so `TryPick*`/`Validate()` fail (degraded JSON pass-through, not the typed path).
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Messages;
|
||||
|
||||
Message response = await client.Messages.Create(parameters);
|
||||
|
||||
// No .ToParam() — reconstruct per variant. Implicit conversions from each
|
||||
// *Param type to ContentBlockParam mean no explicit wrapper.
|
||||
List<ContentBlockParam> assistantContent = [];
|
||||
List<ContentBlockParam> toolResults = [];
|
||||
foreach (ContentBlock block in response.Content)
|
||||
{
|
||||
if (block.TryPickText(out TextBlock? text))
|
||||
{
|
||||
assistantContent.Add(new TextBlockParam { Text = text.Text });
|
||||
}
|
||||
else if (block.TryPickThinking(out ThinkingBlock? thinking))
|
||||
{
|
||||
// Signature MUST be preserved — the API rejects tampering
|
||||
assistantContent.Add(new ThinkingBlockParam
|
||||
{
|
||||
Thinking = thinking.Thinking,
|
||||
Signature = thinking.Signature,
|
||||
});
|
||||
}
|
||||
else if (block.TryPickRedactedThinking(out RedactedThinkingBlock? redacted))
|
||||
{
|
||||
assistantContent.Add(new RedactedThinkingBlockParam { Data = redacted.Data });
|
||||
}
|
||||
else if (block.TryPickToolUse(out ToolUseBlock? toolUse))
|
||||
{
|
||||
// ToolUseBlock has required Caller; ToolUseBlockParam.Caller is optional — don't copy it
|
||||
assistantContent.Add(new ToolUseBlockParam
|
||||
{
|
||||
ID = toolUse.ID,
|
||||
Name = toolUse.Name,
|
||||
Input = toolUse.Input,
|
||||
});
|
||||
// Execute the tool; collect ONE result per tool_use block — the API
|
||||
// rejects the follow-up if any tool_use ID lacks a matching tool_result.
|
||||
string result = ExecuteYourTool(toolUse.Name, toolUse.Input);
|
||||
toolResults.Add(new ToolResultBlockParam
|
||||
{
|
||||
ToolUseID = toolUse.ID,
|
||||
Content = result,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Follow-up: prior messages + assistant echo + user tool_result(s)
|
||||
List<MessageParam> followUpMessages =
|
||||
[
|
||||
.. parameters.Messages,
|
||||
new() { Role = Role.Assistant, Content = assistantContent },
|
||||
new() { Role = Role.User, Content = toolResults },
|
||||
];
|
||||
```
|
||||
|
||||
`ToolResultBlockParam` has no tuple constructor — use the object initializer. `Content` is a string-or-list union; a plain `string` implicitly converts.
|
||||
|
||||
---
|
||||
|
||||
## Context Editing / Compaction (Beta)
|
||||
|
||||
**Beta-namespace prefix is inconsistent** (source-verified against `src/Anthropic/Models/Beta/Messages/*.cs` @ 12.9.0). No prefix: `MessageCreateParams`, `MessageCountTokensParams`, `Role`. **Everything else has the `Beta` prefix**: `BetaMessageParam`, `BetaMessage`, `BetaContentBlock`, `BetaToolUseBlock`, all block param types. The unprefixed `Role` WILL collide with `Anthropic.Models.Messages.Role` if you import both namespaces (CS0104). Safest: import only Beta; if mixing, alias the beta `Role`:
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Beta.Messages;
|
||||
using NonBeta = Anthropic.Models.Messages; // only if you also need non-beta types
|
||||
// Now: MessageCreateParams, BetaMessageParam, Role (beta's), NonBeta.Role (if needed)
|
||||
```
|
||||
|
||||
|
||||
`BetaMessage.Content` is `IReadOnlyList<BetaContentBlock>` — a 15-variant discriminated union. Narrow with `TryPick*`. **Response `BetaContentBlock` is NOT assignable to param `BetaContentBlockParam`** — there's no `.ToParam()` in C#. Round-trip by converting each block:
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Beta.Messages;
|
||||
|
||||
var betaParams = new MessageCreateParams // no Beta prefix — one of only 2 unprefixed
|
||||
{
|
||||
Model = Model.ClaudeOpus4_6,
|
||||
MaxTokens = 16000,
|
||||
Betas = ["compact-2026-01-12"],
|
||||
ContextManagement = new BetaContextManagementConfig
|
||||
{
|
||||
Edits = [new BetaCompact20260112Edit()],
|
||||
},
|
||||
Messages = messages,
|
||||
};
|
||||
BetaMessage resp = await client.Beta.Messages.Create(betaParams);
|
||||
|
||||
foreach (BetaContentBlock block in resp.Content)
|
||||
{
|
||||
if (block.TryPickCompaction(out BetaCompactionBlock? compaction))
|
||||
{
|
||||
// Content is nullable — compaction can fail server-side
|
||||
Console.WriteLine($"compaction summary: {compaction.Content}");
|
||||
}
|
||||
}
|
||||
|
||||
// Context-edit metadata lives on a separate nullable field
|
||||
if (resp.ContextManagement is { } ctx)
|
||||
{
|
||||
foreach (var edit in ctx.AppliedEdits)
|
||||
Console.WriteLine($"cleared {edit.ClearedInputTokens} tokens");
|
||||
}
|
||||
|
||||
// ROUND-TRIP: BetaMessageParam.Content is BetaMessageParamContent (a string|list
|
||||
// union). It implicit-converts from List<BetaContentBlockParam>, NOT from the
|
||||
// response's IReadOnlyList<BetaContentBlock>. Convert each block:
|
||||
List<BetaContentBlockParam> paramBlocks = [];
|
||||
foreach (var b in resp.Content)
|
||||
{
|
||||
if (b.TryPickText(out var t)) paramBlocks.Add(new BetaTextBlockParam { Text = t.Text });
|
||||
else if (b.TryPickCompaction(out var c)) paramBlocks.Add(new BetaCompactionBlockParam { Content = c.Content });
|
||||
// ... other variants as needed
|
||||
}
|
||||
messages.Add(new BetaMessageParam { Role = Role.Assistant, Content = paramBlocks });
|
||||
```
|
||||
|
||||
All 15 `BetaContentBlock.TryPick*` variants: `Text`, `Thinking`, `RedactedThinking`, `ToolUse`, `ServerToolUse`, `WebSearchToolResult`, `WebFetchToolResult`, `CodeExecutionToolResult`, `BashCodeExecutionToolResult`, `TextEditorCodeExecutionToolResult`, `ToolSearchToolResult`, `McpToolUse`, `McpToolResult`, `ContainerUpload`, `Compaction`.
|
||||
|
||||
**`BetaToolUseBlock.Input` is `IReadOnlyDictionary<string, JsonElement>`** — index by key then call the `JsonElement` extractor:
|
||||
|
||||
```csharp
|
||||
if (block.TryPickToolUse(out BetaToolUseBlock? tu))
|
||||
{
|
||||
int a = tu.Input["a"].GetInt32();
|
||||
string s = tu.Input["name"].GetString()!;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Effort Parameter
|
||||
|
||||
Effort is nested under `OutputConfig`, NOT a top-level property. `ApiEnum<string, Effort>` has an implicit conversion from the enum, so assign `Effort.High` directly.
|
||||
|
||||
```csharp
|
||||
OutputConfig = new OutputConfig { Effort = Effort.High },
|
||||
```
|
||||
|
||||
Values: `Effort.Low`, `Effort.Medium`, `Effort.High`, `Effort.Max`. Combine with `Thinking = new ThinkingConfigAdaptive()` for cost-quality control.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
`System` takes `MessageCreateParamsSystem?` — a union of `string` or `List<TextBlockParam>`. There is no `SystemTextBlockParam`; use plain `TextBlockParam`. The implicit conversion needs the concrete `List<TextBlockParam>` type (array literals won't convert). For placement patterns and the silent-invalidator audit checklist, see `shared/prompt-caching.md`.
|
||||
|
||||
```csharp
|
||||
System = new List<TextBlockParam> {
|
||||
new() {
|
||||
Text = longSystemPrompt,
|
||||
CacheControl = new CacheControlEphemeral(), // auto-sets Type = "ephemeral"
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
Optional `Ttl` on `CacheControlEphemeral`: `new() { Ttl = Ttl.Ttl1h }` or `Ttl.Ttl5m`. `CacheControl` also exists on `Tool.CacheControl` and top-level `MessageCreateParams.CacheControl`.
|
||||
|
||||
Verify hits via `response.Usage.CacheCreationInputTokens` / `response.Usage.CacheReadInputTokens`.
|
||||
|
||||
---
|
||||
|
||||
## Token Counting
|
||||
|
||||
```csharp
|
||||
MessageTokensCount result = await client.Messages.CountTokens(new MessageCountTokensParams {
|
||||
Model = Model.ClaudeOpus4_6,
|
||||
Messages = [new() { Role = Role.User, Content = "Hello" }],
|
||||
});
|
||||
long tokens = result.InputTokens;
|
||||
```
|
||||
|
||||
`MessageCountTokensParams.Tools` uses a different union type (`MessageCountTokensTool`) than `MessageCreateParams.Tools` (`ToolUnion`) — if you're passing tools, the compiler will tell you when it matters.
|
||||
|
||||
---
|
||||
|
||||
## Structured Output
|
||||
|
||||
```csharp
|
||||
OutputConfig = new OutputConfig {
|
||||
Format = new JsonOutputFormat {
|
||||
Schema = new Dictionary<string, JsonElement> {
|
||||
["type"] = JsonSerializer.SerializeToElement("object"),
|
||||
["properties"] = JsonSerializer.SerializeToElement(
|
||||
new { name = new { type = "string" } }),
|
||||
["required"] = JsonSerializer.SerializeToElement(new[] { "name" }),
|
||||
},
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
`JsonOutputFormat.Type` is auto-set to `"json_schema"` by the constructor. `Schema` is `required`.
|
||||
|
||||
---
|
||||
|
||||
## PDF / Document Input
|
||||
|
||||
`DocumentBlockParam` takes a `DocumentBlockParamSource` union: `Base64PdfSource` / `UrlPdfSource` / `PlainTextSource` / `ContentBlockSource`. `Base64PdfSource` auto-sets `MediaType = "application/pdf"` and `Type = "base64"`.
|
||||
|
||||
```csharp
|
||||
new MessageParam {
|
||||
Role = Role.User,
|
||||
Content = new List<ContentBlockParam> {
|
||||
new DocumentBlockParam { Source = new Base64PdfSource { Data = base64String } },
|
||||
new TextBlockParam { Text = "Summarize this PDF" },
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools
|
||||
|
||||
Web search, bash, text editor, and code execution are built-in server tools. Type names are version-suffixed; constructors auto-set `name`/`type`. All implicit-convert to `ToolUnion`.
|
||||
|
||||
```csharp
|
||||
Tools = [
|
||||
new WebSearchTool20260209(),
|
||||
new ToolBash20250124(),
|
||||
new ToolTextEditor20250728(),
|
||||
new CodeExecutionTool20260120(),
|
||||
],
|
||||
```
|
||||
|
||||
Also available: `WebFetchTool20260209`, `MemoryTool20250818`. `WebSearchTool20260209` optionals: `AllowedDomains`, `BlockedDomains`, `MaxUses`, `UserLocation`.
|
||||
|
||||
---
|
||||
|
||||
## Files API (Beta)
|
||||
|
||||
Files live under `client.Beta.Files` (namespace `Anthropic.Models.Beta.Files`). `BinaryContent` implicit-converts from `Stream` and `byte[]`.
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Beta.Files;
|
||||
using Anthropic.Models.Beta.Messages;
|
||||
|
||||
FileMetadata meta = await client.Beta.Files.Upload(
|
||||
new FileUploadParams { File = File.OpenRead("doc.pdf") });
|
||||
|
||||
// Referencing the uploaded file requires Beta message types:
|
||||
new BetaRequestDocumentBlock {
|
||||
Source = new BetaFileDocumentSource { FileID = meta.ID },
|
||||
}
|
||||
```
|
||||
|
||||
The non-beta `DocumentBlockParamSource` union has no file-ID variant — file references need `client.Beta.Messages.Create()`.
|
||||
|
||||
---
|
||||
|
||||
## Tool Runner (Beta)
|
||||
|
||||
The C# SDK provides a `BetaToolRunner` for automatic tool execution loops. Define tools with raw JSON schemas, and the runner handles the API call → tool execution → result feedback loop.
|
||||
|
||||
```csharp
|
||||
using Anthropic.Models.Beta.Messages;
|
||||
|
||||
// Define tools and create params as shown in the Tool Use section above,
|
||||
// but using the beta namespace types (BetaToolUnion, etc.)
|
||||
var runner = client.Beta.Messages.ToolRunner(betaParams);
|
||||
|
||||
await foreach (BetaMessage message in runner)
|
||||
{
|
||||
foreach (var block in message.Content)
|
||||
{
|
||||
if (block.TryPickText(out var text))
|
||||
{
|
||||
Console.WriteLine(text.Text);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stop Details
|
||||
|
||||
When `StopReason` is `"refusal"`, the response includes structured `StopDetails`:
|
||||
|
||||
```csharp
|
||||
if (response.StopReason == "refusal" && response.StopDetails is { } details)
|
||||
{
|
||||
Console.WriteLine($"Category: {details.Category}");
|
||||
Console.WriteLine($"Explanation: {details.Explanation}");
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Managed Agents (Beta)
|
||||
|
||||
The C# SDK supports Managed Agents via `client.Beta.Agents`, `client.Beta.Sessions`, `client.Beta.Environments`, and related namespaces. See `shared/managed-agents-overview.md` for the architecture and `curl/managed-agents.md` for the wire-level reference.
|
||||
@@ -1,216 +0,0 @@
|
||||
# Claude API — cURL / Raw HTTP
|
||||
|
||||
Use these examples when the user needs raw HTTP requests or is working in a language without an official SDK.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="your-api-key"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-d '{
|
||||
"model": "claude-opus-4-8",
|
||||
"max_tokens": 16000,
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is the capital of France?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### Parsing the response
|
||||
|
||||
Use `jq` to extract fields from the JSON response. Do not use `grep`/`sed` —
|
||||
JSON strings can contain any character and regex parsing will break on quotes,
|
||||
escapes, or multi-line content.
|
||||
|
||||
```bash
|
||||
# Capture the response, then extract fields
|
||||
response=$(curl -s https://api.anthropic.com/v1/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-d '{"model":"claude-opus-4-8","max_tokens":16000,"messages":[{"role":"user","content":"Hello"}]}')
|
||||
|
||||
# Print the first text block (-r strips the JSON quotes)
|
||||
echo "$response" | jq -r '.content[0].text'
|
||||
|
||||
# Read usage fields
|
||||
input_tokens=$(echo "$response" | jq -r '.usage.input_tokens')
|
||||
output_tokens=$(echo "$response" | jq -r '.usage.output_tokens')
|
||||
|
||||
# Read stop reason (for tool-use loops)
|
||||
stop_reason=$(echo "$response" | jq -r '.stop_reason')
|
||||
|
||||
# Extract all text blocks (content is an array; filter to type=="text")
|
||||
echo "$response" | jq -r '.content[] | select(.type == "text") | .text'
|
||||
```
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Streaming (SSE)
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-d '{
|
||||
"model": "claude-opus-4-8",
|
||||
"max_tokens": 64000,
|
||||
"stream": true,
|
||||
"messages": [{"role": "user", "content": "Write a haiku"}]
|
||||
}'
|
||||
```
|
||||
|
||||
The response is a stream of Server-Sent Events:
|
||||
|
||||
```
|
||||
event: message_start
|
||||
data: {"type":"message_start","message":{"id":"msg_...","type":"message",...}}
|
||||
|
||||
event: content_block_start
|
||||
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}
|
||||
|
||||
event: content_block_stop
|
||||
data: {"type":"content_block_stop","index":0}
|
||||
|
||||
event: message_delta
|
||||
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":12}}
|
||||
|
||||
event: message_stop
|
||||
data: {"type":"message_stop"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tool Use
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-d '{
|
||||
"model": "claude-opus-4-8",
|
||||
"max_tokens": 16000,
|
||||
"tools": [{
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "City name"}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}],
|
||||
"messages": [{"role": "user", "content": "What is the weather in Paris?"}]
|
||||
}'
|
||||
```
|
||||
|
||||
When Claude responds with a `tool_use` block, send the result back:
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-d '{
|
||||
"model": "claude-opus-4-8",
|
||||
"max_tokens": 16000,
|
||||
"tools": [{
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "City name"}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}],
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is the weather in Paris?"},
|
||||
{"role": "assistant", "content": [
|
||||
{"type": "text", "text": "Let me check the weather."},
|
||||
{"type": "tool_use", "id": "toolu_abc123", "name": "get_weather", "input": {"location": "Paris"}}
|
||||
]},
|
||||
{"role": "user", "content": [
|
||||
{"type": "tool_result", "tool_use_id": "toolu_abc123", "content": "72°F and sunny"}
|
||||
]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
Put `cache_control` on the last block of the stable prefix. See `shared/prompt-caching.md` for placement patterns and the silent-invalidator audit checklist.
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-d '{
|
||||
"model": "claude-opus-4-8",
|
||||
"max_tokens": 16000,
|
||||
"system": [
|
||||
{"type": "text", "text": "<large shared prompt...>", "cache_control": {"type": "ephemeral"}}
|
||||
],
|
||||
"messages": [{"role": "user", "content": "Summarize the key points"}]
|
||||
}'
|
||||
```
|
||||
|
||||
For 1-hour TTL: `"cache_control": {"type": "ephemeral", "ttl": "1h"}`. Top-level `"cache_control"` on the request body auto-places on the last cacheable block. Verify hits via the response `usage.cache_creation_input_tokens` / `usage.cache_read_input_tokens` fields.
|
||||
|
||||
---
|
||||
|
||||
## Extended Thinking
|
||||
|
||||
> **Fable 5, Opus 4.8, Opus 4.7, Opus 4.6, and Sonnet 4.6:** Use adaptive thinking. `budget_tokens` is removed on Fable 5, Opus 4.8, and 4.7 (400 if sent); deprecated on Opus 4.6 and Sonnet 4.6.
|
||||
> **Older models:** Use `"type": "enabled"` with `"budget_tokens": N` (must be < `max_tokens`, min 1024).
|
||||
|
||||
```bash
|
||||
# Fable 5 / Opus 4.8 / 4.7 / 4.6: adaptive thinking (recommended)
|
||||
curl https://api.anthropic.com/v1/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-d '{
|
||||
"model": "claude-opus-4-8",
|
||||
"max_tokens": 16000,
|
||||
"thinking": {
|
||||
"type": "adaptive"
|
||||
},
|
||||
"output_config": {
|
||||
"effort": "high"
|
||||
},
|
||||
"messages": [{"role": "user", "content": "Solve this step by step..."}]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Required Headers
|
||||
|
||||
| Header | Value | Description |
|
||||
| ------------------- | ------------------ | -------------------------- |
|
||||
| `Content-Type` | `application/json` | Required |
|
||||
| `x-api-key` | Your API key | Authentication |
|
||||
| `anthropic-version` | `2023-06-01` | API version |
|
||||
| `anthropic-beta` | Beta feature IDs | Required for beta features |
|
||||
@@ -1,338 +0,0 @@
|
||||
# Managed Agents — cURL / Raw HTTP
|
||||
|
||||
Use these examples when the user needs raw HTTP requests or is working without an SDK.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="your-api-key"
|
||||
|
||||
# Common headers
|
||||
HEADERS=(
|
||||
-H "Content-Type: application/json"
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY"
|
||||
-H "anthropic-version: 2023-06-01"
|
||||
-H "anthropic-beta: managed-agents-2026-04-01"
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Environment
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.anthropic.com/v1/environments \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"name": "my-dev-env",
|
||||
"config": {
|
||||
"type": "cloud",
|
||||
"networking": { "type": "unrestricted" }
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### With restricted networking
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.anthropic.com/v1/environments \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"name": "restricted-env",
|
||||
"config": {
|
||||
"type": "cloud",
|
||||
"networking": {
|
||||
"type": "limited",
|
||||
"allow_package_managers": true,
|
||||
"allow_mcp_servers": true,
|
||||
"allowed_hosts": ["api.example.com"]
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Agent (required first step)
|
||||
|
||||
> ⚠️ **There is no inline agent config.** Under `managed-agents-2026-04-01`, `model`/`system`/`tools` are top-level fields on `POST /v1/agents`, not on the session. Always create the agent first — the session only takes `"agent": {"type": "agent", "id": "..."}`.
|
||||
|
||||
### Minimal
|
||||
|
||||
```bash
|
||||
# 1. Create the agent
|
||||
curl -X POST https://api.anthropic.com/v1/agents \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"name": "Coding Assistant",
|
||||
"model": "claude-opus-4-8",
|
||||
"tools": [{ "type": "agent_toolset_20260401" }]
|
||||
}'
|
||||
# → { "id": "agent_abc123", ... }
|
||||
|
||||
# 2. Start a session
|
||||
curl -X POST https://api.anthropic.com/v1/sessions \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"agent": { "type": "agent", "id": "agent_abc123", "version": "1772585501101368014" },
|
||||
"environment_id": "env_abc123"
|
||||
}'
|
||||
```
|
||||
|
||||
### With system prompt, custom tools, and GitHub repo
|
||||
|
||||
```bash
|
||||
# 1. Create the agent
|
||||
curl -X POST https://api.anthropic.com/v1/agents \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"name": "Code Reviewer",
|
||||
"model": "claude-opus-4-8",
|
||||
"system": "You are a senior code reviewer. Be thorough and constructive.",
|
||||
"tools": [
|
||||
{ "type": "agent_toolset_20260401" },
|
||||
{
|
||||
"type": "custom",
|
||||
"name": "run_linter",
|
||||
"description": "Run the project linter on a file",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file_path": { "type": "string", "description": "Path to lint" }
|
||||
},
|
||||
"required": ["file_path"]
|
||||
}
|
||||
}
|
||||
]
|
||||
}'
|
||||
|
||||
# 2. Start a session with the repo mounted
|
||||
curl -X POST https://api.anthropic.com/v1/sessions \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"agent": { "type": "agent", "id": "agent_abc123", "version": "1772585501101368014" },
|
||||
"environment_id": "env_abc123",
|
||||
"title": "Code review session",
|
||||
"resources": [
|
||||
{
|
||||
"type": "github_repository",
|
||||
"url": "https://github.com/owner/repo",
|
||||
"mount_path": "/workspace/repo",
|
||||
"authorization_token": "ghp_...",
|
||||
"branch": "feature-branch"
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Send a User Message
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.anthropic.com/v1/sessions/$SESSION_ID/events \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"events": [
|
||||
{
|
||||
"type": "user.message",
|
||||
"content": [{ "type": "text", "text": "Review the auth module for security issues" }]
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stream Events (SSE)
|
||||
|
||||
```bash
|
||||
curl -N https://api.anthropic.com/v1/sessions/$SESSION_ID/events/stream \
|
||||
"${HEADERS[@]}"
|
||||
```
|
||||
|
||||
Response format:
|
||||
|
||||
```
|
||||
event: session.status_running
|
||||
data: {"type":"session.status_running","id":"sevt_...","processed_at":"..."}
|
||||
|
||||
event: agent.message
|
||||
data: {"type":"agent.message","id":"sevt_...","content":[{"type":"text","text":"I'll review..."}],"processed_at":"..."}
|
||||
|
||||
event: session.status_idle
|
||||
data: {"type":"session.status_idle","id":"sevt_...","processed_at":"..."}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Poll Events
|
||||
|
||||
```bash
|
||||
# Get all events
|
||||
curl https://api.anthropic.com/v1/sessions/$SESSION_ID/events \
|
||||
"${HEADERS[@]}"
|
||||
|
||||
# Paginated — get next page of events
|
||||
curl "https://api.anthropic.com/v1/sessions/$SESSION_ID/events?page=page_abc123" \
|
||||
"${HEADERS[@]}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provide Custom Tool Result
|
||||
|
||||
When the agent calls a custom tool, send the result back:
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.anthropic.com/v1/sessions/$SESSION_ID/events \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"events": [
|
||||
{
|
||||
"type": "user.custom_tool_result",
|
||||
"custom_tool_use_id": "sevt_abc123",
|
||||
"content": [{ "type": "text", "text": "No linting errors found." }]
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Interrupt a Running Session
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.anthropic.com/v1/sessions/$SESSION_ID/events \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"events": [
|
||||
{
|
||||
"type": "interrupt"
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Get Session Details
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/sessions/$SESSION_ID \
|
||||
"${HEADERS[@]}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List Sessions
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/sessions \
|
||||
"${HEADERS[@]}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delete a Session
|
||||
|
||||
```bash
|
||||
curl -X DELETE https://api.anthropic.com/v1/sessions/$SESSION_ID \
|
||||
"${HEADERS[@]}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.anthropic.com/v1/files \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-H "anthropic-beta: files-api-2025-04-14" \
|
||||
-F "file=@path/to/file.txt"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List and Download Session Files
|
||||
|
||||
List files the agent wrote to `/mnt/session/outputs/` during a session, then download them.
|
||||
|
||||
```bash
|
||||
# List files associated with a session
|
||||
curl "https://api.anthropic.com/v1/files?scope_id=$SESSION_ID" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-H "anthropic-beta: files-api-2025-04-14,managed-agents-2026-04-01"
|
||||
|
||||
# Download a specific file
|
||||
curl "https://api.anthropic.com/v1/files/$FILE_ID/content" \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-H "anthropic-beta: files-api-2025-04-14,managed-agents-2026-04-01" \
|
||||
-o downloaded_file.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List Agents
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/agents \
|
||||
"${HEADERS[@]}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Integration
|
||||
|
||||
```bash
|
||||
# 1. Agent declares MCP server (no auth here — auth goes in a vault)
|
||||
curl -X POST https://api.anthropic.com/v1/agents \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"name": "MCP Agent",
|
||||
"model": "claude-opus-4-8",
|
||||
"mcp_servers": [
|
||||
{ "type": "url", "name": "my-tools", "url": "https://my-mcp-server.example.com/sse" }
|
||||
],
|
||||
"tools": [
|
||||
{ "type": "agent_toolset_20260401" },
|
||||
{ "type": "mcp_toolset", "mcp_server_name": "my-tools" }
|
||||
]
|
||||
}'
|
||||
|
||||
# 2. Session attaches vault containing credentials for that MCP server URL
|
||||
curl -X POST https://api.anthropic.com/v1/sessions \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"agent": "agent_abc123",
|
||||
"environment_id": "env_abc123",
|
||||
"vault_ids": ["vlt_abc123"]
|
||||
}'
|
||||
```
|
||||
|
||||
See `shared/managed-agents-tools.md` §Vaults for creating vaults and adding credentials.
|
||||
|
||||
---
|
||||
|
||||
## Tool Configuration
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.anthropic.com/v1/agents \
|
||||
"${HEADERS[@]}" \
|
||||
-d '{
|
||||
"name": "Restricted Agent",
|
||||
"model": "claude-opus-4-8",
|
||||
"tools": [
|
||||
{
|
||||
"type": "agent_toolset_20260401",
|
||||
"default_config": { "enabled": true },
|
||||
"configs": [
|
||||
{ "name": "bash", "enabled": false }
|
||||
]
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
@@ -1,440 +0,0 @@
|
||||
# Claude API — Go
|
||||
|
||||
> **Note:** The Go SDK supports the Claude API and beta tool use with `BetaToolRunner`. Agent SDK is not yet available for Go.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
go get github.com/anthropics/anthropic-sdk-go
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```go
|
||||
import (
|
||||
"github.com/anthropics/anthropic-sdk-go"
|
||||
"github.com/anthropics/anthropic-sdk-go/option"
|
||||
)
|
||||
|
||||
// Default (uses ANTHROPIC_API_KEY env var)
|
||||
client := anthropic.NewClient()
|
||||
|
||||
// Explicit API key
|
||||
client := anthropic.NewClient(
|
||||
option.WithAPIKey("your-api-key"),
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Model Constants
|
||||
|
||||
The Go SDK provides typed model constants: `anthropic.ModelClaudeFable5`, `anthropic.ModelClaudeOpus4_8`, `anthropic.ModelClaudeOpus4_7`, `anthropic.ModelClaudeSonnet4_6`, `anthropic.ModelClaudeHaiku4_5_20251001`. Use `ModelClaudeOpus4_8` unless the user specifies otherwise; if they ask for Fable or the most powerful model, use `anthropic.ModelClaudeFable5` (see `shared/models.md` for the full resolution table).
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```go
|
||||
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
|
||||
Model: anthropic.ModelClaudeOpus4_8,
|
||||
MaxTokens: 16000,
|
||||
Messages: []anthropic.MessageParam{
|
||||
anthropic.NewUserMessage(anthropic.NewTextBlock("What is the capital of France?")),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
for _, block := range response.Content {
|
||||
switch variant := block.AsAny().(type) {
|
||||
case anthropic.TextBlock:
|
||||
fmt.Println(variant.Text)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming
|
||||
|
||||
```go
|
||||
stream := client.Messages.NewStreaming(context.Background(), anthropic.MessageNewParams{
|
||||
Model: anthropic.ModelClaudeOpus4_6,
|
||||
MaxTokens: 64000,
|
||||
Messages: []anthropic.MessageParam{
|
||||
anthropic.NewUserMessage(anthropic.NewTextBlock("Write a haiku")),
|
||||
},
|
||||
})
|
||||
|
||||
for stream.Next() {
|
||||
event := stream.Current()
|
||||
switch eventVariant := event.AsAny().(type) {
|
||||
case anthropic.ContentBlockDeltaEvent:
|
||||
switch deltaVariant := eventVariant.Delta.AsAny().(type) {
|
||||
case anthropic.TextDelta:
|
||||
fmt.Print(deltaVariant.Text)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := stream.Err(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
**Accumulating the final message** (there is no `GetFinalMessage()` on the stream):
|
||||
|
||||
```go
|
||||
stream := client.Messages.NewStreaming(ctx, params)
|
||||
message := anthropic.Message{}
|
||||
for stream.Next() {
|
||||
message.Accumulate(stream.Current())
|
||||
}
|
||||
if err := stream.Err(); err != nil { log.Fatal(err) }
|
||||
// message.Content now has the complete response
|
||||
```
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Tool Use
|
||||
|
||||
### Tool Runner (Beta — Recommended)
|
||||
|
||||
**Beta:** The Go SDK provides `BetaToolRunner` for automatic tool use loops via the `toolrunner` package.
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"github.com/anthropics/anthropic-sdk-go"
|
||||
"github.com/anthropics/anthropic-sdk-go/toolrunner"
|
||||
)
|
||||
|
||||
// Define tool input with jsonschema tags for automatic schema generation
|
||||
type GetWeatherInput struct {
|
||||
City string `json:"city" jsonschema:"required,description=The city name"`
|
||||
}
|
||||
|
||||
// Create a tool with automatic schema generation from struct tags
|
||||
weatherTool, err := toolrunner.NewBetaToolFromJSONSchema(
|
||||
"get_weather",
|
||||
"Get current weather for a city",
|
||||
func(ctx context.Context, input GetWeatherInput) (anthropic.BetaToolResultBlockParamContentUnion, error) {
|
||||
return anthropic.BetaToolResultBlockParamContentUnion{
|
||||
OfText: &anthropic.BetaTextBlockParam{
|
||||
Text: fmt.Sprintf("The weather in %s is sunny, 72°F", input.City),
|
||||
},
|
||||
}, nil
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Create a tool runner that handles the conversation loop automatically
|
||||
runner := client.Beta.Messages.NewToolRunner(
|
||||
[]anthropic.BetaTool{weatherTool},
|
||||
anthropic.BetaToolRunnerParams{
|
||||
BetaMessageNewParams: anthropic.BetaMessageNewParams{
|
||||
Model: anthropic.ModelClaudeOpus4_6,
|
||||
MaxTokens: 16000,
|
||||
Messages: []anthropic.BetaMessageParam{
|
||||
anthropic.NewBetaUserMessage(anthropic.NewBetaTextBlock("What's the weather in Paris?")),
|
||||
},
|
||||
},
|
||||
MaxIterations: 5,
|
||||
},
|
||||
)
|
||||
|
||||
// Run until Claude produces a final response
|
||||
message, err := runner.RunToCompletion(context.Background())
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// RunToCompletion returns *BetaMessage; content is []BetaContentBlockUnion.
|
||||
// Narrow via AsAny() switch — note the Beta-namespace types (BetaTextBlock,
|
||||
// not TextBlock):
|
||||
for _, block := range message.Content {
|
||||
switch block := block.AsAny().(type) {
|
||||
case anthropic.BetaTextBlock:
|
||||
fmt.Println(block.Text)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key features of the Go tool runner:**
|
||||
|
||||
- Automatic schema generation from Go structs via `jsonschema` tags
|
||||
- `RunToCompletion()` for simple one-shot usage
|
||||
- `All()` iterator for processing each message in the conversation
|
||||
- `NextMessage()` for step-by-step iteration
|
||||
- Streaming variant via `NewToolRunnerStreaming()` with `AllStreaming()`
|
||||
|
||||
### Manual Loop
|
||||
|
||||
For fine-grained control over the agentic loop, define tools with `ToolParam`, check `StopReason`, execute tools yourself, and feed `tool_result` blocks back. This is the pattern when you need to intercept, validate, or log tool calls.
|
||||
|
||||
Derived from `anthropic-sdk-go/examples/tools/main.go`.
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"github.com/anthropics/anthropic-sdk-go"
|
||||
)
|
||||
|
||||
func main() {
|
||||
client := anthropic.NewClient()
|
||||
|
||||
// 1. Define tools. ToolParam.InputSchema uses a map, no struct tags needed.
|
||||
addTool := anthropic.ToolParam{
|
||||
Name: "add",
|
||||
Description: anthropic.String("Add two integers"),
|
||||
InputSchema: anthropic.ToolInputSchemaParam{
|
||||
Properties: map[string]any{
|
||||
"a": map[string]any{"type": "integer"},
|
||||
"b": map[string]any{"type": "integer"},
|
||||
},
|
||||
},
|
||||
}
|
||||
// ToolParam must be wrapped in ToolUnionParam for the Tools slice
|
||||
tools := []anthropic.ToolUnionParam{{OfTool: &addTool}}
|
||||
|
||||
messages := []anthropic.MessageParam{
|
||||
anthropic.NewUserMessage(anthropic.NewTextBlock("What is 2 + 3?")),
|
||||
}
|
||||
|
||||
for {
|
||||
resp, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
|
||||
Model: anthropic.ModelClaudeSonnet4_6,
|
||||
MaxTokens: 16000,
|
||||
Messages: messages,
|
||||
Tools: tools,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// 2. Append the assistant response to history BEFORE processing tool calls.
|
||||
// resp.ToParam() converts Message → MessageParam in one call.
|
||||
messages = append(messages, resp.ToParam())
|
||||
|
||||
// 3. Walk content blocks. ContentBlockUnion is a flattened struct;
|
||||
// use block.AsAny().(type) to switch on the actual variant.
|
||||
toolResults := []anthropic.ContentBlockParamUnion{}
|
||||
for _, block := range resp.Content {
|
||||
switch variant := block.AsAny().(type) {
|
||||
case anthropic.TextBlock:
|
||||
fmt.Println(variant.Text)
|
||||
case anthropic.ToolUseBlock:
|
||||
// 4. Parse the tool input. Use variant.JSON.Input.Raw() to get the
|
||||
// raw JSON — block.Input is json.RawMessage, not the parsed value.
|
||||
var in struct {
|
||||
A int `json:"a"`
|
||||
B int `json:"b"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(variant.JSON.Input.Raw()), &in); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
result := fmt.Sprintf("%d", in.A+in.B)
|
||||
// 5. NewToolResultBlock(toolUseID, content, isError) builds the
|
||||
// ContentBlockParamUnion for you. block.ID is the tool_use_id.
|
||||
toolResults = append(toolResults,
|
||||
anthropic.NewToolResultBlock(block.ID, result, false))
|
||||
}
|
||||
}
|
||||
|
||||
// 6. Exit when Claude stops asking for tools
|
||||
if resp.StopReason != anthropic.StopReasonToolUse {
|
||||
break
|
||||
}
|
||||
|
||||
// 7. Tool results go in a user message (variadic: all results in one turn)
|
||||
messages = append(messages, anthropic.NewUserMessage(toolResults...))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key API surface:**
|
||||
|
||||
| Symbol | Purpose |
|
||||
|---|---|
|
||||
| `resp.ToParam()` | Convert `Message` response → `MessageParam` for history |
|
||||
| `block.AsAny().(type)` | Type-switch on `ContentBlockUnion` variants |
|
||||
| `variant.JSON.Input.Raw()` | Raw JSON string of tool input (for `json.Unmarshal`) |
|
||||
| `anthropic.NewToolResultBlock(id, content, isError)` | Build `tool_result` block |
|
||||
| `anthropic.NewUserMessage(blocks...)` | Wrap tool results as a user turn |
|
||||
| `anthropic.StopReasonToolUse` | `StopReason` constant to check loop termination |
|
||||
| `anthropic.ToolUnionParam{OfTool: &t}` | Wrap `ToolParam` in the union for `Tools:` |
|
||||
|
||||
---
|
||||
|
||||
## Thinking
|
||||
|
||||
Enable Claude's internal reasoning by setting `Thinking` in `MessageNewParams`. The response will contain `ThinkingBlock` content before the final `TextBlock`.
|
||||
|
||||
**Adaptive thinking is the recommended mode for Claude 4.6+ models.** Claude decides dynamically when and how much to think. Combine with the `effort` parameter for cost-quality control.
|
||||
|
||||
Derived from `anthropic-sdk-go/message.go` (`ThinkingConfigParamUnion`, `ThinkingConfigAdaptiveParam`).
|
||||
|
||||
```go
|
||||
// There is no ThinkingConfigParamOfAdaptive helper — construct the union
|
||||
// struct-literal directly and take the address of the variant.
|
||||
adaptive := anthropic.ThinkingConfigAdaptiveParam{}
|
||||
params := anthropic.MessageNewParams{
|
||||
Model: anthropic.ModelClaudeSonnet4_6,
|
||||
MaxTokens: 16000,
|
||||
Thinking: anthropic.ThinkingConfigParamUnion{OfAdaptive: &adaptive},
|
||||
Messages: []anthropic.MessageParam{
|
||||
anthropic.NewUserMessage(anthropic.NewTextBlock("How many r's in strawberry?")),
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := client.Messages.New(context.Background(), params)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// ThinkingBlock(s) precede TextBlock in content
|
||||
for _, block := range resp.Content {
|
||||
switch b := block.AsAny().(type) {
|
||||
case anthropic.ThinkingBlock:
|
||||
fmt.Println("[thinking]", b.Thinking)
|
||||
case anthropic.TextBlock:
|
||||
fmt.Println(b.Text)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Deprecated:** `ThinkingConfigParamOfEnabled(budgetTokens)` (fixed-budget extended thinking) still works on Claude 4.6 but is deprecated. Use adaptive thinking above.
|
||||
|
||||
To disable: `anthropic.ThinkingConfigParamUnion{OfDisabled: &anthropic.ThinkingConfigDisabledParam{}}`.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
`System` is `[]TextBlockParam`; set `CacheControl` on the last block to cache tools + system together. For placement patterns and the silent-invalidator audit checklist, see `shared/prompt-caching.md`.
|
||||
|
||||
```go
|
||||
System: []anthropic.TextBlockParam{{
|
||||
Text: longSystemPrompt,
|
||||
CacheControl: anthropic.NewCacheControlEphemeralParam(), // default 5m TTL
|
||||
}},
|
||||
```
|
||||
|
||||
For 1-hour TTL: `anthropic.CacheControlEphemeralParam{TTL: anthropic.CacheControlEphemeralTTLTTL1h}`. There's also a top-level `CacheControl` on `MessageNewParams` that auto-places on the last cacheable block.
|
||||
|
||||
Verify hits via `resp.Usage.CacheCreationInputTokens` / `resp.Usage.CacheReadInputTokens`.
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools
|
||||
|
||||
Version-suffixed struct names with `Param` suffix. `Name`/`Type` are `constant.*` types — zero value marshals correctly, so `{}` works. Wrap in `ToolUnionParam` with the matching `Of*` field.
|
||||
|
||||
```go
|
||||
Tools: []anthropic.ToolUnionParam{
|
||||
{OfWebSearchTool20260209: &anthropic.WebSearchTool20260209Param{}},
|
||||
{OfBashTool20250124: &anthropic.ToolBash20250124Param{}},
|
||||
{OfTextEditor20250728: &anthropic.ToolTextEditor20250728Param{}},
|
||||
{OfCodeExecutionTool20260120: &anthropic.CodeExecutionTool20260120Param{}},
|
||||
},
|
||||
```
|
||||
|
||||
Also available: `WebFetchTool20260209Param`, `MemoryTool20250818Param`, `ToolSearchToolBm25_20251119Param`, `ToolSearchToolRegex20251119Param`. For the advisor tool, use `BetaAdvisorTool20260301Param` in the beta namespace.
|
||||
|
||||
---
|
||||
|
||||
## Stop Details
|
||||
|
||||
When `StopReason` is `anthropic.StopReasonRefusal`, the response includes structured `StopDetails`:
|
||||
|
||||
```go
|
||||
if resp.StopReason == anthropic.StopReasonRefusal {
|
||||
fmt.Println("Category:", resp.StopDetails.Category) // "cyber" | "bio" | ""
|
||||
fmt.Println("Explanation:", resp.StopDetails.Explanation)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PDF / Document Input
|
||||
|
||||
`NewDocumentBlock` generic helper accepts any source type. `MediaType`/`Type` are auto-set.
|
||||
|
||||
```go
|
||||
b64 := base64.StdEncoding.EncodeToString(pdfBytes)
|
||||
|
||||
msg := anthropic.NewUserMessage(
|
||||
anthropic.NewDocumentBlock(anthropic.Base64PDFSourceParam{Data: b64}),
|
||||
anthropic.NewTextBlock("Summarize this document"),
|
||||
)
|
||||
```
|
||||
|
||||
Other sources: `URLPDFSourceParam{URL: "https://..."}`, `PlainTextSourceParam{Data: "..."}`.
|
||||
|
||||
---
|
||||
|
||||
## Files API (Beta)
|
||||
|
||||
Under `client.Beta.Files`. Method is **`Upload`** (NOT `New`/`Create`), params struct is `BetaFileUploadParams`. The `File` field takes an `io.Reader`; use `anthropic.File()` to attach a filename + content-type for the multipart encoding.
|
||||
|
||||
```go
|
||||
f, _ := os.Open("./upload_me.txt")
|
||||
defer f.Close()
|
||||
|
||||
meta, err := client.Beta.Files.Upload(ctx, anthropic.BetaFileUploadParams{
|
||||
File: anthropic.File(f, "upload_me.txt", "text/plain"),
|
||||
Betas: []anthropic.AnthropicBeta{anthropic.AnthropicBetaFilesAPI2025_04_14},
|
||||
})
|
||||
// meta.ID is the file_id to reference in subsequent message requests
|
||||
```
|
||||
|
||||
Other `Beta.Files` methods: `List`, `Delete`, `Download`, `GetMetadata`.
|
||||
|
||||
---
|
||||
|
||||
## Context Editing / Compaction (Beta)
|
||||
|
||||
Use `Beta.Messages.New` with `ContextManagement` on `BetaMessageNewParams`. There is no `NewBetaAssistantMessage` — use `.ToParam()` for the round-trip.
|
||||
|
||||
```go
|
||||
params := anthropic.BetaMessageNewParams{
|
||||
Model: anthropic.ModelClaudeOpus4_6, // also supported: ModelClaudeSonnet4_6
|
||||
MaxTokens: 16000,
|
||||
Betas: []anthropic.AnthropicBeta{"compact-2026-01-12"},
|
||||
ContextManagement: anthropic.BetaContextManagementConfigParam{
|
||||
Edits: []anthropic.BetaContextManagementConfigEditUnionParam{
|
||||
{OfCompact20260112: &anthropic.BetaCompact20260112EditParam{}},
|
||||
},
|
||||
},
|
||||
Messages: []anthropic.BetaMessageParam{ /* ... */ },
|
||||
}
|
||||
|
||||
resp, err := client.Beta.Messages.New(ctx, params)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Round-trip: append response to history via .ToParam()
|
||||
params.Messages = append(params.Messages, resp.ToParam())
|
||||
|
||||
// Read compaction blocks from the response
|
||||
for _, block := range resp.Content {
|
||||
if c, ok := block.AsAny().(anthropic.BetaCompactionBlock); ok {
|
||||
fmt.Println("compaction summary:", c.Content)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Other edit types: `BetaClearToolUses20250919EditParam`, `BetaClearThinking20251015EditParam`.
|
||||
@@ -1,561 +0,0 @@
|
||||
# Managed Agents — Go
|
||||
|
||||
> **Bindings not shown here:** This README covers the most common managed-agents flows for Go. If you need a class, method, namespace, field, or behavior that isn't shown, WebFetch the Go SDK repo **or the relevant docs page** from `shared/live-sources.md` rather than guess. Do not extrapolate from cURL shapes or another language's SDK.
|
||||
|
||||
> **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `agents.New` and pass it to every subsequent `sessions.New`; do not call `agents.New` in the request path. The Anthropic CLI is one convenient way to create agents and environments from version-controlled YAML — its URL is in `shared/live-sources.md`. The examples below show in-code creation for completeness; in production the create call belongs in setup, not in the request path.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
go get github.com/anthropics/anthropic-sdk-go
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/anthropics/anthropic-sdk-go"
|
||||
"github.com/anthropics/anthropic-sdk-go/option"
|
||||
)
|
||||
|
||||
// Default (uses ANTHROPIC_API_KEY env var)
|
||||
client := anthropic.NewClient()
|
||||
|
||||
// Explicit API key
|
||||
client := anthropic.NewClient(
|
||||
option.WithAPIKey("your-api-key"),
|
||||
)
|
||||
|
||||
ctx := context.Background()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Environment
|
||||
|
||||
```go
|
||||
environment, err := client.Beta.Environments.New(ctx, anthropic.BetaEnvironmentNewParams{
|
||||
Name: "my-dev-env",
|
||||
Config: anthropic.BetaCloudConfigParams{
|
||||
Networking: anthropic.BetaCloudConfigParamsNetworkingUnion{
|
||||
OfUnrestricted: &anthropic.UnrestrictedNetworkParam{},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Println(environment.ID) // env_...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Agent (required first step)
|
||||
|
||||
> ⚠️ **There is no inline agent config.** `Model`/`System`/`Tools` live on the agent object, not the session. Always start with `Beta.Agents.New()` — the session only takes `Agent: anthropic.BetaSessionNewParamsAgentUnion{OfString: anthropic.String(agent.ID)}` (or the typed `OfBetaManagedAgentsAgents` variant when you need a specific version).
|
||||
|
||||
### Minimal
|
||||
|
||||
```go
|
||||
// 1. Create the agent (reusable, versioned)
|
||||
agent, err := client.Beta.Agents.New(ctx, anthropic.BetaAgentNewParams{
|
||||
Name: "Coding Assistant",
|
||||
Model: anthropic.BetaManagedAgentsModelConfigParams{
|
||||
ID: "claude-opus-4-8",
|
||||
Type: anthropic.BetaManagedAgentsModelConfigParamsTypeModelConfig,
|
||||
},
|
||||
System: anthropic.String("You are a helpful coding assistant."),
|
||||
Tools: []anthropic.BetaAgentNewParamsToolUnion{{
|
||||
OfAgentToolset20260401: &anthropic.BetaManagedAgentsAgentToolset20260401Params{
|
||||
Type: anthropic.BetaManagedAgentsAgentToolset20260401ParamsTypeAgentToolset20260401,
|
||||
},
|
||||
}},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// 2. Start a session
|
||||
session, err := client.Beta.Sessions.New(ctx, anthropic.BetaSessionNewParams{
|
||||
Agent: anthropic.BetaSessionNewParamsAgentUnion{
|
||||
OfBetaManagedAgentsAgents: &anthropic.BetaManagedAgentsAgentParams{
|
||||
Type: anthropic.BetaManagedAgentsAgentParamsTypeAgent,
|
||||
ID: agent.ID,
|
||||
Version: anthropic.Int(agent.Version),
|
||||
},
|
||||
},
|
||||
EnvironmentID: environment.ID,
|
||||
Title: anthropic.String("Quickstart session"),
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Printf("Session ID: %s, status: %s\n", session.ID, session.Status)
|
||||
```
|
||||
|
||||
### Updating an Agent
|
||||
|
||||
Updates create new versions; the agent object is immutable per version.
|
||||
|
||||
```go
|
||||
updatedAgent, err := client.Beta.Agents.Update(ctx, agent.ID, anthropic.BetaAgentUpdateParams{
|
||||
Version: agent.Version,
|
||||
System: anthropic.String("You are a helpful coding agent. Always write tests."),
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Printf("New version: %d\n", updatedAgent.Version)
|
||||
|
||||
// List all versions
|
||||
iter := client.Beta.Agents.Versions.ListAutoPaging(ctx, agent.ID, anthropic.BetaAgentVersionListParams{})
|
||||
for iter.Next() {
|
||||
version := iter.Current()
|
||||
fmt.Printf("Version %d: %s\n", version.Version, version.UpdatedAt.Format(time.RFC3339))
|
||||
}
|
||||
if err := iter.Err(); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Archive the agent
|
||||
_, err = client.Beta.Agents.Archive(ctx, agent.ID, anthropic.BetaAgentArchiveParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Send a User Message
|
||||
|
||||
```go
|
||||
_, err = client.Beta.Sessions.Events.Send(ctx, session.ID, anthropic.BetaSessionEventSendParams{
|
||||
Events: []anthropic.SendEventsParamsUnion{{
|
||||
OfUserMessage: &anthropic.BetaManagedAgentsUserMessageEventParams{
|
||||
Type: anthropic.BetaManagedAgentsUserMessageEventParamsTypeUserMessage,
|
||||
Content: []anthropic.BetaManagedAgentsUserMessageEventParamsContentUnion{{
|
||||
OfText: &anthropic.BetaManagedAgentsTextBlockParam{
|
||||
Type: anthropic.BetaManagedAgentsTextBlockTypeText,
|
||||
Text: "Review the auth module",
|
||||
},
|
||||
}},
|
||||
},
|
||||
}},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
> 💡 **Stream-first:** Open the stream *before* (or concurrently with) sending the message. The stream only delivers events that occur after it opens — stream-after-send means early events arrive buffered in one batch. See [Steering Patterns](../../shared/managed-agents-events.md#steering-patterns).
|
||||
|
||||
---
|
||||
|
||||
## Stream Events (SSE)
|
||||
|
||||
```go
|
||||
// Open the stream first, then send the user message
|
||||
stream := client.Beta.Sessions.Events.StreamEvents(ctx, session.ID, anthropic.BetaSessionEventStreamParams{})
|
||||
defer stream.Close()
|
||||
|
||||
if _, err := client.Beta.Sessions.Events.Send(ctx, session.ID, anthropic.BetaSessionEventSendParams{
|
||||
Events: []anthropic.SendEventsParamsUnion{{
|
||||
OfUserMessage: &anthropic.BetaManagedAgentsUserMessageEventParams{
|
||||
Type: anthropic.BetaManagedAgentsUserMessageEventParamsTypeUserMessage,
|
||||
Content: []anthropic.BetaManagedAgentsUserMessageEventParamsContentUnion{{
|
||||
OfText: &anthropic.BetaManagedAgentsTextBlockParam{
|
||||
Type: anthropic.BetaManagedAgentsTextBlockTypeText,
|
||||
Text: "Summarize the repo README",
|
||||
},
|
||||
}},
|
||||
},
|
||||
}},
|
||||
}); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
events:
|
||||
for stream.Next() {
|
||||
switch event := stream.Current().AsAny().(type) {
|
||||
case anthropic.BetaManagedAgentsAgentMessageEvent:
|
||||
for _, block := range event.Content {
|
||||
fmt.Print(block.Text)
|
||||
}
|
||||
case anthropic.BetaManagedAgentsAgentToolUseEvent:
|
||||
fmt.Printf("\n[Using tool: %s]\n", event.Name)
|
||||
case anthropic.BetaManagedAgentsSessionStatusIdleEvent:
|
||||
break events
|
||||
case anthropic.BetaManagedAgentsSessionErrorEvent:
|
||||
fmt.Printf("\n[Error: %s]\n", event.Error.Message)
|
||||
break events
|
||||
}
|
||||
}
|
||||
if err := stream.Err(); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
### Reconnecting and Tailing
|
||||
|
||||
When reconnecting mid-session, list past events first to dedupe, then tail live events:
|
||||
|
||||
```go
|
||||
stream := client.Beta.Sessions.Events.StreamEvents(ctx, session.ID, anthropic.BetaSessionEventStreamParams{})
|
||||
defer stream.Close()
|
||||
|
||||
// Stream is open and buffering. List history before tailing live.
|
||||
seenEventIDs := map[string]struct{}{}
|
||||
history := client.Beta.Sessions.Events.ListAutoPaging(ctx, session.ID, anthropic.BetaSessionEventListParams{})
|
||||
for history.Next() {
|
||||
seenEventIDs[history.Current().ID] = struct{}{}
|
||||
}
|
||||
if err := history.Err(); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Tail live events, skipping anything already seen
|
||||
tail:
|
||||
for stream.Next() {
|
||||
event := stream.Current()
|
||||
if _, seen := seenEventIDs[event.ID]; seen {
|
||||
continue
|
||||
}
|
||||
seenEventIDs[event.ID] = struct{}{}
|
||||
switch event := event.AsAny().(type) {
|
||||
case anthropic.BetaManagedAgentsAgentMessageEvent:
|
||||
for _, block := range event.Content {
|
||||
fmt.Print(block.Text)
|
||||
}
|
||||
case anthropic.BetaManagedAgentsSessionStatusIdleEvent:
|
||||
break tail
|
||||
}
|
||||
}
|
||||
if err := stream.Err(); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provide Custom Tool Result
|
||||
|
||||
> ℹ️ The Go managed-agents bindings for `user.custom_tool_result` are not yet documented in this skill or in the apps source examples. Refer to `shared/managed-agents-events.md` for the wire format and the `github.com/anthropics/anthropic-sdk-go` repository for the corresponding Go params types.
|
||||
|
||||
---
|
||||
|
||||
## Poll Events
|
||||
|
||||
```go
|
||||
// Auto-paginating iterator
|
||||
iter := client.Beta.Sessions.Events.ListAutoPaging(ctx, session.ID, anthropic.BetaSessionEventListParams{})
|
||||
for iter.Next() {
|
||||
event := iter.Current()
|
||||
fmt.Printf("%s: %s\n", event.Type, event.ID)
|
||||
}
|
||||
if err := iter.Err(); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
```go
|
||||
csvFile, err := os.Open("data.csv")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
defer csvFile.Close()
|
||||
|
||||
file, err := client.Beta.Files.Upload(ctx, anthropic.BetaFileUploadParams{
|
||||
File: csvFile,
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Printf("File ID: %s\n", file.ID)
|
||||
|
||||
// Mount in a session
|
||||
session, err := client.Beta.Sessions.New(ctx, anthropic.BetaSessionNewParams{
|
||||
Agent: anthropic.BetaSessionNewParamsAgentUnion{
|
||||
OfString: anthropic.String(agent.ID),
|
||||
},
|
||||
EnvironmentID: environment.ID,
|
||||
Resources: []anthropic.BetaSessionNewParamsResourceUnion{{
|
||||
OfFile: &anthropic.BetaManagedAgentsFileResourceParams{
|
||||
Type: anthropic.BetaManagedAgentsFileResourceParamsTypeFile,
|
||||
FileID: file.ID,
|
||||
MountPath: anthropic.String("/workspace/data.csv"),
|
||||
},
|
||||
}},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
### Add and Manage Resources on an Existing Session
|
||||
|
||||
```go
|
||||
// Attach an additional file to an open session
|
||||
resource, err := client.Beta.Sessions.Resources.Add(ctx, session.ID, anthropic.BetaSessionResourceAddParams{
|
||||
BetaManagedAgentsFileResourceParams: anthropic.BetaManagedAgentsFileResourceParams{
|
||||
Type: anthropic.BetaManagedAgentsFileResourceParamsTypeFile,
|
||||
FileID: file.ID,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Println(resource.ID) // "sesrsc_01ABC..."
|
||||
|
||||
// List resources on the session
|
||||
listed, err := client.Beta.Sessions.Resources.List(ctx, session.ID, anthropic.BetaSessionResourceListParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
for _, entry := range listed.Data {
|
||||
fmt.Println(entry.ID, entry.Type)
|
||||
}
|
||||
|
||||
// Detach a resource
|
||||
if _, err := client.Beta.Sessions.Resources.Delete(ctx, resource.ID, anthropic.BetaSessionResourceDeleteParams{
|
||||
SessionID: session.ID,
|
||||
}); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List and Download Session Files
|
||||
|
||||
> ℹ️ Listing and downloading files an agent wrote during a session is not yet documented for Go in this skill or in the apps source examples. See `shared/managed-agents-events.md` and the `github.com/anthropics/anthropic-sdk-go` repository for the `Beta.Files.List` and `Beta.Files.Download` Go params types.
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
```go
|
||||
// List environments
|
||||
environments, err := client.Beta.Environments.List(ctx, anthropic.BetaEnvironmentListParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Retrieve a specific environment
|
||||
env, err := client.Beta.Environments.Get(ctx, environment.ID, anthropic.BetaEnvironmentGetParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Archive an environment (read-only, existing sessions continue)
|
||||
_, err = client.Beta.Environments.Archive(ctx, environment.ID, anthropic.BetaEnvironmentArchiveParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Delete an environment (only if no sessions reference it)
|
||||
_, err = client.Beta.Environments.Delete(ctx, environment.ID, anthropic.BetaEnvironmentDeleteParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Delete a session
|
||||
_, err = client.Beta.Sessions.Delete(ctx, session.ID, anthropic.BetaSessionDeleteParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Integration
|
||||
|
||||
```go
|
||||
// Agent declares MCP server (no auth here — auth goes in a vault)
|
||||
agent, err := client.Beta.Agents.New(ctx, anthropic.BetaAgentNewParams{
|
||||
Name: "GitHub Assistant",
|
||||
Model: anthropic.BetaManagedAgentsModelConfigParams{
|
||||
ID: "claude-opus-4-8",
|
||||
Type: anthropic.BetaManagedAgentsModelConfigParamsTypeModelConfig,
|
||||
},
|
||||
MCPServers: []anthropic.BetaManagedAgentsUrlmcpServerParams{{
|
||||
Type: anthropic.BetaManagedAgentsUrlmcpServerParamsTypeURL,
|
||||
Name: "github",
|
||||
URL: "https://api.githubcopilot.com/mcp/",
|
||||
}},
|
||||
Tools: []anthropic.BetaAgentNewParamsToolUnion{
|
||||
{
|
||||
OfAgentToolset20260401: &anthropic.BetaManagedAgentsAgentToolset20260401Params{
|
||||
Type: anthropic.BetaManagedAgentsAgentToolset20260401ParamsTypeAgentToolset20260401,
|
||||
},
|
||||
},
|
||||
{
|
||||
OfMCPToolset: &anthropic.BetaManagedAgentsMCPToolsetParams{
|
||||
Type: anthropic.BetaManagedAgentsMCPToolsetParamsTypeMCPToolset,
|
||||
MCPServerName: "github",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Session attaches vault(s) containing credentials for those MCP server URLs
|
||||
session, err := client.Beta.Sessions.New(ctx, anthropic.BetaSessionNewParams{
|
||||
Agent: anthropic.BetaSessionNewParamsAgentUnion{
|
||||
OfBetaManagedAgentsAgents: &anthropic.BetaManagedAgentsAgentParams{
|
||||
Type: anthropic.BetaManagedAgentsAgentParamsTypeAgent,
|
||||
ID: agent.ID,
|
||||
Version: anthropic.Int(agent.Version),
|
||||
},
|
||||
},
|
||||
EnvironmentID: environment.ID,
|
||||
VaultIDs: []string{vault.ID},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
See `shared/managed-agents-tools.md` §Vaults for creating vaults and adding credentials.
|
||||
|
||||
---
|
||||
|
||||
## Vaults
|
||||
|
||||
```go
|
||||
// Create a vault
|
||||
vault, err := client.Beta.Vaults.New(ctx, anthropic.BetaVaultNewParams{
|
||||
DisplayName: "Alice",
|
||||
Metadata: map[string]string{"external_user_id": "usr_abc123"},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Add an OAuth credential
|
||||
credential, err := client.Beta.Vaults.Credentials.New(ctx, vault.ID, anthropic.BetaVaultCredentialNewParams{
|
||||
DisplayName: anthropic.String("Alice's Slack"),
|
||||
Auth: anthropic.BetaVaultCredentialNewParamsAuthUnion{
|
||||
OfMCPOAuth: &anthropic.BetaManagedAgentsMCPOAuthCreateParams{
|
||||
Type: anthropic.BetaManagedAgentsMCPOAuthCreateParamsTypeMCPOAuth,
|
||||
MCPServerURL: "https://mcp.slack.com/mcp",
|
||||
AccessToken: "xoxp-...",
|
||||
ExpiresAt: anthropic.Time(time.Date(2026, time.April, 15, 0, 0, 0, 0, time.UTC)),
|
||||
Refresh: anthropic.BetaManagedAgentsMCPOAuthRefreshParams{
|
||||
TokenEndpoint: "https://slack.com/api/oauth.v2.access",
|
||||
ClientID: "1234567890.0987654321",
|
||||
Scope: anthropic.String("channels:read chat:write"),
|
||||
RefreshToken: "xoxe-1-...",
|
||||
TokenEndpointAuth: anthropic.BetaManagedAgentsMCPOAuthRefreshParamsTokenEndpointAuthUnion{
|
||||
OfClientSecretPost: &anthropic.BetaManagedAgentsTokenEndpointAuthPostParam{
|
||||
Type: anthropic.BetaManagedAgentsTokenEndpointAuthPostParamTypeClientSecretPost,
|
||||
ClientSecret: "abc123...",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Rotate the credential (e.g., after a token refresh)
|
||||
_, err = client.Beta.Vaults.Credentials.Update(ctx, credential.ID, anthropic.BetaVaultCredentialUpdateParams{
|
||||
VaultID: vault.ID,
|
||||
Auth: anthropic.BetaVaultCredentialUpdateParamsAuthUnion{
|
||||
OfMCPOAuth: &anthropic.BetaManagedAgentsMCPOAuthUpdateParams{
|
||||
Type: anthropic.BetaManagedAgentsMCPOAuthUpdateParamsTypeMCPOAuth,
|
||||
AccessToken: anthropic.String("xoxp-new-..."),
|
||||
ExpiresAt: anthropic.Time(time.Date(2026, time.May, 15, 0, 0, 0, 0, time.UTC)),
|
||||
Refresh: anthropic.BetaManagedAgentsMCPOAuthRefreshUpdateParams{
|
||||
RefreshToken: anthropic.String("xoxe-1-new-..."),
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Archive a vault
|
||||
_, err = client.Beta.Vaults.Archive(ctx, vault.ID, anthropic.BetaVaultArchiveParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GitHub Repository Integration
|
||||
|
||||
Mount a GitHub repository as a session resource (a vault holds the GitHub MCP credential):
|
||||
|
||||
```go
|
||||
session, err := client.Beta.Sessions.New(ctx, anthropic.BetaSessionNewParams{
|
||||
Agent: anthropic.BetaSessionNewParamsAgentUnion{OfString: anthropic.String(agent.ID)},
|
||||
EnvironmentID: environment.ID,
|
||||
VaultIDs: []string{vault.ID},
|
||||
Resources: []anthropic.BetaSessionNewParamsResourceUnion{
|
||||
{
|
||||
OfGitHubRepository: &anthropic.BetaManagedAgentsGitHubRepositoryResourceParams{
|
||||
Type: anthropic.BetaManagedAgentsGitHubRepositoryResourceParamsTypeGitHubRepository,
|
||||
URL: "https://github.com/org/repo",
|
||||
MountPath: anthropic.String("/workspace/repo"),
|
||||
AuthorizationToken: "ghp_your_github_token",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
Multiple repositories on the same session:
|
||||
|
||||
```go
|
||||
resources := []anthropic.BetaSessionNewParamsResourceUnion{
|
||||
{
|
||||
OfGitHubRepository: &anthropic.BetaManagedAgentsGitHubRepositoryResourceParams{
|
||||
Type: anthropic.BetaManagedAgentsGitHubRepositoryResourceParamsTypeGitHubRepository,
|
||||
URL: "https://github.com/org/frontend",
|
||||
MountPath: anthropic.String("/workspace/frontend"),
|
||||
AuthorizationToken: "ghp_your_github_token",
|
||||
},
|
||||
},
|
||||
{
|
||||
OfGitHubRepository: &anthropic.BetaManagedAgentsGitHubRepositoryResourceParams{
|
||||
Type: anthropic.BetaManagedAgentsGitHubRepositoryResourceParamsTypeGitHubRepository,
|
||||
URL: "https://github.com/org/backend",
|
||||
MountPath: anthropic.String("/workspace/backend"),
|
||||
AuthorizationToken: "ghp_your_github_token",
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Rotating a repository's authorization token:
|
||||
|
||||
```go
|
||||
listed, err := client.Beta.Sessions.Resources.List(ctx, session.ID, anthropic.BetaSessionResourceListParams{})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
repoResourceID := listed.Data[0].ID
|
||||
|
||||
_, err = client.Beta.Sessions.Resources.Update(ctx, repoResourceID, anthropic.BetaSessionResourceUpdateParams{
|
||||
SessionID: session.ID,
|
||||
AuthorizationToken: "ghp_your_new_github_token",
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
@@ -1,461 +0,0 @@
|
||||
# Claude API — Java
|
||||
|
||||
> **Note:** The Java SDK supports the Claude API and beta tool use with annotated classes. Agent SDK is not yet available for Java.
|
||||
|
||||
## Installation
|
||||
|
||||
Maven:
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.anthropic</groupId>
|
||||
<artifactId>anthropic-java</artifactId>
|
||||
<version>2.34.0</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
Gradle:
|
||||
|
||||
```groovy
|
||||
implementation("com.anthropic:anthropic-java:2.34.0")
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```java
|
||||
import com.anthropic.client.AnthropicClient;
|
||||
import com.anthropic.client.okhttp.AnthropicOkHttpClient;
|
||||
|
||||
// Default (reads ANTHROPIC_API_KEY from environment)
|
||||
AnthropicClient client = AnthropicOkHttpClient.fromEnv();
|
||||
|
||||
// Explicit API key
|
||||
AnthropicClient client = AnthropicOkHttpClient.builder()
|
||||
.apiKey("your-api-key")
|
||||
.build();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.MessageCreateParams;
|
||||
import com.anthropic.models.messages.Message;
|
||||
import com.anthropic.models.messages.Model;
|
||||
|
||||
MessageCreateParams params = MessageCreateParams.builder()
|
||||
.model(Model.CLAUDE_OPUS_4_6)
|
||||
.maxTokens(16000L)
|
||||
.addUserMessage("What is the capital of France?")
|
||||
.build();
|
||||
|
||||
Message response = client.messages().create(params);
|
||||
response.content().stream()
|
||||
.flatMap(block -> block.text().stream())
|
||||
.forEach(textBlock -> System.out.println(textBlock.text()));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming
|
||||
|
||||
```java
|
||||
import com.anthropic.core.http.StreamResponse;
|
||||
import com.anthropic.models.messages.RawMessageStreamEvent;
|
||||
|
||||
MessageCreateParams params = MessageCreateParams.builder()
|
||||
.model(Model.CLAUDE_OPUS_4_6)
|
||||
.maxTokens(64000L)
|
||||
.addUserMessage("Write a haiku")
|
||||
.build();
|
||||
|
||||
try (StreamResponse<RawMessageStreamEvent> streamResponse = client.messages().createStreaming(params)) {
|
||||
streamResponse.stream()
|
||||
.flatMap(event -> event.contentBlockDelta().stream())
|
||||
.flatMap(deltaEvent -> deltaEvent.delta().text().stream())
|
||||
.forEach(textDelta -> System.out.print(textDelta.text()));
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Thinking
|
||||
|
||||
**Adaptive thinking is the recommended mode for Claude 4.6+ models.** Claude decides dynamically when and how much to think. The builder has a direct `.thinking(ThinkingConfigAdaptive)` overload — no manual union wrapping.
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.ContentBlock;
|
||||
import com.anthropic.models.messages.MessageCreateParams;
|
||||
import com.anthropic.models.messages.Model;
|
||||
import com.anthropic.models.messages.ThinkingConfigAdaptive;
|
||||
|
||||
MessageCreateParams params = MessageCreateParams.builder()
|
||||
.model(Model.CLAUDE_SONNET_4_6)
|
||||
.maxTokens(16000L)
|
||||
.thinking(ThinkingConfigAdaptive.builder().build())
|
||||
.addUserMessage("Solve this step by step: 27 * 453")
|
||||
.build();
|
||||
|
||||
for (ContentBlock block : client.messages().create(params).content()) {
|
||||
block.thinking().ifPresent(t -> System.out.println("[thinking] " + t.thinking()));
|
||||
block.text().ifPresent(t -> System.out.println(t.text()));
|
||||
}
|
||||
```
|
||||
|
||||
> **Deprecated:** `ThinkingConfigEnabled.builder().budgetTokens(N)` (and the `.enabledThinking(N)` shortcut) still works on Claude 4.6 but is deprecated. Use adaptive thinking above.
|
||||
|
||||
`ContentBlock` narrowing: `.thinking()` / `.text()` return `Optional<T>` — use `.ifPresent(...)` or `.stream().flatMap(...)`. Alternative: `isThinking()` / `asThinking()` boolean+unwrap pairs (throws on wrong variant).
|
||||
|
||||
---
|
||||
|
||||
## Tool Use (Beta)
|
||||
|
||||
The Java SDK supports beta tool use with annotated classes. Tool classes implement `Supplier<String>` for automatic execution via `BetaToolRunner`.
|
||||
|
||||
### Tool Runner (automatic loop)
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.messages.MessageCreateParams;
|
||||
import com.anthropic.models.beta.messages.BetaMessage;
|
||||
import com.anthropic.helpers.BetaToolRunner;
|
||||
import com.fasterxml.jackson.annotation.JsonClassDescription;
|
||||
import com.fasterxml.jackson.annotation.JsonPropertyDescription;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
@JsonClassDescription("Get the weather in a given location")
|
||||
static class GetWeather implements Supplier<String> {
|
||||
@JsonPropertyDescription("The city and state, e.g. San Francisco, CA")
|
||||
public String location;
|
||||
|
||||
@Override
|
||||
public String get() {
|
||||
return "The weather in " + location + " is sunny and 72°F";
|
||||
}
|
||||
}
|
||||
|
||||
BetaToolRunner toolRunner = client.beta().messages().toolRunner(
|
||||
MessageCreateParams.builder()
|
||||
.model("claude-opus-4-8")
|
||||
.maxTokens(16000L)
|
||||
.putAdditionalHeader("anthropic-beta", "structured-outputs-2025-11-13")
|
||||
.addTool(GetWeather.class)
|
||||
.addUserMessage("What's the weather in San Francisco?")
|
||||
.build());
|
||||
|
||||
for (BetaMessage message : toolRunner) {
|
||||
System.out.println(message);
|
||||
}
|
||||
```
|
||||
|
||||
### Memory Tool
|
||||
|
||||
The Java SDK provides `BetaMemoryToolHandler` for implementing the memory tool backend. You supply a handler that manages file storage, and the `BetaToolRunner` handles memory tool calls automatically.
|
||||
|
||||
```java
|
||||
import com.anthropic.helpers.BetaMemoryToolHandler;
|
||||
import com.anthropic.helpers.BetaToolRunner;
|
||||
import com.anthropic.models.beta.messages.BetaMemoryTool20250818;
|
||||
import com.anthropic.models.beta.messages.BetaMessage;
|
||||
import com.anthropic.models.beta.messages.MessageCreateParams;
|
||||
import com.anthropic.models.beta.messages.ToolRunnerCreateParams;
|
||||
|
||||
// Implement BetaMemoryToolHandler with your storage backend (e.g., filesystem)
|
||||
BetaMemoryToolHandler memoryHandler = new FileSystemMemoryToolHandler(sandboxRoot);
|
||||
|
||||
MessageCreateParams createParams = MessageCreateParams.builder()
|
||||
.model("claude-opus-4-8")
|
||||
.maxTokens(4096L)
|
||||
.addTool(BetaMemoryTool20250818.builder().build())
|
||||
.addUserMessage("Remember that my favorite color is blue")
|
||||
.build();
|
||||
|
||||
BetaToolRunner toolRunner = client.beta().messages().toolRunner(
|
||||
ToolRunnerCreateParams.builder()
|
||||
.betaMemoryToolHandler(memoryHandler)
|
||||
.initialMessageParams(createParams)
|
||||
.build());
|
||||
|
||||
for (BetaMessage message : toolRunner) {
|
||||
System.out.println(message);
|
||||
}
|
||||
```
|
||||
|
||||
See the [shared memory tool concepts](../shared/tool-use-concepts.md) for more details on the memory tool.
|
||||
|
||||
### Non-Beta Tool Declaration (manual JSON schema)
|
||||
|
||||
`Tool.InputSchema.Properties` is a freeform `Map<String, JsonValue>` wrapper — build property schemas via `putAdditionalProperty`. `type: "object"` is the default. The builder has a direct `.addTool(Tool)` overload that wraps in `ToolUnion` automatically.
|
||||
|
||||
```java
|
||||
import com.anthropic.core.JsonValue;
|
||||
import com.anthropic.models.messages.Tool;
|
||||
|
||||
Tool tool = Tool.builder()
|
||||
.name("get_weather")
|
||||
.description("Get the current weather in a given location")
|
||||
.inputSchema(Tool.InputSchema.builder()
|
||||
.properties(Tool.InputSchema.Properties.builder()
|
||||
.putAdditionalProperty("location", JsonValue.from(Map.of("type", "string")))
|
||||
.build())
|
||||
.required(List.of("location"))
|
||||
.build())
|
||||
.build();
|
||||
|
||||
MessageCreateParams params = MessageCreateParams.builder()
|
||||
.model(Model.CLAUDE_SONNET_4_6)
|
||||
.maxTokens(16000L)
|
||||
.addTool(tool)
|
||||
.addUserMessage("Weather in Paris?")
|
||||
.build();
|
||||
```
|
||||
|
||||
For manual tool loops, handle `tool_use` blocks in the response, send `tool_result` back, loop until `stop_reason` is `"end_turn"`. See [shared tool use concepts](../shared/tool-use-concepts.md).
|
||||
|
||||
### Building `MessageParam` with Content Blocks (Tool Result Round-Trip)
|
||||
|
||||
`MessageParam.Content` is an inner union class (string | list). Use the builder's `.contentOfBlockParams(List<ContentBlockParam>)` alias — there is NO separate `MessageParamContent` class with a static `ofBlockParams`:
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.MessageParam;
|
||||
import com.anthropic.models.messages.ContentBlockParam;
|
||||
import com.anthropic.models.messages.ToolResultBlockParam;
|
||||
|
||||
List<ContentBlockParam> results = List.of(
|
||||
ContentBlockParam.ofToolResult(ToolResultBlockParam.builder()
|
||||
.toolUseId(toolUseBlock.id())
|
||||
.content(yourResultString)
|
||||
.build())
|
||||
);
|
||||
|
||||
MessageParam toolResultMsg = MessageParam.builder()
|
||||
.role(MessageParam.Role.USER)
|
||||
.contentOfBlockParams(results) // builder alias for Content.ofBlockParams(...)
|
||||
.build();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Effort Parameter
|
||||
|
||||
Effort is nested inside `OutputConfig` — there is NO `.effort()` directly on `MessageCreateParams.Builder`.
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.OutputConfig;
|
||||
|
||||
.outputConfig(OutputConfig.builder()
|
||||
.effort(OutputConfig.Effort.HIGH) // or LOW, MEDIUM, MAX
|
||||
.build())
|
||||
```
|
||||
|
||||
Combine with `Thinking = ThinkingConfigAdaptive` for cost-quality control.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
System message as a list of `TextBlockParam` with `CacheControlEphemeral`. Use `.systemOfTextBlockParams(...)` — the plain `.system(String)` overload can't carry cache control. For placement patterns and the silent-invalidator audit checklist, see `shared/prompt-caching.md`.
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.TextBlockParam;
|
||||
import com.anthropic.models.messages.CacheControlEphemeral;
|
||||
|
||||
.systemOfTextBlockParams(List.of(
|
||||
TextBlockParam.builder()
|
||||
.text(longSystemPrompt)
|
||||
.cacheControl(CacheControlEphemeral.builder()
|
||||
.ttl(CacheControlEphemeral.Ttl.TTL_1H) // optional; also TTL_5M
|
||||
.build())
|
||||
.build()))
|
||||
```
|
||||
|
||||
There's also a top-level `.cacheControl(CacheControlEphemeral)` on `MessageCreateParams.Builder` and on `Tool.builder()`.
|
||||
|
||||
Verify hits via `response.usage().cacheCreationInputTokens()` / `response.usage().cacheReadInputTokens()`.
|
||||
|
||||
---
|
||||
|
||||
## Token Counting
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.MessageCountTokensParams;
|
||||
|
||||
long tokens = client.messages().countTokens(
|
||||
MessageCountTokensParams.builder()
|
||||
.model(Model.CLAUDE_SONNET_4_6)
|
||||
.addUserMessage("Hello")
|
||||
.build()
|
||||
).inputTokens();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Structured Output
|
||||
|
||||
The class-based overload auto-derives the JSON schema from your POJO and gives you a typed `.text()` return — no manual schema, no manual parsing.
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.StructuredMessageCreateParams;
|
||||
|
||||
record Book(String title, String author) {}
|
||||
record BookList(List<Book> books) {}
|
||||
|
||||
StructuredMessageCreateParams<BookList> params = MessageCreateParams.builder()
|
||||
.model(Model.CLAUDE_SONNET_4_6)
|
||||
.maxTokens(16000L)
|
||||
.outputConfig(BookList.class) // returns a typed builder
|
||||
.addUserMessage("List 3 classic novels")
|
||||
.build();
|
||||
|
||||
client.messages().create(params).content().stream()
|
||||
.flatMap(cb -> cb.text().stream())
|
||||
.forEach(typed -> {
|
||||
// typed.text() returns BookList, not String
|
||||
for (Book b : typed.text().books()) System.out.println(b.title());
|
||||
});
|
||||
```
|
||||
|
||||
Supports Jackson annotations: `@JsonPropertyDescription`, `@JsonIgnore`, `@ArraySchema(minItems=...)`. Manual schema path: `OutputConfig.builder().format(JsonOutputFormat.builder().schema(...).build())`.
|
||||
|
||||
---
|
||||
|
||||
## PDF / Document Input
|
||||
|
||||
`DocumentBlockParam` builder has source shortcuts. Wrap in `ContentBlockParam.ofDocument()` and pass via `.addUserMessageOfBlockParams()`.
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.DocumentBlockParam;
|
||||
import com.anthropic.models.messages.ContentBlockParam;
|
||||
import com.anthropic.models.messages.TextBlockParam;
|
||||
|
||||
DocumentBlockParam doc = DocumentBlockParam.builder()
|
||||
.base64Source(base64String) // or .urlSource("https://...") or .textSource("...")
|
||||
.title("My Document") // optional
|
||||
.build();
|
||||
|
||||
.addUserMessageOfBlockParams(List.of(
|
||||
ContentBlockParam.ofDocument(doc),
|
||||
ContentBlockParam.ofText(TextBlockParam.builder().text("Summarize this").build())))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools
|
||||
|
||||
Version-suffixed types; `name`/`type` auto-set by builder. Direct `.addTool()` overloads exist for every type — no manual `ToolUnion` wrapping.
|
||||
|
||||
```java
|
||||
import com.anthropic.models.messages.WebSearchTool20260209;
|
||||
import com.anthropic.models.messages.ToolBash20250124;
|
||||
import com.anthropic.models.messages.ToolTextEditor20250728;
|
||||
import com.anthropic.models.messages.CodeExecutionTool20260120;
|
||||
|
||||
.addTool(WebSearchTool20260209.builder()
|
||||
.maxUses(5L) // optional
|
||||
.allowedDomains(List.of("example.com")) // optional
|
||||
.build())
|
||||
.addTool(ToolBash20250124.builder().build())
|
||||
.addTool(ToolTextEditor20250728.builder().build())
|
||||
.addTool(CodeExecutionTool20260120.builder().build())
|
||||
```
|
||||
|
||||
Also available: `WebFetchTool20260209`, `MemoryTool20250818`, `ToolSearchToolBm25_20251119`. For the advisor tool, use `BetaAdvisorTool20260301` in the beta namespace.
|
||||
|
||||
### Beta namespace (MCP, compaction)
|
||||
|
||||
For beta-only features use `com.anthropic.models.beta.messages.*` — class names have a `Beta` prefix AND live in the beta package. The beta `MessageCreateParams.Builder` has direct `.addTool(BetaToolBash20250124)` overloads AND `.addMcpServer()`:
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.messages.MessageCreateParams;
|
||||
import com.anthropic.models.beta.messages.BetaToolBash20250124;
|
||||
import com.anthropic.models.beta.messages.BetaCodeExecutionTool20260120;
|
||||
import com.anthropic.models.beta.messages.BetaRequestMcpServerUrlDefinition;
|
||||
|
||||
MessageCreateParams params = MessageCreateParams.builder()
|
||||
.model(Model.CLAUDE_OPUS_4_6)
|
||||
.maxTokens(16000L)
|
||||
.addBeta("mcp-client-2025-11-20")
|
||||
.addTool(BetaToolBash20250124.builder().build())
|
||||
.addTool(BetaCodeExecutionTool20260120.builder().build())
|
||||
.addMcpServer(BetaRequestMcpServerUrlDefinition.builder()
|
||||
.name("my-server")
|
||||
.url("https://example.com/mcp")
|
||||
.build())
|
||||
.addUserMessage("...")
|
||||
.build();
|
||||
|
||||
client.beta().messages().create(params);
|
||||
```
|
||||
|
||||
`BetaTool*` types are NOT interchangeable with non-beta `Tool*` — pick one namespace per request.
|
||||
|
||||
**Reading server-tool blocks in the response:** `ServerToolUseBlock` has `.id()`, `.name()` (enum), and `._input()` returning raw `JsonValue` — there is NO typed `.input()`. For code execution results, unwrap two levels:
|
||||
|
||||
```java
|
||||
for (ContentBlock block : response.content()) {
|
||||
block.serverToolUse().ifPresent(stu -> {
|
||||
System.out.println("tool: " + stu.name() + " input: " + stu._input());
|
||||
});
|
||||
block.codeExecutionToolResult().ifPresent(r -> {
|
||||
r.content().resultBlock().ifPresent(result -> {
|
||||
System.out.println("stdout: " + result.stdout());
|
||||
System.out.println("stderr: " + result.stderr());
|
||||
System.out.println("exit: " + result.returnCode());
|
||||
});
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stop Details
|
||||
|
||||
When `stopReason()` is `"refusal"`, the response includes structured `stopDetails()`:
|
||||
|
||||
```java
|
||||
response.stopDetails().ifPresent(details -> {
|
||||
System.out.println("Category: " + details.category());
|
||||
System.out.println("Explanation: " + details.explanation());
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Type
|
||||
|
||||
`AnthropicServiceException` exposes `.errorType()` returning `Optional<ErrorType>` for programmatic error classification:
|
||||
|
||||
```java
|
||||
try {
|
||||
client.messages().create(params);
|
||||
} catch (AnthropicServiceException e) {
|
||||
e.errorType().ifPresent(type ->
|
||||
System.out.println("Error type: " + type) // RATE_LIMIT_ERROR, OVERLOADED_ERROR, etc.
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files API (Beta)
|
||||
|
||||
Under `client.beta().files()`. File references in messages need the beta message types (non-beta `DocumentBlockParam.Source` has no file-ID variant).
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.files.FileUploadParams;
|
||||
import com.anthropic.models.beta.files.FileMetadata;
|
||||
import com.anthropic.models.beta.messages.BetaRequestDocumentBlock;
|
||||
import java.nio.file.Paths;
|
||||
|
||||
FileMetadata meta = client.beta().files().upload(
|
||||
FileUploadParams.builder()
|
||||
.file(Paths.get("/path/to/doc.pdf")) // or .file(InputStream) or .file(byte[])
|
||||
.build());
|
||||
|
||||
// Reference in a beta message:
|
||||
BetaRequestDocumentBlock doc = BetaRequestDocumentBlock.builder()
|
||||
.fileSource(meta.id())
|
||||
.build();
|
||||
```
|
||||
|
||||
Other methods: `.list()`, `.delete(String fileId)`, `.download(String fileId)`, `.retrieveMetadata(String fileId)`.
|
||||
@@ -1,442 +0,0 @@
|
||||
# Managed Agents — Java
|
||||
|
||||
> **Bindings not shown here:** This README covers the most common managed-agents flows for Java. If you need a class, method, namespace, field, or behavior that isn't shown, WebFetch the Java SDK repo **or the relevant docs page** from `shared/live-sources.md` rather than guess. Do not extrapolate from cURL shapes or another language's SDK.
|
||||
|
||||
> **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `client.beta().agents().create` and pass it to every subsequent `client.beta().sessions().create`; do not call `agents().create` in the request path. The Anthropic CLI is one convenient way to create agents and environments from version-controlled YAML — its URL is in `shared/live-sources.md`. The examples below show in-code creation for completeness; in production the create call belongs in setup, not in the request path.
|
||||
|
||||
## Installation
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.anthropic</groupId>
|
||||
<artifactId>anthropic-java</artifactId>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```java
|
||||
import com.anthropic.client.okhttp.AnthropicOkHttpClient;
|
||||
|
||||
// Default (uses ANTHROPIC_API_KEY env var)
|
||||
var client = AnthropicOkHttpClient.fromEnv();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Environment
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.environments.BetaCloudConfigParams;
|
||||
import com.anthropic.models.beta.environments.EnvironmentCreateParams;
|
||||
import com.anthropic.models.beta.environments.UnrestrictedNetwork;
|
||||
|
||||
var environment = client.beta().environments().create(EnvironmentCreateParams.builder()
|
||||
.name("my-dev-env")
|
||||
.config(BetaCloudConfigParams.builder()
|
||||
.networking(UnrestrictedNetwork.builder().build())
|
||||
.build())
|
||||
.build());
|
||||
System.out.println("Environment ID: " + environment.id()); // env_...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Agent (required first step)
|
||||
|
||||
> ⚠️ **There is no inline agent config.** Model, system, and tools live on the agent object, not the session. Always start with `client.beta().agents().create()` — the session takes either `.agent(agent.id())` or the typed `BetaManagedAgentsAgentParams.builder()...build()`.
|
||||
|
||||
### Minimal
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.agents.AgentCreateParams;
|
||||
import com.anthropic.models.beta.agents.BetaManagedAgentsAgentToolset20260401Params;
|
||||
import com.anthropic.models.beta.sessions.BetaManagedAgentsAgentParams;
|
||||
import com.anthropic.models.beta.sessions.SessionCreateParams;
|
||||
|
||||
// 1. Create the agent (reusable, versioned)
|
||||
var agent = client.beta().agents().create(AgentCreateParams.builder()
|
||||
.name("Coding Assistant")
|
||||
.model("claude-opus-4-8")
|
||||
.system("You are a helpful coding assistant.")
|
||||
.addTool(BetaManagedAgentsAgentToolset20260401Params.builder()
|
||||
.type(BetaManagedAgentsAgentToolset20260401Params.Type.AGENT_TOOLSET_20260401)
|
||||
.build())
|
||||
.build());
|
||||
|
||||
// 2. Start a session
|
||||
var session = client.beta().sessions().create(SessionCreateParams.builder()
|
||||
.agent(BetaManagedAgentsAgentParams.builder()
|
||||
.type(BetaManagedAgentsAgentParams.Type.AGENT)
|
||||
.id(agent.id())
|
||||
.version(agent.version())
|
||||
.build())
|
||||
.environmentId(environment.id())
|
||||
.title("Quickstart session")
|
||||
.build());
|
||||
System.out.println("Session ID: " + session.id());
|
||||
```
|
||||
|
||||
### Updating an Agent
|
||||
|
||||
Updates create new versions; the agent object is immutable per version.
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.agents.AgentUpdateParams;
|
||||
|
||||
var updatedAgent = client.beta().agents().update(agent.id(), AgentUpdateParams.builder()
|
||||
.version(agent.version())
|
||||
.system("You are a helpful coding agent. Always write tests.")
|
||||
.build());
|
||||
System.out.println("New version: " + updatedAgent.version());
|
||||
|
||||
// List all versions
|
||||
for (var version : client.beta().agents().versions().list(agent.id()).autoPager()) {
|
||||
System.out.println("Version " + version.version() + ": " + version.updatedAt());
|
||||
}
|
||||
|
||||
// Archive the agent
|
||||
var archived = client.beta().agents().archive(agent.id());
|
||||
System.out.println("Archived at: " + archived.archivedAt().orElseThrow());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Send a User Message
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.sessions.events.BetaManagedAgentsUserMessageEventParams;
|
||||
import com.anthropic.models.beta.sessions.events.EventSendParams;
|
||||
|
||||
client.beta().sessions().events().send(session.id(), EventSendParams.builder()
|
||||
.addEvent(BetaManagedAgentsUserMessageEventParams.builder()
|
||||
.type(BetaManagedAgentsUserMessageEventParams.Type.USER_MESSAGE)
|
||||
.addTextContent("Review the auth module")
|
||||
.build())
|
||||
.build());
|
||||
```
|
||||
|
||||
> 💡 **Stream-first:** Open the stream *before* (or concurrently with) sending the message. The stream only delivers events that occur after it opens — stream-after-send means early events arrive buffered in one batch. See [Steering Patterns](../../shared/managed-agents-events.md#steering-patterns).
|
||||
|
||||
---
|
||||
|
||||
## Stream Events (SSE)
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.sessions.events.StreamEvents;
|
||||
|
||||
// Open the stream first, then send the user message
|
||||
try (var stream = client.beta().sessions().events().streamStreaming(session.id())) {
|
||||
client.beta().sessions().events().send(session.id(), EventSendParams.builder()
|
||||
.addEvent(BetaManagedAgentsUserMessageEventParams.builder()
|
||||
.type(BetaManagedAgentsUserMessageEventParams.Type.USER_MESSAGE)
|
||||
.addTextContent("Summarize the repo README")
|
||||
.build())
|
||||
.build());
|
||||
|
||||
for (var event : (Iterable<StreamEvents>) stream.stream()::iterator) {
|
||||
if (event.isAgentMessage()) {
|
||||
event.asAgentMessage().content().forEach(block -> System.out.print(block.text()));
|
||||
} else if (event.isAgentToolUse()) {
|
||||
System.out.println("\n[Using tool: " + event.asAgentToolUse().name() + "]");
|
||||
} else if (event.isSessionStatusIdle()) {
|
||||
break;
|
||||
} else if (event.isSessionError()) {
|
||||
System.out.println("\n[Error]");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Reconnecting and Tailing
|
||||
|
||||
When reconnecting mid-session, list past events first to dedupe, then tail live events. The cross-variant `id` field is read from the raw `_json()` value:
|
||||
|
||||
```java
|
||||
import com.anthropic.core.JsonValue;
|
||||
import java.util.HashSet;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
|
||||
try (var stream = client.beta().sessions().events().streamStreaming(session.id())) {
|
||||
// Stream is open and buffering. List history before tailing live.
|
||||
var seenEventIds = new HashSet<String>();
|
||||
for (var past : client.beta().sessions().events().list(session.id()).autoPager()) {
|
||||
Optional<Map<String, JsonValue>> obj = past._json().orElseThrow().asObject();
|
||||
seenEventIds.add(obj.orElseThrow().get("id").asStringOrThrow());
|
||||
}
|
||||
|
||||
// Tail live events, skipping anything already seen
|
||||
for (var event : (Iterable<StreamEvents>) stream.stream()::iterator) {
|
||||
Optional<Map<String, JsonValue>> obj = event._json().orElseThrow().asObject();
|
||||
if (!seenEventIds.add(obj.orElseThrow().get("id").asStringOrThrow())) continue;
|
||||
if (event.isAgentMessage()) {
|
||||
event.asAgentMessage().content().forEach(block -> System.out.print(block.text()));
|
||||
} else if (event.isSessionStatusIdle()) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provide Custom Tool Result
|
||||
|
||||
> ℹ️ The Java managed-agents bindings for `user.custom_tool_result` are not yet documented in this skill or in the apps source examples. Refer to `shared/managed-agents-events.md` for the wire format and the `anthropic-java` repository for the corresponding params types.
|
||||
|
||||
---
|
||||
|
||||
## Poll Events
|
||||
|
||||
```java
|
||||
for (var event : client.beta().sessions().events().list(session.id()).autoPager()) {
|
||||
System.out.println(event.type() + ": " + event);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.files.FileUploadParams;
|
||||
import com.anthropic.models.beta.sessions.BetaManagedAgentsFileResourceParams;
|
||||
import java.nio.file.Path;
|
||||
|
||||
var dataCsv = Path.of("data.csv");
|
||||
|
||||
var file = client.beta().files().upload(FileUploadParams.builder()
|
||||
.file(dataCsv)
|
||||
.build());
|
||||
System.out.println("File ID: " + file.id());
|
||||
|
||||
// Mount in a session
|
||||
var session = client.beta().sessions().create(SessionCreateParams.builder()
|
||||
.agent(agent.id())
|
||||
.environmentId(environment.id())
|
||||
.addResource(BetaManagedAgentsFileResourceParams.builder()
|
||||
.type(BetaManagedAgentsFileResourceParams.Type.FILE)
|
||||
.fileId(file.id())
|
||||
.mountPath("/workspace/data.csv")
|
||||
.build())
|
||||
.build());
|
||||
```
|
||||
|
||||
### Add and Manage Resources on an Existing Session
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.sessions.resources.ResourceAddParams;
|
||||
import com.anthropic.models.beta.sessions.resources.ResourceDeleteParams;
|
||||
|
||||
// Attach an additional file to an open session
|
||||
var resource = client.beta().sessions().resources().add(session.id(), ResourceAddParams.builder()
|
||||
.betaManagedAgentsFileResourceParams(BetaManagedAgentsFileResourceParams.builder()
|
||||
.type(BetaManagedAgentsFileResourceParams.Type.FILE)
|
||||
.fileId(file.id())
|
||||
.build())
|
||||
.build());
|
||||
System.out.println(resource.id()); // "sesrsc_01ABC..."
|
||||
|
||||
// List resources on the session — entries are a discriminated union
|
||||
var listed = client.beta().sessions().resources().list(session.id());
|
||||
for (var entry : listed.data()) {
|
||||
if (entry.isFile()) {
|
||||
var fileResource = entry.asFile();
|
||||
System.out.println(fileResource.id() + " " + fileResource.type());
|
||||
} else if (entry.isGitHubRepository()) {
|
||||
var repoResource = entry.asGitHubRepository();
|
||||
System.out.println(repoResource.id() + " " + repoResource.type());
|
||||
}
|
||||
}
|
||||
|
||||
// Detach a resource
|
||||
client.beta().sessions().resources().delete(resource.id(), ResourceDeleteParams.builder()
|
||||
.sessionId(session.id())
|
||||
.build());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List and Download Session Files
|
||||
|
||||
> ℹ️ Listing and downloading files an agent wrote during a session is not yet documented for Java in this skill or in the apps source examples. See `shared/managed-agents-events.md` and the `anthropic-java` repository for the file list/download bindings.
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
```java
|
||||
// List environments
|
||||
var environments = client.beta().environments().list();
|
||||
|
||||
// Retrieve a specific environment
|
||||
var env = client.beta().environments().retrieve(environment.id());
|
||||
|
||||
// Archive an environment (read-only, existing sessions continue)
|
||||
client.beta().environments().archive(environment.id());
|
||||
|
||||
// Delete an environment (only if no sessions reference it)
|
||||
client.beta().environments().delete(environment.id());
|
||||
|
||||
// Delete a session
|
||||
client.beta().sessions().delete(session.id());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Integration
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.agents.BetaManagedAgentsMcpToolsetParams;
|
||||
import com.anthropic.models.beta.agents.BetaManagedAgentsUrlmcpServerParams;
|
||||
|
||||
// Agent declares MCP server (no auth here — auth goes in a vault)
|
||||
var agent = client.beta().agents().create(AgentCreateParams.builder()
|
||||
.name("GitHub Assistant")
|
||||
.model("claude-opus-4-8")
|
||||
.addMcpServer(BetaManagedAgentsUrlmcpServerParams.builder()
|
||||
.type(BetaManagedAgentsUrlmcpServerParams.Type.URL)
|
||||
.name("github")
|
||||
.url("https://api.githubcopilot.com/mcp/")
|
||||
.build())
|
||||
.addTool(BetaManagedAgentsAgentToolset20260401Params.builder()
|
||||
.type(BetaManagedAgentsAgentToolset20260401Params.Type.AGENT_TOOLSET_20260401)
|
||||
.build())
|
||||
.addTool(BetaManagedAgentsMcpToolsetParams.builder()
|
||||
.type(BetaManagedAgentsMcpToolsetParams.Type.MCP_TOOLSET)
|
||||
.mcpServerName("github")
|
||||
.build())
|
||||
.build());
|
||||
|
||||
// Session attaches vault(s) containing credentials for those MCP server URLs
|
||||
var session = client.beta().sessions().create(SessionCreateParams.builder()
|
||||
.agent(BetaManagedAgentsAgentParams.builder()
|
||||
.type(BetaManagedAgentsAgentParams.Type.AGENT)
|
||||
.id(agent.id())
|
||||
.version(agent.version())
|
||||
.build())
|
||||
.environmentId(environment.id())
|
||||
.addVaultId(vault.id())
|
||||
.build());
|
||||
```
|
||||
|
||||
See `shared/managed-agents-tools.md` §Vaults for creating vaults and adding credentials.
|
||||
|
||||
---
|
||||
|
||||
## Vaults
|
||||
|
||||
```java
|
||||
import com.anthropic.core.JsonValue;
|
||||
import com.anthropic.models.beta.vaults.VaultCreateParams;
|
||||
import com.anthropic.models.beta.vaults.credentials.BetaManagedAgentsMcpOAuthCreateParams;
|
||||
import com.anthropic.models.beta.vaults.credentials.BetaManagedAgentsMcpOAuthRefreshParams;
|
||||
import com.anthropic.models.beta.vaults.credentials.BetaManagedAgentsMcpOAuthRefreshUpdateParams;
|
||||
import com.anthropic.models.beta.vaults.credentials.BetaManagedAgentsMcpOAuthUpdateParams;
|
||||
import com.anthropic.models.beta.vaults.credentials.CredentialCreateParams;
|
||||
import com.anthropic.models.beta.vaults.credentials.CredentialUpdateParams;
|
||||
import java.time.OffsetDateTime;
|
||||
|
||||
// Create a vault
|
||||
var vault = client.beta().vaults().create(VaultCreateParams.builder()
|
||||
.displayName("Alice")
|
||||
.metadata(VaultCreateParams.Metadata.builder()
|
||||
.putAdditionalProperty("external_user_id", JsonValue.from("usr_abc123"))
|
||||
.build())
|
||||
.build());
|
||||
System.out.println(vault.id()); // "vlt_01ABC..."
|
||||
|
||||
// Add an OAuth credential
|
||||
var credential = client.beta().vaults().credentials().create(vault.id(),
|
||||
CredentialCreateParams.builder()
|
||||
.displayName("Alice's Slack")
|
||||
.auth(BetaManagedAgentsMcpOAuthCreateParams.builder()
|
||||
.type(BetaManagedAgentsMcpOAuthCreateParams.Type.MCP_OAUTH)
|
||||
.mcpServerUrl("https://mcp.slack.com/mcp")
|
||||
.accessToken("xoxp-...")
|
||||
.expiresAt(OffsetDateTime.parse("2026-04-15T00:00:00Z"))
|
||||
.refresh(BetaManagedAgentsMcpOAuthRefreshParams.builder()
|
||||
.tokenEndpoint("https://slack.com/api/oauth.v2.access")
|
||||
.clientId("1234567890.0987654321")
|
||||
.scope("channels:read chat:write")
|
||||
.refreshToken("xoxe-1-...")
|
||||
.clientSecretPostTokenEndpointAuth("abc123...")
|
||||
.build())
|
||||
.build())
|
||||
.build());
|
||||
|
||||
// Rotate the credential (e.g., after a token refresh)
|
||||
client.beta().vaults().credentials().update(credential.id(),
|
||||
CredentialUpdateParams.builder()
|
||||
.vaultId(vault.id())
|
||||
.auth(BetaManagedAgentsMcpOAuthUpdateParams.builder()
|
||||
.type(BetaManagedAgentsMcpOAuthUpdateParams.Type.MCP_OAUTH)
|
||||
.accessToken("xoxp-new-...")
|
||||
.expiresAt(OffsetDateTime.parse("2026-05-15T00:00:00Z"))
|
||||
.refresh(BetaManagedAgentsMcpOAuthRefreshUpdateParams.builder()
|
||||
.refreshToken("xoxe-1-new-...")
|
||||
.build())
|
||||
.build())
|
||||
.build());
|
||||
|
||||
// Archive a vault
|
||||
client.beta().vaults().archive(vault.id());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GitHub Repository Integration
|
||||
|
||||
Mount a GitHub repository as a session resource (a vault holds the GitHub MCP credential):
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.sessions.BetaManagedAgentsGitHubRepositoryResourceParams;
|
||||
|
||||
var session = client.beta().sessions().create(SessionCreateParams.builder()
|
||||
.agent(agent.id())
|
||||
.environmentId(environment.id())
|
||||
.addVaultId(vault.id())
|
||||
.addResource(BetaManagedAgentsGitHubRepositoryResourceParams.builder()
|
||||
.type(BetaManagedAgentsGitHubRepositoryResourceParams.Type.GITHUB_REPOSITORY)
|
||||
.url("https://github.com/org/repo")
|
||||
.mountPath("/workspace/repo")
|
||||
.authorizationToken("ghp_your_github_token")
|
||||
.build())
|
||||
.build());
|
||||
```
|
||||
|
||||
Multiple repositories on the same session:
|
||||
|
||||
```java
|
||||
import java.util.List;
|
||||
|
||||
var resources = List.of(
|
||||
BetaManagedAgentsGitHubRepositoryResourceParams.builder()
|
||||
.type(BetaManagedAgentsGitHubRepositoryResourceParams.Type.GITHUB_REPOSITORY)
|
||||
.url("https://github.com/org/frontend")
|
||||
.mountPath("/workspace/frontend")
|
||||
.authorizationToken("ghp_your_github_token")
|
||||
.build(),
|
||||
BetaManagedAgentsGitHubRepositoryResourceParams.builder()
|
||||
.type(BetaManagedAgentsGitHubRepositoryResourceParams.Type.GITHUB_REPOSITORY)
|
||||
.url("https://github.com/org/backend")
|
||||
.mountPath("/workspace/backend")
|
||||
.authorizationToken("ghp_your_github_token")
|
||||
.build());
|
||||
```
|
||||
|
||||
Rotating a repository's authorization token:
|
||||
|
||||
```java
|
||||
import com.anthropic.models.beta.sessions.resources.ResourceUpdateParams;
|
||||
|
||||
var listed = client.beta().sessions().resources().list(session.id());
|
||||
var repoResourceId = listed.data().get(0).asGitHubRepository().id();
|
||||
|
||||
client.beta().sessions().resources().update(repoResourceId, ResourceUpdateParams.builder()
|
||||
.sessionId(session.id())
|
||||
.authorizationToken("ghp_your_new_github_token")
|
||||
.build());
|
||||
```
|
||||
@@ -1,402 +0,0 @@
|
||||
# Claude API — PHP
|
||||
|
||||
> **Note:** The PHP SDK is the official Anthropic SDK for PHP. A beta tool runner is available via `$client->beta->messages->toolRunner()`. Structured output helpers are supported via `StructuredOutputModel` classes. Agent SDK is not available. Bedrock, Vertex AI, and Foundry clients are supported.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
composer require "anthropic-ai/sdk"
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```php
|
||||
use Anthropic\Client;
|
||||
|
||||
// Using API key from environment variable
|
||||
$client = new Client(apiKey: getenv("ANTHROPIC_API_KEY"));
|
||||
```
|
||||
|
||||
### Amazon Bedrock
|
||||
|
||||
```php
|
||||
use Anthropic\Bedrock;
|
||||
|
||||
// Constructor is private — use the static factory. Reads AWS credentials from env.
|
||||
$client = Bedrock\Client::fromEnvironment(region: 'us-east-1');
|
||||
```
|
||||
|
||||
### Google Vertex AI
|
||||
|
||||
```php
|
||||
use Anthropic\Vertex;
|
||||
|
||||
// Constructor is private. Parameter is `location`, not `region`.
|
||||
$client = Vertex\Client::fromEnvironment(
|
||||
location: 'us-east5',
|
||||
projectId: 'my-project-id',
|
||||
);
|
||||
```
|
||||
|
||||
### Anthropic Foundry
|
||||
|
||||
```php
|
||||
use Anthropic\Foundry;
|
||||
|
||||
// Constructor is private. baseUrl or resource is required.
|
||||
$client = Foundry\Client::withCredentials(
|
||||
authToken: getenv('ANTHROPIC_FOUNDRY_AUTH_TOKEN'),
|
||||
baseUrl: 'https://<resource>.services.ai.azure.com/anthropic',
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```php
|
||||
$message = $client->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
messages: [
|
||||
['role' => 'user', 'content' => 'What is the capital of France?'],
|
||||
],
|
||||
);
|
||||
|
||||
// content is an array of polymorphic blocks (TextBlock, ToolUseBlock,
|
||||
// ThinkingBlock). Accessing ->text on content[0] without checking the block
|
||||
// type will throw if the first block is not a TextBlock (e.g., when extended
|
||||
// thinking is enabled and a ThinkingBlock comes first). Always guard:
|
||||
foreach ($message->content as $block) {
|
||||
if ($block->type === 'text') {
|
||||
echo $block->text;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If you only want the first text block:
|
||||
|
||||
```php
|
||||
foreach ($message->content as $block) {
|
||||
if ($block->type === 'text') {
|
||||
echo $block->text;
|
||||
break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming
|
||||
|
||||
> **Requires SDK v0.5.0+.** v0.4.0 and earlier used a single `$params` array; calling with named parameters throws `Unknown named parameter $model`. Upgrade: `composer require "anthropic-ai/sdk:^0.7"`
|
||||
|
||||
```php
|
||||
use Anthropic\Messages\RawContentBlockDeltaEvent;
|
||||
use Anthropic\Messages\TextDelta;
|
||||
|
||||
$stream = $client->messages->createStream(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 64000,
|
||||
messages: [
|
||||
['role' => 'user', 'content' => 'Write a haiku'],
|
||||
],
|
||||
);
|
||||
|
||||
foreach ($stream as $event) {
|
||||
if ($event instanceof RawContentBlockDeltaEvent && $event->delta instanceof TextDelta) {
|
||||
echo $event->delta->text;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tool Use
|
||||
|
||||
### Tool Runner (Beta)
|
||||
|
||||
**Beta:** The PHP SDK provides a tool runner via `$client->beta->messages->toolRunner()`. Define tools with `BetaRunnableTool` — a definition array plus a `run` closure:
|
||||
|
||||
```php
|
||||
use Anthropic\Lib\Tools\BetaRunnableTool;
|
||||
|
||||
$weatherTool = new BetaRunnableTool(
|
||||
definition: [
|
||||
'name' => 'get_weather',
|
||||
'description' => 'Get the current weather for a location.',
|
||||
'input_schema' => [
|
||||
'type' => 'object',
|
||||
'properties' => [
|
||||
'location' => ['type' => 'string', 'description' => 'City and state'],
|
||||
],
|
||||
'required' => ['location'],
|
||||
],
|
||||
],
|
||||
run: function (array $input): string {
|
||||
return "The weather in {$input['location']} is sunny and 72°F.";
|
||||
},
|
||||
);
|
||||
|
||||
$runner = $client->beta->messages->toolRunner(
|
||||
maxTokens: 16000,
|
||||
messages: [['role' => 'user', 'content' => 'What is the weather in Paris?']],
|
||||
model: 'claude-opus-4-8',
|
||||
tools: [$weatherTool],
|
||||
);
|
||||
|
||||
foreach ($runner as $message) {
|
||||
foreach ($message->content as $block) {
|
||||
if ($block->type === 'text') {
|
||||
echo $block->text;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Manual Loop
|
||||
|
||||
Tools are passed as arrays. **The SDK uses camelCase keys** (`inputSchema`, `toolUseID`, `stopReason`) and auto-maps to the API's snake_case on the wire — since v0.5.0. See [shared tool use concepts](../shared/tool-use-concepts.md) for the loop pattern.
|
||||
|
||||
```php
|
||||
use Anthropic\Messages\ToolUseBlock;
|
||||
|
||||
$tools = [
|
||||
[
|
||||
'name' => 'get_weather',
|
||||
'description' => 'Get the current weather in a given location',
|
||||
'inputSchema' => [ // camelCase, not input_schema
|
||||
'type' => 'object',
|
||||
'properties' => [
|
||||
'location' => ['type' => 'string', 'description' => 'City and state'],
|
||||
],
|
||||
'required' => ['location'],
|
||||
],
|
||||
],
|
||||
];
|
||||
|
||||
$messages = [['role' => 'user', 'content' => 'What is the weather in SF?']];
|
||||
|
||||
$response = $client->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
tools: $tools,
|
||||
messages: $messages,
|
||||
);
|
||||
|
||||
while ($response->stopReason === 'tool_use') { // camelCase property
|
||||
$toolResults = [];
|
||||
foreach ($response->content as $block) {
|
||||
if ($block instanceof ToolUseBlock) {
|
||||
// $block->name : string — tool name to dispatch on
|
||||
// $block->input : array<string,mixed> — parsed JSON input
|
||||
// $block->id : string — pass back as toolUseID
|
||||
$result = executeYourTool($block->name, $block->input);
|
||||
$toolResults[] = [
|
||||
'type' => 'tool_result',
|
||||
'toolUseID' => $block->id, // camelCase, not tool_use_id
|
||||
'content' => $result,
|
||||
];
|
||||
}
|
||||
}
|
||||
|
||||
// Append assistant turn + user turn with tool results
|
||||
$messages[] = ['role' => 'assistant', 'content' => $response->content];
|
||||
$messages[] = ['role' => 'user', 'content' => $toolResults];
|
||||
|
||||
$response = $client->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
tools: $tools,
|
||||
messages: $messages,
|
||||
);
|
||||
}
|
||||
|
||||
// Final text response
|
||||
foreach ($response->content as $block) {
|
||||
if ($block->type === 'text') {
|
||||
echo $block->text;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`$block->type === 'tool_use'` also works; `instanceof ToolUseBlock` narrows for PHPStan.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Extended Thinking
|
||||
|
||||
**Adaptive thinking is the recommended mode for Claude 4.6+ models.** Claude decides dynamically when and how much to think.
|
||||
|
||||
```php
|
||||
use Anthropic\Messages\ThinkingBlock;
|
||||
|
||||
$message = $client->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
thinking: ['type' => 'adaptive'],
|
||||
messages: [
|
||||
['role' => 'user', 'content' => 'Solve: 27 * 453'],
|
||||
],
|
||||
);
|
||||
|
||||
// ThinkingBlock(s) precede TextBlock in content
|
||||
foreach ($message->content as $block) {
|
||||
if ($block instanceof ThinkingBlock) {
|
||||
echo "Thinking:\n{$block->thinking}\n\n";
|
||||
// $block->signature is an opaque string — preserve verbatim if
|
||||
// passing thinking blocks back in multi-turn conversations
|
||||
} elseif ($block->type === 'text') {
|
||||
echo "Answer: {$block->text}\n";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Deprecated:** `['type' => 'enabled', 'budgetTokens' => N]` (fixed-budget extended thinking) still works on Claude 4.6 but is deprecated. Use adaptive thinking above.
|
||||
|
||||
`$block->type === 'thinking'` also works for the check; `instanceof` narrows for PHPStan.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
`system:` takes an array of text blocks; set `cacheControl` on the last block. Array-shape syntax (camelCase keys) is idiomatic. For placement patterns and the silent-invalidator audit checklist, see `shared/prompt-caching.md`.
|
||||
|
||||
```php
|
||||
$message = $client->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
system: [
|
||||
['type' => 'text', 'text' => $longSystemPrompt, 'cacheControl' => ['type' => 'ephemeral']],
|
||||
],
|
||||
messages: [['role' => 'user', 'content' => 'Summarize the key points']],
|
||||
);
|
||||
```
|
||||
|
||||
For 1-hour TTL: `'cacheControl' => ['type' => 'ephemeral', 'ttl' => '1h']`. There's also a top-level `cacheControl:` on `messages->create(...)` that auto-places on the last cacheable block.
|
||||
|
||||
Verify hits via `$message->usage->cacheCreationInputTokens` / `$message->usage->cacheReadInputTokens`.
|
||||
|
||||
---
|
||||
|
||||
## Structured Outputs
|
||||
|
||||
### Using StructuredOutputModel (Recommended)
|
||||
|
||||
Define a PHP class implementing `StructuredOutputModel` and pass it as `outputConfig`:
|
||||
|
||||
```php
|
||||
use Anthropic\Lib\Contracts\StructuredOutputModel;
|
||||
use Anthropic\Lib\Concerns\StructuredOutputModelTrait;
|
||||
use Anthropic\Lib\Attributes\Constrained;
|
||||
|
||||
class Person implements StructuredOutputModel
|
||||
{
|
||||
use StructuredOutputModelTrait;
|
||||
|
||||
#[Constrained(description: 'Full name')]
|
||||
public string $name;
|
||||
|
||||
public int $age;
|
||||
|
||||
public ?string $email = null; // nullable = optional field
|
||||
}
|
||||
|
||||
$message = $client->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
messages: [['role' => 'user', 'content' => 'Generate a profile for Alice, age 30']],
|
||||
outputConfig: ['format' => Person::class],
|
||||
);
|
||||
|
||||
$person = $message->parsedOutput(); // Person instance
|
||||
echo $person->name;
|
||||
```
|
||||
|
||||
Types are inferred from PHP type hints. Use `#[Constrained(description: '...')]` to add descriptions. Nullable properties (`?string`) become optional fields.
|
||||
|
||||
### Raw Schema
|
||||
|
||||
```php
|
||||
$message = $client->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
messages: [['role' => 'user', 'content' => 'Extract: John (john@co.com), Enterprise plan']],
|
||||
outputConfig: [
|
||||
'format' => [
|
||||
'type' => 'json_schema',
|
||||
'schema' => [
|
||||
'type' => 'object',
|
||||
'properties' => [
|
||||
'name' => ['type' => 'string'],
|
||||
'email' => ['type' => 'string'],
|
||||
'plan' => ['type' => 'string'],
|
||||
],
|
||||
'required' => ['name', 'email', 'plan'],
|
||||
'additionalProperties' => false,
|
||||
],
|
||||
],
|
||||
],
|
||||
);
|
||||
|
||||
// First text block contains valid JSON
|
||||
foreach ($message->content as $block) {
|
||||
if ($block->type === 'text') {
|
||||
$data = json_decode($block->text, true);
|
||||
break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Beta Features & Server-Side Tools
|
||||
|
||||
**`betas:` is NOT a param on `$client->messages->create()`** — it only exists on the beta namespace. Use it for features that need an explicit opt-in header:
|
||||
|
||||
```php
|
||||
use Anthropic\Beta\Messages\BetaRequestMCPServerURLDefinition;
|
||||
|
||||
$response = $client->beta->messages->create(
|
||||
model: 'claude-opus-4-8',
|
||||
maxTokens: 16000,
|
||||
mcpServers: [
|
||||
BetaRequestMCPServerURLDefinition::with(
|
||||
name: 'my-server',
|
||||
url: 'https://example.com/mcp',
|
||||
),
|
||||
],
|
||||
betas: ['mcp-client-2025-11-20'], // only valid on ->beta->messages
|
||||
messages: [['role' => 'user', 'content' => 'Use the MCP tools']],
|
||||
);
|
||||
```
|
||||
|
||||
**Server-side tools** (bash, web_search, text_editor, code_execution) are GA and work on both paths — `Anthropic\Messages\ToolBash20250124` / `WebSearchTool20260209` / `ToolTextEditor20250728` / `CodeExecutionTool20260120` for non-beta, `Anthropic\Beta\Messages\BetaToolBash20250124` / `BetaWebSearchTool20260209` / `BetaToolTextEditor20250728` / `BetaCodeExecutionTool20260120` for beta. No `betas:` header needed for these.
|
||||
|
||||
---
|
||||
|
||||
## Stop Details
|
||||
|
||||
When `stopReason` is `'refusal'`, the response includes structured `stopDetails`:
|
||||
|
||||
```php
|
||||
if ($message->stopReason === 'refusal' && $message->stopDetails !== null) {
|
||||
echo "Category: " . $message->stopDetails->category . "\n"; // "cyber" | "bio" | null
|
||||
echo "Explanation: " . $message->stopDetails->explanation . "\n";
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Type
|
||||
|
||||
`APIStatusException` exposes a `->type` property for programmatic error classification:
|
||||
|
||||
```php
|
||||
try {
|
||||
$client->messages->create(...);
|
||||
} catch (\Anthropic\Core\Exceptions\APIStatusException $e) {
|
||||
echo $e->type?->value; // "rate_limit_error", "overloaded_error", etc.
|
||||
}
|
||||
```
|
||||
@@ -1,435 +0,0 @@
|
||||
# Managed Agents — PHP
|
||||
|
||||
> **Bindings not shown here:** This README covers the most common managed-agents flows for PHP. If you need a class, method, namespace, field, or behavior that isn't shown, WebFetch the PHP SDK repo **or the relevant docs page** from `shared/live-sources.md` rather than guess. Do not extrapolate from cURL shapes or another language's SDK.
|
||||
|
||||
> **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `$client->beta->agents->create` and pass it to every subsequent `->sessions->create`; do not call `agents->create` in the request path. The Anthropic CLI is one convenient way to create agents and environments from version-controlled YAML — its URL is in `shared/live-sources.md`. The examples below show in-code creation for completeness; in production the create call belongs in setup, not in the request path.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
composer require "anthropic-ai/sdk"
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```php
|
||||
use Anthropic\Client;
|
||||
|
||||
// Default (uses ANTHROPIC_API_KEY env var)
|
||||
$client = new Client();
|
||||
|
||||
// Explicit API key
|
||||
$client = new Client(apiKey: 'your-api-key');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Environment
|
||||
|
||||
```php
|
||||
$environment = $client->beta->environments->create(
|
||||
name: 'my-dev-env',
|
||||
config: ['type' => 'cloud', 'networking' => ['type' => 'unrestricted']],
|
||||
);
|
||||
echo "Environment ID: {$environment->id}\n"; // env_...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Agent (required first step)
|
||||
|
||||
> ⚠️ **There is no inline agent config.** `model`/`system`/`tools` live on the agent object, not the session. Always start with `$client->beta->agents->create()` — the session takes either `agent: $agent->id` or the typed `BetaManagedAgentsAgentParams::with(type: 'agent', id: $agent->id, version: $agent->version)`.
|
||||
|
||||
### Minimal
|
||||
|
||||
```php
|
||||
use Anthropic\Beta\Agents\BetaManagedAgentsAgentToolset20260401Params;
|
||||
|
||||
// 1. Create the agent (reusable, versioned)
|
||||
$agent = $client->beta->agents->create(
|
||||
name: 'Coding Assistant',
|
||||
model: 'claude-opus-4-8',
|
||||
system: 'You are a helpful coding assistant.',
|
||||
tools: [
|
||||
BetaManagedAgentsAgentToolset20260401Params::with(
|
||||
type: 'agent_toolset_20260401',
|
||||
),
|
||||
],
|
||||
);
|
||||
|
||||
// 2. Start a session
|
||||
$session = $client->beta->sessions->create(
|
||||
agent: ['type' => 'agent', 'id' => $agent->id, 'version' => $agent->version],
|
||||
environmentID: $environment->id,
|
||||
title: 'Quickstart session',
|
||||
);
|
||||
echo "Session ID: {$session->id}\n";
|
||||
```
|
||||
|
||||
### Updating an Agent
|
||||
|
||||
Updates create new versions; the agent object is immutable per version.
|
||||
|
||||
```php
|
||||
$updatedAgent = $client->beta->agents->update(
|
||||
$agent->id,
|
||||
version: $agent->version,
|
||||
system: 'You are a helpful coding agent. Always write tests.',
|
||||
);
|
||||
echo "New version: {$updatedAgent->version}\n";
|
||||
|
||||
// List all versions
|
||||
foreach ($client->beta->agents->versions->list($agent->id)->pagingEachItem() as $version) {
|
||||
echo "Version {$version->version}: {$version->updatedAt->format(DateTimeInterface::ATOM)}\n";
|
||||
}
|
||||
|
||||
// Archive the agent
|
||||
$archived = $client->beta->agents->archive($agent->id);
|
||||
echo "Archived at: {$archived->archivedAt->format(DateTimeInterface::ATOM)}\n";
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Send a User Message
|
||||
|
||||
```php
|
||||
$client->beta->sessions->events->send(
|
||||
$session->id,
|
||||
events: [
|
||||
[
|
||||
'type' => 'user.message',
|
||||
'content' => [['type' => 'text', 'text' => 'Review the auth module']],
|
||||
],
|
||||
],
|
||||
);
|
||||
```
|
||||
|
||||
> 💡 **Stream-first:** Open the stream *before* (or concurrently with) sending the message. The stream only delivers events that occur after it opens — stream-after-send means early events arrive buffered in one batch. See [Steering Patterns](../../shared/managed-agents-events.md#steering-patterns).
|
||||
|
||||
---
|
||||
|
||||
## Stream Events (SSE)
|
||||
|
||||
> ℹ️ **Streaming transporter:** PHP's default buffered PSR-18 client never returns for the open-ended session event stream. Use a streaming Guzzle transporter for `streamStream()` calls — other calls keep the default client.
|
||||
|
||||
```php
|
||||
$streamingClient = new GuzzleHttp\Client(['stream' => true]);
|
||||
|
||||
// Open the stream first, then send the user message
|
||||
$stream = $client->beta->sessions->events->streamStream(
|
||||
$session->id,
|
||||
requestOptions: ['transporter' => $streamingClient],
|
||||
);
|
||||
$client->beta->sessions->events->send(
|
||||
$session->id,
|
||||
events: [
|
||||
[
|
||||
'type' => 'user.message',
|
||||
'content' => [['type' => 'text', 'text' => 'Summarize the repo README']],
|
||||
],
|
||||
],
|
||||
);
|
||||
|
||||
foreach ($stream as $event) {
|
||||
match ($event->type) {
|
||||
'agent.message' => array_walk(
|
||||
$event->content,
|
||||
static fn($block) => $block->type === 'text' ? print($block->text) : null,
|
||||
),
|
||||
'agent.tool_use' => print("\n[Using tool: {$event->name}]\n"),
|
||||
'session.error' => printf("\n[Error: %s]", $event->error?->message ?? 'unknown'),
|
||||
default => null,
|
||||
};
|
||||
if ($event->type === 'session.status_idle' || $event->type === 'session.error') {
|
||||
break;
|
||||
}
|
||||
}
|
||||
$stream->close();
|
||||
```
|
||||
|
||||
### Reconnecting and Tailing
|
||||
|
||||
When reconnecting mid-session, list past events first to dedupe, then tail live events:
|
||||
|
||||
```php
|
||||
$stream = $client->beta->sessions->events->streamStream(
|
||||
$session->id,
|
||||
requestOptions: ['transporter' => $streamingClient],
|
||||
);
|
||||
|
||||
// Stream is open and buffering. List history before tailing live.
|
||||
$seenEventIds = [];
|
||||
foreach ($client->beta->sessions->events->list($session->id)->pagingEachItem() as $event) {
|
||||
$seenEventIds[$event->id] = true;
|
||||
}
|
||||
|
||||
// Tail live events, skipping anything already seen
|
||||
foreach ($stream as $event) {
|
||||
if (isset($seenEventIds[$event->id])) {
|
||||
continue;
|
||||
}
|
||||
$seenEventIds[$event->id] = true;
|
||||
match ($event->type) {
|
||||
'agent.message' => array_walk(
|
||||
$event->content,
|
||||
static fn($block) => $block->type === 'text' ? print($block->text) : null,
|
||||
),
|
||||
default => null,
|
||||
};
|
||||
if ($event->type === 'session.status_idle') {
|
||||
break;
|
||||
}
|
||||
}
|
||||
$stream->close();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provide Custom Tool Result
|
||||
|
||||
> ℹ️ The PHP managed-agents bindings for `user.custom_tool_result` are not yet documented in this skill or in the apps source examples. Refer to `shared/managed-agents-events.md` for the wire format and the `anthropic-ai/sdk` PHP repository for the corresponding params.
|
||||
|
||||
---
|
||||
|
||||
## Poll Events
|
||||
|
||||
```php
|
||||
foreach ($client->beta->sessions->events->list($session->id)->pagingEachItem() as $event) {
|
||||
echo "{$event->type}: {$event->id}\n";
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
> ℹ️ **PHP file upload:** The PHP SDK's beta managed-agents file upload binding is not shown in the apps source examples; the canonical PHP example uses raw cURL against `POST /v1/files`. If your codebase prefers the SDK, WebFetch the `anthropic-ai/sdk` PHP repository for the latest binding before writing code.
|
||||
|
||||
```php
|
||||
use Anthropic\Beta\Sessions\BetaManagedAgentsFileResourceParams;
|
||||
|
||||
// Raw cURL upload (canonical example from the apps source)
|
||||
$csvPath = 'data.csv';
|
||||
$ch = curl_init('https://api.anthropic.com/v1/files');
|
||||
curl_setopt_array($ch, [
|
||||
CURLOPT_RETURNTRANSFER => true,
|
||||
CURLOPT_POST => true,
|
||||
CURLOPT_HTTPHEADER => [
|
||||
'x-api-key: ' . getenv('ANTHROPIC_API_KEY'),
|
||||
'anthropic-version: 2023-06-01',
|
||||
'anthropic-beta: files-api-2025-04-14',
|
||||
],
|
||||
CURLOPT_POSTFIELDS => ['file' => new CURLFile($csvPath, 'text/csv', 'data.csv')],
|
||||
]);
|
||||
$file = json_decode(curl_exec($ch));
|
||||
echo "File ID: {$file->id}\n";
|
||||
|
||||
// Mount in a session
|
||||
$session = $client->beta->sessions->create(
|
||||
agent: $agent->id,
|
||||
environmentID: $environment->id,
|
||||
resources: [
|
||||
BetaManagedAgentsFileResourceParams::with(
|
||||
type: 'file',
|
||||
fileID: $file->id,
|
||||
mountPath: '/workspace/data.csv',
|
||||
),
|
||||
],
|
||||
);
|
||||
```
|
||||
|
||||
### Add and Manage Resources on an Existing Session
|
||||
|
||||
```php
|
||||
// Attach an additional file to an open session
|
||||
$resource = $client->beta->sessions->resources->add(
|
||||
$session->id,
|
||||
type: 'file',
|
||||
fileID: $file->id,
|
||||
);
|
||||
echo "{$resource->id}\n"; // "sesrsc_01ABC..."
|
||||
|
||||
// List resources on the session
|
||||
$listed = $client->beta->sessions->resources->list($session->id);
|
||||
foreach ($listed->data as $entry) {
|
||||
echo "{$entry->id} {$entry->type}\n";
|
||||
}
|
||||
|
||||
// Detach a resource
|
||||
$client->beta->sessions->resources->delete($resource->id, sessionID: $session->id);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List and Download Session Files
|
||||
|
||||
> ℹ️ Listing and downloading files an agent wrote during a session is not yet documented for PHP in this skill or in the apps source examples. See `shared/managed-agents-events.md` and the `anthropic-ai/sdk` PHP repository for the file list/download bindings.
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
```php
|
||||
// List environments
|
||||
$environments = $client->beta->environments->list();
|
||||
|
||||
// Retrieve a specific environment
|
||||
$env = $client->beta->environments->retrieve($environment->id);
|
||||
|
||||
// Archive an environment (read-only, existing sessions continue)
|
||||
$client->beta->environments->archive($environment->id);
|
||||
|
||||
// Delete an environment (only if no sessions reference it)
|
||||
$client->beta->environments->delete($environment->id);
|
||||
|
||||
// Delete a session
|
||||
$client->beta->sessions->delete($session->id);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Integration
|
||||
|
||||
```php
|
||||
use Anthropic\Beta\Agents\BetaManagedAgentsAgentToolset20260401Params;
|
||||
use Anthropic\Beta\Agents\BetaManagedAgentsMCPToolsetParams;
|
||||
use Anthropic\Beta\Agents\BetaManagedAgentsUrlmcpServerParams;
|
||||
use Anthropic\Beta\Sessions\BetaManagedAgentsAgentParams;
|
||||
|
||||
// Agent declares MCP server (no auth here — auth goes in a vault)
|
||||
$agent = $client->beta->agents->create(
|
||||
name: 'GitHub Assistant',
|
||||
model: 'claude-opus-4-8',
|
||||
mcpServers: [
|
||||
BetaManagedAgentsUrlmcpServerParams::with(
|
||||
type: 'url',
|
||||
name: 'github',
|
||||
url: 'https://api.githubcopilot.com/mcp/',
|
||||
),
|
||||
],
|
||||
tools: [
|
||||
BetaManagedAgentsAgentToolset20260401Params::with(type: 'agent_toolset_20260401'),
|
||||
BetaManagedAgentsMCPToolsetParams::with(
|
||||
type: 'mcp_toolset',
|
||||
mcpServerName: 'github',
|
||||
),
|
||||
],
|
||||
);
|
||||
|
||||
// Session attaches vault(s) containing credentials for those MCP server URLs
|
||||
$session = $client->beta->sessions->create(
|
||||
agent: BetaManagedAgentsAgentParams::with(
|
||||
type: 'agent',
|
||||
id: $agent->id,
|
||||
version: $agent->version,
|
||||
),
|
||||
environmentID: $environment->id,
|
||||
vaultIDs: [$vault->id],
|
||||
);
|
||||
```
|
||||
|
||||
See `shared/managed-agents-tools.md` §Vaults for creating vaults and adding credentials.
|
||||
|
||||
---
|
||||
|
||||
## Vaults
|
||||
|
||||
```php
|
||||
// Create a vault
|
||||
$vault = $client->beta->vaults->create(
|
||||
displayName: 'Alice',
|
||||
metadata: ['external_user_id' => 'usr_abc123'],
|
||||
);
|
||||
echo $vault->id . "\n"; // "vlt_01ABC..."
|
||||
|
||||
// Add an OAuth credential
|
||||
$credential = $client->beta->vaults->credentials->create(
|
||||
vaultID: $vault->id,
|
||||
displayName: "Alice's Slack",
|
||||
auth: [
|
||||
'type' => 'mcp_oauth',
|
||||
'mcp_server_url' => 'https://mcp.slack.com/mcp',
|
||||
'access_token' => 'xoxp-...',
|
||||
'expires_at' => '2026-04-15T00:00:00Z',
|
||||
'refresh' => [
|
||||
'token_endpoint' => 'https://slack.com/api/oauth.v2.access',
|
||||
'client_id' => '1234567890.0987654321',
|
||||
'scope' => 'channels:read chat:write',
|
||||
'refresh_token' => 'xoxe-1-...',
|
||||
'token_endpoint_auth' => [
|
||||
'type' => 'client_secret_post',
|
||||
'client_secret' => 'abc123...',
|
||||
],
|
||||
],
|
||||
],
|
||||
);
|
||||
|
||||
// Rotate the credential (e.g., after a token refresh)
|
||||
$client->beta->vaults->credentials->update(
|
||||
$credential->id,
|
||||
vaultID: $vault->id,
|
||||
auth: [
|
||||
'type' => 'mcp_oauth',
|
||||
'access_token' => 'xoxp-new-...',
|
||||
'expires_at' => '2026-05-15T00:00:00Z',
|
||||
'refresh' => ['refresh_token' => 'xoxe-1-new-...'],
|
||||
],
|
||||
);
|
||||
|
||||
// Archive a vault
|
||||
$client->beta->vaults->archive($vault->id);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GitHub Repository Integration
|
||||
|
||||
Mount a GitHub repository as a session resource (a vault holds the GitHub MCP credential):
|
||||
|
||||
```php
|
||||
$session = $client->beta->sessions->create(
|
||||
agent: $agent->id,
|
||||
environmentID: $environment->id,
|
||||
vaultIDs: [$vault->id],
|
||||
resources: [
|
||||
[
|
||||
'type' => 'github_repository',
|
||||
'url' => 'https://github.com/org/repo',
|
||||
'mountPath' => '/workspace/repo',
|
||||
'authorizationToken' => 'ghp_your_github_token',
|
||||
],
|
||||
],
|
||||
);
|
||||
```
|
||||
|
||||
Multiple repositories on the same session:
|
||||
|
||||
```php
|
||||
$resources = [
|
||||
[
|
||||
'type' => 'github_repository',
|
||||
'url' => 'https://github.com/org/frontend',
|
||||
'mountPath' => '/workspace/frontend',
|
||||
'authorizationToken' => 'ghp_your_github_token',
|
||||
],
|
||||
[
|
||||
'type' => 'github_repository',
|
||||
'url' => 'https://github.com/org/backend',
|
||||
'mountPath' => '/workspace/backend',
|
||||
'authorizationToken' => 'ghp_your_github_token',
|
||||
],
|
||||
];
|
||||
```
|
||||
|
||||
Rotating a repository's authorization token:
|
||||
|
||||
```php
|
||||
$listed = $client->beta->sessions->resources->list($session->id);
|
||||
$repoResourceId = $listed->data[0]->id;
|
||||
|
||||
$client->beta->sessions->resources->update(
|
||||
$repoResourceId,
|
||||
sessionID: $session->id,
|
||||
authorizationToken: 'ghp_your_new_github_token',
|
||||
);
|
||||
```
|
||||
@@ -1,536 +0,0 @@
|
||||
# Claude API — Python
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install anthropic
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
# Default — resolves credentials from the environment:
|
||||
# ANTHROPIC_API_KEY, or ANTHROPIC_AUTH_TOKEN, or an `ant auth login` profile.
|
||||
# Prefer this for local dev; don't hardcode a key.
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
# Explicit API key (only when you must inject a specific key)
|
||||
client = anthropic.Anthropic(api_key="your-api-key")
|
||||
|
||||
# Async client
|
||||
async_client = anthropic.AsyncAnthropic()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Client Configuration
|
||||
|
||||
### Per-request overrides
|
||||
|
||||
Use `with_options()` to override client settings for a single call without mutating the client:
|
||||
|
||||
```python
|
||||
client.with_options(timeout=5.0, max_retries=5).messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=1024,
|
||||
messages=[{"role": "user", "content": "Hello"}],
|
||||
)
|
||||
```
|
||||
|
||||
### Timeouts
|
||||
|
||||
Default request timeout is 10 minutes. Pass a float (seconds) or an `httpx.Timeout` for granular control. On timeout the SDK raises `anthropic.APITimeoutError` (and retries per `max_retries`).
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
client = anthropic.Anthropic(timeout=20.0)
|
||||
client = anthropic.Anthropic(
|
||||
timeout=httpx.Timeout(60.0, read=5.0, write=10.0, connect=2.0),
|
||||
)
|
||||
```
|
||||
|
||||
### Retries
|
||||
|
||||
The SDK auto-retries connection errors, 408, 409, 429, and ≥500 with exponential backoff (default 2 retries). Set `max_retries` on the client or via `with_options()`; `max_retries=0` disables.
|
||||
|
||||
### Async performance (aiohttp backend)
|
||||
|
||||
For high-concurrency async workloads, install `anthropic[aiohttp]` and pass `DefaultAioHttpClient` instead of the default httpx backend:
|
||||
|
||||
```python
|
||||
from anthropic import AsyncAnthropic, DefaultAioHttpClient
|
||||
|
||||
async with AsyncAnthropic(http_client=DefaultAioHttpClient()) as client:
|
||||
...
|
||||
```
|
||||
|
||||
### Custom HTTP client (proxy, base URL)
|
||||
|
||||
Use `DefaultHttpxClient` / `DefaultAsyncHttpxClient` — not raw `httpx.Client` — so the SDK's default timeouts and connection limits are preserved:
|
||||
|
||||
```python
|
||||
from anthropic import Anthropic, DefaultHttpxClient
|
||||
|
||||
client = Anthropic(
|
||||
base_url="http://my.test.server.example.com:8083", # or ANTHROPIC_BASE_URL env var
|
||||
http_client=DefaultHttpxClient(proxy="http://my.test.proxy.example.com"),
|
||||
)
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Set `ANTHROPIC_LOG=debug` (or `info`) to enable SDK logging via the standard `logging` module.
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[
|
||||
{"role": "user", "content": "What is the capital of France?"}
|
||||
]
|
||||
)
|
||||
# response.content is a list of content block objects (TextBlock, ThinkingBlock,
|
||||
# ToolUseBlock, ...). Check .type before accessing .text.
|
||||
for block in response.content:
|
||||
if block.type == "text":
|
||||
print(block.text)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## System Prompts
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
system="You are a helpful coding assistant. Always provide examples in Python.",
|
||||
messages=[{"role": "user", "content": "How do I read a JSON file?"}]
|
||||
)
|
||||
```
|
||||
|
||||
### Mid-conversation system messages (beta, model-gated)
|
||||
|
||||
For operator instructions that arrive mid-conversation (mode switches, injected state), append `{"role": "system", ...}` to `messages` instead of editing top-level `system` — this preserves the cached prefix and carries operator authority. Must follow a user message; cannot be `messages[0]`. Unsupported models return a 400 (`role 'system' is not supported on this model`). See `shared/prompt-caching.md` for when to use this vs. top-level `system`.
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model=MODEL_ID, # must support mid-conversation system messages
|
||||
max_tokens=16000,
|
||||
system=[{"type": "text", "text": STABLE_SYSTEM, "cache_control": {"type": "ephemeral"}}],
|
||||
messages=history + [
|
||||
{"role": "user", "content": user_message},
|
||||
{"role": "system", "content": "Terse mode enabled — keep responses under 40 words."},
|
||||
],
|
||||
extra_headers={"anthropic-beta": "mid-conversation-system-2026-04-07"},
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Vision (Images)
|
||||
|
||||
### Base64
|
||||
|
||||
```python
|
||||
import base64
|
||||
|
||||
with open("image.png", "rb") as f:
|
||||
image_data = base64.standard_b64encode(f.read()).decode("utf-8")
|
||||
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image",
|
||||
"source": {
|
||||
"type": "base64",
|
||||
"media_type": "image/png",
|
||||
"data": image_data
|
||||
}
|
||||
},
|
||||
{"type": "text", "text": "What's in this image?"}
|
||||
]
|
||||
}]
|
||||
)
|
||||
```
|
||||
|
||||
### URL
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image",
|
||||
"source": {
|
||||
"type": "url",
|
||||
"url": "https://example.com/image.png"
|
||||
}
|
||||
},
|
||||
{"type": "text", "text": "Describe this image"}
|
||||
]
|
||||
}]
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
Cache large context to reduce costs (up to 90% savings). **Caching is a prefix match** — any byte change anywhere in the prefix invalidates everything after it. For placement patterns, architectural guidance (frozen system prompt, deterministic tool order, where to put volatile content), and the silent-invalidator audit checklist, read `shared/prompt-caching.md`.
|
||||
|
||||
### Automatic Caching (Recommended)
|
||||
|
||||
Use top-level `cache_control` to automatically cache the last cacheable block in the request — no need to annotate individual content blocks:
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
cache_control={"type": "ephemeral"}, # auto-caches the last cacheable block
|
||||
system="You are an expert on this large document...",
|
||||
messages=[{"role": "user", "content": "Summarize the key points"}]
|
||||
)
|
||||
```
|
||||
|
||||
### Manual Cache Control
|
||||
|
||||
For fine-grained control, add `cache_control` to specific content blocks:
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
system=[{
|
||||
"type": "text",
|
||||
"text": "You are an expert on this large document...",
|
||||
"cache_control": {"type": "ephemeral"} # default TTL is 5 minutes
|
||||
}],
|
||||
messages=[{"role": "user", "content": "Summarize the key points"}]
|
||||
)
|
||||
|
||||
# With explicit TTL (time-to-live)
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
system=[{
|
||||
"type": "text",
|
||||
"text": "You are an expert on this large document...",
|
||||
"cache_control": {"type": "ephemeral", "ttl": "1h"} # 1 hour TTL
|
||||
}],
|
||||
messages=[{"role": "user", "content": "Summarize the key points"}]
|
||||
)
|
||||
```
|
||||
|
||||
### Verifying Cache Hits
|
||||
|
||||
```python
|
||||
print(response.usage.cache_creation_input_tokens) # tokens written to cache (~1.25x cost)
|
||||
print(response.usage.cache_read_input_tokens) # tokens served from cache (~0.1x cost)
|
||||
print(response.usage.input_tokens) # uncached tokens (full cost)
|
||||
```
|
||||
|
||||
If `cache_read_input_tokens` is zero across repeated identical-prefix requests, a silent invalidator is at work — `datetime.now()` or a UUID in the system prompt, unsorted `json.dumps()`, or a varying tool set. See `shared/prompt-caching.md` for the full audit table.
|
||||
|
||||
---
|
||||
|
||||
## Extended Thinking
|
||||
|
||||
> **Fable 5, Opus 4.8, Opus 4.7, Opus 4.6, and Sonnet 4.6:** Use adaptive thinking. `budget_tokens` is removed on Fable 5, Opus 4.8, and 4.7 (400 if sent); deprecated on Opus 4.6 and Sonnet 4.6.
|
||||
> **Older models:** Use `thinking: {type: "enabled", budget_tokens: N}` (must be < `max_tokens`, min 1024).
|
||||
|
||||
```python
|
||||
# Fable 5 / Opus 4.8 / 4.7 / 4.6: adaptive thinking (recommended)
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
thinking={"type": "adaptive"},
|
||||
output_config={"effort": "high"}, # low | medium | high | max
|
||||
messages=[{"role": "user", "content": "Solve this step by step..."}]
|
||||
)
|
||||
|
||||
# Access thinking and response
|
||||
for block in response.content:
|
||||
if block.type == "thinking":
|
||||
print(f"Thinking: {block.thinking}")
|
||||
elif block.type == "text":
|
||||
print(f"Response: {block.text}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
try:
|
||||
response = client.messages.create(...)
|
||||
except anthropic.BadRequestError as e:
|
||||
print(f"Bad request: {e.message}")
|
||||
except anthropic.AuthenticationError:
|
||||
print("Invalid API key")
|
||||
except anthropic.PermissionDeniedError:
|
||||
print("API key lacks required permissions")
|
||||
except anthropic.NotFoundError:
|
||||
print("Invalid model or endpoint")
|
||||
except anthropic.RateLimitError as e:
|
||||
retry_after = int(e.response.headers.get("retry-after", "60"))
|
||||
print(f"Rate limited. Retry after {retry_after}s.")
|
||||
except anthropic.APIStatusError as e:
|
||||
if e.status_code >= 500:
|
||||
print(f"Server error ({e.status_code}). Retry later.")
|
||||
else:
|
||||
print(f"API error: {e.message}")
|
||||
except anthropic.APIConnectionError:
|
||||
print("Network error. Check internet connection.")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Response Helpers
|
||||
|
||||
Every response object exposes `_request_id` (populated from the `request-id` header) — log it when reporting failures to Anthropic. Despite the underscore prefix, this property is public.
|
||||
|
||||
```python
|
||||
message = client.messages.create(...)
|
||||
print(message._request_id) # req_018EeWyXxfu5pfWkrYcMdjWG
|
||||
print(message.to_json()) # serialize the Pydantic model
|
||||
print(message.to_dict()) # plain dict
|
||||
```
|
||||
|
||||
To access raw headers or other response metadata, use `.with_raw_response`:
|
||||
|
||||
```python
|
||||
raw = client.messages.with_raw_response.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=1024,
|
||||
messages=[{"role": "user", "content": "Hello"}],
|
||||
)
|
||||
print(raw.headers.get("request-id"))
|
||||
message = raw.parse() # the Message object messages.create() would have returned
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Multi-Turn Conversations
|
||||
|
||||
The API is stateless — send the full conversation history each time.
|
||||
|
||||
```python
|
||||
class ConversationManager:
|
||||
"""Manage multi-turn conversations with the Claude API."""
|
||||
|
||||
def __init__(self, client: anthropic.Anthropic, model: str, system: str = None):
|
||||
self.client = client
|
||||
self.model = model
|
||||
self.system = system
|
||||
self.messages = []
|
||||
|
||||
def send(self, user_message: str, **kwargs) -> str:
|
||||
"""Send a message and get a response."""
|
||||
self.messages.append({"role": "user", "content": user_message})
|
||||
|
||||
response = self.client.messages.create(
|
||||
model=self.model,
|
||||
max_tokens=kwargs.get("max_tokens", 16000),
|
||||
system=self.system,
|
||||
messages=self.messages,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
assistant_message = next(
|
||||
(b.text for b in response.content if b.type == "text"), ""
|
||||
)
|
||||
self.messages.append({"role": "assistant", "content": assistant_message})
|
||||
|
||||
return assistant_message
|
||||
|
||||
# Usage
|
||||
conversation = ConversationManager(
|
||||
client=anthropic.Anthropic(),
|
||||
model="claude-opus-4-8",
|
||||
system="You are a helpful assistant."
|
||||
)
|
||||
|
||||
response1 = conversation.send("My name is Alice.")
|
||||
response2 = conversation.send("What's my name?") # Claude remembers "Alice"
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
|
||||
- Consecutive same-role messages are allowed — the API combines them into a single turn
|
||||
- First message must be `user`
|
||||
- `role: "system"` messages are allowed mid-conversation under the `mid-conversation-system-2026-04-07` beta on supporting models — see § Mid-conversation system messages above
|
||||
|
||||
---
|
||||
|
||||
### Compaction (long conversations)
|
||||
|
||||
> **Beta, Fable 5, Opus 4.8, Opus 4.7, Opus 4.6, and Sonnet 4.6.** When conversations approach the 200K context window, compaction automatically summarizes earlier context server-side. The API returns a `compaction` block; you must pass it back on subsequent requests — append `response.content`, not just the text.
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
messages = []
|
||||
|
||||
def chat(user_message: str) -> str:
|
||||
messages.append({"role": "user", "content": user_message})
|
||||
|
||||
response = client.beta.messages.create(
|
||||
betas=["compact-2026-01-12"],
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=messages,
|
||||
context_management={
|
||||
"edits": [{"type": "compact_20260112"}]
|
||||
}
|
||||
)
|
||||
|
||||
# Append full content — compaction blocks must be preserved
|
||||
messages.append({"role": "assistant", "content": response.content})
|
||||
|
||||
return next(block.text for block in response.content if block.type == "text")
|
||||
|
||||
# Compaction triggers automatically when context grows large
|
||||
print(chat("Help me build a Python web scraper"))
|
||||
print(chat("Add support for JavaScript-rendered pages"))
|
||||
print(chat("Now add rate limiting and error handling"))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stop Reasons
|
||||
|
||||
The `stop_reason` field in the response indicates why the model stopped generating:
|
||||
|
||||
| Value | Meaning |
|
||||
|-------|---------|
|
||||
| `end_turn` | Claude finished its response naturally |
|
||||
| `max_tokens` | Hit the `max_tokens` limit — increase it or use streaming |
|
||||
| `stop_sequence` | Hit a custom stop sequence |
|
||||
| `tool_use` | Claude wants to call a tool — execute it and continue |
|
||||
| `pause_turn` | Model paused and can be resumed (agentic flows) |
|
||||
| `refusal` | Claude refused for safety reasons — check `stop_details` |
|
||||
|
||||
### Structured Stop Details
|
||||
|
||||
When `stop_reason` is `"refusal"`, the response includes a `stop_details` object with structured information about the refusal:
|
||||
|
||||
```python
|
||||
if response.stop_reason == "refusal" and response.stop_details:
|
||||
print(f"Category: {response.stop_details.category}") # "cyber" | "bio" | None
|
||||
print(f"Explanation: {response.stop_details.explanation}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost Optimization Strategies
|
||||
|
||||
### 1. Use Prompt Caching for Repeated Context
|
||||
|
||||
```python
|
||||
# Automatic caching (simplest — caches the last cacheable block)
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
cache_control={"type": "ephemeral"},
|
||||
system=large_document_text, # e.g., 50KB of context
|
||||
messages=[{"role": "user", "content": "Summarize the key points"}]
|
||||
)
|
||||
|
||||
# First request: full cost
|
||||
# Subsequent requests: ~90% cheaper for cached portion
|
||||
```
|
||||
|
||||
### 2. Choose the Right Model
|
||||
|
||||
```python
|
||||
# Default to Opus for most tasks
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8", # $5.00/$25.00 per 1M tokens
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Explain quantum computing"}]
|
||||
)
|
||||
|
||||
# Use Sonnet for high-volume production workloads
|
||||
standard_response = client.messages.create(
|
||||
model="claude-sonnet-4-6", # $3.00/$15.00 per 1M tokens
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Summarize this document"}]
|
||||
)
|
||||
|
||||
# Use Haiku only for simple, speed-critical tasks
|
||||
simple_response = client.messages.create(
|
||||
model="claude-haiku-4-5", # $1.00/$5.00 per 1M tokens
|
||||
max_tokens=256,
|
||||
messages=[{"role": "user", "content": "Classify this as positive or negative"}]
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Use Token Counting Before Requests
|
||||
|
||||
```python
|
||||
count_response = client.messages.count_tokens(
|
||||
model="claude-opus-4-8",
|
||||
messages=messages,
|
||||
system=system
|
||||
)
|
||||
|
||||
estimated_input_cost = count_response.input_tokens * 0.000005 # $5/1M tokens
|
||||
print(f"Estimated input cost: ${estimated_input_cost:.4f}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Retry with Exponential Backoff
|
||||
|
||||
> **Note:** The Anthropic SDK automatically retries rate limit (429) and server errors (5xx) with exponential backoff. You can configure this with `max_retries` (default: 2). Only implement custom retry logic if you need behavior beyond what the SDK provides.
|
||||
|
||||
```python
|
||||
import time
|
||||
import random
|
||||
import anthropic
|
||||
|
||||
def call_with_retry(
|
||||
client: anthropic.Anthropic,
|
||||
max_retries: int = 5,
|
||||
base_delay: float = 1.0,
|
||||
max_delay: float = 60.0,
|
||||
**kwargs
|
||||
):
|
||||
"""Call the API with exponential backoff retry."""
|
||||
last_exception = None
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
return client.messages.create(**kwargs)
|
||||
except anthropic.RateLimitError as e:
|
||||
last_exception = e
|
||||
except anthropic.APIStatusError as e:
|
||||
if e.status_code >= 500:
|
||||
last_exception = e
|
||||
else:
|
||||
raise # Client errors (4xx except 429) should not be retried
|
||||
|
||||
delay = min(base_delay * (2 ** attempt) + random.uniform(0, 1), max_delay)
|
||||
print(f"Retry {attempt + 1}/{max_retries} after {delay:.1f}s")
|
||||
time.sleep(delay)
|
||||
|
||||
raise last_exception
|
||||
```
|
||||
@@ -1,198 +0,0 @@
|
||||
# Message Batches API — Python
|
||||
|
||||
The Batches API (`POST /v1/messages/batches`) processes Messages API requests asynchronously at 50% of standard prices.
|
||||
|
||||
## Key Facts
|
||||
|
||||
- Up to 100,000 requests or 256 MB per batch
|
||||
- Most batches complete within 1 hour; maximum 24 hours
|
||||
- Results available for 29 days after creation
|
||||
- 50% cost reduction on all token usage
|
||||
- All Messages API features supported (vision, tools, caching, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Create a Batch
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
|
||||
from anthropic.types.messages.batch_create_params import Request
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
message_batch = client.messages.batches.create(
|
||||
requests=[
|
||||
Request(
|
||||
custom_id="request-1",
|
||||
params=MessageCreateParamsNonStreaming(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Summarize climate change impacts"}]
|
||||
)
|
||||
),
|
||||
Request(
|
||||
custom_id="request-2",
|
||||
params=MessageCreateParamsNonStreaming(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Explain quantum computing basics"}]
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
print(f"Batch ID: {message_batch.id}")
|
||||
print(f"Status: {message_batch.processing_status}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Poll for Completion
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
while True:
|
||||
batch = client.messages.batches.retrieve(message_batch.id)
|
||||
if batch.processing_status == "ended":
|
||||
break
|
||||
print(f"Status: {batch.processing_status}, processing: {batch.request_counts.processing}")
|
||||
time.sleep(60)
|
||||
|
||||
print("Batch complete!")
|
||||
print(f"Succeeded: {batch.request_counts.succeeded}")
|
||||
print(f"Errored: {batch.request_counts.errored}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Retrieve Results
|
||||
|
||||
> **Note:** Examples below use `match/case` syntax, requiring Python 3.10+. For earlier versions, use `if/elif` chains instead.
|
||||
|
||||
```python
|
||||
for result in client.messages.batches.results(message_batch.id):
|
||||
match result.result.type:
|
||||
case "succeeded":
|
||||
msg = result.result.message
|
||||
text = next((b.text for b in msg.content if b.type == "text"), "")
|
||||
print(f"[{result.custom_id}] {text[:100]}")
|
||||
case "errored":
|
||||
if result.result.error.type == "invalid_request":
|
||||
print(f"[{result.custom_id}] Validation error - fix request and retry")
|
||||
else:
|
||||
print(f"[{result.custom_id}] Server error - safe to retry")
|
||||
case "canceled":
|
||||
print(f"[{result.custom_id}] Canceled")
|
||||
case "expired":
|
||||
print(f"[{result.custom_id}] Expired - resubmit")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cancel a Batch
|
||||
|
||||
```python
|
||||
cancelled = client.messages.batches.cancel(message_batch.id)
|
||||
print(f"Status: {cancelled.processing_status}") # "canceling"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List Batches (auto-pagination)
|
||||
|
||||
Iterating the return value of any `list()` call auto-paginates across all pages — do not index into `.data` if you want the full set:
|
||||
|
||||
```python
|
||||
for batch in client.messages.batches.list(limit=20):
|
||||
print(batch.id, batch.processing_status)
|
||||
```
|
||||
|
||||
For manual control, use `first_page.has_next_page()` / `first_page.get_next_page()` / `first_page.next_page_info()`; `first_page.data` holds the current page's items and `first_page.last_id` is the cursor.
|
||||
|
||||
---
|
||||
|
||||
## Batch with Prompt Caching
|
||||
|
||||
```python
|
||||
shared_system = [
|
||||
{"type": "text", "text": "You are a literary analyst."},
|
||||
{
|
||||
"type": "text",
|
||||
"text": large_document_text, # Shared across all requests
|
||||
"cache_control": {"type": "ephemeral"}
|
||||
}
|
||||
]
|
||||
|
||||
message_batch = client.messages.batches.create(
|
||||
requests=[
|
||||
Request(
|
||||
custom_id=f"analysis-{i}",
|
||||
params=MessageCreateParamsNonStreaming(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
system=shared_system,
|
||||
messages=[{"role": "user", "content": question}]
|
||||
)
|
||||
)
|
||||
for i, question in enumerate(questions)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full End-to-End Example
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
import time
|
||||
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
|
||||
from anthropic.types.messages.batch_create_params import Request
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
# 1. Prepare requests
|
||||
items_to_classify = [
|
||||
"The product quality is excellent!",
|
||||
"Terrible customer service, never again.",
|
||||
"It's okay, nothing special.",
|
||||
]
|
||||
|
||||
requests = [
|
||||
Request(
|
||||
custom_id=f"classify-{i}",
|
||||
params=MessageCreateParamsNonStreaming(
|
||||
model="claude-haiku-4-5",
|
||||
max_tokens=50,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": f"Classify as positive/negative/neutral (one word): {text}"
|
||||
}]
|
||||
)
|
||||
)
|
||||
for i, text in enumerate(items_to_classify)
|
||||
]
|
||||
|
||||
# 2. Create batch
|
||||
batch = client.messages.batches.create(requests=requests)
|
||||
print(f"Created batch: {batch.id}")
|
||||
|
||||
# 3. Wait for completion
|
||||
while True:
|
||||
batch = client.messages.batches.retrieve(batch.id)
|
||||
if batch.processing_status == "ended":
|
||||
break
|
||||
time.sleep(10)
|
||||
|
||||
# 4. Collect results
|
||||
results = {}
|
||||
for result in client.messages.batches.results(batch.id):
|
||||
if result.result.type == "succeeded":
|
||||
msg = result.result.message
|
||||
results[result.custom_id] = next((b.text for b in msg.content if b.type == "text"), "")
|
||||
|
||||
for custom_id, classification in sorted(results.items()):
|
||||
print(f"{custom_id}: {classification}")
|
||||
```
|
||||
@@ -1,170 +0,0 @@
|
||||
# Files API — Python
|
||||
|
||||
The Files API uploads files for use in Messages API requests. Reference files via `file_id` in content blocks, avoiding re-uploads across multiple API calls.
|
||||
|
||||
**Beta:** Pass `betas=["files-api-2025-04-14"]` in your API calls (the SDK sets the required header automatically).
|
||||
|
||||
## Key Facts
|
||||
|
||||
- Maximum file size: 500 MB
|
||||
- Total storage: 100 GB per organization
|
||||
- Files persist until deleted
|
||||
- File operations (upload, list, delete) are free; content used in messages is billed as input tokens
|
||||
- Not available on Amazon Bedrock or Google Vertex AI
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
The `file` argument accepts a `(filename, content, content_type)` tuple, a `pathlib.Path` (or any `PathLike` — read for you, async-safe with `AsyncAnthropic`), or an open binary file object.
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
from pathlib import Path
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
uploaded = client.beta.files.upload(
|
||||
file=("report.pdf", open("report.pdf", "rb"), "application/pdf"),
|
||||
)
|
||||
# or: client.beta.files.upload(file=Path("report.pdf"))
|
||||
print(f"File ID: {uploaded.id}")
|
||||
print(f"Size: {uploaded.size_bytes} bytes")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Use a File in Messages
|
||||
|
||||
### PDF / Text Document
|
||||
|
||||
```python
|
||||
response = client.beta.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Summarize the key findings in this report."},
|
||||
{
|
||||
"type": "document",
|
||||
"source": {"type": "file", "file_id": uploaded.id},
|
||||
"title": "Q4 Report", # optional
|
||||
"citations": {"enabled": True} # optional, enables citations
|
||||
}
|
||||
]
|
||||
}],
|
||||
betas=["files-api-2025-04-14"],
|
||||
)
|
||||
for block in response.content:
|
||||
if block.type == "text":
|
||||
print(block.text)
|
||||
```
|
||||
|
||||
### Image
|
||||
|
||||
```python
|
||||
image_file = client.beta.files.upload(
|
||||
file=("photo.png", open("photo.png", "rb"), "image/png"),
|
||||
)
|
||||
|
||||
response = client.beta.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What's in this image?"},
|
||||
{
|
||||
"type": "image",
|
||||
"source": {"type": "file", "file_id": image_file.id}
|
||||
}
|
||||
]
|
||||
}],
|
||||
betas=["files-api-2025-04-14"],
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manage Files
|
||||
|
||||
### List Files
|
||||
|
||||
Iterate the list result directly — the SDK auto-paginates across all pages. Only use `.data` if you want the first page only.
|
||||
|
||||
```python
|
||||
for f in client.beta.files.list():
|
||||
print(f"{f.id}: {f.filename} ({f.size_bytes} bytes)")
|
||||
```
|
||||
|
||||
### Get File Metadata
|
||||
|
||||
```python
|
||||
file_info = client.beta.files.retrieve_metadata("file_011CNha8iCJcU1wXNR6q4V8w")
|
||||
print(f"Filename: {file_info.filename}")
|
||||
print(f"MIME type: {file_info.mime_type}")
|
||||
```
|
||||
|
||||
### Delete a File
|
||||
|
||||
```python
|
||||
client.beta.files.delete("file_011CNha8iCJcU1wXNR6q4V8w")
|
||||
```
|
||||
|
||||
### Download a File
|
||||
|
||||
Only files created by the code execution tool or skills can be downloaded (not user-uploaded files).
|
||||
|
||||
```python
|
||||
file_content = client.beta.files.download("file_011CNha8iCJcU1wXNR6q4V8w")
|
||||
file_content.write_to_file("output.txt")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full End-to-End Example
|
||||
|
||||
Upload a document once, ask multiple questions about it:
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
# 1. Upload once
|
||||
uploaded = client.beta.files.upload(
|
||||
file=("contract.pdf", open("contract.pdf", "rb"), "application/pdf"),
|
||||
)
|
||||
print(f"Uploaded: {uploaded.id}")
|
||||
|
||||
# 2. Ask multiple questions using the same file_id
|
||||
questions = [
|
||||
"What are the key terms and conditions?",
|
||||
"What is the termination clause?",
|
||||
"Summarize the payment schedule.",
|
||||
]
|
||||
|
||||
for question in questions:
|
||||
response = client.beta.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": question},
|
||||
{
|
||||
"type": "document",
|
||||
"source": {"type": "file", "file_id": uploaded.id}
|
||||
}
|
||||
]
|
||||
}],
|
||||
betas=["files-api-2025-04-14"],
|
||||
)
|
||||
print(f"\nQ: {question}")
|
||||
text = next((b.text for b in response.content if b.type == "text"), "")
|
||||
print(f"A: {text[:200]}")
|
||||
|
||||
# 3. Clean up when done
|
||||
client.beta.files.delete(uploaded.id)
|
||||
```
|
||||
@@ -1,179 +0,0 @@
|
||||
# Streaming — Python
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
with client.messages.stream(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=64000,
|
||||
messages=[{"role": "user", "content": "Write a story"}]
|
||||
) as stream:
|
||||
for text in stream.text_stream:
|
||||
print(text, end="", flush=True)
|
||||
```
|
||||
|
||||
### Async
|
||||
|
||||
```python
|
||||
async with async_client.messages.stream(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=64000,
|
||||
messages=[{"role": "user", "content": "Write a story"}]
|
||||
) as stream:
|
||||
async for text in stream.text_stream:
|
||||
print(text, end="", flush=True)
|
||||
```
|
||||
|
||||
### Low-level: `stream=True`
|
||||
|
||||
`messages.stream()` (above) is the recommended helper — it accumulates state and exposes `text_stream` / `get_final_message()`. If you only need the raw event iterator and want lower memory use, pass `stream=True` to `messages.create()` instead:
|
||||
|
||||
```python
|
||||
for event in client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=64000,
|
||||
messages=[{"role": "user", "content": "Write a story"}],
|
||||
stream=True,
|
||||
):
|
||||
print(event.type)
|
||||
```
|
||||
|
||||
No final-message accumulation is done for you in this form.
|
||||
|
||||
---
|
||||
|
||||
## Handling Different Content Types
|
||||
|
||||
Claude may return text, thinking blocks, or tool use. Handle each appropriately:
|
||||
|
||||
> **Fable 5 / Opus 4.8 / Opus 4.7 / Opus 4.6:** Use `thinking: {type: "adaptive"}`. On older models, use `thinking: {type: "enabled", budget_tokens: N}` instead.
|
||||
|
||||
```python
|
||||
with client.messages.stream(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=64000,
|
||||
thinking={"type": "adaptive"},
|
||||
messages=[{"role": "user", "content": "Analyze this problem"}]
|
||||
) as stream:
|
||||
for event in stream:
|
||||
if event.type == "content_block_start":
|
||||
if event.content_block.type == "thinking":
|
||||
print("\n[Thinking...]")
|
||||
elif event.content_block.type == "text":
|
||||
print("\n[Response:]")
|
||||
|
||||
elif event.type == "content_block_delta":
|
||||
if event.delta.type == "thinking_delta":
|
||||
print(event.delta.thinking, end="", flush=True)
|
||||
elif event.delta.type == "text_delta":
|
||||
print(event.delta.text, end="", flush=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming with Tool Use
|
||||
|
||||
The Python tool runner currently returns complete messages. Use streaming for individual API calls within a manual loop if you need per-token streaming with tools:
|
||||
|
||||
```python
|
||||
with client.messages.stream(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=64000,
|
||||
tools=tools,
|
||||
messages=messages
|
||||
) as stream:
|
||||
for text in stream.text_stream:
|
||||
print(text, end="", flush=True)
|
||||
|
||||
response = stream.get_final_message()
|
||||
# Continue with tool execution if response.stop_reason == "tool_use"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting the Final Message
|
||||
|
||||
```python
|
||||
with client.messages.stream(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=64000,
|
||||
messages=[{"role": "user", "content": "Hello"}]
|
||||
) as stream:
|
||||
for text in stream.text_stream:
|
||||
print(text, end="", flush=True)
|
||||
|
||||
# Get full message after streaming
|
||||
final_message = stream.get_final_message()
|
||||
print(f"\n\nTokens used: {final_message.usage.output_tokens}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming with Progress Updates
|
||||
|
||||
```python
|
||||
def stream_with_progress(client, **kwargs):
|
||||
"""Stream a response with progress updates."""
|
||||
total_tokens = 0
|
||||
content_parts = []
|
||||
|
||||
with client.messages.stream(**kwargs) as stream:
|
||||
for event in stream:
|
||||
if event.type == "content_block_delta":
|
||||
if event.delta.type == "text_delta":
|
||||
text = event.delta.text
|
||||
content_parts.append(text)
|
||||
print(text, end="", flush=True)
|
||||
|
||||
elif event.type == "message_delta":
|
||||
if event.usage and event.usage.output_tokens is not None:
|
||||
total_tokens = event.usage.output_tokens
|
||||
|
||||
final_message = stream.get_final_message()
|
||||
|
||||
print(f"\n\n[Tokens used: {total_tokens}]")
|
||||
return "".join(content_parts)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling in Streams
|
||||
|
||||
```python
|
||||
try:
|
||||
with client.messages.stream(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=64000,
|
||||
messages=[{"role": "user", "content": "Write a story"}]
|
||||
) as stream:
|
||||
for text in stream.text_stream:
|
||||
print(text, end="", flush=True)
|
||||
except anthropic.APIConnectionError:
|
||||
print("\nConnection lost. Please retry.")
|
||||
except anthropic.RateLimitError:
|
||||
print("\nRate limited. Please wait and retry.")
|
||||
except anthropic.APIStatusError as e:
|
||||
print(f"\nAPI error: {e.status_code}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stream Event Types
|
||||
|
||||
| Event Type | Description | When it fires |
|
||||
| --------------------- | --------------------------- | --------------------------------- |
|
||||
| `message_start` | Contains message metadata | Once at the beginning |
|
||||
| `content_block_start` | New content block beginning | When a text/tool_use block starts |
|
||||
| `content_block_delta` | Incremental content update | For each token/chunk |
|
||||
| `content_block_stop` | Content block complete | When a block finishes |
|
||||
| `message_delta` | Message-level updates | Contains `stop_reason`, usage |
|
||||
| `message_stop` | Message complete | Once at the end |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always flush output** — Use `flush=True` to show tokens immediately
|
||||
2. **Handle partial responses** — If the stream is interrupted, you may have incomplete content
|
||||
3. **Track token usage** — The `message_delta` event contains usage information
|
||||
4. **Use timeouts** — Set appropriate timeouts for your application
|
||||
5. **Default to streaming** — Use `.get_final_message()` to get the complete response even when streaming, giving you timeout protection without needing to handle individual events
|
||||
6. **Large `max_tokens` without streaming raises `ValueError`** — The SDK refuses non-streaming requests it estimates will exceed ~10 minutes (idle connections drop). Pass `stream=True` / use `messages.stream()`, or explicitly override `timeout`, to suppress the guard.
|
||||
@@ -1,590 +0,0 @@
|
||||
# Tool Use — Python
|
||||
|
||||
For conceptual overview (tool definitions, tool choice, tips), see [shared/tool-use-concepts.md](../../shared/tool-use-concepts.md).
|
||||
|
||||
## Tool Runner (Recommended)
|
||||
|
||||
**Beta:** The tool runner is in beta in the Python SDK.
|
||||
|
||||
Use the `@beta_tool` decorator to define tools as typed functions, then pass them to `client.beta.messages.tool_runner()`:
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
from anthropic import beta_tool
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
@beta_tool
|
||||
def get_weather(location: str, unit: str = "celsius") -> str:
|
||||
"""Get current weather for a location.
|
||||
|
||||
Args:
|
||||
location: City and state, e.g., San Francisco, CA.
|
||||
unit: Temperature unit, either "celsius" or "fahrenheit".
|
||||
"""
|
||||
# Your implementation here
|
||||
return f"72°F and sunny in {location}"
|
||||
|
||||
# The tool runner handles the agentic loop automatically
|
||||
runner = client.beta.messages.tool_runner(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
tools=[get_weather],
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
)
|
||||
|
||||
# Each iteration yields a BetaMessage; iteration stops when Claude is done
|
||||
for message in runner:
|
||||
print(message)
|
||||
```
|
||||
|
||||
For async usage, use `@beta_async_tool` with `async def` functions.
|
||||
|
||||
**Key benefits of the tool runner:**
|
||||
|
||||
- No manual loop — the SDK handles calling tools and feeding results back
|
||||
- Type-safe tool inputs via decorators
|
||||
- Tool schemas are generated automatically from function signatures
|
||||
- Iteration stops automatically when Claude has no more tool calls
|
||||
|
||||
---
|
||||
|
||||
## MCP Tool Conversion Helpers
|
||||
|
||||
**Beta.** Convert [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) tools, prompts, and resources to Anthropic API types for use with the tool runner. Requires `pip install anthropic[mcp]` (Python 3.10+).
|
||||
|
||||
> **Note:** The Claude API also supports an `mcp_servers` parameter that lets Claude connect directly to remote MCP servers. Use these helpers instead when you need local MCP servers, prompts, resources, or more control over the MCP connection.
|
||||
|
||||
### MCP Tools with Tool Runner
|
||||
|
||||
```python
|
||||
from anthropic import AsyncAnthropic
|
||||
from anthropic.lib.tools.mcp import async_mcp_tool
|
||||
from mcp import ClientSession
|
||||
from mcp.client.stdio import stdio_client, StdioServerParameters
|
||||
|
||||
client = AsyncAnthropic()
|
||||
|
||||
async with stdio_client(StdioServerParameters(command="mcp-server")) as (read, write):
|
||||
async with ClientSession(read, write) as mcp_client:
|
||||
await mcp_client.initialize()
|
||||
|
||||
tools_result = await mcp_client.list_tools()
|
||||
# tool_runner is sync — returns the runner, not a coroutine
|
||||
runner = client.beta.messages.tool_runner(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Use the available tools"}],
|
||||
tools=[async_mcp_tool(t, mcp_client) for t in tools_result.tools],
|
||||
)
|
||||
async for message in runner:
|
||||
print(message)
|
||||
```
|
||||
|
||||
For sync usage, use `mcp_tool` instead of `async_mcp_tool`.
|
||||
|
||||
### MCP Prompts
|
||||
|
||||
```python
|
||||
from anthropic.lib.tools.mcp import mcp_message
|
||||
|
||||
prompt = await mcp_client.get_prompt(name="my-prompt")
|
||||
response = await client.beta.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[mcp_message(m) for m in prompt.messages],
|
||||
)
|
||||
```
|
||||
|
||||
### MCP Resources as Content
|
||||
|
||||
```python
|
||||
from anthropic.lib.tools.mcp import mcp_resource_to_content
|
||||
|
||||
resource = await mcp_client.read_resource(uri="file:///path/to/doc.txt")
|
||||
response = await client.beta.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
mcp_resource_to_content(resource),
|
||||
{"type": "text", "text": "Summarize this document"},
|
||||
],
|
||||
}],
|
||||
)
|
||||
```
|
||||
|
||||
### Upload MCP Resources as Files
|
||||
|
||||
```python
|
||||
from anthropic.lib.tools.mcp import mcp_resource_to_file
|
||||
|
||||
resource = await mcp_client.read_resource(uri="file:///path/to/data.json")
|
||||
uploaded = await client.beta.files.upload(file=mcp_resource_to_file(resource))
|
||||
```
|
||||
|
||||
Conversion functions raise `UnsupportedMCPValueError` if an MCP value cannot be converted (e.g., unsupported content types like audio, unsupported MIME types).
|
||||
|
||||
---
|
||||
|
||||
## Manual Agentic Loop
|
||||
|
||||
Use this when you need fine-grained control over the loop (e.g., custom logging, conditional tool execution, human-in-the-loop approval):
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
tools = [...] # Your tool definitions
|
||||
messages = [{"role": "user", "content": user_input}]
|
||||
|
||||
# Agentic loop: keep going until Claude stops calling tools
|
||||
while True:
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
tools=tools,
|
||||
messages=messages
|
||||
)
|
||||
|
||||
# If Claude is done (no more tool calls), break
|
||||
if response.stop_reason == "end_turn":
|
||||
break
|
||||
|
||||
# Server-side tool hit iteration limit; re-send to continue
|
||||
if response.stop_reason == "pause_turn":
|
||||
messages = [
|
||||
{"role": "user", "content": user_input},
|
||||
{"role": "assistant", "content": response.content},
|
||||
]
|
||||
continue
|
||||
|
||||
# Extract tool use blocks from the response
|
||||
tool_use_blocks = [b for b in response.content if b.type == "tool_use"]
|
||||
|
||||
# Append assistant's response (including tool_use blocks)
|
||||
messages.append({"role": "assistant", "content": response.content})
|
||||
|
||||
# Execute each tool and collect results
|
||||
tool_results = []
|
||||
for tool in tool_use_blocks:
|
||||
result = execute_tool(tool.name, tool.input) # Your implementation
|
||||
tool_results.append({
|
||||
"type": "tool_result",
|
||||
"tool_use_id": tool.id, # Must match the tool_use block's id
|
||||
"content": result
|
||||
})
|
||||
|
||||
# Append tool results as a user message
|
||||
messages.append({"role": "user", "content": tool_results})
|
||||
|
||||
# Final response text
|
||||
final_text = next(b.text for b in response.content if b.type == "text")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Handling Tool Results
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
tools=tools,
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}]
|
||||
)
|
||||
|
||||
for block in response.content:
|
||||
if block.type == "tool_use":
|
||||
tool_name = block.name
|
||||
tool_input = block.input
|
||||
tool_use_id = block.id
|
||||
|
||||
result = execute_tool(tool_name, tool_input)
|
||||
|
||||
followup = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
tools=tools,
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Paris?"},
|
||||
{"role": "assistant", "content": response.content},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": tool_use_id,
|
||||
"content": result
|
||||
}]
|
||||
}
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Multiple Tool Calls
|
||||
|
||||
```python
|
||||
tool_results = []
|
||||
|
||||
for block in response.content:
|
||||
if block.type == "tool_use":
|
||||
result = execute_tool(block.name, block.input)
|
||||
tool_results.append({
|
||||
"type": "tool_result",
|
||||
"tool_use_id": block.id,
|
||||
"content": result
|
||||
})
|
||||
|
||||
# Send all results back at once
|
||||
if tool_results:
|
||||
followup = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
tools=tools,
|
||||
messages=[
|
||||
*previous_messages,
|
||||
{"role": "assistant", "content": response.content},
|
||||
{"role": "user", "content": tool_results}
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling in Tool Results
|
||||
|
||||
```python
|
||||
tool_result = {
|
||||
"type": "tool_result",
|
||||
"tool_use_id": tool_use_id,
|
||||
"content": "Error: Location 'xyz' not found. Please provide a valid city name.",
|
||||
"is_error": True
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tool Choice
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
tools=tools,
|
||||
tool_choice={"type": "tool", "name": "get_weather"}, # Force specific tool
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}]
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code Execution
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Calculate the mean and standard deviation of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
|
||||
}],
|
||||
tools=[{
|
||||
"type": "code_execution_20260120",
|
||||
"name": "code_execution"
|
||||
}]
|
||||
)
|
||||
|
||||
for block in response.content:
|
||||
if block.type == "text":
|
||||
print(block.text)
|
||||
elif block.type == "bash_code_execution_tool_result":
|
||||
print(f"stdout: {block.content.stdout}")
|
||||
```
|
||||
|
||||
### Upload Files for Analysis
|
||||
|
||||
```python
|
||||
# 1. Upload a file
|
||||
uploaded = client.beta.files.upload(file=open("sales_data.csv", "rb"))
|
||||
|
||||
# 2. Pass to code execution via container_upload block
|
||||
# Code execution is GA; Files API is still beta (pass via extra_headers)
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
extra_headers={"anthropic-beta": "files-api-2025-04-14"},
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Analyze this sales data. Show trends and create a visualization."},
|
||||
{"type": "container_upload", "file_id": uploaded.id}
|
||||
]
|
||||
}],
|
||||
tools=[{"type": "code_execution_20260120", "name": "code_execution"}]
|
||||
)
|
||||
```
|
||||
|
||||
### Retrieve Generated Files
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
OUTPUT_DIR = "./claude_outputs"
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
|
||||
for block in response.content:
|
||||
if block.type == "bash_code_execution_tool_result":
|
||||
result = block.content
|
||||
if result.type == "bash_code_execution_result" and result.content:
|
||||
for file_ref in result.content:
|
||||
if file_ref.type == "bash_code_execution_output":
|
||||
metadata = client.beta.files.retrieve_metadata(file_ref.file_id)
|
||||
file_content = client.beta.files.download(file_ref.file_id)
|
||||
# Use basename to prevent path traversal; validate result
|
||||
safe_name = os.path.basename(metadata.filename)
|
||||
if not safe_name or safe_name in (".", ".."):
|
||||
print(f"Skipping invalid filename: {metadata.filename}")
|
||||
continue
|
||||
output_path = os.path.join(OUTPUT_DIR, safe_name)
|
||||
file_content.write_to_file(output_path)
|
||||
print(f"Saved: {output_path}")
|
||||
```
|
||||
|
||||
### Container Reuse
|
||||
|
||||
```python
|
||||
# First request: set up environment
|
||||
response1 = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Install tabulate and create data.json with sample data"}],
|
||||
tools=[{"type": "code_execution_20260120", "name": "code_execution"}]
|
||||
)
|
||||
|
||||
# Get container ID from response
|
||||
container_id = response1.container.id
|
||||
|
||||
# Second request: reuse the same container
|
||||
response2 = client.messages.create(
|
||||
container=container_id,
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Read data.json and display as a formatted table"}],
|
||||
tools=[{"type": "code_execution_20260120", "name": "code_execution"}]
|
||||
)
|
||||
```
|
||||
|
||||
### Response Structure
|
||||
|
||||
```python
|
||||
for block in response.content:
|
||||
if block.type == "text":
|
||||
print(block.text) # Claude's explanation
|
||||
elif block.type == "server_tool_use":
|
||||
print(f"Running: {block.name} - {block.input}") # What Claude is doing
|
||||
elif block.type == "bash_code_execution_tool_result":
|
||||
result = block.content
|
||||
if result.type == "bash_code_execution_result":
|
||||
if result.return_code == 0:
|
||||
print(f"Output: {result.stdout}")
|
||||
else:
|
||||
print(f"Error: {result.stderr}")
|
||||
else:
|
||||
print(f"Tool error: {result.error_code}")
|
||||
elif block.type == "text_editor_code_execution_tool_result":
|
||||
print(f"File operation: {block.content}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Memory Tool
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Remember that my preferred language is Python."}],
|
||||
tools=[{"type": "memory_20250818", "name": "memory"}],
|
||||
)
|
||||
```
|
||||
|
||||
### SDK Memory Helper
|
||||
|
||||
Subclass `BetaAbstractMemoryTool`:
|
||||
|
||||
```python
|
||||
from anthropic.lib.tools import BetaAbstractMemoryTool
|
||||
|
||||
class MyMemoryTool(BetaAbstractMemoryTool):
|
||||
def view(self, command): ...
|
||||
def create(self, command): ...
|
||||
def str_replace(self, command): ...
|
||||
def insert(self, command): ...
|
||||
def delete(self, command): ...
|
||||
def rename(self, command): ...
|
||||
|
||||
memory = MyMemoryTool()
|
||||
|
||||
# Use with tool runner
|
||||
runner = client.beta.messages.tool_runner(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
tools=[memory],
|
||||
messages=[{"role": "user", "content": "Remember my preferences"}],
|
||||
)
|
||||
|
||||
for message in runner:
|
||||
print(message)
|
||||
```
|
||||
|
||||
For full implementation examples, use WebFetch:
|
||||
|
||||
- `https://github.com/anthropics/anthropic-sdk-python/blob/main/examples/memory/basic.py`
|
||||
|
||||
---
|
||||
|
||||
## Structured Outputs
|
||||
|
||||
### JSON Outputs (Pydantic — Recommended)
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from typing import List
|
||||
import anthropic
|
||||
|
||||
class ContactInfo(BaseModel):
|
||||
name: str
|
||||
email: str
|
||||
plan: str
|
||||
interests: List[str]
|
||||
demo_requested: bool
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
response = client.messages.parse(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Extract: Jane Doe (jane@co.com) wants Enterprise, interested in API and SDKs, wants a demo."
|
||||
}],
|
||||
output_format=ContactInfo,
|
||||
)
|
||||
|
||||
# response.parsed_output is a validated ContactInfo instance
|
||||
contact = response.parsed_output
|
||||
print(contact.name) # "Jane Doe"
|
||||
print(contact.interests) # ["API", "SDKs"]
|
||||
```
|
||||
|
||||
### Raw Schema
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Extract info: John Smith (john@example.com) wants the Enterprise plan."
|
||||
}],
|
||||
output_config={
|
||||
"format": {
|
||||
"type": "json_schema",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string"},
|
||||
"email": {"type": "string"},
|
||||
"plan": {"type": "string"},
|
||||
"demo_requested": {"type": "boolean"}
|
||||
},
|
||||
"required": ["name", "email", "plan", "demo_requested"],
|
||||
"additionalProperties": False
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
import json
|
||||
# output_config.format guarantees the first block is text with valid JSON
|
||||
text = next(b.text for b in response.content if b.type == "text")
|
||||
data = json.loads(text)
|
||||
```
|
||||
|
||||
### Strict Tool Use
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Book a flight to Tokyo for 2 passengers on March 15"}],
|
||||
tools=[{
|
||||
"name": "book_flight",
|
||||
"description": "Book a flight to a destination",
|
||||
"strict": True,
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"destination": {"type": "string"},
|
||||
"date": {"type": "string", "format": "date"},
|
||||
"passengers": {"type": "integer", "enum": [1, 2, 3, 4, 5, 6, 7, 8]}
|
||||
},
|
||||
"required": ["destination", "date", "passengers"],
|
||||
"additionalProperties": False
|
||||
}
|
||||
}]
|
||||
)
|
||||
```
|
||||
|
||||
### Using Both Together
|
||||
|
||||
```python
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=16000,
|
||||
messages=[{"role": "user", "content": "Plan a trip to Paris next month"}],
|
||||
output_config={
|
||||
"format": {
|
||||
"type": "json_schema",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"summary": {"type": "string"},
|
||||
"next_steps": {"type": "array", "items": {"type": "string"}}
|
||||
},
|
||||
"required": ["summary", "next_steps"],
|
||||
"additionalProperties": False
|
||||
}
|
||||
}
|
||||
},
|
||||
tools=[{
|
||||
"name": "search_flights",
|
||||
"description": "Search for available flights",
|
||||
"strict": True,
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"destination": {"type": "string"},
|
||||
"date": {"type": "string", "format": "date"}
|
||||
},
|
||||
"required": ["destination", "date"],
|
||||
"additionalProperties": False
|
||||
}
|
||||
}]
|
||||
)
|
||||
```
|
||||
@@ -1,334 +0,0 @@
|
||||
# Managed Agents — Python
|
||||
|
||||
> **Bindings not shown here:** This README covers the most common managed-agents flows for Python. If you need a class, method, namespace, field, or behavior that isn't shown, WebFetch the Python SDK repo **or the relevant docs page** from `shared/live-sources.md` rather than guess. Do not extrapolate from cURL shapes or another language's SDK.
|
||||
|
||||
> **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `agents.create` and pass it to every subsequent `sessions.create`; do not call `agents.create` in the request path. The Anthropic CLI is one convenient way to create agents and environments from version-controlled YAML — its URL is in `shared/live-sources.md`. The examples below show in-code creation for completeness; in production the create call belongs in setup, not in the request path.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install anthropic
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
|
||||
# Default — resolves credentials from the environment:
|
||||
# ANTHROPIC_API_KEY, or ANTHROPIC_AUTH_TOKEN, or an `ant auth login` profile.
|
||||
# Prefer this for local dev; don't hardcode a key.
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
# Explicit API key (only when you must inject a specific key)
|
||||
client = anthropic.Anthropic(api_key="your-api-key")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Environment
|
||||
|
||||
```python
|
||||
environment = client.beta.environments.create(
|
||||
name="my-dev-env",
|
||||
config={
|
||||
"type": "cloud",
|
||||
"networking": {"type": "unrestricted"},
|
||||
},
|
||||
)
|
||||
print(environment.id) # env_...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Agent (required first step)
|
||||
|
||||
> ⚠️ **There is no inline agent config.** `model`/`system`/`tools` live on the agent object, not the session. Always start with `agents.create()` — the session only takes `agent={"type": "agent", "id": agent.id}`.
|
||||
|
||||
### Minimal
|
||||
|
||||
```python
|
||||
# 1. Create the agent (reusable, versioned)
|
||||
agent = client.beta.agents.create(
|
||||
name="Coding Assistant",
|
||||
model="claude-opus-4-8",
|
||||
tools=[{"type": "agent_toolset_20260401", "default_config": {"enabled": True}}],
|
||||
)
|
||||
|
||||
# 2. Start a session
|
||||
session = client.beta.sessions.create(
|
||||
agent={"type": "agent", "id": agent.id, "version": agent.version},
|
||||
environment_id=environment.id,
|
||||
)
|
||||
print(session.id, session.status)
|
||||
```
|
||||
|
||||
### With system prompt and custom tools
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
agent = client.beta.agents.create(
|
||||
name="Code Reviewer",
|
||||
model="claude-opus-4-8",
|
||||
system="You are a senior code reviewer.",
|
||||
tools=[
|
||||
{"type": "agent_toolset_20260401"},
|
||||
{
|
||||
"type": "custom",
|
||||
"name": "run_tests",
|
||||
"description": "Run the test suite",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"test_path": {"type": "string", "description": "Path to test file"}
|
||||
},
|
||||
"required": ["test_path"],
|
||||
},
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
session = client.beta.sessions.create(
|
||||
agent={"type": "agent", "id": agent.id, "version": agent.version},
|
||||
environment_id=environment.id,
|
||||
title="Code review session",
|
||||
resources=[
|
||||
{
|
||||
"type": "github_repository",
|
||||
"url": "https://github.com/owner/repo",
|
||||
"mount_path": "/workspace/repo",
|
||||
"authorization_token": os.environ["GITHUB_TOKEN"],
|
||||
"branch": "main",
|
||||
}
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Send a User Message
|
||||
|
||||
```python
|
||||
client.beta.sessions.events.send(
|
||||
session_id=session.id,
|
||||
events=[
|
||||
{
|
||||
"type": "user.message",
|
||||
"content": [{"type": "text", "text": "Review the auth module"}],
|
||||
}
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
> 💡 **Stream-first:** Open the stream *before* (or concurrently with) sending the message. The stream only delivers events that occur after it opens — stream-after-send means early events arrive buffered in one batch. See [Steering Patterns](../../shared/managed-agents-events.md#steering-patterns).
|
||||
|
||||
---
|
||||
|
||||
## Stream Events (SSE)
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# Stream-first: open stream, then send while stream is live
|
||||
with client.beta.sessions.events.stream(
|
||||
session_id=session.id,
|
||||
) as stream:
|
||||
client.beta.sessions.events.send(
|
||||
session_id=session.id,
|
||||
events=[{"type": "user.message", "content": [{"type": "text", "text": "..."}]}],
|
||||
)
|
||||
for event in stream:
|
||||
... # process events
|
||||
|
||||
# Standalone stream iteration:
|
||||
with client.beta.sessions.events.stream(
|
||||
session_id=session.id,
|
||||
) as stream:
|
||||
for event in stream:
|
||||
if event.type == "agent.message":
|
||||
for block in event.content:
|
||||
if block.type == "text":
|
||||
print(block.text, end="", flush=True)
|
||||
elif event.type == "agent.custom_tool_use":
|
||||
# Custom tool invocation — session is now idle
|
||||
print(f"\nCustom tool call: {event.name}")
|
||||
print(f"Input: {json.dumps(event.input)}")
|
||||
# Send result back (see below)
|
||||
elif event.type == "session.status_idle":
|
||||
print("\n--- Agent idle ---")
|
||||
elif event.type == "session.status_terminated":
|
||||
print("\n--- Session terminated ---")
|
||||
break
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provide Custom Tool Result
|
||||
|
||||
```python
|
||||
client.beta.sessions.events.send(
|
||||
session_id=session.id,
|
||||
events=[
|
||||
{
|
||||
"type": "user.custom_tool_result",
|
||||
"custom_tool_use_id": "sevt_abc123",
|
||||
"content": [{"type": "text", "text": "All 42 tests passed."}],
|
||||
}
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Poll Events
|
||||
|
||||
```python
|
||||
events = client.beta.sessions.events.list(
|
||||
session_id=session.id,
|
||||
)
|
||||
for event in events.data:
|
||||
print(f"{event.type}: {event.id}")
|
||||
```
|
||||
|
||||
> ⚠️ **Prefer the SDK over raw `requests`/`httpx`.** If you hand-roll a poll loop, don't assume `timeout=(5, 60)` or `httpx.Timeout(120)` caps total call duration — both are **per-chunk** read timeouts (reset on every byte), so a trickling response can block forever. For a hard wall-clock deadline, track `time.monotonic()` at the loop level and bail explicitly, or wrap with `asyncio.wait_for()`. See [Receiving Events](../../shared/managed-agents-events.md#receiving-events).
|
||||
|
||||
---
|
||||
|
||||
## Full Streaming Loop with Custom Tools
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
|
||||
def run_custom_tool(tool_name: str, tool_input: dict) -> str:
|
||||
"""Execute a custom tool and return the result."""
|
||||
if tool_name == "run_tests":
|
||||
# Your tool implementation here
|
||||
return "All tests passed."
|
||||
return f"Unknown tool: {tool_name}"
|
||||
|
||||
|
||||
def run_session(client, session_id: str):
|
||||
"""Stream events and handle custom tool calls."""
|
||||
while True:
|
||||
with client.beta.sessions.events.stream(
|
||||
session_id=session_id,
|
||||
) as stream:
|
||||
tool_calls = []
|
||||
for event in stream:
|
||||
if event.type == "agent.message":
|
||||
for block in event.content:
|
||||
if block.type == "text":
|
||||
print(block.text, end="", flush=True)
|
||||
elif event.type == "agent.custom_tool_use":
|
||||
tool_calls.append(event)
|
||||
elif event.type == "session.status_idle":
|
||||
break
|
||||
elif event.type == "session.status_terminated":
|
||||
return
|
||||
|
||||
if not tool_calls:
|
||||
break
|
||||
|
||||
# Process custom tool calls
|
||||
results = []
|
||||
for call in tool_calls:
|
||||
result = run_custom_tool(call.name, call.input)
|
||||
results.append({
|
||||
"type": "user.custom_tool_result",
|
||||
"custom_tool_use_id": call.id,
|
||||
"content": [{"type": "text", "text": result}],
|
||||
})
|
||||
|
||||
client.beta.sessions.events.send(
|
||||
session_id=session_id,
|
||||
events=results,
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
```python
|
||||
with open("data.csv", "rb") as f:
|
||||
file = client.beta.files.upload(
|
||||
file=f,
|
||||
)
|
||||
|
||||
# Use in a session
|
||||
session = client.beta.sessions.create(
|
||||
agent={"type": "agent", "id": agent.id, "version": agent.version},
|
||||
environment_id=environment.id,
|
||||
resources=[{"type": "file", "file_id": file.id, "mount_path": "/workspace/data.csv"}],
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List and Download Session Files
|
||||
|
||||
List files the agent wrote to `/mnt/session/outputs/` during a session, then download them.
|
||||
|
||||
```python
|
||||
# List files associated with a session
|
||||
files = client.beta.files.list(
|
||||
scope_id=session.id,
|
||||
betas=["managed-agents-2026-04-01"],
|
||||
)
|
||||
for f in files.data:
|
||||
print(f.filename, f.size_bytes)
|
||||
# Download each file and save to disk
|
||||
file_content = client.beta.files.download(f.id)
|
||||
file_content.write_to_file(f.filename)
|
||||
```
|
||||
|
||||
> 💡 There's a brief indexing lag (~1–3s) between `session.status_idle` and output files appearing in `files.list`. Retry once or twice if the list is empty.
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
```python
|
||||
# Get session details
|
||||
session = client.beta.sessions.retrieve(session_id="sesn_011CZxAbc123Def456")
|
||||
print(session.status, session.usage)
|
||||
|
||||
# List sessions
|
||||
sessions = client.beta.sessions.list()
|
||||
|
||||
# Delete a session
|
||||
client.beta.sessions.delete(session_id="sesn_011CZxAbc123Def456")
|
||||
|
||||
# Archive a session
|
||||
client.beta.sessions.archive(session_id="sesn_011CZxAbc123Def456")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Integration
|
||||
|
||||
```python
|
||||
# Agent declares MCP server (no auth here — auth goes in a vault)
|
||||
agent = client.beta.agents.create(
|
||||
name="MCP Agent",
|
||||
model="claude-opus-4-8",
|
||||
mcp_servers=[
|
||||
{"type": "url", "name": "my-tools", "url": "https://my-mcp-server.example.com/sse"},
|
||||
],
|
||||
tools=[
|
||||
{"type": "agent_toolset_20260401", "default_config": {"enabled": True}},
|
||||
{"type": "mcp_toolset", "mcp_server_name": "my-tools"},
|
||||
],
|
||||
)
|
||||
|
||||
# Session attaches vault(s) containing credentials for those MCP server URLs
|
||||
session = client.beta.sessions.create(
|
||||
agent=agent.id,
|
||||
environment_id=environment.id,
|
||||
vault_ids=[vault.id],
|
||||
)
|
||||
```
|
||||
|
||||
See `shared/managed-agents-tools.md` §Vaults for creating vaults and adding credentials.
|
||||
@@ -1,140 +0,0 @@
|
||||
# Claude API — Ruby
|
||||
|
||||
> **Note:** The Ruby SDK supports the Claude API. A tool runner is available in beta via `client.beta.messages.tool_runner()`. Agent SDK is not yet available for Ruby.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
gem install anthropic
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```ruby
|
||||
require "anthropic"
|
||||
|
||||
# Default (uses ANTHROPIC_API_KEY env var)
|
||||
client = Anthropic::Client.new
|
||||
|
||||
# Explicit API key
|
||||
client = Anthropic::Client.new(api_key: "your-api-key")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```ruby
|
||||
message = client.messages.create(
|
||||
model: :"claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{ role: "user", content: "What is the capital of France?" }
|
||||
]
|
||||
)
|
||||
# content is an array of polymorphic block objects (TextBlock, ThinkingBlock,
|
||||
# ToolUseBlock, ...). .type is a Symbol — compare with :text, not "text".
|
||||
# .text raises NoMethodError on non-TextBlock entries.
|
||||
message.content.each do |block|
|
||||
puts block.text if block.type == :text
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming
|
||||
|
||||
```ruby
|
||||
stream = client.messages.stream(
|
||||
model: :"claude-opus-4-8",
|
||||
max_tokens: 64000,
|
||||
messages: [{ role: "user", content: "Write a haiku" }]
|
||||
)
|
||||
|
||||
stream.text.each { |text| print(text) }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tool Use
|
||||
|
||||
The Ruby SDK supports tool use via raw JSON schema definitions and also provides a beta tool runner for automatic tool execution.
|
||||
|
||||
### Tool Runner (Beta)
|
||||
|
||||
```ruby
|
||||
class GetWeatherInput < Anthropic::BaseModel
|
||||
required :location, String, doc: "City and state, e.g. San Francisco, CA"
|
||||
end
|
||||
|
||||
class GetWeather < Anthropic::BaseTool
|
||||
doc "Get the current weather for a location"
|
||||
|
||||
input_schema GetWeatherInput
|
||||
|
||||
def call(input)
|
||||
"The weather in #{input.location} is sunny and 72°F."
|
||||
end
|
||||
end
|
||||
|
||||
client.beta.messages.tool_runner(
|
||||
model: :"claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: [GetWeather.new],
|
||||
messages: [{ role: "user", content: "What's the weather in San Francisco?" }]
|
||||
).each_message do |message|
|
||||
puts message.content
|
||||
end
|
||||
```
|
||||
|
||||
### Manual Loop
|
||||
|
||||
See the [shared tool use concepts](../shared/tool-use-concepts.md) for the tool definition format and agentic loop pattern.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
`system_:` (trailing underscore — avoids shadowing `Kernel#system`) takes an array of text blocks; set `cache_control` on the last block. Plain hashes work via the `OrHash` type alias. For placement patterns and the silent-invalidator audit checklist, see `shared/prompt-caching.md`.
|
||||
|
||||
```ruby
|
||||
message = client.messages.create(
|
||||
model: :"claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
system_: [
|
||||
{ type: "text", text: long_system_prompt, cache_control: { type: "ephemeral" } }
|
||||
],
|
||||
messages: [{ role: "user", content: "Summarize the key points" }]
|
||||
)
|
||||
```
|
||||
|
||||
For 1-hour TTL: `cache_control: { type: "ephemeral", ttl: "1h" }`. There's also a top-level `cache_control:` on `messages.create` that auto-places on the last cacheable block.
|
||||
|
||||
Verify hits via `message.usage.cache_creation_input_tokens` / `message.usage.cache_read_input_tokens`.
|
||||
|
||||
---
|
||||
|
||||
## Stop Details
|
||||
|
||||
When `stop_reason` is `:refusal`, the response includes structured `stop_details`:
|
||||
|
||||
```ruby
|
||||
if message.stop_reason == :refusal && message.stop_details
|
||||
puts "Category: #{message.stop_details.category}" # :cyber, :bio, or nil
|
||||
puts "Explanation: #{message.stop_details.explanation}"
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Type
|
||||
|
||||
`APIStatusError` exposes a `.type` field for programmatic error classification:
|
||||
|
||||
```ruby
|
||||
begin
|
||||
client.messages.create(...)
|
||||
rescue Anthropic::APIStatusError => e
|
||||
puts e.type # :rate_limit_error, :overloaded_error, etc.
|
||||
end
|
||||
```
|
||||
@@ -1,389 +0,0 @@
|
||||
# Managed Agents — Ruby
|
||||
|
||||
> **Bindings not shown here:** This README covers the most common managed-agents flows for Ruby. If you need a class, method, namespace, field, or behavior that isn't shown, WebFetch the Ruby SDK repo **or the relevant docs page** from `shared/live-sources.md` rather than guess. Do not extrapolate from cURL shapes or another language's SDK.
|
||||
|
||||
> **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `client.beta.agents.create` and pass it to every subsequent `client.beta.sessions.create`; do not call `agents.create` in the request path. The Anthropic CLI is one convenient way to create agents and environments from version-controlled YAML — its URL is in `shared/live-sources.md`. The examples below show in-code creation for completeness; in production the create call belongs in setup, not in the request path.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
gem install anthropic
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```ruby
|
||||
require "anthropic"
|
||||
|
||||
# Default (uses ANTHROPIC_API_KEY env var)
|
||||
client = Anthropic::Client.new
|
||||
|
||||
# Explicit API key
|
||||
client = Anthropic::Client.new(api_key: "your-api-key")
|
||||
```
|
||||
|
||||
> ⚠️ **Trailing underscores:** The Ruby SDK uses `system_:` and `send_(` (trailing underscore) to avoid shadowing `Kernel#system` and `Kernel#send`. Use these forms throughout managed-agents code.
|
||||
|
||||
---
|
||||
|
||||
## Create an Environment
|
||||
|
||||
```ruby
|
||||
environment = client.beta.environments.create(
|
||||
name: "my-dev-env",
|
||||
config: {
|
||||
type: "cloud",
|
||||
networking: {type: "unrestricted"}
|
||||
}
|
||||
)
|
||||
puts "Environment ID: #{environment.id}" # env_...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Agent (required first step)
|
||||
|
||||
> ⚠️ **There is no inline agent config.** `model`/`system_`/`tools` live on the agent object, not the session. Always start with `client.beta.agents.create()` — the session takes either `agent: agent.id` or the typed hash form `agent: {type: "agent", id: agent.id, version: agent.version}`.
|
||||
|
||||
### Minimal
|
||||
|
||||
```ruby
|
||||
# 1. Create the agent (reusable, versioned)
|
||||
agent = client.beta.agents.create(
|
||||
name: "Coding Assistant",
|
||||
model: :"claude-opus-4-8",
|
||||
system_: "You are a helpful coding assistant.",
|
||||
tools: [{type: "agent_toolset_20260401"}]
|
||||
)
|
||||
|
||||
# 2. Start a session
|
||||
session = client.beta.sessions.create(
|
||||
agent: {type: "agent", id: agent.id, version: agent.version},
|
||||
environment_id: environment.id,
|
||||
title: "Quickstart session"
|
||||
)
|
||||
puts "Session ID: #{session.id}"
|
||||
```
|
||||
|
||||
### Updating an Agent
|
||||
|
||||
Updates create new versions; the agent object is immutable per version.
|
||||
|
||||
```ruby
|
||||
updated_agent = client.beta.agents.update(
|
||||
agent.id,
|
||||
version: agent.version,
|
||||
system_: "You are a helpful coding agent. Always write tests."
|
||||
)
|
||||
puts "New version: #{updated_agent.version}"
|
||||
|
||||
# List all versions
|
||||
client.beta.agents.versions.list(agent.id).auto_paging_each do |version|
|
||||
puts "Version #{version.version}: #{version.updated_at.iso8601}"
|
||||
end
|
||||
|
||||
# Archive the agent
|
||||
archived = client.beta.agents.archive(agent.id)
|
||||
puts "Archived at: #{archived.archived_at.iso8601}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Send a User Message
|
||||
|
||||
```ruby
|
||||
client.beta.sessions.events.send_(
|
||||
session.id,
|
||||
events: [{
|
||||
type: "user.message",
|
||||
content: [{type: "text", text: "Review the auth module"}]
|
||||
}]
|
||||
)
|
||||
```
|
||||
|
||||
> 💡 **Stream-first:** Open the stream *before* (or concurrently with) sending the message. The stream only delivers events that occur after it opens — stream-after-send means early events arrive buffered in one batch. See [Steering Patterns](../../shared/managed-agents-events.md#steering-patterns).
|
||||
|
||||
---
|
||||
|
||||
## Stream Events (SSE)
|
||||
|
||||
```ruby
|
||||
# Open the stream first, then send the user message
|
||||
stream = client.beta.sessions.events.stream_events(session.id)
|
||||
|
||||
client.beta.sessions.events.send_(
|
||||
session.id,
|
||||
events: [{
|
||||
type: "user.message",
|
||||
content: [{type: "text", text: "Summarize the repo README"}]
|
||||
}]
|
||||
)
|
||||
|
||||
stream.each do |event|
|
||||
case event.type
|
||||
in :"agent.message"
|
||||
event.content.each { |block| print block.text }
|
||||
in :"agent.tool_use"
|
||||
puts "\n[Using tool: #{event.name}]"
|
||||
in :"session.status_idle"
|
||||
break
|
||||
in :"session.error"
|
||||
puts "\n[Error: #{event.error&.message || "unknown"}]"
|
||||
break
|
||||
else
|
||||
# ignore other event types
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
> ℹ️ Event `.type` is a Symbol (compare with `:"agent.message"`, not `"agent.message"`).
|
||||
|
||||
### Reconnecting and Tailing
|
||||
|
||||
When reconnecting mid-session, list past events first to dedupe, then tail live events:
|
||||
|
||||
```ruby
|
||||
require "set"
|
||||
|
||||
stream = client.beta.sessions.events.stream_events(session.id)
|
||||
|
||||
# Stream is open and buffering. List history before tailing live.
|
||||
seen_event_ids = Set.new
|
||||
client.beta.sessions.events.list(session.id).auto_paging_each { |past| seen_event_ids << past.id }
|
||||
|
||||
# Tail live events, skipping anything already seen
|
||||
stream.each do |event|
|
||||
next if seen_event_ids.include?(event.id)
|
||||
seen_event_ids << event.id
|
||||
case event.type
|
||||
in :"agent.message"
|
||||
event.content.each { |block| print block.text }
|
||||
in :"session.status_idle"
|
||||
break
|
||||
else
|
||||
# ignore other event types
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provide Custom Tool Result
|
||||
|
||||
> ℹ️ The Ruby managed-agents bindings for `user.custom_tool_result` are not yet documented in this skill or in the apps source examples. Refer to `shared/managed-agents-events.md` for the wire format and the `anthropic` Ruby gem repository for the corresponding params.
|
||||
|
||||
---
|
||||
|
||||
## Poll Events
|
||||
|
||||
```ruby
|
||||
client.beta.sessions.events.list(session.id).auto_paging_each do |event|
|
||||
puts "#{event.type}: #{event.id}"
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
```ruby
|
||||
require "pathname"
|
||||
|
||||
file = client.beta.files.upload(file: Pathname("data.csv"))
|
||||
puts "File ID: #{file.id}"
|
||||
|
||||
# Mount in a session
|
||||
session = client.beta.sessions.create(
|
||||
agent: agent.id,
|
||||
environment_id: environment.id,
|
||||
resources: [
|
||||
{
|
||||
type: "file",
|
||||
file_id: file.id,
|
||||
mount_path: "/workspace/data.csv"
|
||||
}
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### Add and Manage Resources on an Existing Session
|
||||
|
||||
```ruby
|
||||
# Attach an additional file to an open session
|
||||
resource = client.beta.sessions.resources.add(
|
||||
session.id,
|
||||
type: "file",
|
||||
file_id: file.id
|
||||
)
|
||||
puts resource.id # "sesrsc_01ABC..."
|
||||
|
||||
# List resources on the session
|
||||
listed = client.beta.sessions.resources.list(session.id)
|
||||
listed.data.each { |entry| puts "#{entry.id} #{entry.type}" }
|
||||
|
||||
# Detach a resource
|
||||
client.beta.sessions.resources.delete(resource.id, session_id: session.id)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List and Download Session Files
|
||||
|
||||
> ℹ️ Listing and downloading files an agent wrote during a session is not yet documented for Ruby in this skill or in the apps source examples. See `shared/managed-agents-events.md` and the `anthropic` Ruby gem repository for the file list/download bindings.
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
```ruby
|
||||
# List environments
|
||||
environments = client.beta.environments.list
|
||||
|
||||
# Retrieve a specific environment
|
||||
env = client.beta.environments.retrieve(environment.id)
|
||||
|
||||
# Archive an environment (read-only, existing sessions continue)
|
||||
client.beta.environments.archive(environment.id)
|
||||
|
||||
# Delete an environment (only if no sessions reference it)
|
||||
client.beta.environments.delete(environment.id)
|
||||
|
||||
# Delete a session
|
||||
client.beta.sessions.delete(session.id)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Integration
|
||||
|
||||
```ruby
|
||||
# Agent declares MCP server (no auth here — auth goes in a vault)
|
||||
agent = client.beta.agents.create(
|
||||
name: "GitHub Assistant",
|
||||
model: :"claude-opus-4-8",
|
||||
mcp_servers: [
|
||||
{
|
||||
type: "url",
|
||||
name: "github",
|
||||
url: "https://api.githubcopilot.com/mcp/"
|
||||
}
|
||||
],
|
||||
tools: [
|
||||
{type: "agent_toolset_20260401"},
|
||||
{type: "mcp_toolset", mcp_server_name: "github"}
|
||||
]
|
||||
)
|
||||
|
||||
# Session attaches vault(s) containing credentials for those MCP server URLs
|
||||
session = client.beta.sessions.create(
|
||||
agent: {type: "agent", id: agent.id, version: agent.version},
|
||||
environment_id: environment.id,
|
||||
vault_ids: [vault.id]
|
||||
)
|
||||
```
|
||||
|
||||
See `shared/managed-agents-tools.md` §Vaults for creating vaults and adding credentials.
|
||||
|
||||
---
|
||||
|
||||
## Vaults
|
||||
|
||||
```ruby
|
||||
# Create a vault
|
||||
vault = client.beta.vaults.create(
|
||||
display_name: "Alice",
|
||||
metadata: {external_user_id: "usr_abc123"}
|
||||
)
|
||||
puts vault.id # "vlt_01ABC..."
|
||||
|
||||
# Add an OAuth credential
|
||||
credential = client.beta.vaults.credentials.create(
|
||||
vault.id,
|
||||
display_name: "Alice's Slack",
|
||||
auth: {
|
||||
type: "mcp_oauth",
|
||||
mcp_server_url: "https://mcp.slack.com/mcp",
|
||||
access_token: "xoxp-...",
|
||||
expires_at: "2026-04-15T00:00:00Z",
|
||||
refresh: {
|
||||
token_endpoint: "https://slack.com/api/oauth.v2.access",
|
||||
client_id: "1234567890.0987654321",
|
||||
scope: "channels:read chat:write",
|
||||
refresh_token: "xoxe-1-...",
|
||||
token_endpoint_auth: {
|
||||
type: "client_secret_post",
|
||||
client_secret: "abc123..."
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
# Rotate the credential (e.g., after a token refresh)
|
||||
client.beta.vaults.credentials.update(
|
||||
credential.id,
|
||||
vault_id: vault.id,
|
||||
auth: {
|
||||
type: "mcp_oauth",
|
||||
access_token: "xoxp-new-...",
|
||||
expires_at: "2026-05-15T00:00:00Z",
|
||||
refresh: {refresh_token: "xoxe-1-new-..."}
|
||||
}
|
||||
)
|
||||
|
||||
# Archive a vault
|
||||
client.beta.vaults.archive(vault.id)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GitHub Repository Integration
|
||||
|
||||
Mount a GitHub repository as a session resource (a vault holds the GitHub MCP credential):
|
||||
|
||||
```ruby
|
||||
session = client.beta.sessions.create(
|
||||
agent: agent.id,
|
||||
environment_id: environment.id,
|
||||
vault_ids: [vault.id],
|
||||
resources: [
|
||||
{
|
||||
type: "github_repository",
|
||||
url: "https://github.com/org/repo",
|
||||
mount_path: "/workspace/repo",
|
||||
authorization_token: "ghp_your_github_token"
|
||||
}
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
Multiple repositories on the same session:
|
||||
|
||||
```ruby
|
||||
resources = [
|
||||
{
|
||||
type: "github_repository",
|
||||
url: "https://github.com/org/frontend",
|
||||
mount_path: "/workspace/frontend",
|
||||
authorization_token: "ghp_your_github_token"
|
||||
},
|
||||
{
|
||||
type: "github_repository",
|
||||
url: "https://github.com/org/backend",
|
||||
mount_path: "/workspace/backend",
|
||||
authorization_token: "ghp_your_github_token"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Rotating a repository's authorization token:
|
||||
|
||||
```ruby
|
||||
listed = client.beta.sessions.resources.list(session.id)
|
||||
repo_resource_id = listed.data.first.id
|
||||
|
||||
client.beta.sessions.resources.update(
|
||||
repo_resource_id,
|
||||
session_id: session.id,
|
||||
authorization_token: "ghp_your_new_github_token"
|
||||
)
|
||||
```
|
||||
@@ -1,101 +0,0 @@
|
||||
# Agent Design Patterns
|
||||
|
||||
This file covers decision heuristics for building agents on the Claude API: which primitives to reach for, how to design your tool surface, and how to manage context and cost over long runs. For per-tool mechanics and code examples, see `tool-use-concepts.md` and the language-specific folders.
|
||||
|
||||
---
|
||||
|
||||
## Model Parameters
|
||||
|
||||
| Parameter | When to use it | What to expect |
|
||||
| --- | --- | --- |
|
||||
| **Adaptive thinking** (`thinking: {type: "adaptive"}`) | When you want Claude to control when and how much to think. | Claude determines thinking depth per request and automatically interleaves thinking between tool calls. No token budget to tune. |
|
||||
| **Effort** (`output_config: {effort: ...}`) | When adjusting the tradeoff between thoroughness and token efficiency. | Lower effort → fewer and more-consolidated tool calls, less preamble, terser confirmations. `medium` is often a favorable balance. Use `max` when correctness matters more than cost. |
|
||||
|
||||
See `SKILL.md` §Thinking & Effort for model support and parameter details.
|
||||
|
||||
---
|
||||
|
||||
## Designing Your Tool Surface
|
||||
|
||||
### Bash vs. dedicated tools
|
||||
|
||||
Claude doesn't know your application's security boundary, approval policy, or UX surface. Claude emits tool calls; your harness handles them. The shape of those tool calls determines what the harness can do.
|
||||
|
||||
A **bash tool** gives Claude broad programmatic leverage — it can perform almost any action. But it gives the harness only an opaque command string, the same shape for every action. Promoting an action to a **dedicated tool** gives the harness an action-specific hook with typed arguments it can intercept, gate, render, or audit.
|
||||
|
||||
**When to promote an action to a dedicated tool:**
|
||||
|
||||
- **Security boundary.** Actions that require gating are natural candidates. Reversibility is a useful criterion: hard-to-reverse actions (external API calls, sending messages, deleting data) can be gated behind user confirmation. A `send_email` tool is easy to gate; `bash -c "curl -X POST ..."` is not.
|
||||
- **Staleness checks.** A dedicated `edit` tool can reject writes if the file changed since Claude last read it. Bash can't enforce that invariant.
|
||||
- **Rendering.** Some actions benefit from custom UI. Claude Code promotes question-asking to a tool so it can render as a modal, present options, and block the agent loop until answered.
|
||||
- **Scheduling.** Read-only tools like `glob` and `grep` can be marked parallel-safe. When the same actions run through bash, the harness can't tell a parallel-safe `grep` from a parallel-unsafe `git push`, so it must serialize.
|
||||
|
||||
**Rule of thumb:** Start with bash for breadth. Promote to dedicated tools when you need to gate, render, audit, or parallelize the action.
|
||||
|
||||
---
|
||||
|
||||
## Anthropic-Provided Tools
|
||||
|
||||
| Tool | Side | When to use it | What to expect |
|
||||
| --- | --- | --- | --- |
|
||||
| **Bash** | Client | Claude needs to execute shell commands. | Claude emits commands; your harness executes them. Reference implementation provided. |
|
||||
| **Text editor** | Client | Claude needs to read or edit files. | Claude views, creates, and edits files via your implementation. Reference implementation provided. |
|
||||
| **Computer use** | Client or Server | Claude needs to interact with GUIs, web apps, or visual interfaces. | Claude takes screenshots and issues mouse/keyboard commands. Can be self-hosted (you run the environment) or Anthropic-hosted. |
|
||||
| **Code execution** | Server | Claude needs to run code in a sandbox you don't want to manage. | Anthropic-hosted container with built-in file and bash sub-tools. No client-side execution. |
|
||||
| **Web search / fetch** | Server | Claude needs information past its training cutoff (news, current events, recent docs) or the content of a specific URL. | Claude issues a query or URL; Anthropic executes it and returns results with citations. |
|
||||
| **Memory** | Client | Claude needs to save context across sessions. | Claude reads/writes a `/memories` directory. You implement the storage backend. |
|
||||
|
||||
**Client-side** tools are defined by Anthropic (name, schema, Claude's usage pattern) but executed by your harness. Anthropic provides reference implementations. **Server-side** tools run entirely on Anthropic infrastructure — declare them in `tools` and Claude handles the rest.
|
||||
|
||||
---
|
||||
|
||||
## Composing Tool Calls: Programmatic Tool Calling
|
||||
|
||||
With standard tool use, each tool call is a round trip: Claude calls the tool, the result lands in Claude's context, Claude reasons about it, then calls the next tool. Three sequential actions (read profile → look up orders → check inventory) means three round trips. Each adds latency and tokens, and most of the intermediate data is never needed again.
|
||||
|
||||
**Programmatic tool calling (PTC)** lets Claude compose those calls into a script instead. The script runs in the code execution container. When the script calls a tool, the container pauses, the call is executed (client-side or server-side), and the result returns to the running code — not to Claude's context. The script processes it with normal control flow (loops, filters, branches). Only the script's final output returns to Claude.
|
||||
|
||||
| When to use it | What to expect |
|
||||
| --- | --- |
|
||||
| Many sequential tool calls, or large intermediate results you want filtered before they hit the context window. | Claude writes code that invokes tools as functions. Runs in the code execution container. Token cost scales with final output, not intermediate results. |
|
||||
|
||||
---
|
||||
|
||||
## Scaling the Tool and Instruction Set
|
||||
|
||||
| Feature | When to use it | What to expect |
|
||||
| --- | --- | --- |
|
||||
| **Tool search** | Many tools available, but only a few relevant per request. Don't want all schemas in context upfront. | Claude searches the tool set and loads only relevant schemas. Tool definitions are appended, not swapped — preserves cache (see Caching below). |
|
||||
| **Skills** | Task-specific instructions Claude should load only when relevant. | Each skill is a folder with a `SKILL.md`. The skill's description sits in context by default; Claude reads the full file when the task calls for it. |
|
||||
|
||||
Both patterns keep the fixed context small and load detail on demand.
|
||||
|
||||
---
|
||||
|
||||
## Long-Running Agents: Managing Context
|
||||
|
||||
| Pattern | When to use it | What to expect |
|
||||
| --- | --- | --- |
|
||||
| **Context editing** | Context grows stale over many turns (old tool results, completed thinking). | Tool results and thinking blocks are cleared based on configurable thresholds. Keeps the transcript lean without summarizing. |
|
||||
| **Compaction** | Conversation likely to reach or exceed the context window limit. | Earlier context is summarized into a compaction block server-side. See `SKILL.md` §Compaction for the critical `response.content` handling. |
|
||||
| **Memory** | State must persist across sessions (not just within one conversation). | Claude reads/writes files in a memory directory. Survives process restarts. |
|
||||
|
||||
**Choosing between them:** Context editing and compaction operate within a session — editing prunes stale turns, compaction summarizes when you're near the limit. Memory is for cross-session persistence. Many long-running agents use all three.
|
||||
|
||||
---
|
||||
|
||||
## Caching for Agents
|
||||
|
||||
**Read `prompt-caching.md` first.** It covers the prefix-match invariant, breakpoint placement, the silent-invalidator audit, and why changing tools or models mid-session breaks the cache. This section covers only the agent-specific workarounds for those constraints.
|
||||
|
||||
| Constraint (from `prompt-caching.md`) | Agent-specific workaround |
|
||||
| --- | --- |
|
||||
| Editing the system prompt mid-session invalidates the cache. | Append a `{"role": "system", ...}` message to `messages[]` instead (beta, on supporting models — see `prompt-caching.md` § Mid-conversation system messages). The cached prefix stays intact, and the model treats it as an operator-authority instruction rather than user text. On models that don't support it, fall back to a `<system-reminder>` text block in the user turn. |
|
||||
| Switching models mid-session invalidates the cache. | Spawn a **subagent** with the cheaper model for the sub-task; keep the main loop on one model. Claude Code's Explore subagents use Haiku this way. |
|
||||
| Adding/removing tools mid-session invalidates the cache. | Use **tool search** for dynamic discovery — it appends tool schemas rather than swapping them, so the existing prefix is preserved. |
|
||||
|
||||
For multi-turn breakpoint placement, use top-level auto-caching — see `prompt-caching.md` §Placement patterns.
|
||||
|
||||
---
|
||||
|
||||
For live documentation on any of these features, see `live-sources.md`.
|
||||
@@ -1,246 +0,0 @@
|
||||
# Anthropic CLI (`ant`)
|
||||
|
||||
The `ant` CLI exposes every Claude API resource as a shell subcommand. Compared to `curl`: request bodies are built from typed flags or piped YAML instead of hand-written JSON, `@path` inlines file contents into any string field, `--transform` extracts fields with a GJSON path (no `jq`), list endpoints auto-paginate (cap total results with `--max-items N`; `--limit` only sets the server page size), and the `beta:` prefix auto-sets the right `anthropic-beta` header.
|
||||
|
||||
## When to use the CLI vs the SDK
|
||||
|
||||
**CLI for the control plane, SDK for the data plane.** Agents and environments are relatively static resources you define, configure, and debug with `ant` — check the YAML into your repo, apply from CI, inspect from a terminal. Sessions are dynamic and driven by your application through the SDK — create per task, stream events, react to tool calls, integrate into your product. Both hit the same API; the split is about where the call lives, not what's possible.
|
||||
|
||||
| | Control plane → `ant` | Data plane → SDK |
|
||||
|---|---|---|
|
||||
| Resources | agents, environments, skills, vaults, files | sessions, events |
|
||||
| Cadence | Once per deploy / ad-hoc | Every task / every turn |
|
||||
| Lives in | `*.yaml` in your repo + CI + terminal | Application code |
|
||||
| Typical calls | `create < agent.yaml`, `update --version N`, `list`, `retrieve`, `archive`, `--debug` | `sessions.create()`, `events.stream()`, `events.send()` |
|
||||
|
||||
## Install and auth
|
||||
|
||||
```sh
|
||||
# macOS
|
||||
brew install anthropics/tap/ant
|
||||
xattr -d com.apple.quarantine "$(brew --prefix)/bin/ant"
|
||||
|
||||
# Linux / WSL — pick the release from github.com/anthropics/anthropic-cli/releases
|
||||
curl -fsSL "https://github.com/anthropics/anthropic-cli/releases/download/v${VERSION}/ant_${VERSION}_$(uname -s | tr A-Z a-z)_$(uname -m | sed -e s/x86_64/amd64/ -e s/aarch64/arm64/).tar.gz" \
|
||||
| sudo tar -xz -C /usr/local/bin ant
|
||||
|
||||
# Or from source (Go 1.22+)
|
||||
go install github.com/anthropics/anthropic-cli/cmd/ant@latest
|
||||
```
|
||||
|
||||
**Auth** — the CLI resolves credentials the same way the SDKs do (first match wins): explicit flags, then `ANTHROPIC_API_KEY`, then `ANTHROPIC_AUTH_TOKEN`, then the `ANTHROPIC_PROFILE`-selected or active profile, then Workload Identity Federation env vars, then the default profile on disk. Override the host with `ANTHROPIC_BASE_URL` or `--base-url`.
|
||||
|
||||
- **API key**: set `ANTHROPIC_API_KEY` in the environment.
|
||||
- **OAuth profile** (no static key to manage): `ant auth login` opens a browser, exchanges for a short-lived token, and stores a profile under `$ANTHROPIC_CONFIG_DIR` (default `~/.config/anthropic/` on Linux/macOS, `%APPDATA%\Anthropic` on Windows — `configs/<profile>.json` for settings, `credentials/<profile>.json` for tokens). Subsequent `ant` (and SDK) calls pick it up automatically — a bare `Anthropic()` client works after login, but scripts that read `ANTHROPIC_API_KEY` directly do not. Claude Code and the Claude Agent SDK honor the same profile resolution. `ant auth status` shows which credential source and profile won (it reports status only — don't script against its exit code as a health check); `ant auth logout` clears the active profile (`--all` for every profile). On a remote host without a browser, `ant auth login --no-browser` prints the authorize URL and accepts the code back in the terminal.
|
||||
- **Non-interactive workloads** (CI, servers, containers): interactive login is for development on your own machine — use Workload Identity Federation instead (see the authentication docs via `shared/live-sources.md`).
|
||||
|
||||
> **The #1 auth trap:** profiles are only consulted when no API key is set. A stale exported `ANTHROPIC_API_KEY` silently overrides every profile — requests hit whatever org/workspace that key is scoped to. `ant auth status` shows which source won; unset the key (or per-command: `env -u ANTHROPIC_API_KEY ant …`) before relying on a profile. Truly **unset** it — an empty `ANTHROPIC_API_KEY=""` still wins its precedence slot and authenticates with an empty key. The same shadowing applies in reverse to Claude Code: after `ant auth login`, Claude Code may warn about an auth conflict between the profile and its own `/login` credential — keep one (use the profile and `/logout` in Claude Code, or `ant auth logout` to keep Claude Code's own login).
|
||||
|
||||
**Named profiles** — an interactive-login token is bound to a single org+workspace, and the API only shows resources belonging to that workspace. If an agent, session, or file you created "disappears", the usual cause is a token scoped to a different workspace than the one that created it (`ant auth status` shows the active workspace). Multi-workspace work means one profile per workspace:
|
||||
|
||||
```sh
|
||||
ant auth login --profile <name> # creates the profile if it doesn't exist; org/workspace picker in browser
|
||||
ant auth login --profile <name> --workspace-id wrkspc_01... # bind directly, skip the picker
|
||||
ant profile activate <name> # switch the default profile
|
||||
ant --profile <name> models list # one-off; equivalent: ANTHROPIC_PROFILE=<name> ant models list
|
||||
ant profile list # inspect
|
||||
ant profile set workspace_id wrkspc_01... --profile <name> # edit config keys (workspace_id, base_url, organization_id, …)
|
||||
```
|
||||
|
||||
`ant profile set` edits an existing profile's config — it never creates one, and it does **not** rebind already-issued credentials; run `ant auth login` again under that profile to mint a token for the new target. Pointing `ANTHROPIC_PROFILE` at a profile that doesn't exist is an error, not a fall-through. Refresh tokens eventually hard-expire (they don't slide with use) — when a previously working profile starts failing auth, re-run `ant auth login` before debugging anything else.
|
||||
|
||||
**Scopes** — a profile's OAuth scope set is requested at login (`--scope`) and persists on the profile (`scope` is also a `profile set` config key; like other config edits, changing it requires a fresh `ant auth login` to take effect). Privileged scopes — e.g. `org:admin` for organization-administration endpoints — are **not** in the default scope set: pass the full set you want explicitly (`ant auth login --profile admin --scope "... org:admin"`), and the server grants a privileged scope only if your role actually has it. Because the scope set rides on every token the profile mints, keep privileged work on a dedicated profile (`admin` vs `default`) and do day-to-day inference on the unprivileged one, switching with `--profile`/`ANTHROPIC_PROFILE`. Check `ant auth login --help` for the current scope list, and `ant auth status` to see what the active token carries.
|
||||
|
||||
To hand the active credential to a subprocess or raw-HTTP script:
|
||||
|
||||
```sh
|
||||
# Bare access token — for curl's Authorization header
|
||||
curl https://api.anthropic.com/v1/messages \
|
||||
-H "Authorization: Bearer $(ant auth print-credentials --access-token)" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-H "anthropic-beta: oauth-2025-04-20" \
|
||||
-H "content-type: application/json" \
|
||||
-d '{"model": "claude-opus-4-8", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
|
||||
|
||||
# .env format — sets ANTHROPIC_AUTH_TOKEN (and ANTHROPIC_BASE_URL if the profile has one).
|
||||
# Output is bare KEY=value (no `export`), so use `set -a` to auto-export for child processes:
|
||||
set -a; eval "$(ant auth print-credentials --env)"; set +a
|
||||
python my_script.py # SDK picks up ANTHROPIC_AUTH_TOKEN
|
||||
```
|
||||
|
||||
OAuth tokens go on `Authorization: Bearer` (not `x-api-key:`) **plus the `anthropic-beta: oauth-2025-04-20` header** — converting a raw curl/httpx script from an API key is a header change, not a key swap. The beta header requirement is endpoint-dependent (some endpoints happen to work without it; `/v1/messages` does not) — always send it so requests don't break when you switch endpoints. The token is short-lived and not auto-refreshed when passed via env var, so re-run `print-credentials` before it expires for long-running scripts (`print-credentials` itself refreshes the token if needed). If both `ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` are set, the SDKs send both and the API rejects the request — unset `ANTHROPIC_API_KEY` before `eval`ing the `--env` output.
|
||||
|
||||
**Foot-gun:** `ant auth print-credentials` with **no flags** prints the entire credentials JSON, not the bare token — putting that in an `Authorization` header yields an empty response or HTTP/2 protocol error. Always use `--access-token` for headers (it always reads the named/active profile; a set `ANTHROPIC_API_KEY` doesn't override credential printing).
|
||||
|
||||
## Command structure
|
||||
|
||||
```
|
||||
ant <resource>[:<subresource>] <action> [flags]
|
||||
```
|
||||
|
||||
Beta resources (agents, sessions, environments, deployments, skills, vaults, memory stores) live under `beta:` — the CLI auto-sends the right `anthropic-beta` header, so don't pass it yourself unless overriding with `--beta <header>`. For self-hosted environments, `ant beta:worker poll/run` and `ant beta:environments:work stats/stop` drive and monitor the work queue — see `shared/managed-agents-self-hosted-sandboxes.md`.
|
||||
|
||||
```sh
|
||||
ant models list
|
||||
ant messages create --model claude-opus-4-8 --max-tokens 1024 --message '{role: user, content: "Hello"}'
|
||||
ant beta:agents retrieve --agent-id agent_01...
|
||||
ant beta:sessions:events list --session-id session_01...
|
||||
```
|
||||
|
||||
`ant --help` lists resources; append `--help` to any subcommand for its flags.
|
||||
|
||||
## Global flags
|
||||
|
||||
| Flag | Purpose |
|
||||
| --- | --- |
|
||||
| `--format` | `auto` (default: pretty if TTY, compact if piped), `json`, `jsonl`, `yaml`, `pretty`, `raw`, `explore` (interactive TUI) |
|
||||
| `--transform` | GJSON path applied to the response (per-item on list endpoints). Not applied when `--format raw`. |
|
||||
| `-r`, `--raw-output` | If the transformed result is a string, print it without quotes (jq semantics). Pair with `--transform` for scalar capture. |
|
||||
| `--max-items` | Cap total results returned from auto-paginating list endpoints (distinct from `--limit`, which is the server page size). |
|
||||
| `--format-error` / `--transform-error` | Same as `--format`/`--transform`, applied to error responses. `-r` does not apply to the error path — use `--format-error yaml` for unquoted error scalars. |
|
||||
| `--base-url` | Override API host |
|
||||
| `--debug` | Print full HTTP request + response to stderr (API key redacted) |
|
||||
|
||||
## Output — `--transform` + `--format`
|
||||
|
||||
`--transform` takes a [GJSON path](https://github.com/tidwall/gjson/blob/master/SYNTAX.md). On list endpoints it runs **per item**, not on the envelope.
|
||||
|
||||
```sh
|
||||
ant beta:agents list --transform '{id,name,model}' --format jsonl
|
||||
```
|
||||
|
||||
**Extract a scalar for shell use:** pair `--transform` with `-r` (`--raw-output` — prints strings unquoted, jq-style):
|
||||
|
||||
```sh
|
||||
AGENT_ID=$(ant beta:agents create --name "My Agent" --model '{id: claude-sonnet-4-6}' \
|
||||
--transform id -r)
|
||||
```
|
||||
|
||||
## Input — flags, stdin, `@file`
|
||||
|
||||
**Flags** — scalar fields map directly. Structured fields accept relaxed-YAML syntax (unquoted keys) or strict JSON. Repeatable flags build arrays (each `--tool`, `--event`, `--message` appends one element):
|
||||
|
||||
```sh
|
||||
ant beta:agents create \
|
||||
--name "Research Agent" \
|
||||
--model '{id: claude-opus-4-8}' \
|
||||
--tool '{type: agent_toolset_20260401}' \
|
||||
--tool '{type: custom, name: search_docs, input_schema: {type: object, properties: {query: {type: string}}}}'
|
||||
```
|
||||
|
||||
**Stdin** — pipe a full JSON or YAML body. Merged with flags; flags win on conflict (for array fields, any flag **replaces** the stdin array entirely — it does not append). Quote the heredoc delimiter (`<<'YAML'`) to disable shell expansion inside the body:
|
||||
|
||||
```sh
|
||||
ant beta:agents create <<'YAML'
|
||||
name: Research Agent
|
||||
model: claude-opus-4-8
|
||||
system: |
|
||||
You are a research assistant. Cite sources for every claim.
|
||||
tools:
|
||||
- type: agent_toolset_20260401
|
||||
YAML
|
||||
```
|
||||
|
||||
**`@file` references** — inline a file's contents into any string-valued field. Inside structured flag values, quote the path. Binary files are auto-base64'd; force with `@file://` (text) or `@data://` (base64). Escape a literal leading `@` as `\@`.
|
||||
|
||||
```sh
|
||||
ant beta:agents create --name "Researcher" --model '{id: claude-sonnet-4-6}' --system @./prompts/researcher.txt
|
||||
|
||||
ant messages create --model claude-opus-4-8 --max-tokens 1024 \
|
||||
--message '{role: user, content: [
|
||||
{type: document, source: {type: base64, media_type: application/pdf, data: "@./scan.pdf"}},
|
||||
{type: text, text: "Extract the text from this scanned document."}
|
||||
]}' \
|
||||
--transform 'content.0.text' -r
|
||||
```
|
||||
|
||||
Flags that natively take a file path (e.g. `--file` on `beta:files upload`) accept a bare path without `@`.
|
||||
|
||||
## Version-controlled Managed Agents resources
|
||||
|
||||
This is the recommended flow for defining agents and environments — check the YAML into your repo and sync via `create` (first time) / `update` (thereafter). See `shared/managed-agents-core.md` for the field reference.
|
||||
|
||||
```yaml
|
||||
# summarizer.agent.yaml
|
||||
name: Summarizer
|
||||
model: claude-sonnet-4-6
|
||||
system: |
|
||||
You are a helpful assistant that writes concise summaries.
|
||||
tools:
|
||||
- type: agent_toolset_20260401
|
||||
```
|
||||
|
||||
```sh
|
||||
# Create (once) — capture the ID
|
||||
AGENT_ID=$(ant beta:agents create < summarizer.agent.yaml --transform id -r)
|
||||
|
||||
# Update (CI) — needs ID + current version (optimistic lock)
|
||||
ant beta:agents update --agent-id "$AGENT_ID" --version 1 < summarizer.agent.yaml
|
||||
```
|
||||
|
||||
Same pattern for environments (`ant beta:environments create|update < env.yaml`), then start a session with both IDs:
|
||||
|
||||
```sh
|
||||
ant beta:sessions create --agent "$AGENT_ID" --environment-id "$ENV_ID" --title "Task"
|
||||
ant beta:sessions:events send --session-id "$SID" \
|
||||
--event '{type: user.message, content: [{type: text, text: "Summarize X"}]}'
|
||||
ant beta:sessions:events list --session-id "$SID" --transform 'content.0.text' -r
|
||||
ant beta:sessions:events stream --session-id "$SID" # live event stream
|
||||
```
|
||||
|
||||
### Interactive session loop (stream-before-send)
|
||||
|
||||
`ant beta:sessions:events stream` only delivers events emitted *after* the stream opens — so open it **before** sending the kickoff to avoid missing early events. Use process substitution to hold the stream on a file descriptor, send, then read:
|
||||
|
||||
```sh
|
||||
exec {stream}< <(ant beta:sessions:events stream --session-id "$SID" \
|
||||
--transform '{type,text:content.#(type=="text").text,err:error.message}' --format yaml)
|
||||
|
||||
ant beta:sessions:events send --session-id "$SID" > /dev/null <<'YAML'
|
||||
events:
|
||||
- type: user.message
|
||||
content:
|
||||
- type: text
|
||||
text: Summarize the repo README
|
||||
YAML
|
||||
|
||||
type=
|
||||
while IFS= read -r -u "$stream" line; do
|
||||
case "$line" in
|
||||
type:\ session.status_idle) break ;;
|
||||
type:\ session.error)
|
||||
IFS= read -r -u "$stream" next || next=
|
||||
case "$next" in err:\ *) msg=${next#err: } ;; *) msg=unknown ;; esac
|
||||
printf '\n[Error: %s]\n' "$msg"; break ;;
|
||||
type:\ *) type=${line#type: } ;;
|
||||
text:*)
|
||||
[[ $type == agent.message ]] || continue
|
||||
val=${line#text: }
|
||||
case "$val" in '|-'|'|') ;; *) printf '%s' "$val" ;; esac ;;
|
||||
\ \ *)
|
||||
if [[ $type == agent.message ]]; then printf '%s\n' "${line# }"; fi ;;
|
||||
esac
|
||||
done
|
||||
exec {stream}<&-
|
||||
```
|
||||
|
||||
This works for interactive exploration and demos. For application code that needs to react to `agent.tool_use` / `agent.custom_tool_use` events, reconnect after drops, or dedup against `events.list`, use the SDK — see `shared/managed-agents-client-patterns.md`.
|
||||
|
||||
## Scripting patterns
|
||||
|
||||
`--transform id -r` on a list endpoint emits one bare ID per line — compose with `xargs`, or use `--max-items N` to bound the result set without piping through `head`:
|
||||
|
||||
```sh
|
||||
FIRST=$(ant beta:agents list --transform id -r --max-items 1)
|
||||
ant beta:agents:versions list --agent-id "$FIRST" --transform '{version,created_at}' --format jsonl
|
||||
```
|
||||
|
||||
Error shaping mirrors the success path (note: `-r` does not apply to error output — use `--format-error yaml` for an unquoted scalar here):
|
||||
|
||||
```sh
|
||||
ant beta:agents retrieve --agent-id bogus --transform-error error.message --format-error yaml 2>&1
|
||||
```
|
||||
|
||||
Shell completion: `ant @completion {zsh|bash|fish|powershell}`.
|
||||
|
||||
For the full, always-current reference (including per-endpoint flags), WebFetch the **Anthropic CLI** URL in `shared/live-sources.md`.
|
||||
@@ -1,59 +0,0 @@
|
||||
# Claude Platform on AWS
|
||||
|
||||
**Anthropic-operated** access to the Claude Developer Platform through AWS infrastructure — SigV4 authentication, AWS IAM access control, and AWS Marketplace billing. Because Anthropic operates it, **the API surface matches first-party with same-day parity**: Managed Agents, server-side tools, batches, Files, and every feature in this skill work the same way (**except self-hosted sandboxes** — `config:{type:"self_hosted"}` is not available here; use `cloud`). Model IDs are the bare first-party strings (`claude-opus-4-8`, `claude-sonnet-4-6`) — **no provider prefix**.
|
||||
|
||||
> **Not the same as Amazon Bedrock.** Bedrock is partner-operated (AWS runs the service; release schedules vary, feature subset, `anthropic.`-prefixed model IDs). Claude Platform on AWS and Bedrock coexist; pick by whether you need AWS-native IAM/billing with full Anthropic API parity (this page) vs. Bedrock's own ecosystem.
|
||||
|
||||
---
|
||||
|
||||
## Client & install
|
||||
|
||||
| Language | Install | Client |
|
||||
|---|---|---|
|
||||
| Python | `pip install -U "anthropic[aws]"` | `from anthropic import AnthropicAWS` → `AnthropicAWS()` |
|
||||
| TypeScript | `npm install @anthropic-ai/aws-sdk` | `import AnthropicAws from "@anthropic-ai/aws-sdk"` → `new AnthropicAws()` |
|
||||
| Go | `go get github.com/anthropics/anthropic-sdk-go` | `import anthropicaws "github.com/anthropics/anthropic-sdk-go/aws"` → `anthropicaws.NewClient(ctx, anthropicaws.ClientConfig{})` |
|
||||
| C# | `dotnet add package Anthropic.Aws` | `new AnthropicAwsClient()` |
|
||||
| Java | See SDK repo in `shared/live-sources.md` | See SDK repo in `shared/live-sources.md` |
|
||||
| Ruby | `gem install anthropic aws-sdk-core` | See SDK repo in `shared/live-sources.md` |
|
||||
| PHP | `composer require anthropic-ai/sdk aws/aws-sdk-php` | See SDK repo in `shared/live-sources.md` |
|
||||
|
||||
After construction, **use the client exactly as you would `Anthropic()`** — `client.messages.create(...)`, `client.beta.sessions.*`, etc., with bare model IDs.
|
||||
|
||||
```python
|
||||
from anthropic import AnthropicAWS
|
||||
|
||||
client = AnthropicAWS() # region + workspace_id from env; see below
|
||||
client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=1024,
|
||||
messages=[{"role": "user", "content": "Hello"}],
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Required configuration
|
||||
|
||||
Two values must be available (constructor args or environment) — **there is no default fallback** for either:
|
||||
|
||||
| Value | Env var | Notes |
|
||||
|---|---|---|
|
||||
| AWS region | `AWS_REGION` | Required. Unlike `AnthropicBedrock`, there is no `us-east-1` fallback. |
|
||||
| Workspace ID | `ANTHROPIC_AWS_WORKSPACE_ID` | Required. Routes requests to your Claude workspace. |
|
||||
|
||||
Endpoint pattern: `https://aws-external-anthropic.{region}.api.aws/v1/...`. Requests are SigV4-signed with service name `aws-external-anthropic`.
|
||||
|
||||
## Authentication
|
||||
|
||||
The client resolves AWS credentials via the standard precedence chain: explicit constructor args → environment (`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`/`AWS_SESSION_TOKEN`) → shared profile → assumed role / instance metadata.
|
||||
|
||||
**Short-term API keys** are also supported for cases where SigV4 isn't practical (e.g., browser, simple scripts). Mint one with the per-language token-generator package; pass it as `api_key` on the client. Lifetime is the **lesser of** the requested duration, the underlying credential's expiry, and **12 hours**. For package names and IAM details, WebFetch the Claude Platform on AWS page in `shared/live-sources.md`.
|
||||
|
||||
---
|
||||
|
||||
## What to tell users
|
||||
|
||||
- Treat it as first-party: every section of this skill applies unchanged. Do **not** apply Bedrock's feature-availability mask.
|
||||
- Model IDs are bare (`claude-opus-4-8`). Do **not** add an `anthropic.` prefix.
|
||||
- A missing region or `workspace_id` throws at client-construction time (no request is sent). A **403** means the request reached the server — check for a **wrong** `workspace_id` or a missing IAM action on the principal. See the IAM actions reference in `shared/live-sources.md`.
|
||||
@@ -1,233 +0,0 @@
|
||||
# HTTP Error Codes Reference
|
||||
|
||||
This file documents HTTP error codes returned by the Claude API, their common causes, and how to handle them. For language-specific error handling examples, see the `python/` or `typescript/` folders.
|
||||
|
||||
## Error Code Summary
|
||||
|
||||
| Code | Error Type | Retryable | Common Cause |
|
||||
| ---- | ----------------------- | --------- | ------------------------------------ |
|
||||
| 400 | `invalid_request_error` | No | Invalid request format or parameters |
|
||||
| 401 | `authentication_error` | No | Invalid or missing API key |
|
||||
| 403 | `permission_error` | No | API key lacks permission |
|
||||
| 404 | `not_found_error` | No | Invalid endpoint or model ID |
|
||||
| 413 | `request_too_large` | No | Request exceeds size limits |
|
||||
| 429 | `rate_limit_error` | Yes | Too many requests |
|
||||
| 500 | `api_error` | Yes | Anthropic service issue |
|
||||
| 529 | `overloaded_error` | Yes | API is temporarily overloaded |
|
||||
|
||||
## Detailed Error Information
|
||||
|
||||
### 400 Bad Request
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Malformed JSON in request body
|
||||
- Missing required parameters (`model`, `max_tokens`, `messages`)
|
||||
- Invalid parameter types (e.g., string where integer expected)
|
||||
- Empty messages array
|
||||
- Messages not alternating user/assistant
|
||||
|
||||
**Example error:**
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "invalid_request_error",
|
||||
"message": "messages: roles must alternate between \"user\" and \"assistant\""
|
||||
},
|
||||
"request_id": "req_011CSHoEeqs5C35K2UUqR7Fy"
|
||||
}
|
||||
```
|
||||
|
||||
**Fix:** Validate request structure before sending. Check that:
|
||||
|
||||
- `model` is a valid model ID
|
||||
- `max_tokens` is a positive integer
|
||||
- `messages` array is non-empty and alternates correctly
|
||||
|
||||
---
|
||||
|
||||
### 401 Unauthorized
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Missing `x-api-key` header or `Authorization` header
|
||||
- Invalid API key format
|
||||
- Revoked or deleted API key
|
||||
- OAuth bearer token sent via `x-api-key` instead of `Authorization: Bearer`
|
||||
- Both `ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` set — the SDK sends both headers and the API rejects the request
|
||||
|
||||
**Fix:** Set `ANTHROPIC_API_KEY`, or run `ant auth login` and leave the client constructor empty. For raw HTTP with an OAuth token, use `Authorization: Bearer <token>` (not `x-api-key:`).
|
||||
|
||||
---
|
||||
|
||||
### 403 Forbidden
|
||||
|
||||
**Causes:**
|
||||
|
||||
- API key doesn't have access to the requested model
|
||||
- Organization-level restrictions
|
||||
- Attempting to access beta features without beta access
|
||||
|
||||
**Fix:** Check your API key permissions in the Console. You may need a different API key or to request access to specific features.
|
||||
|
||||
---
|
||||
|
||||
### 404 Not Found
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Typo in model ID (e.g., `claude-sonnet-4.6` instead of `claude-sonnet-4-6`)
|
||||
- Using deprecated model ID
|
||||
- Invalid API endpoint
|
||||
|
||||
**Fix:** Use exact model IDs from the models documentation. You can use aliases (e.g., `claude-opus-4-8`).
|
||||
|
||||
---
|
||||
|
||||
### 413 Request Too Large
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Request body exceeds maximum size
|
||||
- Too many tokens in input
|
||||
- Image data too large
|
||||
|
||||
**Fix:** Reduce input size — truncate conversation history, compress/resize images, or split large documents into chunks.
|
||||
|
||||
---
|
||||
|
||||
### 400 Validation Errors
|
||||
|
||||
Some 400 errors are specifically related to parameter validation:
|
||||
|
||||
- `max_tokens` exceeds model's limit
|
||||
- Invalid `temperature` value (must be 0.0-1.0)
|
||||
- `budget_tokens` >= `max_tokens` in extended thinking
|
||||
- Invalid tool definition schema
|
||||
|
||||
**Model-specific 400s on Fable 5 / Opus 4.8 / 4.7:**
|
||||
|
||||
- `temperature`, `top_p`, `top_k` are removed — sending any of them returns 400. Delete the parameter; see `shared/model-migration.md` → Per-SDK Syntax Reference.
|
||||
- `thinking: {type: "enabled", budget_tokens: N}` is removed — sending it returns 400. Use `thinking: {type: "adaptive"}` instead.
|
||||
- **Fable 5 only:** an explicit `thinking: {type: "disabled"}` returns 400 (it is accepted on Opus 4.8/4.7). Omit the `thinking` param entirely instead.
|
||||
- **Fable 5 only:** if the organization is set to zero data retention (ZDR) — or any retention below the required 30 days — then **all** Fable 5 requests return `400 invalid_request_error`, even with a perfectly valid payload. Check the org's retention configuration before debugging the request body.
|
||||
|
||||
**Common mistake with extended thinking on older models (Opus 4.6 and earlier):**
|
||||
|
||||
```
|
||||
# Wrong: budget_tokens must be < max_tokens
|
||||
thinking: budget_tokens=10000, max_tokens=1000 → Error!
|
||||
|
||||
# Correct
|
||||
thinking: budget_tokens=10000, max_tokens=16000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 429 Rate Limited
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Exceeded requests per minute (RPM)
|
||||
- Exceeded tokens per minute (TPM)
|
||||
- Exceeded tokens per day (TPD)
|
||||
|
||||
**Headers to check:**
|
||||
|
||||
- `retry-after`: Seconds to wait before retrying
|
||||
- `x-ratelimit-limit-*`: Your limits
|
||||
- `x-ratelimit-remaining-*`: Remaining quota
|
||||
|
||||
**Fix:** The Anthropic SDKs automatically retry 429 and 5xx errors with exponential backoff (default: `max_retries=2`). For custom retry behavior, see the language-specific error handling examples.
|
||||
|
||||
---
|
||||
|
||||
### 500 Internal Server Error
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Temporary Anthropic service issue
|
||||
- Bug in API processing
|
||||
|
||||
**Fix:** Retry with exponential backoff. If persistent, check [status.anthropic.com](https://status.anthropic.com).
|
||||
|
||||
---
|
||||
|
||||
### 529 Overloaded
|
||||
|
||||
**Causes:**
|
||||
|
||||
- High API demand
|
||||
- Service capacity reached
|
||||
|
||||
**Fix:** Retry with exponential backoff. Consider using a different model (Haiku is often less loaded), spreading requests over time, or implementing request queuing.
|
||||
|
||||
---
|
||||
|
||||
## Common Mistakes and Fixes
|
||||
|
||||
| Mistake | Error | Fix |
|
||||
| ------------------------------- | ---------------- | ------------------------------------------------------- |
|
||||
| `temperature`/`top_p`/`top_k` on Fable 5 / Opus 4.8 / 4.7 | 400 | Remove the parameter (see `shared/model-migration.md`) |
|
||||
| `budget_tokens` on Fable 5 / Opus 4.8 / 4.7 | 400 | Use `thinking: {type: "adaptive"}` |
|
||||
| `thinking: {type: "disabled"}` on Fable 5 | 400 | Omit the `thinking` param entirely (accepted on Opus 4.8/4.7) |
|
||||
| Org set to ZDR / retention below 30 days (Fable 5) | 400 on every request | Fix the org's data-retention configuration — the payload isn't the problem |
|
||||
| `budget_tokens` >= `max_tokens` (older models) | 400 | Ensure `budget_tokens` < `max_tokens` |
|
||||
| Typo in model ID | 404 | Use valid model ID like `claude-opus-4-8` |
|
||||
| First message is `assistant` | 400 | First message must be `user` |
|
||||
| Consecutive same-role messages | 400 | Alternate `user` and `assistant` |
|
||||
| API key in code | 401 (leaked key) | Use environment variable |
|
||||
| Custom retry needs | 429/5xx | SDK retries automatically; customize with `max_retries` |
|
||||
|
||||
## Typed Exceptions in SDKs
|
||||
|
||||
**Always use the SDK's typed exception classes** instead of checking error messages with string matching. Each HTTP error code maps to a specific exception class:
|
||||
|
||||
| HTTP Code | TypeScript Class | Python Class |
|
||||
| --------- | --------------------------------- | --------------------------------- |
|
||||
| 400 | `Anthropic.BadRequestError` | `anthropic.BadRequestError` |
|
||||
| 401 | `Anthropic.AuthenticationError` | `anthropic.AuthenticationError` |
|
||||
| 403 | `Anthropic.PermissionDeniedError` | `anthropic.PermissionDeniedError` |
|
||||
| 404 | `Anthropic.NotFoundError` | `anthropic.NotFoundError` |
|
||||
| 413 | `Anthropic.RequestTooLargeError` | `anthropic.RequestTooLargeError` |
|
||||
| 429 | `Anthropic.RateLimitError` | `anthropic.RateLimitError` |
|
||||
| 500+ | `Anthropic.InternalServerError` | `anthropic.InternalServerError` |
|
||||
| 529 | `Anthropic.OverloadedError` | `anthropic.OverloadedError` |
|
||||
| Any | `Anthropic.APIError` | `anthropic.APIError` |
|
||||
|
||||
```typescript
|
||||
// ✅ Correct: use typed exceptions
|
||||
try {
|
||||
const response = await client.messages.create({...});
|
||||
} catch (error) {
|
||||
if (error instanceof Anthropic.RateLimitError) {
|
||||
// Handle rate limiting
|
||||
} else if (error instanceof Anthropic.APIError) {
|
||||
console.error(`API error ${error.status}:`, error.message);
|
||||
}
|
||||
}
|
||||
|
||||
// ❌ Wrong: don't check error messages with string matching
|
||||
try {
|
||||
const response = await client.messages.create({...});
|
||||
} catch (error) {
|
||||
const msg = error instanceof Error ? error.message : String(error);
|
||||
if (msg.includes("429") || msg.includes("rate_limit")) { ... }
|
||||
}
|
||||
```
|
||||
|
||||
All exception classes extend `Anthropic.APIError`, which has a `status` property. Use `instanceof` checks from most specific to least specific (e.g., check `RateLimitError` before `APIError`).
|
||||
|
||||
### Error `.type` Field
|
||||
|
||||
All `APIStatusError` subclasses now expose a `.type` property (Python: `.type`, TypeScript: `.type`, Java: `.errorType()`, Go: `.Type()`, Ruby: `.type`, PHP: `.type`) that returns the API error type string (e.g., `"invalid_request_error"`, `"authentication_error"`, `"rate_limit_error"`, `"overloaded_error"`). Use this for programmatic error classification when you need finer granularity than the HTTP status code — for example, distinguishing `"billing_error"` from `"permission_error"` (both map to 403).
|
||||
|
||||
```python
|
||||
except anthropic.APIStatusError as e:
|
||||
if e.type == "rate_limit_error":
|
||||
# handle rate limiting
|
||||
elif e.type == "overloaded_error":
|
||||
# handle overload
|
||||
```
|
||||
@@ -1,143 +0,0 @@
|
||||
# Live Documentation Sources
|
||||
|
||||
This file contains WebFetch URLs for fetching current information from platform.claude.com and Agent SDK repositories. Use these when users need the latest data that may have changed since the cached content was last updated.
|
||||
|
||||
## When to Use WebFetch
|
||||
|
||||
- User explicitly asks for "latest" or "current" information
|
||||
- Cached data seems incorrect
|
||||
- User asks about features not covered in cached content
|
||||
- User needs specific API details or examples
|
||||
|
||||
## Claude API Documentation URLs
|
||||
|
||||
### Models & Pricing
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| --------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
|
||||
| Models Overview | `https://platform.claude.com/docs/en/about-claude/models/overview.md` | "Extract current model IDs, context windows, and pricing for all Claude models" |
|
||||
| Migration Guide | `https://platform.claude.com/docs/en/about-claude/models/migration-guide.md` | "Extract breaking changes, deprecated parameters, and per-model migration steps when moving to a newer Claude model" |
|
||||
| Introducing Claude Fable 5 | `https://platform.claude.com/docs/en/about-claude/models/introducing-claude-fable-5.md` | "Extract capabilities, API changes, and availability stages for Claude Fable 5 and Claude Mythos 5" |
|
||||
| Pricing | `https://platform.claude.com/docs/en/pricing.md` | "Extract current pricing per million tokens for input and output" |
|
||||
|
||||
### Core Features
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| ----------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
||||
| Extended Thinking | `https://platform.claude.com/docs/en/build-with-claude/extended-thinking.md` | "Extract extended thinking parameters, budget_tokens requirements, and usage examples" |
|
||||
| Adaptive Thinking | `https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking.md` | "Extract adaptive thinking setup, effort levels, and Claude Opus 4.8 usage examples" |
|
||||
| Effort Parameter | `https://platform.claude.com/docs/en/build-with-claude/effort.md` | "Extract effort levels, cost-quality tradeoffs, and interaction with thinking" |
|
||||
| Tool Use | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview.md` | "Extract tool definition schema, tool_choice options, and handling tool results" |
|
||||
| Streaming | `https://platform.claude.com/docs/en/build-with-claude/streaming.md` | "Extract streaming event types, SDK examples, and best practices" |
|
||||
| Prompt Caching | `https://platform.claude.com/docs/en/build-with-claude/prompt-caching.md` | "Extract cache_control usage, pricing benefits, and implementation examples" |
|
||||
|
||||
### Media & Files
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| ----------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
||||
| Vision | `https://platform.claude.com/docs/en/build-with-claude/vision.md` | "Extract supported image formats, size limits, and code examples" |
|
||||
| PDF Support | `https://platform.claude.com/docs/en/build-with-claude/pdf-support.md` | "Extract PDF handling capabilities, limits, and examples" |
|
||||
|
||||
### API Operations
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| ---------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| Batch Processing | `https://platform.claude.com/docs/en/build-with-claude/batch-processing.md` | "Extract batch API endpoints, request format, and polling for results" |
|
||||
| Files API | `https://platform.claude.com/docs/en/build-with-claude/files.md` | "Extract file upload, download, and referencing in messages, including supported types and beta header" |
|
||||
| Token Counting | `https://platform.claude.com/docs/en/build-with-claude/token-counting.md` | "Extract token counting API usage and examples" |
|
||||
| Rate Limits | `https://platform.claude.com/docs/en/api/rate-limits.md` | "Extract current rate limits by tier and model" |
|
||||
| Errors | `https://platform.claude.com/docs/en/api/errors.md` | "Extract HTTP error codes, meanings, and retry guidance" |
|
||||
| Amazon Bedrock | `https://platform.claude.com/docs/en/build-with-claude/claude-on-amazon-bedrock.md` | "Extract the AnthropicBedrockMantle client per language, `anthropic.`-prefixed model IDs, auth paths, feature availability, and regions" |
|
||||
| Claude Platform on AWS | `https://platform.claude.com/docs/en/build-with-claude/claude-platform-on-aws.md` | "Extract the AnthropicAWS client per language, SigV4 auth, credential precedence, short-term API keys, workspace_id, and region requirements" |
|
||||
| Claude Platform on AWS — IAM actions | `https://platform.claude.com/docs/en/api/claude-platform-on-aws-iam-actions.md` | "Extract the IAM action names, resource ARNs, and policy examples required for each API capability" |
|
||||
|
||||
### Tools
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| -------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| Code Execution | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/code-execution-tool.md` | "Extract code execution tool setup, file upload, container reuse, and response handling" |
|
||||
| Computer Use | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use.md` | "Extract computer use tool setup, capabilities, and implementation examples" |
|
||||
| Bash Tool | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/bash-tool.md` | "Extract bash tool schema, reference implementation, and security considerations" |
|
||||
| Text Editor | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/text-editor-tool.md` | "Extract text editor tool commands, schema, and reference implementation" |
|
||||
| Memory Tool | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool.md` | "Extract memory tool commands, directory structure, and implementation patterns" |
|
||||
| Tool Search | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.md` | "Extract tool search setup, when to use, and cache interaction" |
|
||||
| Programmatic Tool Calling | `https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling.md` | "Extract PTC setup, script execution model, and tool invocation from code" |
|
||||
| Skills | `https://platform.claude.com/docs/en/agents-and-tools/skills.md` | "Extract skill folder structure, SKILL.md format, and loading behavior" |
|
||||
|
||||
### Advanced Features
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| ------------------ | ----------------------------------------------------------------------------- | --------------------------------------------------- |
|
||||
| Structured Outputs | `https://platform.claude.com/docs/en/build-with-claude/structured-outputs.md` | "Extract output_config.format usage and schema enforcement" |
|
||||
| Compaction | `https://platform.claude.com/docs/en/build-with-claude/compaction.md` | "Extract compaction setup, trigger config, and streaming with compaction" |
|
||||
| Context Editing | `https://platform.claude.com/docs/en/build-with-claude/context-editing.md` | "Extract context editing thresholds, what gets cleared, and configuration" |
|
||||
| Citations | `https://platform.claude.com/docs/en/build-with-claude/citations.md` | "Extract citation format and implementation" |
|
||||
| Context Windows | `https://platform.claude.com/docs/en/build-with-claude/context-windows.md` | "Extract context window sizes and token management" |
|
||||
|
||||
### Managed Agents
|
||||
|
||||
Use these when a managed-agents binding, behavior, or wire-level detail isn't covered in the cached `shared/managed-agents-*.md` concept files or in `{lang}/managed-agents/README.md`.
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| --------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| Overview | `https://platform.claude.com/docs/en/managed-agents/overview.md` | "Extract the high-level architecture and how agents/sessions/environments/vaults fit together" |
|
||||
| Quickstart | `https://platform.claude.com/docs/en/managed-agents/quickstart.md` | "Extract the minimal end-to-end agent → environment → session → stream code path" |
|
||||
| Agent Setup | `https://platform.claude.com/docs/en/managed-agents/agent-setup.md` | "Extract agent create/update/list-versions/archive lifecycle and parameters" |
|
||||
| Define Outcomes | `https://platform.claude.com/docs/en/managed-agents/define-outcomes.md` | "Extract outcome definitions, evaluation hooks, and success criteria configuration" |
|
||||
| Sessions | `https://platform.claude.com/docs/en/managed-agents/sessions.md` | "Extract session lifecycle, status transitions, idle/terminated semantics, and resume rules" |
|
||||
| Environments | `https://platform.claude.com/docs/en/managed-agents/environments.md` | "Extract environment config (cloud/networking), management endpoints, and reuse model" |
|
||||
| Self-Hosted Sandboxes | `https://platform.claude.com/docs/en/managed-agents/self-hosted-sandboxes.md` | "Extract config:{type:self_hosted}, ANTHROPIC_ENVIRONMENT_KEY, EnvironmentWorker.run/run_one, beta_agent_toolset, ant beta:worker poll/run, webhook-driven wake" |
|
||||
| Self-Hosted Sandboxes — Security | `https://platform.claude.com/docs/en/managed-agents/self-hosted-sandboxes-security.md` | "Extract what the customer owns (hardening, egress, key custody, trust boundaries) vs what Anthropic cannot do" |
|
||||
| Events and Streaming | `https://platform.claude.com/docs/en/managed-agents/events-and-streaming.md` | "Extract event stream types, stream-first ordering, reconnect/dedupe, and steering patterns" |
|
||||
| Tools | `https://platform.claude.com/docs/en/managed-agents/tools.md` | "Extract built-in toolset, custom tool definitions, and tool result wire format" |
|
||||
| Files | `https://platform.claude.com/docs/en/managed-agents/files.md` | "Extract file upload, mount paths, session resources, and listing/downloading session outputs" |
|
||||
| Permission Policies | `https://platform.claude.com/docs/en/managed-agents/permission-policies.md` | "Extract permission policy types (allow/deny/confirm) and per-tool config" |
|
||||
| Multi-Agent | `https://platform.claude.com/docs/en/managed-agents/multi-agent.md` | "Extract multi-agent composition patterns, sub-agent invocation, and result handoff" |
|
||||
| Observability | `https://platform.claude.com/docs/en/managed-agents/observability.md` | "Extract logging, tracing, and usage telemetry exposed by managed agents" |
|
||||
| Webhooks | `https://platform.claude.com/docs/en/managed-agents/webhooks.md` | "Extract webhook endpoint registration, HMAC signature verification, supported event types, and delivery semantics" |
|
||||
| GitHub | `https://platform.claude.com/docs/en/managed-agents/github.md` | "Extract github_repository resource shape, multi-repo mounting, and token rotation" |
|
||||
| MCP Connector | `https://platform.claude.com/docs/en/managed-agents/mcp-connector.md` | "Extract MCP server declaration on agents and vault-based credential injection at session" |
|
||||
| Vaults | `https://platform.claude.com/docs/en/managed-agents/vaults.md` | "Extract vault create, credential add/rotate, OAuth refresh shape, and archive" |
|
||||
| Skills | `https://platform.claude.com/docs/en/managed-agents/skills.md` | "Extract skill packaging and loading model for managed agents" |
|
||||
| Memory | `https://platform.claude.com/docs/en/managed-agents/memory.md` | "Extract memory resource shape, scoping, and lifecycle" |
|
||||
| Onboarding | `https://platform.claude.com/docs/en/managed-agents/onboarding.md` | "Extract first-run setup, prerequisites, and account/region requirements" |
|
||||
| Cloud Containers | `https://platform.claude.com/docs/en/managed-agents/cloud-containers.md` | "Extract cloud container runtime, image config, and network/storage knobs" |
|
||||
| Migration | `https://platform.claude.com/docs/en/managed-agents/migration.md` | "Extract migration paths from earlier APIs/preview shapes to GA managed agents" |
|
||||
|
||||
### Anthropic CLI
|
||||
|
||||
The `ant` CLI provides terminal access to the Claude API. Every API resource is exposed as a subcommand. It is one convenient way to create agents, environments, sessions, and other resources from version-controlled YAML, and to inspect responses interactively.
|
||||
|
||||
| Topic | URL | Extraction Prompt |
|
||||
| ------------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
||||
| Anthropic CLI | `https://platform.claude.com/docs/en/api/sdks/cli.md` | "Extract CLI install, authentication, command structure, and the beta:agents/environments/sessions commands" |
|
||||
| Authentication overview | `https://platform.claude.com/docs/en/manage-claude/authentication.md` | "Extract the credential options (API keys, interactive OAuth login, Workload Identity Federation) and when to use each" |
|
||||
| WIF reference | `https://platform.claude.com/docs/en/manage-claude/wif-reference.md` | "Extract credential precedence order, the profile configuration file schema, and the configuration directory layout" |
|
||||
|
||||
---
|
||||
|
||||
## Claude API SDK Repositories
|
||||
|
||||
WebFetch these when a binding (class, method, namespace, field) isn't covered in the cached `{lang}/` skill files or in the managed-agents docs above. The SDKs include beta managed-agents support for `/v1/agents`, `/v1/sessions`, `/v1/environments`, and related resources — search the repo for `BetaManagedAgents`, `beta.agents`, `beta.sessions`, or the equivalent namespace for that language.
|
||||
|
||||
| SDK | URL | Extraction Prompt |
|
||||
| ---------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
|
||||
| Python | `https://github.com/anthropics/anthropic-sdk-python` | "Extract beta managed-agents namespaces, classes, and method signatures (`client.beta.agents`, `client.beta.sessions`)" |
|
||||
| TypeScript | `https://github.com/anthropics/anthropic-sdk-typescript` | "Extract beta managed-agents namespaces, classes, and method signatures (`client.beta.agents`, `client.beta.sessions`)" |
|
||||
| Java | `https://github.com/anthropics/anthropic-sdk-java` | "Extract beta managed-agents classes, builders, and method signatures (`client.beta().agents()`, `BetaManagedAgents*`)" |
|
||||
| Go | `https://github.com/anthropics/anthropic-sdk-go` | "Extract beta managed-agents types and method signatures (`client.Beta.Agents`, `BetaManagedAgents*` event types)" |
|
||||
| Ruby | `https://github.com/anthropics/anthropic-sdk-ruby` | "Extract beta managed-agents methods and parameter shapes (`client.beta.agents`, `client.beta.sessions`)" |
|
||||
| C# | `https://github.com/anthropics/anthropic-sdk-csharp` | "Extract beta managed-agents classes and method signatures (NuGet package, `BetaManagedAgents*` types)" |
|
||||
| PHP | `https://github.com/anthropics/anthropic-sdk-php` | "Extract beta managed-agents classes and method signatures (`$client->beta->agents`, `BetaManagedAgents*` params)" |
|
||||
|
||||
Each SDK repo also ships runnable programs under `examples/` — including the refusal-fallback / `fallbacks` examples (client-side middleware registration, fallback state, server-side `fallbacks` param). Fetch those for exact per-language syntax instead of translating another language's example.
|
||||
|
||||
---
|
||||
|
||||
## Fallback Strategy
|
||||
|
||||
If WebFetch fails (network issues, URL changed):
|
||||
|
||||
1. Use cached content from the language-specific files (note the cache date)
|
||||
2. Inform user the data may be outdated
|
||||
3. Suggest they check platform.claude.com or the GitHub repos directly
|
||||
@@ -1,422 +0,0 @@
|
||||
# Managed Agents — Endpoint Reference
|
||||
|
||||
All endpoints require `x-api-key` and `anthropic-version: 2023-06-01` headers. Managed Agents endpoints additionally require the `anthropic-beta` header.
|
||||
|
||||
## Beta Headers
|
||||
|
||||
```
|
||||
anthropic-beta: managed-agents-2026-04-01
|
||||
```
|
||||
|
||||
The SDK adds this header automatically for all `client.beta.{agents,environments,sessions,vaults,memory_stores,deployments,deployment_runs}.*` calls. Skills endpoints use `skills-2025-10-02`; Files endpoints use `files-api-2025-04-14`.
|
||||
|
||||
---
|
||||
|
||||
## SDK Method Reference
|
||||
|
||||
All resources are under the `beta` namespace. Python and TypeScript share identical method names.
|
||||
|
||||
| Resource | Python / TypeScript (`client.beta.*`) | Go (`client.Beta.*`) |
|
||||
| --- | --- | --- |
|
||||
| Agents | `agents.create` / `retrieve` / `update` / `list` / `archive` | `Agents.New` / `Get` / `Update` / `List` / `Archive` |
|
||||
| Agent Versions | `agents.versions.list` | `Agents.Versions.List` |
|
||||
| Environments | `environments.create` / `retrieve` / `update` / `list` / `delete` / `archive` | `Environments.New` / `Get` / `Update` / `List` / `Delete` / `Archive` |
|
||||
| Environment Work (self-hosted) | `environments.work.poller` / `stats` / `stop` | See `shared/managed-agents-self-hosted-sandboxes.md` |
|
||||
| Sessions | `sessions.create` / `retrieve` / `update` / `list` / `delete` / `archive` | `Sessions.New` / `Get` / `Update` / `List` / `Delete` / `Archive` |
|
||||
| Session Events | `sessions.events.list` / `send` / `stream` | `Sessions.Events.List` / `Send` / `StreamEvents` |
|
||||
| Session Threads | `sessions.threads.list` / `retrieve` / `archive`; `sessions.threads.events.list` / `stream` | `Sessions.Threads.List` / `Get` / `Archive`; `Sessions.Threads.Events.List` / `StreamEvents` |
|
||||
| Session Resources | `sessions.resources.add` / `retrieve` / `update` / `list` / `delete` | `Sessions.Resources.Add` / `Get` / `Update` / `List` / `Delete` |
|
||||
| Deployments | `deployments.create` / `pause` / `unpause` / `archive` / `run` | Not yet documented — WebFetch the SDK repo (`shared/live-sources.md`) |
|
||||
| Deployment Runs | `deployment_runs.list` (TS: `deploymentRuns.list`) | Not yet documented — WebFetch the SDK repo (`shared/live-sources.md`) |
|
||||
| Vaults | `vaults.create` / `retrieve` / `update` / `list` / `delete` / `archive` | `Vaults.New` / `Get` / `Update` / `List` / `Delete` / `Archive` |
|
||||
| Credentials | `vaults.credentials.create` / `retrieve` / `update` / `list` / `delete` / `archive` / `mcp_oauth_validate` | `Vaults.Credentials.New` / `Get` / `Update` / `List` / `Delete` / `Archive` / `McpOauthValidate` |
|
||||
| Memory Stores | `memory_stores.create` / `retrieve` / `update` / `list` / `delete` / `archive` | `MemoryStores.New` / `Get` / `Update` / `List` / `Delete` / `Archive` |
|
||||
| Memories | `memory_stores.memories.create` / `retrieve` / `update` / `list` / `delete` | `MemoryStores.Memories.New` / `Get` / `Update` / `List` / `Delete` |
|
||||
| Memory Versions | `memory_stores.memory_versions.list` / `retrieve` / `redact` | `MemoryStores.MemoryVersions.List` / `Get` / `Redact` |
|
||||
|
||||
**Naming quirks to watch for:**
|
||||
- Agents and Session Threads have **no delete** — only `archive`. Archive is **permanent**: the agent becomes read-only, new sessions cannot reference it, and there is no unarchive. Confirm with the user before archiving a production agent. Environments, Sessions, Vaults, Credentials, and Memory Stores have both `delete` and `archive`; Session Resources, Files, Skills, and Memories are `delete`-only; Memory Versions have neither — only `redact`.
|
||||
- Session resources use `add` (not `create`).
|
||||
- Go's event stream is `StreamEvents` (not `Stream`).
|
||||
- The self-hosted worker is **not** under `client.beta.*` — it's `EnvironmentWorker` from `anthropic.lib.environments` / `@anthropic-ai/sdk/helpers/beta/environments`; only `environments.work.poller/stats/stop` are client methods.
|
||||
|
||||
**Agent shorthand:** `agent` on session create accepts either a bare string (`agent="agent_abc123"` — uses latest version) or the full reference object (`{type: "agent", id: "agent_abc123", version: 123}`).
|
||||
|
||||
**Model shorthand:** `model` on agent create accepts either a bare string (`model="claude-opus-4-8"` — uses `standard` speed) or the full config object (`{id: "claude-opus-4-6", speed: "fast"}`). Note: `speed: "fast"` is only supported on Opus 4.6.
|
||||
|
||||
---
|
||||
|
||||
## Agents
|
||||
|
||||
**Step one of every flow.** Sessions require a pre-created agent — there is no inline agent config under `managed-agents-2026-04-01`.
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/agents` | ListAgents | List agents |
|
||||
| `POST` | `/v1/agents` | CreateAgent | Create a saved agent configuration |
|
||||
| `GET` | `/v1/agents/{agent_id}` | GetAgent | Get agent details |
|
||||
| `POST` | `/v1/agents/{agent_id}` | UpdateAgent | Update agent configuration |
|
||||
| `POST` | `/v1/agents/{agent_id}/archive` | ArchiveAgent | Archive an agent. Makes it **read-only**; existing sessions continue, new sessions cannot reference it. No unarchive — this is the terminal state. |
|
||||
| `GET` | `/v1/agents/{agent_id}/versions` | ListAgentVersions | List agent versions |
|
||||
|
||||
## Sessions
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/sessions` | ListSessions | List sessions (paginated) |
|
||||
| `POST` | `/v1/sessions` | CreateSession | Create a new session |
|
||||
| `GET` | `/v1/sessions/{session_id}` | GetSession | Get session details |
|
||||
| `POST` | `/v1/sessions/{session_id}` | UpdateSession | Update session `metadata`/`title`, or `agent.tools`/`agent.mcp_servers`/`vault_ids` (session-local override; session must be `idle`). See `shared/managed-agents-core.md` → Updating the agent configuration mid-session. |
|
||||
| `DELETE` | `/v1/sessions/{session_id}` | DeleteSession | Delete a session |
|
||||
| `POST` | `/v1/sessions/{session_id}/archive` | ArchiveSession | Archive a session |
|
||||
|
||||
## Events
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/sessions/{session_id}/events` | ListEvents | List events (polling, paginated) |
|
||||
| `POST` | `/v1/sessions/{session_id}/events` | SendEvents | Send events (user message, tool result) |
|
||||
| `GET` | `/v1/sessions/{session_id}/events/stream` | StreamEvents | Stream events via SSE |
|
||||
|
||||
## Session Threads
|
||||
|
||||
Per-subagent event streams in multiagent sessions. See `shared/managed-agents-multiagent.md`.
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/sessions/{session_id}/threads` | ListThreads | List threads (paginated) |
|
||||
| `GET` | `/v1/sessions/{session_id}/threads/{thread_id}` | GetThread | Retrieve one thread (carries `agent` snapshot, `status`, `parent_thread_id`, `stats`, `usage`) |
|
||||
| `POST` | `/v1/sessions/{session_id}/threads/{thread_id}/archive` | ArchiveThread | Archive a thread |
|
||||
| `GET` | `/v1/sessions/{session_id}/threads/{thread_id}/events` | ListThreadEvents | List past events for one thread (paginated) |
|
||||
| `GET` | `/v1/sessions/{session_id}/threads/{thread_id}/stream` | StreamThreadEvents | Stream one thread via SSE (SDK: `threads.events.stream`) |
|
||||
|
||||
## Session Resources
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------------- | ---------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/sessions/{session_id}/resources` | ListResources | List resources attached to session |
|
||||
| `POST` | `/v1/sessions/{session_id}/resources` | AddResource | Attach `file` or `github_repository` resource (SDK method: `add`, not `create`). `memory_store` resources attach at session-create time only. |
|
||||
| `GET` | `/v1/sessions/{session_id}/resources/{resource_id}` | GetResource | Get a single resource |
|
||||
| `POST` | `/v1/sessions/{session_id}/resources/{resource_id}` | UpdateResource | Update resource |
|
||||
| `DELETE` | `/v1/sessions/{session_id}/resources/{resource_id}` | DeleteResource | Remove resource from session |
|
||||
|
||||
## Environments
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ---------------------------------------------------------------- | -------------------- | ----------------------------------- |
|
||||
| `POST` | `/v1/environments` | CreateEnvironment | Create environment |
|
||||
| `GET` | `/v1/environments` | ListEnvironments | List environments |
|
||||
| `GET` | `/v1/environments/{environment_id}` | GetEnvironment | Get environment details |
|
||||
| `POST` | `/v1/environments/{environment_id}` | UpdateEnvironment | Update environment |
|
||||
| `DELETE` | `/v1/environments/{environment_id}` | DeleteEnvironment | Delete environment. Returns 204. |
|
||||
| `POST` | `/v1/environments/{environment_id}/archive` | ArchiveEnvironment | Archive environment. Makes it **read-only**; existing sessions continue, new sessions cannot reference it. No unarchive — this is the terminal state. |
|
||||
| `GET` | `/v1/environments/{environment_id}/work/stats` | WorkQueueStats | Self-hosted work-queue depth/pending/workers. `x-api-key` auth. See `shared/managed-agents-self-hosted-sandboxes.md`. |
|
||||
| `POST` | `/v1/environments/{environment_id}/work/{work_id}/stop` | StopWork | Self-hosted: stop a claimed work item. `x-api-key` auth. |
|
||||
|
||||
For `type: "self_hosted"`, `config` is the bare `{"type": "self_hosted"}` — `networking` and `packages` do not apply.
|
||||
|
||||
## Deployments
|
||||
|
||||
Scheduled deployments (`depl_` IDs) run an agent on a recurring cron schedule — each firing creates a session. See `shared/managed-agents-scheduled-deployments.md` for the conceptual guide (cron/DST semantics, failure behavior, lifecycle).
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `POST` | `/v1/deployments` | CreateDeployment | Create a scheduled deployment |
|
||||
| `POST` | `/v1/deployments/{deployment_id}/pause` | PauseDeployment | Suppress scheduled triggers (reversible; manual runs still allowed) |
|
||||
| `POST` | `/v1/deployments/{deployment_id}/unpause` | UnpauseDeployment | Resume from the next occurrence (no backfill) |
|
||||
| `POST` | `/v1/deployments/{deployment_id}/archive` | ArchiveDeployment | **Terminal** — schedule stops, deployment becomes immutable |
|
||||
| `POST` | `/v1/deployments/{deployment_id}/run` | RunDeployment | Trigger a manual run immediately (`trigger_context.type: "manual"`); works while paused |
|
||||
|
||||
## Deployment Runs
|
||||
|
||||
Each trigger attempt (scheduled or manual) writes a `deployment_run` record (`drun_` IDs) carrying either the created `session_id` or an `error.type` (`environment_archived`, `agent_archived`, `vault_not_found`, `session_rate_limited`, `service_unavailable`).
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/deployment_runs?deployment_id=...` | ListDeploymentRuns | List runs for a deployment (paginated; filter failures with `has_error=true`) |
|
||||
|
||||
## Vaults
|
||||
|
||||
Vaults store credentials that Anthropic manages on your behalf — MCP credentials (OAuth with auto-refresh, or static bearer tokens) and `environment_variable` credentials substituted into outbound requests at egress. Attach to sessions via `vault_ids`. See `managed-agents-tools.md` §Vaults for the conceptual guide and credential shapes.
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `POST` | `/v1/vaults` | CreateVault | Create a vault |
|
||||
| `GET` | `/v1/vaults` | ListVaults | List vaults |
|
||||
| `GET` | `/v1/vaults/{vault_id}` | GetVault | Get vault details |
|
||||
| `POST` | `/v1/vaults/{vault_id}` | UpdateVault | Update vault |
|
||||
| `DELETE` | `/v1/vaults/{vault_id}` | DeleteVault | Delete vault |
|
||||
| `POST` | `/v1/vaults/{vault_id}/archive` | ArchiveVault | Archive vault |
|
||||
|
||||
## Credentials
|
||||
|
||||
Credentials are individual secrets stored inside a vault.
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ----------------------------------------------------------------- | ------------------ | ---------------------------- |
|
||||
| `POST` | `/v1/vaults/{vault_id}/credentials` | CreateCredential | Create a credential |
|
||||
| `GET` | `/v1/vaults/{vault_id}/credentials` | ListCredentials | List credentials in vault |
|
||||
| `GET` | `/v1/vaults/{vault_id}/credentials/{credential_id}` | GetCredential | Get credential metadata |
|
||||
| `POST` | `/v1/vaults/{vault_id}/credentials/{credential_id}` | UpdateCredential | Update credential |
|
||||
| `DELETE` | `/v1/vaults/{vault_id}/credentials/{credential_id}` | DeleteCredential | Delete credential |
|
||||
| `POST` | `/v1/vaults/{vault_id}/credentials/{credential_id}/archive` | ArchiveCredential | Archive credential |
|
||||
| `POST` | `/v1/vaults/{vault_id}/credentials/{credential_id}/mcp_oauth_validate` | McpOauthValidate | Validate an MCP OAuth credential |
|
||||
|
||||
## Memory Stores
|
||||
|
||||
Workspace-scoped persistent memory that survives across sessions. Attach to a session via a `{"type": "memory_store", "memory_store_id": ...}` entry in `resources[]` (session-create time only). See `shared/managed-agents-memory.md` for the conceptual guide, the FUSE-mount agent interface, preconditions, and versioning.
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ------------------ | ---------------------------------------- |
|
||||
| `POST` | `/v1/memory_stores` | CreateMemoryStore | Create a store (`name`, `description`, `metadata`) |
|
||||
| `GET` | `/v1/memory_stores` | ListMemoryStores | List stores (`include_archived`, `created_at_{gte,lte}`) |
|
||||
| `GET` | `/v1/memory_stores/{memory_store_id}` | GetMemoryStore | Get store details |
|
||||
| `POST` | `/v1/memory_stores/{memory_store_id}` | UpdateMemoryStore | Update store |
|
||||
| `DELETE` | `/v1/memory_stores/{memory_store_id}` | DeleteMemoryStore | Delete store |
|
||||
| `POST` | `/v1/memory_stores/{memory_store_id}/archive` | ArchiveMemoryStore | Archive store. Makes it **read-only**; existing sessions continue, new sessions cannot reference it. No unarchive. |
|
||||
|
||||
## Memories
|
||||
|
||||
Individual text documents inside a store (≤ 100KB each). `create` creates at a `path` and returns `409` (`memory_path_conflict_error`, with `conflicting_memory_id`) if the path is occupied; `update` mutates by `mem_...` ID (rename and/or content). Only `update` accepts a `precondition` (`{"type": "content_sha256", "content_sha256": ...}`) — on mismatch returns `409` (`memory_precondition_failed_error`). List endpoints accept `view: "basic"|"full"` (controls whether `content` is populated; `retrieve` defaults to `full`).
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ----------------------------------------------------------------- | -------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/memory_stores/{memory_store_id}/memories` | ListMemories | Returns `Memory \| MemoryPrefix`; filter by `path_prefix`, `depth`, `order_by`/`order` |
|
||||
| `POST` | `/v1/memory_stores/{memory_store_id}/memories` | CreateMemory | Create at `path` (SDK: `memories.create`); `409 memory_path_conflict_error` if occupied |
|
||||
| `GET` | `/v1/memory_stores/{memory_store_id}/memories/{memory_id}` | GetMemory | Read one memory (defaults to `view="full"`) |
|
||||
| `PATCH` | `/v1/memory_stores/{memory_store_id}/memories/{memory_id}` | UpdateMemory | Change `content`, `path`, or both by ID; optional `precondition` |
|
||||
| `DELETE` | `/v1/memory_stores/{memory_store_id}/memories/{memory_id}` | DeleteMemory | Delete (optional `expected_content_sha256`) |
|
||||
|
||||
## Memory Versions
|
||||
|
||||
Immutable per-mutation snapshots (`memver_...`) — the audit and rollback surface. `operation` ∈ `created` / `modified` / `deleted`.
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ----------------------------------------------------------------------------- | --------------------- | ---------------------------------------- |
|
||||
| `GET` | `/v1/memory_stores/{memory_store_id}/memory_versions` | ListMemoryVersions | Newest-first; filter by `memory_id`, `operation`, `session_id`, `api_key_id`, `created_at_{gte,lte}` |
|
||||
| `GET` | `/v1/memory_stores/{memory_store_id}/memory_versions/{version_id}` | GetMemoryVersion | List fields + full `content` |
|
||||
| `POST` | `/v1/memory_stores/{memory_store_id}/memory_versions/{version_id}/redact` | RedactMemoryVersion | Clear `content`/`content_sha256`/`content_size_bytes`/`path`; preserve actor + timestamps |
|
||||
|
||||
## Files
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | ------------------------------------------------ | ---------------- | ---------------------------------------- |
|
||||
| `POST` | `/v1/files` | UploadFile | Upload a file |
|
||||
| `GET` | `/v1/files` | ListFiles | List files |
|
||||
| `GET` | `/v1/files/{file_id}` | GetFile | Get file metadata (SDK method: `retrieve_metadata`) |
|
||||
| `GET` | `/v1/files/{file_id}/content` | DownloadFile | Download file content |
|
||||
| `DELETE` | `/v1/files/{file_id}` | DeleteFile | Delete a file |
|
||||
|
||||
## Skills
|
||||
|
||||
| Method | Path | Operation | Description |
|
||||
| -------- | --------------------------------------------------------------- | ------------------ | ---------------------------- |
|
||||
| `POST` | `/v1/skills` | CreateSkill | Create a skill |
|
||||
| `GET` | `/v1/skills` | ListSkills | List skills |
|
||||
| `GET` | `/v1/skills/{skill_id}` | GetSkill | Get skill details |
|
||||
| `DELETE` | `/v1/skills/{skill_id}` | DeleteSkill | Delete a skill |
|
||||
| `POST` | `/v1/skills/{skill_id}/versions` | CreateVersion | Create skill version |
|
||||
| `GET` | `/v1/skills/{skill_id}/versions` | ListVersions | List skill versions |
|
||||
| `GET` | `/v1/skills/{skill_id}/versions/{version}` | GetVersion | Get skill version |
|
||||
| `DELETE` | `/v1/skills/{skill_id}/versions/{version}` | DeleteVersion | Delete skill version |
|
||||
|
||||
---
|
||||
|
||||
## Request/Response Schema Quick Reference
|
||||
|
||||
### CreateAgent Request Body
|
||||
|
||||
**Always start here.** `model`, `system`, `tools`, `mcp_servers`, `skills` are top-level fields on this object — they do NOT go on the session.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "string (required, 1-256 chars)",
|
||||
"model": "claude-opus-4-8 (required — bare string, or {id, speed} object)",
|
||||
"description": "string (optional, up to 2048 chars)",
|
||||
"system": "string (optional, up to 100,000 chars)",
|
||||
"tools": [
|
||||
{ "type": "agent_toolset_20260401" }
|
||||
],
|
||||
"skills": [
|
||||
{ "type": "anthropic", "skill_id": "xlsx" },
|
||||
{ "type": "custom", "skill_id": "skill_abc123", "version": "1" }
|
||||
],
|
||||
"mcp_servers": [
|
||||
{
|
||||
"type": "url",
|
||||
"name": "github",
|
||||
"url": "https://api.githubcopilot.com/mcp/"
|
||||
}
|
||||
],
|
||||
"multiagent": {
|
||||
"type": "coordinator",
|
||||
"agents": [
|
||||
"agent_abc123",
|
||||
{ "type": "agent", "id": "agent_def456", "version": 4 },
|
||||
{ "type": "self" }
|
||||
]
|
||||
},
|
||||
"metadata": {
|
||||
"key": "value (max 16 pairs, keys ≤64 chars, values ≤512 chars)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> Limits: `tools` max 128, `skills` max 20, `mcp_servers` max 20 (unique names). `multiagent.agents` 1–20 entries (string ID | `{type:"agent",id,version?}` | `{type:"self"}`) — see `shared/managed-agents-multiagent.md`.
|
||||
|
||||
### CreateSession Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"agent": "agent_abc123 (required — string shorthand for latest version, or {type: \"agent\", id, version} object)",
|
||||
"environment_id": "env_abc123 (required)",
|
||||
"title": "string (optional)",
|
||||
"resources": [
|
||||
{
|
||||
"type": "github_repository",
|
||||
"url": "https://github.com/owner/repo (required)",
|
||||
"authorization_token": "ghp_... (required)",
|
||||
"mount_path": "/workspace/repo (optional — defaults to /workspace/<repo-name>)",
|
||||
"checkout": { "type": "branch", "name": "main" }
|
||||
}
|
||||
],
|
||||
"vault_ids": ["vlt_abc123 (optional — vault credentials: MCP auth + environment variables)"],
|
||||
"metadata": {
|
||||
"key": "value"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> The `agent` field accepts only a string ID or `{type: "agent", id, version}` — `model`/`system`/`tools` live on the agent, not here.
|
||||
>
|
||||
> **`checkout`** accepts `{type: "branch", name: "..."}` or `{type: "commit", sha: "..."}`. Omit for the repo's default branch.
|
||||
|
||||
### CreateEnvironment Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "string (required)",
|
||||
"description": "string (optional)",
|
||||
"config": {
|
||||
"type": "cloud | self_hosted",
|
||||
"networking": {
|
||||
"type": "unrestricted | limited (union — see SDK types)"
|
||||
},
|
||||
"packages": { }
|
||||
},
|
||||
"metadata": { "key": "value" }
|
||||
}
|
||||
```
|
||||
|
||||
### CreateDeployment Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Weekly compliance scan",
|
||||
"agent": "agent_abc123 (required — same shapes as CreateSession)",
|
||||
"environment_id": "env_abc123 (required)",
|
||||
"initial_events": [
|
||||
{ "type": "user.message", "content": [{ "type": "text", "text": "Run the weekly compliance scan." }] }
|
||||
],
|
||||
"schedule": {
|
||||
"type": "cron",
|
||||
"expression": "0 20 * * 5",
|
||||
"timezone": "America/New_York"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> Optional session config (`resources`, `vault_ids`, etc.) is supported the same way as on CreateSession. Response includes `status`, `paused_reason`, and `schedule.upcoming_runs_at` (next fire times). See `shared/managed-agents-scheduled-deployments.md`.
|
||||
|
||||
### SendEvents Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"events": [
|
||||
{
|
||||
"type": "user.message",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Hello"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
> `system.message` events (update the system prompt between turns) use the same envelope with `type: "system.message"` — Claude Opus 4.8 only; see `shared/managed-agents-events.md` § Updating the system prompt mid-session.
|
||||
|
||||
### Define Outcome Event
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "user.define_outcome",
|
||||
"description": "Build a DCF model for Costco in .xlsx",
|
||||
"rubric": { "type": "file", "file_id": "file_01..." },
|
||||
"max_iterations": 5
|
||||
}
|
||||
```
|
||||
|
||||
> `rubric` is required: `{type: "text", content}` or `{type: "file", file_id}`. `max_iterations` default 3, max 20. Echoed back with `outcome_id` + `processed_at`. See `shared/managed-agents-outcomes.md`.
|
||||
|
||||
### Tool Result Event
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "user.custom_tool_result",
|
||||
"custom_tool_use_id": "sevt_abc123",
|
||||
"content": [{ "type": "text", "text": "Result data" }],
|
||||
"is_error": false
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
Managed Agents endpoints use the standard Anthropic API error format. Errors are returned with an HTTP status code and a JSON body containing `type`, `error`, and `request_id`:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "invalid_request_error",
|
||||
"message": "Description of what went wrong"
|
||||
},
|
||||
"request_id": "req_011CRv1W3XQ8XpFikNYG7RnE"
|
||||
}
|
||||
```
|
||||
|
||||
Include the `request_id` when reporting issues to Anthropic — it lets us trace the request end-to-end. The inner `error.type` is one of the following:
|
||||
|
||||
| Status | Error type | Description |
|
||||
|---|---|---|
|
||||
| 400 | `invalid_request_error` | The request was malformed or missing required parameters |
|
||||
| 401 | `authentication_error` | Invalid or missing API key |
|
||||
| 403 | `permission_error` | The API key doesn't have permission for this operation |
|
||||
| 404 | `not_found_error` | The requested resource doesn't exist |
|
||||
| 409 | `invalid_request_error` | The request conflicts with the resource's current state (e.g., sending to an archived session) |
|
||||
| 413 | `request_too_large` | The request body exceeds the maximum allowed size |
|
||||
| 429 | `rate_limit_error` | Too many requests — check rate limit headers for retry timing |
|
||||
| 500 | `api_error` | An internal server error occurred |
|
||||
| 529 | `overloaded_error` | The service is temporarily overloaded — retry with backoff |
|
||||
|
||||
Note that `409 Conflict` carries `error.type: "invalid_request_error"` (there is no separate `conflict_error` type); inspect both the HTTP status and the `message` to distinguish conflicts from other invalid requests.
|
||||
|
||||
---
|
||||
|
||||
## Rate Limits
|
||||
|
||||
Managed Agents endpoints have per-organization request-per-minute (RPM) limits, separate from your [Messages API token limits](https://platform.claude.com/docs/en/api/rate-limits). Model inference inside a session still draws from your organization's standard ITPM/OTPM limits.
|
||||
|
||||
| Endpoint group | Scope | RPM | Max concurrent |
|
||||
|---|---|---|---|
|
||||
| Create operations (Agents, Sessions, Vaults) | organization | 300 | — |
|
||||
| All other operations (Agents, Sessions, Vaults) | organization | 600 | — |
|
||||
| All operations (Environments) | organization | 60 | 5 |
|
||||
|
||||
Files and Skills endpoints use the standard tier-based [rate limits](https://platform.claude.com/docs/en/api/rate-limits).
|
||||
|
||||
When a limit is exceeded the API returns `429` with a `rate_limit_error` (see [Error Handling](#error-handling) for the response envelope) and a `retry-after` header indicating how many seconds to wait before retrying. The Anthropic SDK reads this header and retries automatically.
|
||||
@@ -1,211 +0,0 @@
|
||||
# Managed Agents — Common Client Patterns
|
||||
|
||||
Patterns you'll write on the client side when driving a Managed Agent session, grounded in working SDK examples.
|
||||
|
||||
Code samples are TypeScript — Python and cURL follow the same shape; see `python/managed-agents/README.md` and `curl/managed-agents.md` for equivalents.
|
||||
|
||||
---
|
||||
|
||||
## 1. Lossless stream reconnect
|
||||
|
||||
**Problem:** SSE has no replay. If the connection drops mid-session, a naive reconnect re-opens the stream from "now" and you silently miss every event emitted in between.
|
||||
|
||||
**Solution:** on reconnect, fetch the full event history via `events.list()` *before* consuming the live stream, and dedupe on event ID as the live stream catches up.
|
||||
|
||||
```ts
|
||||
const seenEventIds = new Set<string>()
|
||||
const stream = await client.beta.sessions.events.stream(session.id)
|
||||
|
||||
// Stream is now open and buffering server-side. Read history first.
|
||||
for await (const event of client.beta.sessions.events.list(session.id)) {
|
||||
seenEventIds.add(event.id)
|
||||
handle(event)
|
||||
}
|
||||
|
||||
// Tail the live stream. Dedupe only gates handle() — terminal checks must run
|
||||
// even for already-seen events, or a terminal event that was in the history
|
||||
// response gets skipped by `continue` and the loop never exits.
|
||||
for await (const event of stream) {
|
||||
if (!seenEventIds.has(event.id)) {
|
||||
seenEventIds.add(event.id)
|
||||
handle(event)
|
||||
}
|
||||
if (event.type === 'session.status_terminated') break
|
||||
if (event.type === 'session.status_idle' && event.stop_reason.type !== 'requires_action') break
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. `processed_at` — queued vs processed
|
||||
|
||||
Every event on the stream carries `processed_at` (ISO 8601). For client-sent events (`user.message`, `user.interrupt`, `user.tool_confirmation`, `user.custom_tool_result`) it's `null` when the event has been queued but not yet picked up by the agent, and populated once the agent processes it. The same event appears on the stream twice — once with `processed_at: null`, once with a timestamp.
|
||||
|
||||
```ts
|
||||
for await (const event of stream) {
|
||||
if (event.type === 'user.message') {
|
||||
if (event.processed_at == null) onQueued(event.id)
|
||||
else onProcessed(event.id, event.processed_at)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use this to drive pending → acknowledged UI state for anything you send. How you map a locally-rendered optimistic message to the server-assigned `event.id` is application-specific (typically via the return value of `events.send()` or FIFO ordering).
|
||||
|
||||
---
|
||||
|
||||
## 3. Interrupt a running session
|
||||
|
||||
Send `user.interrupt` as a normal event. The session keeps running until it reaches a safe boundary, then goes idle.
|
||||
|
||||
```ts
|
||||
await client.beta.sessions.events.send(session.id, {
|
||||
events: [{ type: 'user.interrupt' }],
|
||||
})
|
||||
|
||||
// Drain until the session is truly done — see Pattern 5 for the full gate.
|
||||
for await (const event of stream) {
|
||||
if (event.type === 'session.status_terminated') break
|
||||
if (
|
||||
event.type === 'session.status_idle' &&
|
||||
event.stop_reason.type !== 'requires_action'
|
||||
) break
|
||||
}
|
||||
```
|
||||
|
||||
Reference: `interrupt.ts` — sends the interrupt the moment it sees `span.model_request_start`, drains to idle, then verifies via `sessions.retrieve()`.
|
||||
|
||||
---
|
||||
|
||||
## 4. `tool_confirmation` round-trip
|
||||
|
||||
When the agent has `permission_policy: { type: 'always_ask' }`, any call to that tool fires an `agent.tool_use` event with `evaluated_permission === 'ask'` and the session goes idle waiting for a decision. Respond with `user.tool_confirmation`.
|
||||
|
||||
```ts
|
||||
for await (const event of stream) {
|
||||
if (event.type === 'agent.tool_use' && event.evaluated_permission === 'ask') {
|
||||
await client.beta.sessions.events.send(session.id, {
|
||||
events: [{
|
||||
type: 'user.tool_confirmation',
|
||||
tool_use_id: event.id, // not a toolu_ id — use event.id
|
||||
result: 'allow', // or 'deny'
|
||||
// deny_message: '...', // optional, only with result: 'deny'
|
||||
}],
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Key points:
|
||||
- `tool_use_id` is `event.id` (typically `sevt_...`), **not** a `toolu_...` ID.
|
||||
- `result` is `'allow' | 'deny'`. Use `deny_message` to tell the model *why* you denied — it gets surfaced back to the agent.
|
||||
- Multiple pending tools: respond once per `agent.tool_use` event with `evaluated_permission === 'ask'`.
|
||||
|
||||
Reference: `tool-permissions.ts`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Correct idle-break gate
|
||||
|
||||
Do not break on `session.status_idle` alone. The session goes idle transiently — e.g. between parallel tool executions, while waiting for a `user.tool_confirmation`, or while awaiting a `user.custom_tool_result`. Break when idle with a terminal `stop_reason`, or on `session.status_terminated`.
|
||||
|
||||
```ts
|
||||
for await (const event of stream) {
|
||||
handle(event)
|
||||
if (event.type === 'session.status_terminated') break
|
||||
if (event.type === 'session.status_idle') {
|
||||
if (event.stop_reason.type === 'requires_action') continue // waiting on you — handle it
|
||||
break // end_turn or retries_exhausted — both terminal
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`stop_reason.type` values on `session.status_idle`:
|
||||
- `requires_action` — agent is waiting on a client-side event (tool confirmation, custom tool result). Handle it, don't break.
|
||||
- `retries_exhausted` — terminal failure. Break, then check `sessions.retrieve()` for the error state.
|
||||
- `end_turn` — normal completion.
|
||||
|
||||
---
|
||||
|
||||
## 6. Post-idle status-write race
|
||||
|
||||
The SSE stream emits `session.status_idle` slightly before the session's queryable status reflects it. Clients that break on idle and immediately call `sessions.delete()` or `sessions.archive()` will intermittently 400 with "cannot delete/archive while running."
|
||||
|
||||
Poll before cleanup:
|
||||
|
||||
```ts
|
||||
let s
|
||||
for (let i = 0; i < 10; i++) {
|
||||
s = await client.beta.sessions.retrieve(session.id)
|
||||
if (s.status !== 'running') break
|
||||
await new Promise(r => setTimeout(r, 200))
|
||||
}
|
||||
if (s?.status !== 'running') {
|
||||
await client.beta.sessions.archive(session.id)
|
||||
} // else: still running after 2s — don't archive, let it settle or escalate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Stream-first, then send
|
||||
|
||||
Always open the stream **before** sending the kickoff event. Otherwise the agent may process the event and emit the first events before your consumer is attached, and you'll miss them.
|
||||
|
||||
```ts
|
||||
const stream = await client.beta.sessions.events.stream(session.id)
|
||||
await client.beta.sessions.events.send(session.id, {
|
||||
events: [{ type: 'user.message', content: [{ type: 'text', text: 'Hello' }] }],
|
||||
})
|
||||
for await (const event of stream) { /* ... */ }
|
||||
```
|
||||
|
||||
The `Promise.all([stream, send])` shape works too, but stream-first is simpler and has the same effect — the stream starts buffering the moment it's opened.
|
||||
|
||||
---
|
||||
|
||||
## 8. File-mount gotchas
|
||||
|
||||
**The mounted resource has a different `file_id` than the file you uploaded.** Session creation makes a session-scoped copy.
|
||||
|
||||
```ts
|
||||
const uploaded = await client.beta.files.upload({ file })
|
||||
// uploaded.id → the original file
|
||||
const session = await client.beta.sessions.create({
|
||||
/* ... */
|
||||
resources: [{ type: 'file', file_id: uploaded.id, mount_path: '/workspace/data.csv' }],
|
||||
})
|
||||
// session.resources[0].file_id !== uploaded.id ← different IDs
|
||||
```
|
||||
|
||||
Delete the original via `files.delete(uploaded.id)`; the session-scoped copy is garbage-collected with the session. `mount_path` must be absolute — see `shared/managed-agents-environments.md`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Secrets for non-MCP APIs and CLIs — keep them host-side via custom tools
|
||||
|
||||
**Problem:** you want the agent to call a third-party API or run a CLI that needs a secret (API key, token, service-account credential), but you can't or don't want to hand the secret to a vault.
|
||||
|
||||
**First check:** for cloud environments, the first-class answer is now a vault `environment_variable` credential — the agent's shell sees an opaque placeholder and the real secret is substituted at egress. See `shared/managed-agents-tools.md` → Vaults. Use this pattern instead when that doesn't fit: **self-hosted sandboxes** (env-var credentials not yet supported there), clients that reject the placeholder via local format validation, secrets that must never leave your infrastructure, or calls that need host-side binaries.
|
||||
|
||||
**Solution:** move the authenticated call to your side. Declare a custom tool on the agent; when the agent emits `agent.custom_tool_use`, your orchestrator (the process reading the SSE stream) executes the call with its own credentials and responds with `user.custom_tool_result`. The container never sees the key.
|
||||
|
||||
```ts
|
||||
// Agent template: declare the tool, no credentials
|
||||
tools: [{ type: 'custom', name: 'linear_graphql', input_schema: { /* query, vars */ } }]
|
||||
|
||||
// Orchestrator: handle the call with host-side creds
|
||||
for await (const event of stream) {
|
||||
if (event.type === 'agent.custom_tool_use' && event.name === 'linear_graphql') {
|
||||
const result = await linear.request(event.input.query, event.input.vars) // host's key
|
||||
await client.beta.sessions.events.send(session.id, {
|
||||
events: [{ type: 'user.custom_tool_result', tool_use_id: event.id, result }],
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Same shape works for `gh` CLI, local eval scripts, or anything else that needs host-side auth or binaries.
|
||||
|
||||
**Security note:** this does not expose a public endpoint. `agent.custom_tool_use` arrives on the SSE stream your orchestrator already holds open with your Anthropic API key, and `user.custom_tool_result` goes back via `events.send()` under the same key. Your orchestrator is a client, not a server — nothing unauthenticated is listening.
|
||||
|
||||
**Do not embed API keys in the system prompt or user messages as a workaround.** Prompts and messages are stored in the session's event history, returned by `events.list()`, and included in compaction summaries — a secret placed there is durably persisted and readable via the API for the life of the session.
|
||||
@@ -1,252 +0,0 @@
|
||||
# Managed Agents — Core Concepts
|
||||
|
||||
## Architecture
|
||||
|
||||
Managed Agents is built around four core concepts:
|
||||
|
||||
| Concept | Endpoint | What it is |
|
||||
|---|---|---|
|
||||
| **Agent** | `/v1/agents` | A persisted, versioned object defining the agent's capabilities and persona: model, system prompt, tools, MCP servers, skills. **Must be created before starting a session.** See the Agents section below. |
|
||||
| **Session** | `/v1/sessions` | A stateful interaction with an agent. References a pre-created agent by ID + an environment + initial instructions. Produces an event stream. |
|
||||
| **Environment** | `/v1/environments` | A template defining the configuration for container provisioning. |
|
||||
| **Container** | N/A | An isolated compute instance where the agent's **tools** execute (bash, file ops, code). The agent loop does not run here — it runs on Anthropic's orchestration layer and acts on the container via tool calls. |
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ Anthropic orchestration layer │
|
||||
Agent (config) ───────▶│ (agent loop: Claude + tool calls) │
|
||||
└──────────────┬──────────────────────┘
|
||||
│ tool calls
|
||||
▼
|
||||
Environment (template) ──▶ Container (tool execution workspace)
|
||||
│
|
||||
Session ─┤
|
||||
├── Resources (files, repos, memory stores — attached at startup)
|
||||
├── Vault IDs (MCP credential references)
|
||||
└── Conversation (event stream in/out)
|
||||
```
|
||||
|
||||
> **Agent creation is a prerequisite.** Sessions reference a pre-created agent by ID — `model`/`system`/`tools` live on the agent object, never on the session. Every flow starts with `POST /v1/agents`.
|
||||
|
||||
---
|
||||
|
||||
## Session Lifecycle
|
||||
|
||||
```
|
||||
rescheduling → running ↔ idle → terminated
|
||||
```
|
||||
|
||||
| Status | Description |
|
||||
| -------------- | ------------------------------------------------------------------ |
|
||||
| `idle` | Agent has finished the current task, and is awaiting input. It's either waiting for input to continue working via a `user.message` or blocked awaiting a `user.custom_tool_result` or `user.tool_confirmation`. The `stop_reason` attached contains more information about why the Agent has stopped working. |
|
||||
| `running` | Session has starting running, and the Agent is actively doing work. |
|
||||
| `rescheduling` | Session is (re)scheduling after a retryable error has occurred, ready to be picked up by the orchestration system. |
|
||||
| `terminated` | Session has terminated, entering an irreversible and unusable state. |
|
||||
|
||||
- Events can be sent when the session is `running` or `idle`. Messages are queued and processed in order.
|
||||
- The agent transitions `idle → running` when it receives a new event, then back to `idle` when done.
|
||||
- Errors surface as `session.error` events in the stream, not as a status value.
|
||||
|
||||
### Built-in session features
|
||||
|
||||
- **Context compaction** — if you approach max context, the API automatically condenses session history to keep the interaction going
|
||||
- **Prompt caching** — historical repeated tokens are cached, reducing processing time and cost
|
||||
- **Extended thinking** — on by default, returned as `agent.thinking` events
|
||||
|
||||
### Session operations
|
||||
|
||||
| Operation | Notes |
|
||||
|---|---|
|
||||
| List / fetch | Paginated list or single resource by ID |
|
||||
| Update | Only `title` is updatable |
|
||||
| Archive | Session becomes **read-only**. Not reversible. |
|
||||
| Delete | Permanently deletes session, event history, container, and checkpoints. |
|
||||
|
||||
These are ops/inspection calls — typically made from a terminal, not application code. From the shell (see `shared/anthropic-cli.md`):
|
||||
|
||||
```sh
|
||||
ant beta:sessions list --transform '{id,title,status,created_at}' --format jsonl
|
||||
ant beta:sessions retrieve --session-id "$SID"
|
||||
ant beta:sessions:events stream --session-id "$SID" # watch events live
|
||||
ant beta:sessions archive --session-id "$SID"
|
||||
ant beta:sessions delete --session-id "$SID"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sessions
|
||||
|
||||
A session is a running agent instance inside an environment.
|
||||
|
||||
### Session Object
|
||||
|
||||
Key fields returned by the API:
|
||||
|
||||
| Field | Type | Description |
|
||||
| --------------- | -------- | --------------------------------------------------- |
|
||||
| `type` | string | Always `"session"` |
|
||||
| `id` | string | Unique session ID |
|
||||
| `title` | string | Human-readable title |
|
||||
| `status` | string | `idle`, `running`, `rescheduling`, `terminated` |
|
||||
| `created_at` | string | ISO 8601 timestamp |
|
||||
| `updated_at` | string | ISO 8601 timestamp |
|
||||
| `archived_at` | string | ISO 8601 timestamp (nullable) |
|
||||
| `environment_id` | string | Environment ID |
|
||||
| `agent` | object | Agent configuration |
|
||||
| `resources` | array | Attached files, repos, and memory stores |
|
||||
| `metadata` | object | User-provided key-value pairs (max 8 keys) |
|
||||
| `usage` | object | Token usage statistics |
|
||||
|
||||
### Creating a session
|
||||
|
||||
**A session is meaningless without an agent.** Sessions reference a pre-created agent by ID. Create the agent first via `agents.create()`, then reference it:
|
||||
|
||||
```ts
|
||||
// 1. Create the agent (reusable, versioned)
|
||||
const agent = await client.beta.agents.create(
|
||||
{
|
||||
name: "Coding Assistant",
|
||||
model: "claude-opus-4-8",
|
||||
system: "You are a helpful coding agent.",
|
||||
tools: [{ type: "agent_toolset_20260401"}],
|
||||
},
|
||||
);
|
||||
|
||||
// 2. Start a session that references it
|
||||
const session = await client.beta.sessions.create(
|
||||
{
|
||||
agent: agent.id, // string shorthand → latest version. Or: { type: "agent", id: agent.id, version: agent.version }
|
||||
environment_id: environmentId,
|
||||
title: "Hello World Session",
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
> 💡 **Watch it live in Console.** While developing, print a link so you can click through to the session UI instead of parsing the event stream: `https://platform.claude.com/workspaces/{workspace}/sessions/{session.id}`. Use `default` for the workspace slug unless you created the session in a non-default workspace; the sessions list for your workspace is at `https://platform.claude.com/workspaces/default/sessions`. Include this `print`/`console.log` in code you generate for users who are iterating locally.
|
||||
|
||||
**Session creation parameters:**
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
| --------------- | -------- | -------- | ---------------------------------------------- |
|
||||
| `agent` | string or object | **Yes** | String shorthand `"agent_abc123"` (latest version) or `{type: "agent", id, version}` |
|
||||
| `environment_id`| string | **Yes** | Environment ID |
|
||||
| `title` | string | No | Human-readable name (appears in logs/dashboards) |
|
||||
| `resources` | array | No | Files, GitHub repos, or memory stores, attached to the container at startup. Memory stores are session-create-only (not addable via `resources.add()`). |
|
||||
| `vault_ids` | array | No | Vault IDs (`vlt_*`) — MCP credentials with auto-refresh + `environment_variable` secrets substituted at egress. See `shared/managed-agents-tools.md` → Vaults. |
|
||||
| `metadata` | object | No | User-provided key-value pairs |
|
||||
|
||||
**Agent configuration fields** (passed to `agents.create()`, not `sessions.create()`):
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
| ------------- | -------- | -------- | ---------------------------------------------- |
|
||||
| `name` | string | **Yes** | Human-readable name (1-256 chars) |
|
||||
| `model` | string or object | **Yes** | Claude model ID (bare string, or `{id, speed}` object). All Claude 4.5+ models supported. |
|
||||
| `system` | string | No | System prompt — defines the agent's behavior (up to 100K chars) |
|
||||
| `tools` | array | No | Encompasses three kinds: (1) pre-built Claude Agent tools (`agent_toolset_20260401`), (2) MCP tools (`mcp_toolset`), and (3) custom client-side tools. Max 128. |
|
||||
| `mcp_servers` | array | No | MCP server connections — standardized third-party capabilities (e.g. GitHub, Asana). Max 20, unique names. See `shared/managed-agents-tools.md` → MCP Servers. |
|
||||
| `skills` | array | No | Customized "best-practices" context with progressive disclosure. Max 20. See `shared/managed-agents-tools.md` → Skills. |
|
||||
| `description` | string | No | Description of the agent (up to 2048 chars) |
|
||||
| `multiagent` | object | No | `{type: "coordinator", agents: [...]}` — roster this agent may delegate to. See `shared/managed-agents-multiagent.md`. |
|
||||
| `metadata` | object | No | Arbitrary key-value pairs (max 16, keys ≤64 chars, values ≤512 chars) |
|
||||
|
||||
---
|
||||
|
||||
## Agents
|
||||
|
||||
**This is where every Managed Agents flow begins.** The agent object is a persisted, versioned configuration — you create it once, then reference it by ID every time you start a session. No agent → no session.
|
||||
|
||||
### Agent Object
|
||||
|
||||
The API is **flat** — `model`, `system`, `tools` etc. are top-level fields, not wrapped in an `agent:{}` sub-object.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
| ------------------ | -------- | -------- | -------------------------------------------------- |
|
||||
| `name` | string | Yes | Human-readable name |
|
||||
| `model` | string | Yes | Claude model ID |
|
||||
| `system` | string | No | System prompt |
|
||||
| `tools` | array | No | Agent toolset / MCP toolset / custom tools |
|
||||
| `mcp_servers` | array | No | MCP server connections |
|
||||
| `skills` | array | No | Skill references (max 20) |
|
||||
| `description` | string | No | Description of the agent |
|
||||
| `multiagent` | object | No | Coordinator roster — see `shared/managed-agents-multiagent.md` |
|
||||
| `metadata` | object | No | Arbitrary key-value pairs |
|
||||
|
||||
### Lifecycle: create once, run many, update in place
|
||||
|
||||
The agent is a **persistent resource**, not a per-run parameter. The intended pattern:
|
||||
|
||||
```
|
||||
┌─ setup (once) ─────────┐ ┌─ runtime (every invocation) ─┐
|
||||
│ agents.create() │ │ sessions.create( │
|
||||
│ → store agent_id │ ──→ │ agent={type:..., id: ID} │
|
||||
│ in config/env/db │ │ ) │
|
||||
└────────────────────────┘ └──────────────────────────────┘
|
||||
```
|
||||
|
||||
**Anti-pattern:** calling `agents.create()` at the top of every script run. This accumulates orphaned agent objects, pays create latency on every invocation, and defeats the versioning model. If you see `agents.create()` in a function that's called per-request or per-cron-tick, that's wrong — hoist it to one-time setup and persist the ID.
|
||||
|
||||
> **Recommended — define agents and environments as YAML + apply via the `ant` CLI.** The split is **CLI for the control plane, SDK for the data plane**: agents and environments are relatively static resources you manage with `ant` (version-controlled YAML, applied from CI); sessions are dynamic and driven by your application through the SDK. See `shared/anthropic-cli.md` → *Version-controlled Managed Agents resources* for the `ant beta:agents create < agent.yaml` / `update --version N` flow. The SDK `agents.create()` call shown elsewhere in this doc is the in-code equivalent — use it when you need to provision programmatically, but prefer the YAML flow for anything a human maintains.
|
||||
|
||||
### Versioning
|
||||
|
||||
Each `POST /v1/agents/{id}` (update) creates a new immutable version (numeric timestamp, e.g. `1772585501101368014`). The agent's history is append-only — you can't edit a past version.
|
||||
|
||||
**Why version:**
|
||||
- **Reproducibility** — pin a session to a known-good config: `{type: "agent", id, version: 3}`
|
||||
- **Safe iteration** — update the agent without breaking sessions already running on the old version
|
||||
- **Rollback** — if a new system prompt regresses, pin new sessions back to the prior version while you debug
|
||||
|
||||
**`version` is optional.** Omit it (or use the string shorthand `agent="agent_abc123"`) to get the latest version at session-creation time. Pass it explicitly (`{type: "agent", id, version: N}`) to pin for reproducibility.
|
||||
|
||||
**Getting the version to pin:** `agents.create()` and `agents.update()` both return `version` in the response. Store it alongside `agent_id`. To fetch the current latest for an existing agent: `GET /v1/agents/{id}` → `.version`.
|
||||
|
||||
**When to update vs create new:** Update (`POST /v1/agents/{id}`) when it's conceptually the same agent with tweaked behavior (better prompt, extra tool). Create a new agent when it's a different persona/purpose. Rule of thumb: if you'd give it the same `name`, update.
|
||||
|
||||
### Agent Endpoints
|
||||
|
||||
| Operation | Method | Path |
|
||||
| ---------------- | -------- | ------------------------------------- |
|
||||
| Create | `POST` | `/v1/agents` |
|
||||
| List | `GET` | `/v1/agents` |
|
||||
| Get | `GET` | `/v1/agents/{id}` |
|
||||
| Update | `POST` | `/v1/agents/{id}` |
|
||||
| Archive | `POST` | `/v1/agents/{id}/archive` |
|
||||
|
||||
> ⚠️ **Archive is permanent.** Archiving makes the agent read-only: existing sessions continue to run, but **new sessions cannot reference it**, and there is no unarchive. Since agents have no `delete`, this is the terminal lifecycle state. Never archive a production agent as routine cleanup — confirm with the user first.
|
||||
|
||||
### Using an Agent in a Session
|
||||
|
||||
Reference the agent by string ID (latest version) or by object with an explicit version:
|
||||
|
||||
```python
|
||||
# String shorthand — uses the agent's latest version
|
||||
session = client.beta.sessions.create(
|
||||
agent=agent.id,
|
||||
environment_id=environment_id,
|
||||
)
|
||||
|
||||
# Or pin to a specific version (int)
|
||||
session = client.beta.sessions.create(
|
||||
agent={"type": "agent", "id": agent.id, "version": agent.version},
|
||||
environment_id=environment_id,
|
||||
)
|
||||
```
|
||||
|
||||
### Updating the agent configuration mid-session
|
||||
|
||||
`sessions.update()` can change `agent.tools`, `agent.mcp_servers` (including permission policies), and `vault_ids` on an **existing** session. This is a **session-local override** — it does not create a new agent version and does not propagate back to the agent object. The provided arrays are **full replacements**; to append one tool, `GET` the session, modify, and `POST` back. The session must be `idle` — interrupt first if running.
|
||||
|
||||
```python
|
||||
client.beta.sessions.update(
|
||||
session.id,
|
||||
agent={
|
||||
"tools": [
|
||||
{"type": "agent_toolset_20260401"},
|
||||
{"type": "mcp_toolset", "mcp_server_name": "linear"},
|
||||
],
|
||||
"mcp_servers": [{"type": "url", "name": "linear", "url": "https://mcp.linear.app/sse"}],
|
||||
},
|
||||
vault_ids=["vlt_..."],
|
||||
)
|
||||
```
|
||||
|
||||
@@ -1,219 +0,0 @@
|
||||
# Managed Agents — Environments & Resources
|
||||
|
||||
## Environments
|
||||
|
||||
Creating a session requires an `environment_id`. Environments are **reusable configuration templates** for spinning up containers in Anthropic's infrastructure — you might create different environments for different use cases (e.g. data visualization vs web development, with different package sets). Anthropic handles scaling, container lifecycle, and work orchestration.
|
||||
|
||||
**Environment names must be unique.** Creating an environment with an existing name returns 409.
|
||||
|
||||
### Networking
|
||||
|
||||
| Network Policy | Description |
|
||||
| ---------------- | ------------------------------------------------------------- |
|
||||
| `unrestricted` | Full egress (except legal blocklist) |
|
||||
| `limited` | Deny-by-default; opt in via `allowed_hosts` / `allow_package_managers` / `allow_mcp_servers` |
|
||||
|
||||
```json
|
||||
{
|
||||
"networking": {
|
||||
"type": "limited",
|
||||
"allow_package_managers": true,
|
||||
"allow_mcp_servers": true,
|
||||
"allowed_hosts": ["api.example.com"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
All three `limited` fields are optional. `allow_package_managers` (default `false`) permits PyPI/npm/etc.; `allow_mcp_servers` (default `false`) permits the agent's configured MCP server endpoints without listing them in `allowed_hosts`.
|
||||
|
||||
**MCP caveat:** Under `limited` networking, either set `allow_mcp_servers: true` or add each MCP server domain to `allowed_hosts`. Otherwise the container can't reach them and tools silently fail.
|
||||
|
||||
### Creating an environment
|
||||
|
||||
The SDK adds `managed-agents-2026-04-01` automatically. TypeScript:
|
||||
|
||||
```ts
|
||||
const env = await client.beta.environments.create({
|
||||
name: "my_env",
|
||||
config: {
|
||||
type: "cloud",
|
||||
networking: { type: "unrestricted" },
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### Self-hosted sandboxes
|
||||
|
||||
To run tool execution in **your own infrastructure** instead of Anthropic's, set `config: {type: "self_hosted"}` — the agent loop stays on Anthropic's side, but `bash` / file ops / code execute in a container you control via an outbound-polling worker. The `networking` block does not apply (you control egress). Resource mounting (`file`, `github_repository`) and memory stores behave differently — see `shared/managed-agents-self-hosted-sandboxes.md` for the worker, credentials, and cloud-vs-self-hosted comparison.
|
||||
|
||||
### Environment CRUD
|
||||
|
||||
| Operation | Method | Path | Notes |
|
||||
| ---------------- | -------- | ------------------------------------------ | ----- |
|
||||
| Create | `POST` | `/v1/environments` | |
|
||||
| List | `GET` | `/v1/environments` | Paginated (`limit`, `after_id`, `before_id`) |
|
||||
| Get | `GET` | `/v1/environments/{id}` | |
|
||||
| Update | `POST` | `/v1/environments/{id}` | Changes apply only to **new** containers; existing sessions keep their original config |
|
||||
| Delete | `DELETE` | `/v1/environments/{id}` | Returns 204. |
|
||||
| Archive | `POST` | `/v1/environments/{id}/archive` | Makes it **read-only**; existing sessions continue, new sessions cannot reference it. No unarchive — terminal state. |
|
||||
|
||||
---
|
||||
|
||||
## Resources
|
||||
|
||||
Attach files, GitHub repositories, and memory stores to a session. **Session creation blocks until all resources are mounted** — the container won't go `running` until every file and repo is in place. Max **999 file resources** per session. Multiple GitHub repositories per session are supported. For `type: "memory_store"` resources (persistent cross-session memory — max 8 per session), see `shared/managed-agents-memory.md`.
|
||||
|
||||
### File Uploads (input — host → agent)
|
||||
|
||||
Upload a file first via the Files API, then reference by `file_id` + `mount_path`:
|
||||
|
||||
```ts
|
||||
// 1. Upload
|
||||
const file = await client.beta.files.upload({
|
||||
file: fs.createReadStream("data.csv"),
|
||||
});
|
||||
|
||||
// 2. Attach as a session resource
|
||||
const session = await client.beta.sessions.create({
|
||||
agent: agent.id,
|
||||
environment_id: envId,
|
||||
resources: [
|
||||
{ type: "file", file_id: file.id, mount_path: "/workspace/data.csv" }
|
||||
],
|
||||
});
|
||||
```
|
||||
|
||||
**`mount_path` is required** and must be absolute. Parent directories are created automatically. Agent working directory defaults to `/workspace`. Files are mounted read-only — the agent writes modified versions to new paths.
|
||||
|
||||
### Session outputs (output — agent → host)
|
||||
|
||||
The agent can write files to `/mnt/session/outputs/` during a session. These are automatically captured by the Files API and can be listed and downloaded afterwards:
|
||||
|
||||
```ts
|
||||
// After the turn completes, list output files scoped to this session:
|
||||
for await (const f of client.beta.files.list({
|
||||
scope_id: session.id,
|
||||
betas: ["managed-agents-2026-04-01"],
|
||||
})) {
|
||||
console.log(f.filename, f.size_bytes);
|
||||
const resp = await client.beta.files.download(f.id);
|
||||
const text = await resp.text();
|
||||
}
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- The `write` tool (or `bash`) must be enabled for the agent to create output files.
|
||||
- Session-scoped `files.list` / `files.download` captures outputs written to `/mnt/session/outputs/`.
|
||||
- The filter parameter is **`scope_id`** (REST query param `?scope_id=<session_id>`). The SDK's files resource auto-adds only the `files-api-2025-04-14` header, so pass `betas: ["managed-agents-2026-04-01"]` explicitly (or both headers on raw HTTP) — without it the API may reject `scope_id` as an unknown field. Requires `@anthropic-ai/sdk` ≥ 0.88.0 / `anthropic` (Python) ≥ 0.92.0 — older versions don't type `scope_id`. The `ant` CLI does **not** expose this flag yet; use the SDK or curl.
|
||||
- Pass the session ID returned by `sessions.create()` verbatim (e.g. `sesn_011CZx...`) — the API validates the prefix.
|
||||
- There's a brief indexing lag (~1–3s) between `session.status_idle` and output files appearing in `files.list`. Retry once or twice if empty.
|
||||
|
||||
> **Fallback when `scope_id` filtering is unavailable** (older SDK, or endpoint returns an error): send a follow-up `user.message` asking the agent to `read` each file under `/mnt/session/outputs/` and return the contents. The agent streams the file bodies back as `agent.message` text. This works for text files only and costs output tokens — use it to unblock, not as the primary path.
|
||||
|
||||
This gives you a bidirectional file bridge: upload reference data in, download agent artifacts out.
|
||||
|
||||
### GitHub Repositories
|
||||
|
||||
Clones a GitHub repository into the session container during initialization, before the agent begins execution. The agent can read, edit, commit, and push via `bash` (`git`). Multiple repositories per session are supported — add one `resources` entry per repo. Repositories are cached, so future sessions that use the same repository start faster.
|
||||
|
||||
Repositories are attached for the lifetime of the session — to change which repositories are mounted, create a new session. You **can** rotate a repository's `authorization_token` on a running session via `client.beta.sessions.resources.update(resource_id, {session_id, authorization_token})`; the resource `id` is returned at session creation and by `resources.list()`.
|
||||
|
||||
**Fields:**
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `type` | ✅ | `"github_repository"` |
|
||||
| `url` | ✅ | The GitHub repository URL |
|
||||
| `authorization_token` | ✅ | GitHub Personal Access Token with repository access. **Never echoed in API responses.** |
|
||||
| `mount_path` | ❌ | Path where the repository will be cloned. Defaults to `/workspace/<repo-name>`. |
|
||||
| `checkout` | ❌ | `{type: "branch", name: "..."}` or `{type: "commit", sha: "..."}`. Defaults to the repo's default branch. |
|
||||
|
||||
**Token permission levels** (fine-grained PATs):
|
||||
- `Contents: Read` — clone only
|
||||
- `Contents: Read and write` — push changes and create pull requests
|
||||
|
||||
**How auth works:** `authorization_token` is never placed inside the container. `git pull` / `git push` and GitHub REST calls against the attached repository are routed through an Anthropic-side git proxy that injects the token after the request leaves the sandbox. Code running in the container — including anything the agent writes — cannot read or exfiltrate it.
|
||||
|
||||
> ‼️ **To generate pull requests** you also need GitHub **MCP server** access — the `github_repository` resource gives filesystem + git access only. See `shared/managed-agents-tools.md` → MCP Servers. The PR workflow is: edit files in the mounted repo → push branch via `bash` (authenticated via the git proxy using `authorization_token`) → create PR via the MCP `create_pull_request` tool (authenticated via the vault).
|
||||
|
||||
**TypeScript:**
|
||||
|
||||
```ts
|
||||
// 1. Create the agent — declare GitHub MCP (no auth here)
|
||||
const agent = await client.beta.agents.create(
|
||||
{
|
||||
name: 'GitHub Agent',
|
||||
model: 'claude-opus-4-8',
|
||||
mcp_servers: [
|
||||
{ type: 'url', name: 'github', url: 'https://api.githubcopilot.com/mcp/' },
|
||||
],
|
||||
tools: [
|
||||
{ type: 'agent_toolset_20260401', default_config: { enabled: true } },
|
||||
{ type: 'mcp_toolset', mcp_server_name: 'github' },
|
||||
],
|
||||
},
|
||||
);
|
||||
|
||||
// 2. Start a session — attach vault for MCP auth + mount the repo
|
||||
const session = await client.beta.sessions.create({
|
||||
agent: agent.id,
|
||||
environment_id: envId,
|
||||
vault_ids: [vaultId], // vault contains the GitHub MCP OAuth credential
|
||||
resources: [
|
||||
{
|
||||
type: 'github_repository',
|
||||
url: 'https://github.com/owner/repo',
|
||||
authorization_token: process.env.GITHUB_TOKEN, // repo clone token (≠ MCP auth)
|
||||
checkout: { type: 'branch', name: 'main' },
|
||||
},
|
||||
],
|
||||
});
|
||||
```
|
||||
|
||||
**Python:**
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
agent = client.beta.agents.create(
|
||||
name="GitHub Agent",
|
||||
model="claude-opus-4-8",
|
||||
mcp_servers=[{
|
||||
"type": "url",
|
||||
"name": "github",
|
||||
"url": "https://api.githubcopilot.com/mcp/",
|
||||
}],
|
||||
tools=[
|
||||
{"type": "agent_toolset_20260401", "default_config": {"enabled": True}},
|
||||
{"type": "mcp_toolset", "mcp_server_name": "github"},
|
||||
],
|
||||
)
|
||||
|
||||
session = client.beta.sessions.create(
|
||||
agent=agent.id,
|
||||
environment_id=env_id,
|
||||
vault_ids=[vault_id], # vault contains the GitHub MCP OAuth credential
|
||||
resources=[{
|
||||
"type": "github_repository",
|
||||
"url": "https://github.com/owner/repo",
|
||||
"authorization_token": os.environ["GITHUB_TOKEN"], # repo clone token (≠ MCP auth)
|
||||
"checkout": {"type": "branch", "name": "main"},
|
||||
}],
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files API
|
||||
|
||||
Upload and manage files for use as session resources, and download files the agent wrote to `/mnt/session/outputs/`.
|
||||
|
||||
| Operation | Method | Path | SDK |
|
||||
| ---------------- | -------- | ------------------------------------- | --- |
|
||||
| Upload | `POST` | `/v1/files` | `client.beta.files.upload({ file })` |
|
||||
| List | `GET` | `/v1/files?scope_id=...` | `client.beta.files.list({ scope_id, betas: ["managed-agents-2026-04-01"] })` |
|
||||
| Get Metadata | `GET` | `/v1/files/{id}` | `client.beta.files.retrieveMetadata(id)` |
|
||||
| Download | `GET` | `/v1/files/{id}/content` | `client.beta.files.download(id)` → `Response` |
|
||||
| Delete | `DELETE` | `/v1/files/{id}` | `client.beta.files.delete(id)` |
|
||||
|
||||
The `scope_id` filter on List scopes the results to files written to `/mnt/session/outputs/` by that session. Without the filter, you get all files uploaded to your account.
|
||||
@@ -1,220 +0,0 @@
|
||||
# Managed Agents — Events & Steering
|
||||
|
||||
## Events
|
||||
|
||||
### Sending Events
|
||||
|
||||
Send events to a session via `POST /v1/sessions/{id}/events`.
|
||||
|
||||
| Event Type | When to Send |
|
||||
| ------------------------- | --------------------------------------------------- |
|
||||
| `user.message` | Send a user message |
|
||||
| `user.interrupt` | Interrupt the agent while it's running |
|
||||
| `user.tool_confirmation` | Approve/deny a tool call (when `always_ask` policy) |
|
||||
| `user.custom_tool_result` | Provide result for a custom tool call |
|
||||
| `user.define_outcome` | Start a rubric-graded iterate loop — see `shared/managed-agents-outcomes.md` |
|
||||
| `system.message` | Update the agent's system prompt between turns — **Claude Opus 4.8 only**; see § Updating the system prompt mid-session |
|
||||
|
||||
#### Updating the system prompt mid-session (`system.message`)
|
||||
|
||||
Unlike the `system` field on the agent definition (fixed at session creation), a `system.message` event changes the system prompt **as the session progresses** — a different persona, revised constraints, or runtime-fetched context that should shape behavior going forward:
|
||||
|
||||
```python
|
||||
client.beta.sessions.events.send(
|
||||
session.id,
|
||||
events=[
|
||||
{
|
||||
"type": "system.message",
|
||||
"content": [
|
||||
{"type": "text", "text": "The user's current timezone is America/New_York."},
|
||||
],
|
||||
},
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
Constraints:
|
||||
|
||||
- **Claude Opus 4.8 only.** If any model configured on the agent does not support mid-conversation system injection, the event is rejected with a `model_does_not_support_mid_conversation_system` validation error.
|
||||
- **Cannot be sent while the session is idle with `stop_reason: requires_action`** (blocked on `user.custom_tool_result` / `user.tool_confirmation`).
|
||||
- `content` accepts 1–1000 text items.
|
||||
|
||||
### Receiving Events
|
||||
|
||||
Three methods:
|
||||
|
||||
1. **Streaming (SSE)**: `GET /v1/sessions/{id}/events/stream` — real-time Server-Sent Events. **Long-lived** — the server sends periodic heartbeats to keep the connection alive.
|
||||
2. **Polling**: `GET /v1/sessions/{id}/events` — paginated event list (query params: `limit` default 1000, `page`). **Returns immediately** — this is a plain paginated GET, not a long-poll.
|
||||
3. **Webhooks**: Anthropic POSTs session state transitions to your HTTPS endpoint — thin payloads (IDs only), HMAC-signed, Console-registered. See `shared/managed-agents-webhooks.md`.
|
||||
|
||||
All received events carry `id`, `type`, and `processed_at` (ISO 8601; `null` if not yet processed by the agent).
|
||||
|
||||
> ⚠️ **Robust polling (raw HTTP).** If you bypass the SDK and roll your own poll loop, don't rely on `requests` or `httpx` timeouts as wall-clock caps — they're **per-chunk** read timeouts, reset every time a byte arrives. A trickling response (heartbeats, a wedged chunked-encoding body, a misbehaving proxy) can keep the call blocked indefinitely even with `timeout=(5, 60)` or `httpx.Timeout(120)`. Neither library has a "total wall-clock" timeout built in. For a hard deadline: track `time.monotonic()` at the loop level and break/cancel if a single request exceeds your budget (e.g. via a watchdog thread, or `asyncio.wait_for()` around async httpx). **Prefer the SDK** — `client.beta.sessions.events.stream()` and `client.beta.sessions.events.list()` handle timeout + retry sanely.
|
||||
>
|
||||
> If `GET /v1/sessions/{id}/events` (paginated) ever hangs after headers, you've likely hit `GET /v1/sessions/{id}/events` by mistake or a server-side stall — report it; don't treat it as a client-config problem.
|
||||
|
||||
### Event Types (Received)
|
||||
|
||||
Event types use dot notation, grouped by namespace:
|
||||
|
||||
| Event Type | Description |
|
||||
| --- | --- |
|
||||
| `agent.message` | Agent text output |
|
||||
| `agent.thinking` | Extended thinking blocks |
|
||||
| `agent.tool_use` | Agent used a built-in tool (`agent_toolset_20260401`) |
|
||||
| `agent.tool_result` | Result from a built-in tool |
|
||||
| `agent.mcp_tool_use` | Agent used an MCP tool |
|
||||
| `agent.mcp_tool_result` | Result from an MCP tool |
|
||||
| `agent.custom_tool_use` | Agent invoked a custom tool — session goes idle, you respond with `user.custom_tool_result` |
|
||||
| `agent.thread_context_compacted` | Conversation context was compacted |
|
||||
| `session.status_idle` | Agent has finished the current task, and is awaiting input. It's either waiting for input to continue working via a `user.message` or blocked awaiting a `user.custom_tool_result` or `user.tool_confirmation`. The `stop_reason` attached contains more information about why the Agent has stopped working. |
|
||||
| `session.status_running` | Session has starting running, and the Agent is actively doing work. |
|
||||
| `session.status_rescheduled` | Session is (re)scheduling after a retryable error has occurred, ready to be picked up by the orchestration system. |
|
||||
| `session.status_terminated` | Session has terminated, entering an irreversible and unusable state. |
|
||||
| `session.error` | Error occurred during processing |
|
||||
| `span.model_request_start` | Model inference started |
|
||||
| `span.model_request_end` | Model inference completed |
|
||||
| `span.outcome_evaluation_start` / `_ongoing` / `_end` | Grader progress for outcome-oriented sessions — see `shared/managed-agents-outcomes.md` |
|
||||
| `session.thread_created` | Subagent thread spawned (multiagent) — see `shared/managed-agents-multiagent.md` |
|
||||
| `session.thread_status_running` / `_idle` / `_rescheduled` / `_terminated` | Subagent thread status transitions (multiagent). `_idle` carries `stop_reason`. |
|
||||
| `agent.thread_message_sent` / `_received` | Cross-thread message, carries `to_session_thread_id` / `from_session_thread_id` (multiagent) |
|
||||
|
||||
The stream also echoes back user-sent events (`user.message`, `user.interrupt`, `user.tool_confirmation`, `user.custom_tool_result`, `user.define_outcome`).
|
||||
|
||||
---
|
||||
|
||||
## Steering Patterns
|
||||
|
||||
Practical patterns for driving a session via the events surface.
|
||||
|
||||
### Stream-first ordering
|
||||
|
||||
**Open the stream before sending events.** The stream only delivers events that occur *after* it's opened — it does not replay current state or historical events. If you send a message first and open the stream second, early events (including fast status transitions) arrive buffered in a single batch and you lose the ability to react to them in real time.
|
||||
|
||||
```ts
|
||||
// ✅ Correct — stream and send concurrently
|
||||
const [response] = await Promise.all([
|
||||
streamEvents(sessionId), // opens SSE connection
|
||||
sendMessage(sessionId, text),
|
||||
]);
|
||||
|
||||
// ❌ Wrong — events before stream opens arrive as a single buffered batch
|
||||
await sendMessage(sessionId, text);
|
||||
const response = await streamEvents(sessionId);
|
||||
```
|
||||
|
||||
**For full history,** use `GET /v1/sessions/{id}/events` (paginated list) — the stream only gives you live events from connection onward.
|
||||
|
||||
### Reconnecting after a dropped stream
|
||||
|
||||
**The SSE stream has no replay.** If your connection drops (httpx read timeout, network blip) and you reconnect, you only get events emitted *after* reconnection. Any events emitted during the gap are lost from the stream.
|
||||
|
||||
**The consolidation pattern:** on every (re)connect, overlap the stream with a history fetch and dedupe by event ID:
|
||||
|
||||
```python
|
||||
def connect_with_consolidation(client, session_id):
|
||||
# 1. Open the SSE stream first
|
||||
stream = client.beta.sessions.events.stream(session_id=session_id)
|
||||
|
||||
# 2. Fetch history to cover any gap
|
||||
history = client.beta.sessions.events.list(
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
# 3. Yield history first, then stream — dedupe by event.id
|
||||
seen = set()
|
||||
for ev in history.data:
|
||||
seen.add(ev.id)
|
||||
yield ev
|
||||
for ev in stream:
|
||||
if ev.id not in seen:
|
||||
seen.add(ev.id)
|
||||
yield ev
|
||||
```
|
||||
|
||||
### Message queuing
|
||||
|
||||
**You don't have to wait for a response before sending the next message.** User events are queued server-side and processed in order. This is useful for chat bridges where the user sends rapid follow-ups:
|
||||
|
||||
```ts
|
||||
// All three go into one session; agent processes them in order
|
||||
await sendMessage(sessionId, "Summarize the README");
|
||||
await sendMessage(sessionId, "Actually also check the CONTRIBUTING guide");
|
||||
await sendMessage(sessionId, "And compare the two");
|
||||
// Stream once — agent responds to all three as a coherent turn
|
||||
```
|
||||
|
||||
Events can be sent up to the Session at any time. There is no need to wait on a specific session status to enqueue new events via `client.beta.sessions.events.send()`
|
||||
|
||||
### Interrupt
|
||||
|
||||
An `interrupt` event **jumps the queue** (ahead of any pending user messages) and forces the session into `idle`. Use this for "stop" / "nevermind" / "cancel" commands:
|
||||
|
||||
```ts
|
||||
await client.beta.sessions.events.send(sessionId, {
|
||||
events: [{ type: 'interrupt' }],
|
||||
});
|
||||
```
|
||||
|
||||
The agent stops mid-task. It does not see the interrupt as a message — it just halts. Send a follow-up `user` event to explain what to do instead. If an outcome is active, the interrupt also marks `span.outcome_evaluation_end.result: "interrupted"` (see `shared/managed-agents-outcomes.md`).
|
||||
|
||||
> **Note**: Interrupt events may have empty IDs in the current implementation. When troubleshooting, use the `processed_at` timestamp along with surrounding event IDs.
|
||||
|
||||
### Event payloads
|
||||
|
||||
some events carry useful metadata beyond the status change itself:
|
||||
|
||||
`session.status_idle` — includes a `stop_reason` field which elaborates on why the session stopped and what type of further action is required by the user.
|
||||
```json
|
||||
{
|
||||
"id": "sevt_456",
|
||||
"processed_at": "2026-04-07T04:27:43.197Z",
|
||||
"stop_reason": {
|
||||
"event_ids": [
|
||||
"sevt_123"
|
||||
],
|
||||
"type": "requires_action"
|
||||
},
|
||||
"type": "status_idle"
|
||||
}
|
||||
```
|
||||
|
||||
`span.model_request_end` contains a `model_usage` field for cost tracking and efficiency analysis:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "span.model_request_end",
|
||||
"id": "sevt_456",
|
||||
"is_error": false,
|
||||
"model_request_start_id": "sevt_123",
|
||||
"model_usage": {
|
||||
"cache_creation_input_tokens": 0,
|
||||
"cache_read_input_tokens": 6656,
|
||||
"input_tokens": 3571,
|
||||
"output_tokens": 727
|
||||
},
|
||||
"processed_at": "2026-04-07T04:11:32.189Z"
|
||||
}
|
||||
```
|
||||
|
||||
**`agent.thread_context_compacted`** — emitted when the conversation history was summarized to fit context. Includes `pre_compaction_tokens` so you know how much was squeezed:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sevt_abc123",
|
||||
"processed_at": "2026-03-24T14:05:15.787Z",
|
||||
"type": "agent.thread_context_compacted"
|
||||
}
|
||||
```
|
||||
|
||||
### Archive
|
||||
|
||||
When done with a session, archive it to free resources:
|
||||
|
||||
```ts
|
||||
await client.beta.sessions.archive(sessionId);
|
||||
```
|
||||
|
||||
> Archiving a **session** is routine cleanup — sessions are per-run and disposable. **Do not generalize this to agents or environments**: those are persistent, reusable resources, and archiving them is permanent (no unarchive; new sessions cannot reference them). See `shared/managed-agents-overview.md` → Common Pitfalls.
|
||||
|
||||
|
||||
@@ -1,197 +0,0 @@
|
||||
# Managed Agents — Memory Stores
|
||||
|
||||
> **Public beta.** Memory stores ship under the `managed-agents-2026-04-01` beta header; the SDK sets it automatically on all `client.beta.memory_stores.*` calls. If `client.beta.memory_stores` is missing, upgrade to the latest SDK release.
|
||||
|
||||
Sessions are ephemeral by default — when one ends, anything the agent learned is gone. A **memory store** is a workspace-scoped collection of small text documents that persists across sessions. When a store is attached to a session (via `resources[]`), it is mounted into the container as a filesystem directory; the agent reads and writes it with the ordinary file tools, and a system-prompt note tells it the mount is there.
|
||||
|
||||
Every mutation to a memory produces an immutable **memory version** (`memver_...`), giving you an audit trail and point-in-time rollback/redact.
|
||||
|
||||
## Object model
|
||||
|
||||
| Object | ID prefix | Scope | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Memory store | `memstore_...` | Workspace | Attach to sessions via `resources[]` |
|
||||
| Memory | `mem_...` | Store | One text file, addressed by `path` (≤ 100KB each — prefer many small files) |
|
||||
| Memory version | `memver_...` | Memory | Immutable snapshot per mutation; `operation` ∈ `created` / `modified` / `deleted` |
|
||||
|
||||
## Create a store
|
||||
|
||||
`description` is passed to the agent so it knows what the store contains — write it for the model, not for humans.
|
||||
|
||||
```python
|
||||
store = client.beta.memory_stores.create(
|
||||
name="User Preferences",
|
||||
description="Per-user preferences and project context.",
|
||||
)
|
||||
print(store.id) # memstore_01Hx...
|
||||
```
|
||||
|
||||
Other SDKs: TypeScript `client.beta.memoryStores.create({...})`; Go `client.Beta.MemoryStores.New(ctx, ...)`. See `shared/managed-agents-api-reference.md` → SDK Method Reference for the full per-language table.
|
||||
|
||||
Stores support `retrieve` / `update` / `list` (with `include_archived`, `created_at_{gte,lte}` filters) / `delete` / **`archive`**. Archive makes the store read-only — existing session attachments continue, new sessions cannot reference it; no unarchive.
|
||||
|
||||
### Seed with content (optional)
|
||||
|
||||
Pre-load reference material before any session runs. `memories.create` creates a memory at the given `path`; if a memory already exists there the call returns `409` (`memory_path_conflict_error`, with the `conflicting_memory_id`). The store ID is the first positional argument.
|
||||
|
||||
```python
|
||||
client.beta.memory_stores.memories.create(
|
||||
store.id,
|
||||
path="/formatting_standards.md",
|
||||
content="All reports use GAAP formatting. Dates are ISO-8601...",
|
||||
)
|
||||
```
|
||||
|
||||
## Attach to a session
|
||||
|
||||
Memory stores go in the session's `resources[]` array alongside `file` and `github_repository` resources (see `shared/managed-agents-environments.md` → Resources). Memory stores attach at **session create time only** — `sessions.resources.add()` does not accept `memory_store`.
|
||||
|
||||
```python
|
||||
session = client.beta.sessions.create(
|
||||
agent=agent.id,
|
||||
environment_id=environment.id,
|
||||
resources=[
|
||||
{
|
||||
"type": "memory_store",
|
||||
"memory_store_id": store.id,
|
||||
"access": "read_write", # or "read_only"; default is "read_write"
|
||||
"instructions": "User preferences and project context. Check before starting any task.",
|
||||
}
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
| Field | Required | Notes |
|
||||
| --- | --- | --- |
|
||||
| `type` | ✅ | `"memory_store"` |
|
||||
| `memory_store_id` | ✅ | `memstore_...` |
|
||||
| `access` | — | `"read_write"` (default) or `"read_only"` — enforced at the filesystem level on the mount |
|
||||
| `instructions` | — | Session-specific guidance for this store, in addition to the store's `name`/`description`. ≤ 4,096 chars. |
|
||||
|
||||
**Max 8 memory stores per session.** Attach multiple when different slices of memory have different owners or lifecycles — e.g. one read-only shared-reference store plus one read-write per-user store, or one store per end-user/team/project sharing a single agent config.
|
||||
|
||||
### How the agent sees it (FUSE mount)
|
||||
|
||||
Each attached store is mounted in the session container at `/mnt/memory/<store-name>/`. The agent interacts with it using the standard file tools (`bash`, `read`, `write`, `edit`, `glob`, `grep`) — there are no dedicated memory tools. `access: "read_only"` makes the mount read-only at the filesystem level; `"read_write"` allows the agent to create, edit, and delete files under it. A short description of each mount (name, path, `instructions`, access) is automatically injected into the system prompt so the agent knows the store exists without you having to mention it.
|
||||
|
||||
Writes the agent makes under the mount are persisted back to the store and produce memory versions just like host-side `memories.update` calls.
|
||||
|
||||
## Manage memories directly (host-side)
|
||||
|
||||
Use these for review workflows, correcting bad memories, or seeding stores out-of-band.
|
||||
|
||||
### List
|
||||
|
||||
Returns `Memory | MemoryPrefix` entries — a `MemoryPrefix` (`type: "memory_prefix"`, just a `path`) is a directory-like node when listing hierarchically. Use `path_prefix` to scope (include a trailing slash: `"/notes/"` matches `/notes/a.md` but not `/notes_backup/old.md`) and `depth` to bound the tree walk. `order_by` / `order` sort the result. Pass `view="full"` to include `content` in each item; the default `"basic"` returns metadata only.
|
||||
|
||||
```python
|
||||
for m in client.beta.memory_stores.memories.list(store.id, path_prefix="/"):
|
||||
if m.type == "memory":
|
||||
print(f"{m.path} ({m.content_size_bytes} bytes, sha={m.content_sha256[:8]})")
|
||||
else: # "memory_prefix"
|
||||
print(f"{m.path}/")
|
||||
```
|
||||
|
||||
### Read
|
||||
|
||||
```python
|
||||
mem = client.beta.memory_stores.memories.retrieve(memory_id, memory_store_id=store.id)
|
||||
print(mem.content)
|
||||
```
|
||||
|
||||
`retrieve` defaults to `view="full"` (content included); `view` matters mainly on list endpoints.
|
||||
|
||||
### Create vs. update
|
||||
|
||||
| Operation | Addressed by | Semantics |
|
||||
| --- | --- | --- |
|
||||
| `memories.create(store_id, path=..., content=...)` | **Path** | Create at `path`. `409` (`memory_path_conflict_error`, includes `conflicting_memory_id`) if the path is already occupied. |
|
||||
| `memories.update(mem_id, memory_store_id=..., path=..., content=...)` | **`mem_...` ID** | Mutate existing memory. Change `content`, `path` (rename), or both. Renaming onto an occupied path returns the same `409 memory_path_conflict_error`. |
|
||||
|
||||
```python
|
||||
mem = client.beta.memory_stores.memories.create(
|
||||
store.id,
|
||||
path="/preferences/formatting.md",
|
||||
content="Always use tabs, not spaces.",
|
||||
)
|
||||
|
||||
client.beta.memory_stores.memories.update(
|
||||
mem.id,
|
||||
memory_store_id=store.id,
|
||||
path="/archive/2026_q1_formatting.md", # rename
|
||||
)
|
||||
```
|
||||
|
||||
### Optimistic concurrency (precondition on `update`)
|
||||
|
||||
`memories.update` accepts a `precondition` so you can read → modify → write back without clobbering a concurrent writer. The only supported type is `content_sha256`. On mismatch the API returns `409` (`memory_precondition_failed_error`) — re-read and retry against fresh state.
|
||||
|
||||
```python
|
||||
client.beta.memory_stores.memories.update(
|
||||
mem.id,
|
||||
memory_store_id=store.id,
|
||||
content="CORRECTED: Always use 2-space indentation.",
|
||||
precondition={"type": "content_sha256", "content_sha256": mem.content_sha256},
|
||||
)
|
||||
```
|
||||
|
||||
### Delete
|
||||
|
||||
```python
|
||||
client.beta.memory_stores.memories.delete(mem.id, memory_store_id=store.id)
|
||||
```
|
||||
|
||||
Pass `expected_content_sha256` for a conditional delete.
|
||||
|
||||
## Audit and rollback — memory versions
|
||||
|
||||
Every mutation creates an immutable `memver_...` snapshot. Versions accumulate for the lifetime of the parent memory; `memories.retrieve` always returns the current head, the version endpoints give you history.
|
||||
|
||||
| Operation that triggers it | `operation` field on the version |
|
||||
| --- | --- |
|
||||
| `memories.create` at a new path | `"created"` |
|
||||
| `memories.update` changing `content`, `path`, or both (or an agent-side write to the mount) | `"modified"` |
|
||||
| `memories.delete` | `"deleted"` |
|
||||
|
||||
Each version also records `created_by` — an actor object with `type` ∈ `session_actor` / `api_actor` / `user_actor` — and, after redaction, `redacted_at` + `redacted_by`.
|
||||
|
||||
### List versions
|
||||
|
||||
Newest-first, paginated. Filter by `memory_id`, `operation`, `session_id`, `api_key_id`, or `created_at_gte` / `created_at_lte`. Pass `view="full"` to include `content`; default is metadata-only.
|
||||
|
||||
```python
|
||||
for v in client.beta.memory_stores.memory_versions.list(store.id, memory_id=mem.id):
|
||||
print(f"{v.id}: {v.operation}")
|
||||
```
|
||||
|
||||
### Retrieve a version
|
||||
|
||||
```python
|
||||
version = client.beta.memory_stores.memory_versions.retrieve(
|
||||
version_id, memory_store_id=store.id
|
||||
)
|
||||
print(version.content)
|
||||
```
|
||||
|
||||
### Redact a version
|
||||
|
||||
Scrubs content from a historical version while preserving the audit trail (actor + timestamps). Clears `content`, `content_sha256`, `content_size_bytes`, and `path`; everything else stays. Use for leaked secrets, PII, or user-deletion requests.
|
||||
|
||||
```python
|
||||
client.beta.memory_stores.memory_versions.redact(version_id, memory_store_id=store.id)
|
||||
```
|
||||
|
||||
## Endpoint reference
|
||||
|
||||
See `shared/managed-agents-api-reference.md` → Memory Stores / Memories / Memory Versions for the full HTTP method/path tables. Raw HTTP base path:
|
||||
|
||||
```
|
||||
POST /v1/memory_stores
|
||||
POST /v1/memory_stores/{memory_store_id}/archive
|
||||
GET /v1/memory_stores/{memory_store_id}/memories
|
||||
PATCH /v1/memory_stores/{memory_store_id}/memories/{memory_id}
|
||||
GET /v1/memory_stores/{memory_store_id}/memory_versions
|
||||
POST /v1/memory_stores/{memory_store_id}/memory_versions/{version_id}/redact
|
||||
```
|
||||
|
||||
For cURL examples and the CLI (`ant beta:memory-stores ...`), WebFetch the Memory URL in `shared/live-sources.md` → Managed Agents.
|
||||
@@ -1,99 +0,0 @@
|
||||
# Managed Agents — Multiagent Sessions
|
||||
|
||||
A coordinator agent can delegate to other agents within one session. All agents **share the container and filesystem**; each runs in its own **thread** — a context-isolated event stream with its own conversation history, model, system prompt, tools, MCP servers, and skills (from that agent's own config). Threads are persistent: the coordinator can send a follow-up to a subagent it called earlier and that subagent retains its prior turns.
|
||||
|
||||
The SDK sets the `managed-agents-2026-04-01` beta header automatically on all `client.beta.{agents,sessions}.*` calls; no additional header is required for multiagent.
|
||||
|
||||
---
|
||||
|
||||
## Declare the roster on the coordinator
|
||||
|
||||
`multiagent` is a **top-level field** on `agents.create()` / `agents.update()` — **not** a `tools[]` entry. `agents` lists 1–20 roster entries. Nothing changes on `sessions.create()` — the roster is resolved from the coordinator's config.
|
||||
|
||||
```python
|
||||
orchestrator = client.beta.agents.create(
|
||||
name="Engineering Lead",
|
||||
model="claude-opus-4-8",
|
||||
system="You coordinate engineering work. Delegate code review to the reviewer and test writing to the test agent.",
|
||||
tools=[{"type": "agent_toolset_20260401"}],
|
||||
multiagent={
|
||||
"type": "coordinator",
|
||||
"agents": [
|
||||
reviewer.id, # bare string — latest version
|
||||
{"type": "agent", "id": test_writer.id, "version": 4}, # pinned version
|
||||
{"type": "self"}, # the coordinator itself
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
session = client.beta.sessions.create(agent=orchestrator.id, environment_id=env.id)
|
||||
```
|
||||
|
||||
| Roster entry | Shape | Notes |
|
||||
|---|---|---|
|
||||
| String shorthand | `"agent_abc123"` | References the latest version of a stored agent. |
|
||||
| Agent reference | `{type: "agent", id, version?}` | Omit `version` to pin the latest at coordinator save time. |
|
||||
| Self | `{type: "self"}` | The coordinator can spawn copies of itself. |
|
||||
|
||||
Up to **20 unique agents** in the roster; the coordinator may spawn **multiple copies** of each. **One level of delegation only** — depth > 1 is ignored.
|
||||
|
||||
---
|
||||
|
||||
## Threads
|
||||
|
||||
The session-level event stream is the **primary thread** — it shows the coordinator's trace plus a condensed view of subagent activity (thread status transitions and cross-thread messages, not every subagent tool call). Drill into a specific subagent via the per-thread endpoints:
|
||||
|
||||
| Operation | HTTP | SDK (`client.beta.sessions.threads.*`) |
|
||||
|---|---|---|
|
||||
| List threads | `GET /v1/sessions/{sid}/threads` | `.list(session_id)` |
|
||||
| Retrieve one | `GET /v1/sessions/{sid}/threads/{tid}` | `.retrieve(thread_id, session_id=...)` |
|
||||
| Archive | `POST /v1/sessions/{sid}/threads/{tid}/archive` | `.archive(thread_id, session_id=...)` |
|
||||
| List thread events | `GET /v1/sessions/{sid}/threads/{tid}/events` | `.events.list(thread_id, session_id=...)` |
|
||||
| Stream thread events | `GET /v1/sessions/{sid}/threads/{tid}/stream` | `.events.stream(thread_id, session_id=...)` |
|
||||
|
||||
Each `SessionThread` carries `id`, `status` (`running` | `idle` | `rescheduling` | `terminated`), `agent` (a resolved snapshot of the agent config — `id`, `name`, `model`, `system`, `tools`, `skills`, `mcp_servers`, `version`), `parent_thread_id` (null for the primary thread, which is included in the list), `archived_at`, and optional `stats`/`usage`. **Session status aggregates thread statuses** — if any thread is `running`, `session.status` is `running`. Max **25 concurrent threads**. When draining a per-thread stream, break on `session.thread_status_idle` (and check its `stop_reason` as you would for the session-level idle).
|
||||
|
||||
---
|
||||
|
||||
## Multiagent events (on the session stream)
|
||||
|
||||
| Event | Payload highlights | Meaning |
|
||||
|---|---|---|
|
||||
| `session.thread_created` | `session_thread_id`, `agent_name` | A new thread was created. |
|
||||
| `session.thread_status_running` | `session_thread_id`, `agent_name` | Thread started activity. |
|
||||
| `session.thread_status_idle` | `session_thread_id`, `agent_name`, **`stop_reason`** | Thread is awaiting input. Inspect `stop_reason` (same shape as `session.status_idle.stop_reason`). |
|
||||
| `session.thread_status_rescheduled` | `session_thread_id`, `agent_name` | Thread is rescheduling after a retryable error. |
|
||||
| `session.thread_status_terminated` | `session_thread_id`, `agent_name` | Thread was archived or hit a terminal error. |
|
||||
| `agent.thread_message_sent` | `to_session_thread_id`, `to_agent_name`, `content` | Coordinator sent a follow-up to another thread. |
|
||||
| `agent.thread_message_received` | `from_session_thread_id`, `from_agent_name`, `content` | An agent delivered its result to the coordinator. |
|
||||
|
||||
---
|
||||
|
||||
## Tool permissions and custom tools from subagent threads
|
||||
|
||||
When a subagent needs your client (an `always_ask` confirmation, or a custom tool result), the request is **cross-posted to the primary thread** with `session_thread_id` identifying the originating thread — so you only need to watch the session stream. Reply with `user.tool_confirmation` (carrying `tool_use_id`) or `user.custom_tool_result` (carrying `custom_tool_use_id`), and **echo the `session_thread_id` from the originating event** (the SDK param type and docstring expect it). The server also routes by the tool-use ID, so the echo is belt-and-suspenders rather than load-bearing — but include it.
|
||||
|
||||
```python
|
||||
for event_id in stop.event_ids:
|
||||
pending = events_by_id[event_id]
|
||||
confirmation = {
|
||||
"type": "user.tool_confirmation",
|
||||
"tool_use_id": event_id,
|
||||
"result": "allow",
|
||||
}
|
||||
if pending.session_thread_id is not None:
|
||||
confirmation["session_thread_id"] = pending.session_thread_id
|
||||
client.beta.sessions.events.send(session.id, events=[confirmation])
|
||||
```
|
||||
|
||||
The same pattern applies to `user.custom_tool_result`.
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Don't put the roster on `sessions.create()` or in `tools[]`.** `multiagent` is a top-level agent field; update the coordinator, then start a session that references it.
|
||||
- **Don't assume shared context.** Threads share the filesystem but not conversation history or tools. If the coordinator needs a subagent to act on something, it must say so in the delegated message (or write it to disk).
|
||||
- **Depth > 1 is ignored.** A subagent's own `multiagent` roster (if any) doesn't cascade — only the session's coordinator delegates.
|
||||
|
||||
For per-language bindings beyond Python, WebFetch `https://platform.claude.com/docs/en/managed-agents/multi-agent.md` (see `shared/live-sources.md`).
|
||||
@@ -1,144 +0,0 @@
|
||||
# Managed Agents — Onboarding Flow
|
||||
|
||||
> **Invoked via `/claude-api managed-agents-onboard`?** You're in the right place. Run the interview below — don't summarize it back to the user, ask the questions.
|
||||
|
||||
Use this when a user wants to set up a Managed Agent from scratch: **branch on know-vs-explore → configure the template → set up the session → pre-flight viability check → emit working code.** The pre-flight check (§3) is not optional — a setup missing a tool, credential, or data access it needs will fail mid-run, and the gap is usually visible at setup time.
|
||||
|
||||
> Read `shared/managed-agents-core.md` alongside this — it has full detail for each knob. This doc is the interview script, not the reference.
|
||||
|
||||
---
|
||||
|
||||
Claude Managed Agents is a hosted agent: Anthropic runs the agent loop on its orchestration layer and provisions a sandboxed container per session where the agent's tools execute (or, with a `self_hosted` environment, your own worker runs the tools — see `shared/managed-agents-self-hosted-sandboxes.md`). You supply the agent config and the environment config; the harness — event stream, sandbox orchestration, prompt caching, context compaction, and extended thinking — is handled for you.
|
||||
|
||||
**What you supply:**
|
||||
- **An agent config** — tools, skills, model, system prompt. Reusable and versioned.
|
||||
- **An environment config** — the sandbox your agent's tools execute in (`cloud`: networking, packages; or `self_hosted`: your own infra). Reusable across agents.
|
||||
|
||||
Each run of the agent is a **session**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Know or explore?
|
||||
|
||||
Ask the user:
|
||||
|
||||
> Do you already know the agent you want to build, or would you like to explore some common patterns first?
|
||||
|
||||
### Explore path — show the patterns
|
||||
|
||||
Four shapes, same runtime code path (`sessions.create()` → `sessions.events.send()` → stream). Only the trigger and sink differ.
|
||||
|
||||
| Pattern | Trigger | Example |
|
||||
|---|---|---|
|
||||
| Event-triggered | Webhook | GitHub PR push → CMA (GitHub tool) → Slack |
|
||||
| Scheduled | Cron | Daily brief: browser + GitHub + Jira → CMA → Slack |
|
||||
| Fire-and-forget PR | Human | Slack slash-command → CMA (GitHub tool) → PR passing CI |
|
||||
| Research + dashboard | Human | Topic → CMA (web search + `frontend-design` skill) → HTML dashboard |
|
||||
|
||||
Ask which shape fits, then continue with the Know path using it as the reference.
|
||||
|
||||
### Know path — configure template
|
||||
|
||||
Three rounds. Batch the questions in each round; don't ask them one at a time.
|
||||
|
||||
**Round A — Tools.** Start here; it's the most concrete part. Three types; ask which the user wants (any combination):
|
||||
|
||||
| Type | What it is | How to guide |
|
||||
|---|---|---|
|
||||
| **Prebuilt Claude Agent tools** (`agent_toolset_20260401`) | Ready-to-use: `bash`, `read`, `write`, `edit`, `glob`, `grep`, `web_fetch`, `web_search`. Enable all at once, or individually via `enabled: true/false`. | Recommend enabling the full toolset. List the 8 tools so the user knows what they're getting. Full detail: `shared/managed-agents-tools.md` → Agent Toolset. |
|
||||
| **MCP tools** | Third-party integrations (GitHub, Linear, Asana, etc.) via `mcp_toolset`. Credentials live in a vault, not inline. | Ask which services. For each, walk through MCP server URL + vault credentials. Full detail: `shared/managed-agents-tools.md` → MCP Servers + Vaults. |
|
||||
| **Custom tools** | The user's own app handles these tool calls — agent fires `agent.custom_tool_use`, the app sends a result message back. | Ask for each tool: name, description, input schema. The app code that handles the event is *their* code — don't generate it. Full detail: `shared/managed-agents-tools.md` → Custom Tools. |
|
||||
|
||||
**Round B — Skills, files, and repos.** What the agent has on hand when it starts.
|
||||
|
||||
*Skills* — two types; both work the same way — Claude auto-uses them when relevant. Max 20 per agent.
|
||||
- [ ] **Pre-built Agent Skills**: `xlsx`, `docx`, `pptx`, `pdf`. Reference by name.
|
||||
- [ ] **Custom Skills**: skills uploaded to the user's org via the Skills API. Reference by `skill_id` + optional `version`. If the skill doesn't exist yet, walk the user through `POST /v1/skills` + `POST /v1/skills/{id}/versions` (beta header `skills-2025-10-02`). Full detail: `shared/managed-agents-tools.md` → Skills + Skills API.
|
||||
|
||||
*GitHub repositories* — any repos the agent needs on-disk? For each:
|
||||
- [ ] Repo URL (`https://github.com/org/repo`)
|
||||
- [ ] `authorization_token` (PAT or GitHub App token scoped to the repo)
|
||||
- [ ] Optional `mount_path` (defaults to `/workspace/<repo-name>`) and `checkout` (branch or SHA)
|
||||
|
||||
Emit as `resources: [{type: "github_repository", url, authorization_token, ...}]`. Full detail: `shared/managed-agents-environments.md` → GitHub Repositories.
|
||||
|
||||
> ‼️ **PR creation needs the GitHub MCP server too.** `github_repository` gives filesystem access only — to open PRs, also attach the GitHub MCP server in Round A and credential it via a vault. The workflow is: edit files in the mounted repo → push branch via `bash` → create PR via the MCP `create_pull_request` tool.
|
||||
|
||||
*Files* — any local files to seed the session with? For each:
|
||||
- [ ] Upload via the Files API → persist `file_id`
|
||||
- [ ] Choose a `mount_path` — absolute, e.g. `/workspace/data.csv` (parents auto-created; files mount read-only)
|
||||
|
||||
Emit as `resources: [{type: "file", file_id, mount_path}]`. Max 999 file resources. Agent working directory defaults to `/workspace`. Full detail: `shared/managed-agents-environments.md` → Files API.
|
||||
|
||||
**Round C — Identity, success criteria, environment:**
|
||||
- [ ] Name?
|
||||
- [ ] Job (one or two sentences — becomes the system prompt)?
|
||||
- [ ] **What does "done" look like?** Push for concrete, checkable success criteria — not "a good report" but "a CSV with a numeric `price` column per SKU." Explicit criteria give the agent a clear target and let you verify the result; vague ones leave it guessing what "done" means. If they're gradeable, plan to wire an **Outcome** in §2 so the harness grades-and-revises against them. See `shared/managed-agents-outcomes.md`.
|
||||
- [ ] Networking: unrestricted internet from the container, or lock egress to specific hosts? (If locked, MCP server domains must be in `allowed_hosts` or tools silently fail.)
|
||||
- [ ] Model? (default `claude-opus-4-8`)
|
||||
|
||||
---
|
||||
|
||||
## 2. Set up the session
|
||||
|
||||
Per-run. Points at the agent + environment, attaches credentials, kicks off.
|
||||
|
||||
**Vault credentials** (if the agent declared MCP servers, or the job needs an API key for a CLI/SDK/direct API call):
|
||||
- [ ] Existing vault, or create one? (`client.beta.vaults.create()` + `vaults.credentials.create()`)
|
||||
|
||||
Credentials are write-only. MCP credentials are matched to MCP servers by URL and auto-refreshed; `environment_variable` credentials are substituted into outbound requests at egress (the sandbox sees only a placeholder). See `shared/managed-agents-tools.md` → Vaults.
|
||||
|
||||
**Kickoff — pick one:**
|
||||
- [ ] **Conversational:** a first `user.message` to the agent.
|
||||
- [ ] **Outcome-graded** (recommended when §Round C produced checkable criteria): send a `user.define_outcome` with a rubric *instead of* a `user.message` — the harness iterates and grades against the rubric until satisfied. Don't send both. See `shared/managed-agents-outcomes.md`.
|
||||
|
||||
Session creation blocks until all resources mount. Open the event stream before sending the kickoff. Stream is SSE; break on `session.status_terminated`, or on `session.status_idle` with a terminal `stop_reason` — i.e. anything except `requires_action`, which fires transiently while the session waits on a tool confirmation or custom-tool result (see `shared/managed-agents-client-patterns.md` Pattern 5). Usage lands on `span.model_request_end`. Agent-written artifacts end up in `/mnt/session/outputs/` — download via `files.list({scope_id: session.id, betas: ["managed-agents-2026-04-01"]})`.
|
||||
|
||||
**Console escape hatch.** In the runtime block you emit, print the session's Console URL right after `sessions.create()` so the user can watch it in the UI while iterating: `print(f"Watch in Console: https://platform.claude.com/workspaces/default/sessions/{session.id}")` (swap `default` for the user's workspace slug if they named one).
|
||||
|
||||
---
|
||||
|
||||
## 3. Pre-flight viability check — reconcile the job against the resources
|
||||
|
||||
**Do this before emitting any code.** A common, avoidable failure is an under-resourced run: the ask is clear, but the agent is missing a tool, a credential, data access, or the context to act. The agent discovers the gap a few turns in, flails, and gives up — burning the budget to produce nothing. The gap is usually visible at setup time. Catch it here, not after the session fails.
|
||||
|
||||
Walk the stated job clause by clause. For each action the agent must take, confirm a resource covers it — and name the gap out loud if one doesn't:
|
||||
|
||||
| Gap class | Check | If missing |
|
||||
|---|---|---|
|
||||
| **Tool / integration** (most catchable upfront — config is statically inspectable) | Every verb in the job maps to an enabled tool or MCP server. "Triage tickets" → a ticketing MCP server; "open a PR" → GitHub MCP server (a `github_repository` mount alone can't open PRs); "search the web" → `web_search` enabled in the toolset. | Add the tool/MCP server in §Round A, or cut the ask from the job. |
|
||||
| **Credential / access** | Every MCP server has a vault credential attached (§2). Every external host the job touches is reachable — networking `unrestricted`, or the host is in `allowed_hosts`. | Create/attach the vault; widen `allowed_hosts`. These don't fail until runtime — the smoke-test in §4 is how you surface them cheaply. |
|
||||
| **Data** | Every file, dataset, or repo the job references is mounted as a `resource` (file, `github_repository`, or memory store). | Upload + mount it in §Round B, or tell the agent where to fetch it from. |
|
||||
| **Prompt quality / criteria** | The job is specific enough to act on, and "done" is checkable (§Round C). | Tighten the job; wire an Outcome. |
|
||||
|
||||
State any unmet gaps to the user and resolve them before generating code. Don't emit a config you already know is under-resourced — an agent can't complete a task it lacks the tools, credentials, or data for.
|
||||
|
||||
---
|
||||
|
||||
## 4. Emit the code
|
||||
|
||||
Go straight from the last interview answer to the code — no preamble about the setup-vs-runtime split, no "the critical thing to internalize…", no lecture about `agents.create()` being one-time. The two-block structure below already shows that; don't narrate it. Generate **two clearly-separated blocks**:
|
||||
|
||||
**Block 1 — Setup (run once, store the IDs).** Prefer emitting this as **YAML files + `ant` CLI commands** — agents and environments are version-controlled definitions, and the CLI flow is what users should check into their repo and run from CI. Fall back to SDK code only if the user explicitly wants setup in-language or the `ant` CLI is unavailable.
|
||||
|
||||
Emit:
|
||||
1. `<name>.agent.yaml` with everything from §Round A–C (flat: `name`, `model`, `system`, `tools`, `mcp_servers`, `skills`)
|
||||
2. `<name>.environment.yaml` with §Round C networking
|
||||
3. The apply commands:
|
||||
```sh
|
||||
AGENT_ID=$(ant beta:agents create < <name>.agent.yaml --transform id -r)
|
||||
ENV_ID=$(ant beta:environments create < <name>.environment.yaml --transform id -r)
|
||||
# CI sync: ant beta:agents update --agent-id "$AGENT_ID" --version N < <name>.agent.yaml
|
||||
```
|
||||
|
||||
See `shared/anthropic-cli.md` for the full CLI reference. If emitting SDK code instead, label it `# ONE-TIME SETUP — run once, save the IDs to config/.env` and call `environments.create()` → `agents.create()`.
|
||||
|
||||
**Block 2 — Runtime (run on every invocation).** This is SDK code in the detected language (Python/TS/cURL — see SKILL.md → Language Detection). The runtime path needs to react programmatically to events (tool confirmations, custom tool results, reconnect), which is SDK territory — don't emit shell loops here.
|
||||
1. Load `env_id` + `agent_id` from config/env
|
||||
2. `sessions.create(agent=AGENT_ID, environment_id=ENV_ID, resources=[...], vault_ids=[...])` — this blocks until resources mount, so a bad file/repo mount surfaces *here*, before any tokens are spent.
|
||||
3. **Smoke-test first when the job depends on MCP servers, credentials, or reachable hosts.** Credential and MCP-connectivity failures don't surface at `sessions.create()` — only when the agent first tries to use them. Send one cheap probe turn ("Confirm you can reach <service> and list 1–2 items; don't start the task yet"), check it succeeded, *then* send the real kickoff. A few hundred tokens here beats a runaway session that flails on a missing credential and gives up. Skip for agents with no external dependencies.
|
||||
4. Open stream, `events.send()` the kickoff (a `user.message`, or a `user.define_outcome` if §2 chose the outcome-graded path), loop until `session.status_terminated` or `session.status_idle && stop_reason.type !== 'requires_action'` (see `shared/managed-agents-client-patterns.md` Pattern 5 for the full gate — do not break on bare `session.status_idle`)
|
||||
|
||||
> ⚠️ **Never emit `agents.create()` and `sessions.create()` in the same unguarded block.** That teaches the user to create a new agent on every run — the #1 anti-pattern. If they need a single script, wrap agent creation in `if not os.getenv("AGENT_ID"):`.
|
||||
|
||||
Pull exact syntax from `python/managed-agents/README.md`, `typescript/managed-agents/README.md`, or `curl/managed-agents.md`. Don't invent field names.
|
||||
@@ -1,106 +0,0 @@
|
||||
# Managed Agents — Outcomes
|
||||
|
||||
An **outcome** elevates a session from *conversation* to *work*: you state what "done" looks like, and the harness runs an iterate → grade → revise loop until the artifact meets the rubric, hits `max_iterations`, or is interrupted. A separate **grader** (independent context window) scores each iteration against your rubric and feeds per-criterion gaps back to the agent.
|
||||
|
||||
The SDK sets the `managed-agents-2026-04-01` beta header automatically on all `client.beta.sessions.*` calls; no additional header is required for outcomes.
|
||||
|
||||
---
|
||||
|
||||
## The `user.define_outcome` event
|
||||
|
||||
Outcomes are not a field on `sessions.create()`. You create a normal session, then send a `user.define_outcome` event. The agent starts working on receipt — **do not also send a `user.message`** to kick it off.
|
||||
|
||||
```python
|
||||
session = client.beta.sessions.create(
|
||||
agent=AGENT_ID,
|
||||
environment_id=ENVIRONMENT_ID,
|
||||
title="Financial analysis on Costco",
|
||||
)
|
||||
|
||||
client.beta.sessions.events.send(
|
||||
session_id=session.id,
|
||||
events=[
|
||||
{
|
||||
"type": "user.define_outcome",
|
||||
"description": "Build a DCF model for Costco in .xlsx",
|
||||
"rubric": {"type": "text", "content": RUBRIC_MD},
|
||||
# or: "rubric": {"type": "file", "file_id": rubric.id}
|
||||
"max_iterations": 5, # optional; default 3, max 20
|
||||
}
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `type` | `"user.define_outcome"` | |
|
||||
| `description` | string | The task. This is what the agent works toward — no separate `user.message` needed. |
|
||||
| `rubric` | `{type: "text", content}` \| `{type: "file", file_id}` | **Required.** Markdown with explicit, independently gradeable criteria. Upload once via `client.beta.files.upload(...)` (beta `files-api-2025-04-14`) to reuse across sessions. |
|
||||
| `max_iterations` | int | Optional. Default **3**, max **20**. |
|
||||
|
||||
The event is echoed back on the stream with a server-assigned `outcome_id` and `processed_at`.
|
||||
|
||||
> **Writing rubrics.** Use explicit, gradeable criteria ("CSV has a numeric `price` column"), not vibes ("data looks good") — the grader scores each criterion independently, so vague criteria produce noisy loops. If you don't have a rubric, have Claude analyze a known-good artifact and turn that analysis into one.
|
||||
|
||||
---
|
||||
|
||||
## Outcome-specific events
|
||||
|
||||
These appear on the standard event stream (`sessions.events.stream` / `.list`) alongside the usual `agent.*` / `session.*` events.
|
||||
|
||||
| Event | Payload highlights | Meaning |
|
||||
|---|---|---|
|
||||
| `span.outcome_evaluation_start` | `outcome_id`, `iteration` (0-indexed) | Grader began scoring iteration *N*. |
|
||||
| `span.outcome_evaluation_ongoing` | `outcome_id` | Heartbeat while the grader runs. Grader reasoning is opaque — you see *that* it's working, not *what* it's thinking. |
|
||||
| `span.outcome_evaluation_end` | `outcome_evaluation_start_id`, `outcome_id`, `iteration`, `result`, `explanation`, `usage` | Grader finished one iteration. `result` drives what happens next (table below). |
|
||||
|
||||
### `span.outcome_evaluation_end.result`
|
||||
|
||||
| `result` | Next |
|
||||
|---|---|
|
||||
| `satisfied` | Session → `idle`. Terminal for this outcome. |
|
||||
| `needs_revision` | Agent starts another iteration. |
|
||||
| `max_iterations_reached` | No further grader cycles. Agent may run one final revision, then session → `idle`. |
|
||||
| `failed` | Session → `idle`. Rubric fundamentally doesn't match the task (e.g. description and rubric contradict). |
|
||||
| `interrupted` | Only emitted if `_start` had already fired before a `user.interrupt` arrived. |
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "span.outcome_evaluation_end",
|
||||
"id": "sevt_01jkl...",
|
||||
"outcome_evaluation_start_id": "sevt_01def...",
|
||||
"outcome_id": "outc_01a...",
|
||||
"result": "satisfied",
|
||||
"explanation": "All 12 criteria met: revenue projections use 5 years of historical data, ...",
|
||||
"iteration": 0,
|
||||
"usage": { "input_tokens": 2400, "output_tokens": 350, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 1800 },
|
||||
"processed_at": "2026-03-25T14:03:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checking status & retrieving deliverables
|
||||
|
||||
**Status** — either watch the stream for `span.outcome_evaluation_end`, or poll the session and read `outcome_evaluations`:
|
||||
|
||||
```python
|
||||
session = client.beta.sessions.retrieve(session.id)
|
||||
for ev in session.outcome_evaluations:
|
||||
print(f"{ev.outcome_id}: {ev.result}") # outc_01a...: satisfied
|
||||
```
|
||||
|
||||
**Deliverables** — the agent writes to `/mnt/session/outputs/`. Once idle, fetch via the Files API with `scope_id=session.id`. This is the same session-outputs mechanism documented in `shared/managed-agents-environments.md` → Session outputs (including the dual-beta-header requirement on `files.list`).
|
||||
|
||||
---
|
||||
|
||||
## Interaction rules & pitfalls
|
||||
|
||||
- **One outcome at a time.** Chain by sending the next `user.define_outcome` only after the previous one's terminal `span.outcome_evaluation_end` (`satisfied` / `max_iterations_reached` / `failed` / `interrupted`). The session retains history across chained outcomes.
|
||||
- **Steering is allowed but optional.** You *may* send `user.message` events mid-outcome to nudge direction, but the agent already knows to keep working until terminal — don't send "keep going" prompts.
|
||||
- **`user.interrupt` pauses the current outcome** — it marks `result: "interrupted"` and leaves the session `idle`, ready for a new outcome or conversational turn.
|
||||
- **After terminal, the session is reusable** — continue conversationally or define a new outcome.
|
||||
- **Outcome ≠ session-create field.** Don't put `outcome`, `rubric`, or `description` on `sessions.create()` — outcomes are always sent as a `user.define_outcome` event.
|
||||
- **Idle-break gate is unchanged.** In your drain loop, keep using `event.type === 'session.status_idle' && event.stop_reason?.type !== 'requires_action'` — do **not** gate on `span.outcome_evaluation_end` alone (on `needs_revision` the session keeps running). See `shared/managed-agents-client-patterns.md` Pattern 5.
|
||||
|
||||
For the raw HTTP shapes and per-language SDK bindings beyond Python, WebFetch `https://platform.claude.com/docs/en/managed-agents/define-outcomes.md` (see `shared/live-sources.md`).
|
||||
@@ -1,71 +0,0 @@
|
||||
# Managed Agents — Overview
|
||||
|
||||
Managed Agents provisions a container per session as the agent's workspace. The agent loop runs on Anthropic's orchestration layer; the container is where the agent's *tools* execute — bash commands, file operations, code. You create a persisted **Agent** config (model, system prompt, tools, MCP servers, skills), then start **Sessions** that reference it. The session streams events back to you; you send user messages and tool results in.
|
||||
|
||||
## ⚠️ THE MANDATORY FLOW: Agent (once) → Session (every run)
|
||||
|
||||
**Why agents are separate objects: versioning.** An agent is a persisted, versioned config — every update creates a new immutable version, and sessions pin to a version at creation time. This lets you iterate on the agent (tweak the prompt, add a tool) without breaking sessions already running, roll back if a change regresses, and A/B test versions side-by-side. None of that works if you `agents.create()` fresh on every run.
|
||||
|
||||
Every session references a pre-created `/v1/agents` object. Create the agent once, store the ID, and reuse it across runs.
|
||||
|
||||
| Step | Call | Frequency |
|
||||
|---|---|---|
|
||||
| 1 | `POST /v1/agents` — `model`, `system`, `tools`, `mcp_servers`, `skills` live here | **ONCE.** Store `agent.id` **and** `agent.version`. |
|
||||
| 2 | `POST /v1/sessions` — `agent: "agent_abc123"` or `{type: "agent", id, version}` | **Every run.** String shorthand uses latest version. |
|
||||
|
||||
If you're about to write `sessions.create()` with `model`, `system`, or `tools` on the session body — **stop**. Those fields live on `agents.create()`. The session takes a *pointer* only.
|
||||
|
||||
**When generating code, separate setup from runtime.** `agents.create()` belongs in a setup script (or a guarded `if agent_id is None:` block), not at the top of the hot path. If the user's code calls `agents.create()` on every invocation, they're accumulating orphaned agents and paying the create latency for nothing. The correct shape is: create once → persist the ID (config file, env var, secrets manager) → every run loads the ID and calls `sessions.create()`.
|
||||
|
||||
**To change the agent's behavior, use `POST /v1/agents/{id}` — don't create a new one.** Each update bumps the version; running sessions keep their pinned version, new sessions get the latest (or pin explicitly via `{type: "agent", id, version}`). See `shared/managed-agents-core.md` → Agents → Versioning. To change `tools`/`mcp_servers`/`vault_ids` on **one running session** without touching the agent object, use `sessions.update()` — see `shared/managed-agents-core.md` → Updating the agent configuration mid-session.
|
||||
|
||||
## Beta Headers
|
||||
|
||||
Managed Agents is in beta. The SDK sets required beta headers automatically:
|
||||
|
||||
| Beta Header | What it enables |
|
||||
| ------------------------------ | ---------------------------------------------------- |
|
||||
| `managed-agents-2026-04-01` | Agents, Environments, Sessions, Events, Session Resources, Session Threads, Outcomes, Multiagent, Vaults, Credentials, Memory Stores, Deployments |
|
||||
| `skills-2025-10-02` | Skills API (for managing custom skill definitions) |
|
||||
| `files-api-2025-04-14` | Files API for file uploads |
|
||||
|
||||
**Which beta header goes where:** The SDK sets `managed-agents-2026-04-01` automatically on `client.beta.{agents,environments,sessions,vaults,memory_stores,deployments,deployment_runs}.*` calls, and `files-api-2025-04-14` / `skills-2025-10-02` automatically on `client.beta.files.*` / `client.beta.skills.*` calls. You do NOT need to add the Skills or Files beta header when calling Managed Agents endpoints. **Exception — session-scoped file listing:** `client.beta.files.list({scope_id: session.id})` is a Files endpoint that takes a Managed Agents parameter, so it needs **both** headers. Pass `betas: ["managed-agents-2026-04-01"]` explicitly on that call (the SDK adds the Files header; you add the Managed Agents one). See `shared/managed-agents-environments.md` → Session outputs.
|
||||
|
||||
|
||||
## Reading Guide
|
||||
|
||||
| User wants to... | Read these files |
|
||||
| -------------------------------------- | ------------------------------------------------------- |
|
||||
| **Get started from scratch / "help me set up an agent"** | `shared/managed-agents-onboarding.md` — guided interview (WHERE→WHO→WHAT→WATCH), then emit code |
|
||||
| Understand how the API works | `shared/managed-agents-core.md` |
|
||||
| See the full endpoint reference | `shared/managed-agents-api-reference.md` |
|
||||
| **Create an agent** (required first step) | `shared/managed-agents-core.md` (Agents section) + language file |
|
||||
| Update/version an agent | `shared/managed-agents-core.md` (Agents → Versioning) — update, don't re-create |
|
||||
| Create a session | `shared/managed-agents-core.md` + `{lang}/managed-agents/README.md` |
|
||||
| Configure tools and permissions | `shared/managed-agents-tools.md` |
|
||||
| Set up MCP servers | `shared/managed-agents-tools.md` (MCP Servers section) |
|
||||
| Stream events / handle tool_use | `shared/managed-agents-events.md` + language file |
|
||||
| Get notified of session state changes via webhook (no polling) | `shared/managed-agents-webhooks.md` — Console-registered endpoint, HMAC verify, thin payload + fetch |
|
||||
| Define an outcome / rubric-graded iterate loop | `shared/managed-agents-outcomes.md` — `user.define_outcome` event, grader, `span.outcome_evaluation_*` events |
|
||||
| Coordinate multiple agents / subagents / threads | `shared/managed-agents-multiagent.md` — `multiagent: {type: "coordinator", agents: [...]}` on the agent, session threads, cross-posted tool confirmations |
|
||||
| Set up environments | `shared/managed-agents-environments.md` + language file |
|
||||
| Run tool execution in your own infra / VPC (self-hosted sandbox) | `shared/managed-agents-self-hosted-sandboxes.md` — `config:{type:"self_hosted"}`, `ANTHROPIC_ENVIRONMENT_KEY`, `EnvironmentWorker.run()` / `ant beta:worker poll` |
|
||||
| Upload files / attach repos | `shared/managed-agents-environments.md` (Resources) |
|
||||
| Give agents persistent memory across sessions | `shared/managed-agents-memory.md` — memory stores, `memory_store` session resource, preconditions, versions/redact |
|
||||
| Define agents/environments as version-controlled YAML; drive the API from the shell | `shared/anthropic-cli.md` — `ant beta:agents create < agent.yaml`, `--transform`, `@file` inlining |
|
||||
| Store credentials (MCP auth, API keys for CLIs/SDKs) | `shared/managed-agents-tools.md` (Vaults section) — `mcp_oauth` / `static_bearer` / `environment_variable` |
|
||||
| Call a non-MCP API / CLI that needs a secret | `shared/managed-agents-tools.md` (Vaults section) — `environment_variable` credential, substituted at egress. If that doesn't fit (e.g. self-hosted sandboxes), `shared/managed-agents-client-patterns.md` Pattern 9 keeps the secret host-side via a custom tool |
|
||||
| Run an agent on a recurring cron schedule | `shared/managed-agents-scheduled-deployments.md` — deployments, deployment runs, pause/auto-pause |
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Agent FIRST, then session — NO EXCEPTIONS** — the session's `agent` field accepts **only** a string ID or `{type: "agent", id, version}`. `model`, `system`, `tools`, `mcp_servers`, `skills` are **top-level fields on `POST /v1/agents`**, never on `sessions.create()`. If the user hasn't created an agent, that is step zero of every example.
|
||||
- **Agent ONCE, not every run** — `agents.create()` is a setup step. Store the returned `agent_id` and reuse it; don't call `agents.create()` at the top of your hot path. If the agent's config needs to change, `POST /v1/agents/{id}` — each update creates a new version, and sessions can pin to a specific version for reproducibility.
|
||||
- **MCP auth goes through vaults** — the agent's `mcp_servers` array declares `{type, name, url}` only (no auth). Credentials live in vaults (`client.beta.vaults.credentials.create`) and attach to sessions via `vault_ids`. Anthropic auto-refreshes OAuth tokens using the stored refresh token. Vaults also hold `environment_variable` credentials for non-MCP services (CLIs, SDKs, direct API calls) — substituted at egress, never visible in the sandbox.
|
||||
- **Reconcile resources before the first run** — a session with a clear ask but a missing tool, credential, data mount, or context will discover the gap mid-run, then flail and give up. Before creating the session, check that every action in the task maps to a configured tool/MCP server, every MCP server has a vault credential, and every referenced file/host is mounted/reachable. When helping a user set one up, run the reconciliation in `shared/managed-agents-onboarding.md` → §3 Pre-flight viability check.
|
||||
- **Stream to get events** — `GET /v1/sessions/{id}/events/stream` is the primary way to receive agent output in real-time.
|
||||
- **SSE stream has no replay — reconnect with consolidation** — if the stream drops while a `agent.tool_use`, `agent.mcp_tool_use`, or `agent.custom_tool_use` is pending resolution (`user.tool_confirmation` for the first two, `user.custom_tool_result` for the last one), the session deadlocks (client disconnects → session idles → reconnect happens → no client resolution happens). On every (re)connect: open stream with `GET /v1/sessions/{id}/events/stream` , fetch `GET /v1/sessions/{id}/events`, dedupe by event ID, then proceed. See `shared/managed-agents-events.md` → Reconnecting after a dropped stream.
|
||||
- **Don't trust HTTP-library timeouts as wall-clock caps** — `requests` `timeout=(c, r)` and `httpx.Timeout(n)` are *per-chunk* read timeouts; they reset every byte, so a trickling connection can block indefinitely. For a hard deadline on raw-HTTP polling, track `time.monotonic()` at the loop level and bail explicitly. Prefer the SDK's `sessions.events.stream()` / `session.events.list()` over hand-rolled HTTP. See `shared/managed-agents-events.md` → Receiving Events.
|
||||
- **Messages queue** — you can send events while the session is `running` or `idle`; they're processed in order. No need to wait for a response before sending the next message.
|
||||
- **Environment `config.type` is `"cloud"` or `"self_hosted"`** — `cloud` runs the container on Anthropic's infrastructure; `self_hosted` moves tool execution to your own (see `shared/managed-agents-self-hosted-sandboxes.md`).
|
||||
- **Archive is permanent on every resource** — archiving an agent, environment, session, vault, credential, or memory store makes it read-only with no unarchive. For agents, environments, and memory stores specifically, archived resources cannot be referenced by new sessions (existing sessions continue). Do not call `.archive()` on a production agent, environment, or memory store as cleanup — **always confirm with the user before archiving**.
|
||||
@@ -1,144 +0,0 @@
|
||||
# Managed Agents — Scheduled Deployments
|
||||
|
||||
A **scheduled deployment** runs an agent on a recurring cron schedule — each firing creates a session autonomously. Use it for predictable-cadence work: nightly triage, weekly compliance scans, hourly monitors.
|
||||
|
||||
Requires the `managed-agents-2026-04-01` beta header (the SDK sets it automatically for `client.beta.deployments.*` / `client.beta.deployment_runs.*` calls).
|
||||
|
||||
## Create a deployment
|
||||
|
||||
A deployment bundles everything a session needs (agent, environment, optional files / GitHub / memory stores / vaults) plus a `schedule` and the `initial_events` that kick off each run:
|
||||
|
||||
- `agent` and `environment_id` are required — same shapes as `sessions.create` (see `shared/managed-agents-core.md`).
|
||||
- `initial_events` must contain the starting `user.message`.
|
||||
- `schedule` takes a cron `expression` and an IANA `timezone`. Minute-level granularity is the maximum.
|
||||
|
||||
```bash
|
||||
curl -fsSL https://api.anthropic.com/v1/deployments \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-H "anthropic-beta: managed-agents-2026-04-01" \
|
||||
-H "content-type: application/json" \
|
||||
-d @- <<EOF
|
||||
{
|
||||
"name": "Weekly compliance scan",
|
||||
"agent": "$AGENT_ID",
|
||||
"environment_id": "$ENVIRONMENT_ID",
|
||||
"initial_events": [
|
||||
{"type": "user.message", "content": [{"type": "text", "text": "Run the weekly compliance scan."}]}
|
||||
],
|
||||
"schedule": {
|
||||
"type": "cron",
|
||||
"expression": "0 20 * * 5",
|
||||
"timezone": "America/New_York"
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
```python
|
||||
deployment = client.beta.deployments.create(
|
||||
name="Weekly compliance scan",
|
||||
agent=agent.id,
|
||||
environment_id=environment.id,
|
||||
initial_events=[
|
||||
{
|
||||
"type": "user.message",
|
||||
"content": [{"type": "text", "text": "Run the weekly compliance scan."}],
|
||||
},
|
||||
],
|
||||
schedule={
|
||||
"type": "cron",
|
||||
"expression": "0 20 * * 5",
|
||||
"timezone": "America/New_York",
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
The response is a deployment object (`depl_` ID prefix). Check `schedule.upcoming_runs_at` — the next fire times — to confirm the schedule parses the way you intended:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "depl_01xyz",
|
||||
"status": "active",
|
||||
"paused_reason": null,
|
||||
"schedule": {
|
||||
"type": "cron",
|
||||
"expression": "0 20 * * 5",
|
||||
"timezone": "America/New_York",
|
||||
"last_run_at": null,
|
||||
"upcoming_runs_at": ["2026-05-09T00:00:00Z", "2026-05-16T00:00:00Z", "2026-05-23T00:00:00Z"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Deployments may apply up to **10 seconds of jitter** to distribute load. Maximum **1000 scheduled deployments per organization** (contact Anthropic support for more).
|
||||
|
||||
### Cron and timezone semantics
|
||||
|
||||
- **Expression:** standard POSIX cron (`minute hour day-of-month month day-of-week`).
|
||||
- **Timezone:** IANA identifier (e.g. `"America/Los_Angeles"`).
|
||||
- **DST:** literal wall-clock matching — `"0 20 * * *"` in `America/New_York` fires at 8:00 PM local regardless of EST/EDT.
|
||||
|
||||
> ⚠️ **DST edge:** wall-clock times that don't exist on a spring-forward day (e.g. 2AM) are **skipped**; times that occur twice on a fall-back day **fire twice**. Schedule outside the 1–3AM local window, or use UTC, when missed or duplicate executions are unacceptable.
|
||||
|
||||
## Deployment runs
|
||||
|
||||
Every trigger attempt — successful or not — writes a **deployment run** record (`drun_` prefix), so you can audit failures independent of the session lifecycle. A successful run carries the created `session_id`; follow that session via the event stream (`shared/managed-agents-events.md`) or webhooks (`shared/managed-agents-webhooks.md`) as usual. A failed run carries an `error` whose `type` explains why session creation was rejected.
|
||||
|
||||
```python
|
||||
# All runs for a deployment
|
||||
for run in client.beta.deployment_runs.list(deployment_id=deployment.id):
|
||||
print(run.created_at, run.session_id or run.error.type)
|
||||
|
||||
# Failures only
|
||||
for run in client.beta.deployment_runs.list(deployment_id=deployment.id, has_error=True):
|
||||
print(run.created_at, run.error.type, run.error.message)
|
||||
```
|
||||
|
||||
```typescript
|
||||
for await (const run of client.beta.deploymentRuns.list({
|
||||
deployment_id: deployment.id,
|
||||
has_error: true,
|
||||
})) {
|
||||
console.log(run.created_at, run.error?.type, run.error?.message);
|
||||
}
|
||||
```
|
||||
|
||||
Raw HTTP: `GET /v1/deployment_runs?deployment_id=...&has_error=true`.
|
||||
|
||||
A failed run looks like:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "deployment_run",
|
||||
"id": "drun_01abc124",
|
||||
"deployment_id": "depl_01xyz",
|
||||
"trigger_context": { "type": "schedule", "scheduled_at": "2026-05-09T00:00:00Z" },
|
||||
"session_id": null,
|
||||
"error": { "type": "environment_archived", "message": "environment `env_01abc` is archived" },
|
||||
"agent": { "type": "agent", "id": "agent_01ghi789", "version": 3 },
|
||||
"created_at": "2026-05-09T00:00:01Z"
|
||||
}
|
||||
```
|
||||
|
||||
Error types include `environment_archived`, `agent_archived`, `vault_not_found`, `session_rate_limited`, and `service_unavailable`.
|
||||
|
||||
## Lifecycle: pause / unpause / archive
|
||||
|
||||
| Operation | SDK | Effect |
|
||||
|---|---|---|
|
||||
| Pause | `client.beta.deployments.pause(id)` | Suppresses scheduled triggers go-forward. Sessions already running continue. **Manual runs are still permitted while paused.** Sets `paused_reason: {"type": "manual"}`. |
|
||||
| Unpause | `client.beta.deployments.unpause(id)` | Resumes from the next scheduled occurrence. **Missed triggers are not backfilled.** Clears `paused_reason`. |
|
||||
| Archive | `client.beta.deployments.archive(id)` | **Terminal** — the schedule stops and the deployment can no longer be modified. Use pause for anything reversible. |
|
||||
|
||||
Raw HTTP: `POST /v1/deployments/{deployment_id}/pause` (likewise `/unpause`, `/archive`).
|
||||
|
||||
### Failure behavior
|
||||
|
||||
- **Rate-limited:** recorded immediately as a `session_rate_limited` run, **no retry** — the schedule simply tries again at the next occurrence. (Rate limits on API calls *inside* a session are handled by the session itself.)
|
||||
- **Other failed runs** (e.g. `environment_archived`, `vault_not_found`, `service_unavailable`): the run records the `error.type` — monitor runs and fix the referenced resource, or pause the deployment.
|
||||
- **Agent archived or deleted:** the deployment is automatically **archived** (terminal) and no further sessions are created.
|
||||
|
||||
## Manual runs
|
||||
|
||||
`POST /v1/deployments/{deployment_id}/run` (SDK: `client.beta.deployments.run(id)`) creates a session immediately and writes a run with `trigger_context.type: "manual"`. Use it to **test a deployment before committing to the schedule** — and remember it works even while the deployment is paused.
|
||||
@@ -1,174 +0,0 @@
|
||||
# Managed Agents — Self-Hosted Sandboxes
|
||||
|
||||
With `config.type: "self_hosted"`, the **agent loop stays on Anthropic's orchestration layer** but **tool execution moves to infrastructure you control** — bash, file ops, and code run inside your container, so filesystem contents and network egress never leave your environment. Contrast with `config.type: "cloud"`, where Anthropic runs the container. Connectivity is **outbound-only**: your worker long-polls Anthropic's work queue; Anthropic never dials into your network.
|
||||
|
||||
## Flow
|
||||
|
||||
```
|
||||
1. Create environment: config: {type: "self_hosted"} → env_...
|
||||
2. Generate environment key (Console, on the environment page) → sk-ant-oat01-... as ANTHROPIC_ENVIRONMENT_KEY
|
||||
3. Run a worker: EnvironmentWorker.run() or ant beta:worker poll
|
||||
4. Sessions reference environment_id=env_... exactly as for cloud
|
||||
```
|
||||
|
||||
## Create the environment
|
||||
|
||||
```python
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
environment = client.beta.environments.create(
|
||||
name="self-hosted", config={"type": "self_hosted"}
|
||||
)
|
||||
```
|
||||
|
||||
`{"type": "self_hosted"}` is the entire config — there are no pool, capacity, or networking sub-fields; you control those on your side.
|
||||
|
||||
## Run a worker — SDK (primary path)
|
||||
|
||||
`EnvironmentWorker` wraps the poll → dispatch → tool-execute loop. `.run()` is the always-on loop; `.run_one()` / `.runOne()` handles one work item (for webhook-driven wake).
|
||||
|
||||
**Python — always-on:**
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
from anthropic import AsyncAnthropic
|
||||
from anthropic.lib.environments import EnvironmentWorker
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
environment_key = os.environ["ANTHROPIC_ENVIRONMENT_KEY"]
|
||||
environment_id = os.environ["ANTHROPIC_ENVIRONMENT_ID"]
|
||||
async with AsyncAnthropic(auth_token=environment_key) as client:
|
||||
await EnvironmentWorker(
|
||||
client,
|
||||
environment_id=environment_id,
|
||||
environment_key=environment_key,
|
||||
workdir="/workspace",
|
||||
).run()
|
||||
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**TypeScript — always-on:**
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
import { EnvironmentWorker } from "@anthropic-ai/sdk/helpers/beta/environments";
|
||||
|
||||
const environmentKey = process.env.ANTHROPIC_ENVIRONMENT_KEY!;
|
||||
const environmentId = process.env.ANTHROPIC_ENVIRONMENT_ID!;
|
||||
const client = new Anthropic({ authToken: environmentKey });
|
||||
const ctrl = new AbortController();
|
||||
process.once("SIGTERM", () => ctrl.abort());
|
||||
|
||||
await new EnvironmentWorker({
|
||||
client,
|
||||
environmentId,
|
||||
environmentKey,
|
||||
workdir: "/workspace",
|
||||
signal: ctrl.signal
|
||||
}).run();
|
||||
```
|
||||
|
||||
**Customizing tools.** `EnvironmentWorker` runs the built-in toolset by default. To add or replace tools, use `AgentToolContext(workdir=, client=, session_id=)` with `beta_agent_toolset(env)` / `betaAgentToolset(env)` and pass the resulting tools to the lower-level `tool_runner()`. Skills attached to the agent are downloaded into `{workdir}/skills/<name>/` before tool calls begin (`AgentToolContext` handles this when given `client` and `session_id`). Downloaded skill files are marked executable automatically by the CLI and SDK; if you implement skills download yourself, you set permissions.
|
||||
|
||||
> **Runtime deps:** the SDK helpers require `/bin/bash` at that exact path. The TypeScript SDK additionally requires `unzip`, `tar`, and Node.js 22+. These are resolved at fixed paths and do **not** respect `PATH` overrides.
|
||||
|
||||
## Run a worker — `ant` CLI (fixed tools)
|
||||
|
||||
The `ant` CLI ships a worker with the fixed built-in toolset (`bash`, `read`, `write`, `edit`, `glob`, `grep`). Install per `shared/anthropic-cli.md`, then:
|
||||
|
||||
```sh
|
||||
export ANTHROPIC_ENVIRONMENT_KEY=sk-ant-oat01-...
|
||||
ant beta:worker poll --environment-id env_... --workdir /workspace
|
||||
```
|
||||
|
||||
- `--workdir` is the directory tools operate in (default `.`); tool calls are sandboxed to it.
|
||||
- `--environment-key` overrides the env var.
|
||||
- `--on-work <script>` runs your script per work item (e.g. to spin a fresh container per session — see Container orchestration below).
|
||||
- `--unrestricted-paths`, `--max-idle` (default `60s`), `--log-format` — see `ant beta:worker poll --help`.
|
||||
- Flags fall back to env vars (`ANTHROPIC_ENVIRONMENT_ID`, `ANTHROPIC_ENVIRONMENT_KEY`).
|
||||
- Exits cleanly on SIGTERM/SIGINT after draining in-flight work.
|
||||
- **Fixed toolset** — for custom tools, use the SDK worker above.
|
||||
|
||||
Inside an `--on-work` container, run `ant beta:worker run --workdir <dir>` as the entrypoint.
|
||||
|
||||
## Webhook-driven wake (instead of always-on)
|
||||
|
||||
Register a webhook for `session.status_run_started` (see `shared/managed-agents-webhooks.md`), verify the delivery, then drain one work item with `.run_one()`:
|
||||
|
||||
```python
|
||||
import os
|
||||
import anthropic
|
||||
from anthropic.lib.environments import EnvironmentWorker
|
||||
|
||||
environment_key = os.environ["ANTHROPIC_ENVIRONMENT_KEY"]
|
||||
environment_id = os.environ["ANTHROPIC_ENVIRONMENT_ID"]
|
||||
client = anthropic.AsyncAnthropic(
|
||||
auth_token=environment_key,
|
||||
) # reads ANTHROPIC_WEBHOOK_SIGNING_KEY from env for webhooks.unwrap()
|
||||
|
||||
|
||||
async def handle(raw: bytes, headers: dict[str, str]) -> dict:
|
||||
event = client.beta.webhooks.unwrap(raw.decode(), headers=headers)
|
||||
if event.data.type != "session.status_run_started":
|
||||
return {"status": "ignored"}
|
||||
await EnvironmentWorker(
|
||||
client,
|
||||
environment_id=environment_id,
|
||||
environment_key=environment_key,
|
||||
workdir="/workspace",
|
||||
).run_one()
|
||||
return {"status": "ok"}
|
||||
```
|
||||
|
||||
TypeScript: same shape with `client.beta.webhooks.unwrap(body, {headers})` and `new EnvironmentWorker({...}).runOne()`.
|
||||
|
||||
## Container orchestration (mid-level)
|
||||
|
||||
`EnvironmentWorker.run()` polls and executes tools in the same process. To run each session in its **own** container, use the mid-level poller in a thin orchestrator — Python `client.beta.environments.work.poller(environment_id=, environment_key=, drain=, block_ms=, reclaim_older_than_ms=, auto_stop=)`; TypeScript `new WorkPoller({client, environmentId, environmentKey, autoStop})` from `@anthropic-ai/sdk/helpers/beta/environments` — and, for each yielded `work` item, start a fresh container with these env vars injected, whose entrypoint runs `ant beta:worker run` or an `EnvironmentWorker(...).run_one()`. `block_ms` is 1–999 (or `None` for non-blocking); `reclaim_older_than_ms` re-claims items leased to a dead worker; `drain` stops once the queue is empty; `auto_stop` posts a stop signal after the iterator exits (set `False` when the launched container owns the stop call). **Go's poller has no `auto_stop` opt-out** — it calls `work.Stop` when the handler returns, so block in the handler until the session completes rather than detaching.
|
||||
|
||||
| Env var | Value |
|
||||
|---|---|
|
||||
| `ANTHROPIC_SESSION_ID` | `work.data.id` |
|
||||
| `ANTHROPIC_WORK_ID` | `work.id` |
|
||||
| `ANTHROPIC_ENVIRONMENT_ID` | `work.environment_id` |
|
||||
| `ANTHROPIC_ENVIRONMENT_KEY` | pass through |
|
||||
| `ANTHROPIC_BASE_URL` | pass through |
|
||||
|
||||
Skip items where `work.data.type != "session"`.
|
||||
|
||||
## Monitoring & control
|
||||
|
||||
These are **control-plane** calls — authenticate with `x-api-key` (not the environment key); `managed-agents-2026-04-01` beta header. **Call them from outside the worker host** — setting `ANTHROPIC_API_KEY` on the worker host exposes an organization-scoped credential to agent tool calls.
|
||||
|
||||
| SDK (`client.beta.environments.work.*`) | REST | CLI | Returns |
|
||||
|---|---|---|---|
|
||||
| `stats(environment_id)` | `GET /v1/environments/{id}/work/stats` | `ant beta:environments:work stats` | `{type:"work_queue_stats", depth, pending, oldest_queued_at, workers_polling}` |
|
||||
| `stop(work_id, environment_id=)` | `POST /v1/environments/{id}/work/{work_id}/stop` | `ant beta:environments:work stop` | `work.state` |
|
||||
|
||||
## What changes vs `cloud`
|
||||
|
||||
| Concern | `cloud` | `self_hosted` |
|
||||
|---|---|---|
|
||||
| Container lifecycle, hardening, networking | Anthropic | **You** — run non-root, read-only rootfs, drop caps; egress is whatever your VPC/firewall allows |
|
||||
| `file` / `github_repository` resource mounting | Anthropic mounts into the container | **You** — pass pointers via `sessions.create(metadata={...})` and have your orchestrator fetch/clone before dispatch |
|
||||
| `memory_store` resources | Supported | **Not yet supported** |
|
||||
| Vault `environment_variable` credentials | Supported (substituted at Anthropic-managed egress) | **Not yet supported** — egress is yours, so there's nowhere to substitute the secret. Use MCP credentials or a host-side custom tool (`shared/managed-agents-client-patterns.md` Pattern 9) |
|
||||
| Built-in tools | Via `agent_toolset_20260401` | Supplied by your worker (`EnvironmentWorker` default / `beta_agent_toolset(env)` / `ant` CLI fixed set) |
|
||||
| Skills download | Automatic | `EnvironmentWorker` / `AgentToolContext` fetch into `{workdir}/skills/` (needs `client` + `session_id`) |
|
||||
| Claude Platform on AWS | Supported | **Not available** |
|
||||
| SDK worker helpers | All SDKs | **Python, TypeScript, Go only** (`EnvironmentWorker` / poller not in Java, Ruby, PHP, or C#) — use one of those three or the `ant` CLI |
|
||||
|
||||
## Credentials
|
||||
|
||||
| Credential | Format | Scope |
|
||||
|---|---|---|
|
||||
| `ANTHROPIC_ENVIRONMENT_KEY` | `sk-ant-oat01-...` | One environment's work queue. Generate in Console ("Generate environment key"). Pass as `auth_token=` / `authToken` on the client **and** as `environment_key=` / `environmentKey` on `EnvironmentWorker`. Store in a secrets manager; rotate on exposure. |
|
||||
| `ANTHROPIC_WEBHOOK_SIGNING_KEY` | `whsec_...` | Webhook signature verification (if using webhook-driven wake). The SDK reads this env var automatically for `client.beta.webhooks.unwrap()`. |
|
||||
|
||||
## Security — what you own
|
||||
|
||||
Container hardening; egress restriction (there is no default); `ANTHROPIC_ENVIRONMENT_KEY` custody and rotation; one workspace + environment per trust boundary when running untrusted code; least-privilege for the tool process; log retention and redaction. **Anthropic cannot**: fast-revoke a leaked environment key, verify your image or supply chain, sandbox tool execution inside your container, or enforce retention after tool output reaches your infrastructure. See the Self-Hosted Sandboxes Security page in `shared/live-sources.md` for the full checklist.
|
||||
@@ -1,358 +0,0 @@
|
||||
# Managed Agents — Tools & Skills
|
||||
|
||||
## Tools
|
||||
|
||||
### Server tools vs client tools
|
||||
|
||||
| Type | Who runs it | How it works |
|
||||
|---|---|---|
|
||||
| **Prebuilt Claude Agent tools** (`agent_toolset_20260401`) | Anthropic, on the session's container (for `cloud` envs; for `self_hosted`, **your** worker supplies and runs them — see `shared/managed-agents-self-hosted-sandboxes.md`) | File ops, bash, web search, etc. Enable all at once or configure individually with `enabled: true/false`. |
|
||||
| **MCP tools** (`mcp_toolset`) | Anthropic's orchestration layer | Capabilities exposed by connected MCP servers. Grant access per-server via the toolset. |
|
||||
| **Custom tools** | **You** — your application handles the call and returns results | Agent emits a `agent.custom_tool_use` event, session goes `idle`, you send back a `user.custom_tool_result` event. |
|
||||
|
||||
**Recommendation:** Enable all prebuilt tools via `agent_toolset_20260401`, then disable individually as needed.
|
||||
|
||||
**Versioning:** The toolset is a versioned, static resource. When underlying tools change, a new toolset version is created (hence `_20260401`) so you always know exactly what you're getting.
|
||||
|
||||
### Agent Toolset
|
||||
|
||||
The `agent_toolset_20260401` provides these built-in tools:
|
||||
|
||||
| Tool | Description |
|
||||
| ---------------------- | ---------------------------------------- |
|
||||
| `bash` | Execute bash commands in a shell session |
|
||||
| `read` | Read a file from the local filesystem, including text, images, PDFs, and Jupyter notebooks |
|
||||
| `write` | Write a file to the local filesystem |
|
||||
| `edit` | Perform string replacement in a file |
|
||||
| `glob` | Fast file pattern matching using glob patterns |
|
||||
| `grep` | Text search using regex patterns |
|
||||
| `web_fetch` | Fetch content from a URL |
|
||||
| `web_search` | Search the web for information |
|
||||
|
||||
Enable the full toolset:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
{ "type": "agent_toolset_20260401" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Per-Tool Configuration
|
||||
|
||||
Override defaults for individual tools. This example enables everything except bash:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
{
|
||||
"type": "agent_toolset_20260401",
|
||||
"default_config": { "enabled": true },
|
||||
"configs": [
|
||||
{ "name": "bash", "enabled": false }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| `type` | ✅ | `"agent_toolset_20260401"` |
|
||||
| `default_config` | ❌ | Applied to all tools. `{ "enabled": bool, "permission_policy": {...} }` |
|
||||
| `configs` | ❌ | Per-tool overrides: `[{ "name": "...", "enabled": bool, "permission_policy": {...} }]` |
|
||||
|
||||
### Permission Policies
|
||||
|
||||
Control when server-executed tools (agent toolset + MCP) run automatically vs wait for approval. Does not apply to custom tools.
|
||||
|
||||
| Policy | Behavior |
|
||||
|---|---|
|
||||
| `always_allow` | Tool executes automatically (default) |
|
||||
| `always_ask` | Session emits `session.status_idle` and pauses until you send a `tool_confirmation` event |
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "agent_toolset_20260401",
|
||||
"default_config": {
|
||||
"enabled": true,
|
||||
"permission_policy": { "type": "always_allow" }
|
||||
},
|
||||
"configs": [
|
||||
{ "name": "bash", "permission_policy": { "type": "always_ask" } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Responding to `always_ask`:** Send a `user.tool_confirmation` event with `tool_use_id` from the triggering `agent_tool_use`/`mcp_tool_use` event:
|
||||
|
||||
```json
|
||||
{ "type": "tool_confirmation", "tool_use_id": "sevt_abc123", "result": "allow" }
|
||||
{ "type": "tool_confirmation", "tool_use_id": "sevt_def456", "result": "deny", "message": "Read .env.example instead" }
|
||||
```
|
||||
|
||||
The optional `message` on a deny is delivered to the agent so it can adjust its approach.
|
||||
|
||||
To enable only specific tools, flip the default off and opt-in per tool:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
{
|
||||
"type": "agent_toolset_20260401",
|
||||
"default_config": { "enabled": false },
|
||||
"configs": [
|
||||
{ "name": "bash", "enabled": true },
|
||||
{ "name": "read", "enabled": true }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Custom Tools (Client-Side)
|
||||
|
||||
Custom tools are executed by **your application**, not Anthropic. The flow:
|
||||
|
||||
1. Agent decides to use the tool → session emits a `agent.custom_tool_use` event with inputs
|
||||
2. Session goes `idle` waiting for you
|
||||
3. Your application executes the tool
|
||||
4. You send back a `user.custom_tool_result` event with the output
|
||||
5. Session resumes `running`
|
||||
|
||||
No permission policy needed — you're the one executing.
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
{
|
||||
"type": "custom",
|
||||
"name": "get_weather",
|
||||
"description": "Fetch current weather for a city.",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": { "type": "string", "description": "City name" }
|
||||
},
|
||||
"required": ["city"]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### MCP Servers
|
||||
|
||||
MCP (Model Context Protocol) servers expose standardized third-party capabilities (e.g. Asana, GitHub, Linear). **Configuration is split across agent and vault:**
|
||||
|
||||
1. **Agent creation** declares which servers to connect to (`type`, `name`, `url` — no auth). The agent's `mcp_servers` array has no auth field.
|
||||
2. **Vault** stores the OAuth credentials. Attach via `vault_ids` on session create.
|
||||
|
||||
This keeps secrets out of reusable agent definitions. Each vault credential is tied to one MCP server URL; Anthropic matches credentials to servers by URL.
|
||||
|
||||
**Agent side — declare servers (no auth):**
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| `type` | ✅ | `"url"` |
|
||||
| `name` | ✅ | Unique name — referenced by `mcp_toolset.mcp_server_name` |
|
||||
| `url` | ✅ | The MCP server's endpoint URL (Streamable HTTP transport) |
|
||||
|
||||
```json
|
||||
{
|
||||
"mcp_servers": [
|
||||
{ "type": "url", "name": "linear", "url": "https://mcp.linear.app/mcp" }
|
||||
],
|
||||
"tools": [
|
||||
{ "type": "mcp_toolset", "mcp_server_name": "linear" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Session side — attach vault:**
|
||||
|
||||
```json
|
||||
{
|
||||
"agent": "agent_abc123",
|
||||
"environment_id": "env_abc123",
|
||||
"vault_ids": ["vlt_abc123"]
|
||||
}
|
||||
```
|
||||
|
||||
> 💡 **Per-tool enablement (empirical):** `mcp_toolset` has been observed accepting `default_config: {enabled: false}` + `configs: [{name, enabled: true}]` for an allowlist pattern. The API ref shows only the minimal `{type, mcp_server_name}` form.
|
||||
|
||||
> 💡 **Changing tools/MCP servers on a running session:** `sessions.update()` can replace `agent.tools`, `agent.mcp_servers`, and `vault_ids` while the session is `idle` — a session-local override that doesn't touch the agent object. See `shared/managed-agents-core.md` → Updating the agent configuration mid-session.
|
||||
|
||||
**Large MCP tool outputs.** If an MCP tool returns more than **100K tokens**, the output is automatically offloaded to a file in the sandbox — the agent receives a truncated preview plus the file path and can `read` the full content. No configuration required.
|
||||
|
||||
**Invalid vault credentials don't block session creation.** If a vault credential is invalid for a declared MCP server, the session still creates successfully; a `session.error` event describes the MCP auth failure, and auth retries on the next `session.status_idle` → `session.status_running` transition.
|
||||
|
||||
> ⚠️ **MCP auth tokens ≠ REST API tokens.** Hosted MCP servers (`mcp.notion.com`, `mcp.linear.app`, etc.) typically require **OAuth bearer tokens**, not the service's native API keys. A Notion `ntn_` integration token authenticates against Notion's REST API but will **not** work as a vault credential for the Notion MCP server. These are different auth systems.
|
||||
|
||||
### Vaults — the credential store
|
||||
|
||||
**Vaults** store credentials that Anthropic manages on your behalf. Two credential categories:
|
||||
|
||||
- **MCP credentials** (`mcp_oauth`, `static_bearer`) — keyed by `mcp_server_url`. When the agent connects to a server at that URL, the token is injected automatically. `mcp_oauth` tokens are auto-refreshed via the standard OAuth 2.0 `refresh_token` grant. This is the only way to authenticate MCP servers.
|
||||
- **Environment variables** (`environment_variable`) — keyed by `secret_name` (the env var name). The sandbox sees only an **opaque placeholder**; the real secret is substituted into the outbound request **at egress**. Use this for any service that authenticates through an environment variable: CLIs (`aws`, `gcloud`, `stripe`), SDKs, or direct `curl` calls from the `bash` tool.
|
||||
|
||||
Secret fields you supply (`token`, `access_token`, `refresh_token`, `client_secret`, `secret_value`) are write-only — never returned in API responses.
|
||||
|
||||
#### Credentials and the sandbox
|
||||
|
||||
Vaults store credentials; those credentials **never enter the sandbox**. This is a deliberate security boundary — code running in the sandbox (including anything the agent writes) cannot read or exfiltrate a vaulted credential, even under prompt injection. Instead, credentials are injected by Anthropic-side proxies **after** a request leaves the sandbox:
|
||||
|
||||
- **MCP tool calls** are routed through an Anthropic-side proxy that fetches the credential from the vault and adds it to the outbound request.
|
||||
- **Git operations on attached GitHub repositories** (`git pull`, `git push`, GitHub REST calls) are routed through a git proxy that injects the `github_repository` resource's `authorization_token` the same way.
|
||||
- **Environment-variable credentials** appear in the sandbox as an opaque placeholder; the real value replaces the placeholder at egress, on requests to the credential's allowed hosts only.
|
||||
|
||||
**When vault credentials don't fit** (e.g. self-hosted sandboxes — `environment_variable` is not yet supported there), **register a custom tool:** the agent emits `agent.custom_tool_use`, your orchestrator (which already holds the credential) executes the call and returns `user.custom_tool_result` over the same authenticated event stream. No public endpoint is exposed; the sandbox never sees the secret. See `shared/managed-agents-client-patterns.md` → Pattern 9.
|
||||
|
||||
**Do not put API keys in the system prompt or user messages as a workaround** — they persist in the session's event history.
|
||||
|
||||
> Formerly known internally as TATs (Tool/Tenant Access Tokens).
|
||||
|
||||
**Flow:**
|
||||
|
||||
1. Create a vault (`client.beta.vaults.create(...)`) — one per tenant/user, or one shared, depending on your model
|
||||
2. Add credentials to it (`client.beta.vaults.credentials.create(...)`) — MCP credentials are keyed by MCP server URL; environment-variable credentials by `secret_name`
|
||||
3. Reference the vault on session create via `vault_ids: ["vlt_..."]`
|
||||
4. Anthropic auto-refreshes OAuth tokens before they expire and substitutes secrets at runtime
|
||||
|
||||
**MCP OAuth credential shape**:
|
||||
|
||||
```json
|
||||
{
|
||||
"display_name": "Notion (workspace-foo)",
|
||||
"auth": {
|
||||
"type": "mcp_oauth",
|
||||
"mcp_server_url": "https://mcp.notion.com/mcp",
|
||||
"access_token": "<current access token>",
|
||||
"expires_at": "2026-04-02T14:00:00Z",
|
||||
"refresh": {
|
||||
"refresh_token": "<refresh token>",
|
||||
"client_id": "<your OAuth client_id>",
|
||||
"token_endpoint": "https://api.notion.com/v1/oauth/token",
|
||||
"token_endpoint_auth": { "type": "none" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `refresh` block is what enables auto-refresh — `token_endpoint` is where Anthropic posts the `refresh_token` grant. `token_endpoint_auth` is a discriminated union:
|
||||
|
||||
| `type` | Shape | Use when |
|
||||
|---|---|---|
|
||||
| `"none"` | `{type: "none"}` | Public OAuth client (no secret) |
|
||||
| `"client_secret_basic"` | `{type: "client_secret_basic", client_secret: "..."}` | Confidential client, secret via HTTP Basic auth |
|
||||
| `"client_secret_post"` | `{type: "client_secret_post", client_secret: "..."}` | Confidential client, secret in request body |
|
||||
|
||||
Omit `refresh` entirely if you only have an access token with no refresh capability — it'll work until it expires, then the agent loses access.
|
||||
|
||||
> 💡 **Getting an OAuth token.** How you obtain the initial access and refresh tokens depends on the MCP server — consult its documentation. Once you have them, store them in a vault credential using the shape above; Anthropic auto-refreshes via the `refresh.token_endpoint` from there.
|
||||
|
||||
**Environment-variable credential shape**:
|
||||
|
||||
```json
|
||||
{
|
||||
"display_name": "Twilio API key for sandbox",
|
||||
"auth": {
|
||||
"type": "environment_variable",
|
||||
"secret_name": "TWILIO_API_KEY",
|
||||
"secret_value": "sk-your-secret-here",
|
||||
"networking": {
|
||||
"type": "limited",
|
||||
"allowed_hosts": ["api.twilio.com", "*.twilio.com"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`networking.allowed_hosts` controls which outbound hosts the secret can be substituted for — `{"type": "limited", "allowed_hosts": [...]}` or `{"type": "unrestricted"}` if you can't enumerate the domains in advance. Limiting is strongly recommended: it prevents the key from ever being sent to unauthorized hosts.
|
||||
|
||||
> ⚠️ **Two networking layers, both required.** `networking.allowed_hosts` on the credential controls which requests *use the secret*, not which requests are *allowed*. The agent must also be able to reach the domain at the **environment level** (`unrestricted`, or the host listed in the environment's `allowed_hosts` — see `shared/managed-agents-environments.md`). A domain missing from either layer means the secret-substituted request fails.
|
||||
|
||||
> ⚠️ **Client-side validation caveat.** Substitution happens at egress, not inside the sandbox — clients that validate the credential *format* locally before making a network request (e.g. a CLI that checks the key starts with `sk-`) will see the opaque placeholder and may fail at startup. If a client rejects the credential before any network call, that's why.
|
||||
|
||||
> 💡 **Scope the key minimally.** The agent can do anything the key allows; a key with broader permissions than the task needs increases the blast radius if the agent behaves unexpectedly.
|
||||
|
||||
**Not supported with self-hosted sandboxes** — `environment_variable` credentials require Anthropic-managed egress. See `shared/managed-agents-self-hosted-sandboxes.md`.
|
||||
|
||||
**Constraints (all credential types):**
|
||||
|
||||
- **Unique key per vault.** `mcp_server_url` (MCP credentials) and `secret_name` (environment-variable credentials) must be unique among active credentials in a vault; duplicates return a 409.
|
||||
- **Keys are immutable.** Secret values and `display_name` can be updated (rotation); to change `mcp_server_url`, `secret_name`, `token_endpoint`, or `client_id`, archive the credential and create a new one. Archiving purges the secret and frees the key for a replacement.
|
||||
- **Maximum 20 credentials per vault.**
|
||||
- Credentials are stored as provided and **not validated until session runtime** — an invalid credential surfaces as an authentication or downstream error during the session, which is emitted but does not block the session from continuing.
|
||||
|
||||
**Scoping:** Vaults are workspace-scoped. Anyone with developer+ role in the API workspace can create, read (metadata only — secrets are write-only), and attach vaults. `vault_ids` can be set at session **create** time but not via session update (the SDK docstring says "Not yet supported; requests setting this field are rejected").
|
||||
|
||||
---
|
||||
|
||||
## Skills
|
||||
|
||||
Skills are reusable, filesystem-based resources that provide your agent with domain-specific expertise: workflows, context, and best practices that transform general-purpose agents into specialists. Unlike prompts (conversation-level instructions for one-off tasks), skills load on-demand and eliminate the need to repeatedly provide the same guidance across multiple conversations.
|
||||
|
||||
Two types — both work the same way; the agent automatically uses them when relevant to the task at hand:
|
||||
|
||||
| Type | What it is |
|
||||
|---|---|
|
||||
| **Pre-built Anthropic skills** | Common document tasks (PowerPoint, Excel, Word, PDF). Reference by name (e.g. `xlsx`). |
|
||||
| **Custom skills** | Skills you've created in your organization via the Skills API. Reference by `skill_id` + optional `version`. |
|
||||
|
||||
**Max 20 skills per agent.** Agent creation uses `managed-agents-2026-04-01`; the separate Skills API (for managing custom skill definitions) uses `skills-2025-10-02`.
|
||||
|
||||
### Enabling skills on a session
|
||||
|
||||
Skills are attached to the **agent** definition via `agents.create()`:
|
||||
|
||||
```ts
|
||||
const agent = await client.beta.agents.create(
|
||||
{
|
||||
name: "Financial Agent",
|
||||
model: "claude-opus-4-8",
|
||||
system: "You are a financial analysis agent.",
|
||||
skills: [
|
||||
{ type: "anthropic", skill_id: "xlsx" },
|
||||
{ type: "custom", skill_id: "skill_abc123", version: "latest" },
|
||||
],
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
Python:
|
||||
|
||||
```python
|
||||
agent = client.beta.agents.create(
|
||||
name="Financial Agent",
|
||||
model="claude-opus-4-8",
|
||||
system="You are a financial analysis agent.",
|
||||
skills=[
|
||||
{"type": "anthropic", "skill_id": "xlsx"},
|
||||
{"type": "custom", "skill_id": "skill_abc123", "version": "latest"},
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
**Skill reference fields:**
|
||||
|
||||
| Field | Anthropic skill | Custom skill |
|
||||
|---|---|---|
|
||||
| `type` | `"anthropic"` | `"custom"` |
|
||||
| `skill_id` | Skill name (e.g. `"xlsx"`, `"docx"`, `"pptx"`, `"pdf"`) | Skill ID from Skills API (e.g. `"skill_abc123"`) |
|
||||
| `version` | — | `"latest"` or a specific version number |
|
||||
|
||||
### Skills API
|
||||
|
||||
| Operation | Method | Path |
|
||||
| --------------------- | -------- | ----------------------------------------------- |
|
||||
| Create Skill | `POST` | `/v1/skills` |
|
||||
| List Skills | `GET` | `/v1/skills` |
|
||||
| Get Skill | `GET` | `/v1/skills/{id}` |
|
||||
| Delete Skill | `DELETE` | `/v1/skills/{id}` |
|
||||
| Create Version | `POST` | `/v1/skills/{id}/versions` |
|
||||
| List Versions | `GET` | `/v1/skills/{id}/versions` |
|
||||
| Get Version | `GET` | `/v1/skills/{id}/versions/{version}` |
|
||||
| Delete Version | `DELETE` | `/v1/skills/{id}/versions/{version}` |
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
# Managed Agents — Webhooks
|
||||
|
||||
Anthropic can POST to your HTTPS endpoint when a Managed Agents resource changes state — an alternative to holding an SSE stream or polling. Payloads are **thin** (event type + resource IDs only); on receipt, fetch the resource for current state. Every delivery is HMAC-signed.
|
||||
|
||||
> **Direction matters.** This page covers *Anthropic → you* notifications about session/vault state. It does **not** cover *third-party → you* webhooks that *trigger* a session (e.g. a GitHub push handler that calls `sessions.create()`) — that's ordinary application code on your side with no Anthropic-specific wire format.
|
||||
|
||||
---
|
||||
|
||||
## Register an endpoint (Console only)
|
||||
|
||||
Console → **Manage → Webhooks**. There is no programmatic endpoint-management API yet. Secret rotation is supported from the same page.
|
||||
|
||||
| Field | Constraint |
|
||||
|---|---|
|
||||
| URL | HTTPS on port 443, publicly resolvable hostname |
|
||||
| Event types | Subscribe per `data.type` — you only receive subscribed types (plus test events) |
|
||||
| Signing secret | `whsec_`-prefixed, 32 bytes, **shown once at creation** — store it |
|
||||
|
||||
---
|
||||
|
||||
## Verify the signature
|
||||
|
||||
Every delivery is HMAC-signed. **Use the SDK's `client.beta.webhooks.unwrap()`** — it verifies the signature, rejects payloads more than ~5 minutes old, and returns the parsed event. It reads the `whsec_` secret from `ANTHROPIC_WEBHOOK_SIGNING_KEY`.
|
||||
|
||||
```python
|
||||
import anthropic
|
||||
from flask import Flask, request
|
||||
|
||||
client = anthropic.Anthropic() # reads ANTHROPIC_WEBHOOK_SIGNING_KEY from env
|
||||
app = Flask(__name__)
|
||||
|
||||
|
||||
@app.route("/webhook", methods=["POST"])
|
||||
def webhook():
|
||||
try:
|
||||
event = client.beta.webhooks.unwrap(
|
||||
request.get_data(as_text=True),
|
||||
headers=dict(request.headers),
|
||||
)
|
||||
except Exception:
|
||||
return "invalid signature", 400
|
||||
|
||||
if event.id in seen_event_ids: # dedupe retries — id is per-event, not per-delivery
|
||||
return "", 204
|
||||
seen_event_ids.add(event.id)
|
||||
|
||||
match event.data.type:
|
||||
case "session.status_idled":
|
||||
session = client.beta.sessions.retrieve(event.data.id)
|
||||
notify_user(session)
|
||||
case "vault_credential.refresh_failed":
|
||||
alert_oncall(event.data.id)
|
||||
|
||||
return "", 204
|
||||
```
|
||||
|
||||
Pass the **raw request body** to `unwrap()` — frameworks that re-serialize JSON (Express `.json()`, Flask `.get_json()`) change the bytes and break the MAC. For other languages, look up the `beta.webhooks.unwrap` binding in the SDK repo (`shared/live-sources.md`); don't hand-roll verification.
|
||||
|
||||
---
|
||||
|
||||
## Payload envelope
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "event",
|
||||
"id": "event_01ABC...",
|
||||
"created_at": "2026-03-18T14:05:22Z",
|
||||
"data": {
|
||||
"type": "session.status_idled",
|
||||
"id": "session_01XYZ...",
|
||||
"organization_id": "8a3d2f1e-...",
|
||||
"workspace_id": "c7b0e4d9-..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Switch on `data.type`, fetch the resource by `data.id`, return any **2xx** to acknowledge. `created_at` is when the *state transition* happened, not when the webhook fired.
|
||||
|
||||
---
|
||||
|
||||
## Supported `data.type` values
|
||||
|
||||
| `data.type` | Fires when |
|
||||
|---|---|
|
||||
| `session.status_scheduled` | Session created and ready to accept events |
|
||||
| `session.status_run_started` | Agent execution kicked off (every transition to `running`) |
|
||||
| `session.status_idled` | Agent awaiting input (tool approval, custom tool result, or next message) |
|
||||
| `session.status_terminated` | Session hit a terminal error |
|
||||
| `session.thread_created` | Multiagent: coordinator opened a new subagent thread |
|
||||
| `session.thread_idled` | Multiagent: a subagent thread is waiting for input |
|
||||
| `session.outcome_evaluation_ended` | Outcome grader finished one iteration |
|
||||
| `vault.archived` | Vault was archived |
|
||||
| `vault.created` | Vault was created |
|
||||
| `vault.deleted` | Vault was deleted |
|
||||
| `vault_credential.archived` | Vault credential was archived |
|
||||
| `vault_credential.created` | Vault credential was created |
|
||||
| `vault_credential.deleted` | Vault credential was deleted |
|
||||
| `vault_credential.refresh_failed` | MCP OAuth vault credential failed to refresh |
|
||||
|
||||
> These are **webhook** `data.type` values — a separate namespace from SSE event types (`session.status_idle`, `span.outcome_evaluation_end`, etc. in `shared/managed-agents-events.md`). Don't reuse SSE constants in webhook handlers.
|
||||
|
||||
---
|
||||
|
||||
## Delivery behavior & pitfalls
|
||||
|
||||
- **No ordering guarantee.** `session.status_idled` may arrive before `session.outcome_evaluation_ended` even if the evaluation finished first. Sort by envelope `created_at` if order matters.
|
||||
- **Retries carry the same `event.id`.** At least one retry on non-2xx. Dedupe on `event.id`.
|
||||
- **3xx is failure.** Redirects are not followed — update the URL in Console if your endpoint moves.
|
||||
- **Auto-disable** after ~20 consecutive failed deliveries, or immediately if the hostname resolves to a private IP or returns a redirect. Re-enable manually in Console.
|
||||
- **Thin payload is intentional.** Don't expect `stop_reason`, `outcome_evaluations`, credential secrets, etc. on the webhook body — fetch the resource.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,132 +0,0 @@
|
||||
# Claude Model Catalog
|
||||
|
||||
**Only use exact model IDs listed in this file.** Never guess or construct model IDs — incorrect IDs will cause API errors. Use aliases wherever available. For the latest information, WebFetch the Models Overview URL in `shared/live-sources.md`, or query the Models API directly (see Programmatic Model Discovery below).
|
||||
|
||||
## Programmatic Model Discovery
|
||||
|
||||
For **live** capability data — context window, max output tokens, feature support (thinking, vision, effort, structured outputs, etc.) — query the Models API instead of relying on the cached tables below. Use this when the user asks "what's the context window for X", "does model X support vision/thinking/effort", "which models support feature Y", or wants to select a model by capability at runtime.
|
||||
|
||||
```python
|
||||
m = client.models.retrieve("claude-opus-4-8")
|
||||
m.id # "claude-opus-4-8"
|
||||
m.display_name # "Claude Opus 4.8"
|
||||
m.max_input_tokens # context window (int)
|
||||
m.max_tokens # max output tokens (int)
|
||||
|
||||
# capabilities is an untyped nested dict — bracket access, check ["supported"] at the leaf
|
||||
caps = m.capabilities
|
||||
caps["image_input"]["supported"] # vision
|
||||
caps["thinking"]["types"]["adaptive"]["supported"] # adaptive thinking
|
||||
caps["effort"]["max"]["supported"] # effort: max (also low/medium/high)
|
||||
caps["structured_outputs"]["supported"]
|
||||
caps["context_management"]["compact_20260112"]["supported"]
|
||||
|
||||
# filter across all models — iterate the page object directly (auto-paginates); do NOT use .data
|
||||
[m for m in client.models.list()
|
||||
if m.capabilities["thinking"]["types"]["adaptive"]["supported"]
|
||||
and m.max_input_tokens >= 200_000]
|
||||
```
|
||||
|
||||
Top-level fields (`id`, `display_name`, `max_input_tokens`, `max_tokens`) are typed attributes. `capabilities` is a dict — use bracket access, not attribute access. The API returns the full capability tree for every model with `supported: true/false` at each leaf, so bracket chains are safe without `.get()` guards. TypeScript SDK: same method names, also auto-paginates on iteration.
|
||||
|
||||
### Raw HTTP
|
||||
|
||||
```bash
|
||||
curl https://api.anthropic.com/v1/models/claude-opus-4-8 \
|
||||
-H "x-api-key: $ANTHROPIC_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01"
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "claude-opus-4-8",
|
||||
"display_name": "Claude Opus 4.8",
|
||||
"max_input_tokens": 1000000,
|
||||
"max_tokens": 128000,
|
||||
"capabilities": {
|
||||
"image_input": {"supported": true},
|
||||
"structured_outputs": {"supported": true},
|
||||
"thinking": {"supported": true, "types": {"enabled": {"supported": false}, "adaptive": {"supported": true}}},
|
||||
"effort": {"supported": true, "low": {"supported": true}, …, "max": {"supported": true}},
|
||||
…
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Current Models (recommended)
|
||||
|
||||
| Friendly Name | Alias (use this) | Full ID | Context | Max Output | Status |
|
||||
|-------------------|---------------------|-------------------------------|----------------|------------|--------|
|
||||
| Claude Fable 5 | `claude-fable-5` | — | 1M | 128K | Active |
|
||||
| Claude Mythos 5 | `claude-mythos-5` | — | 1M | 128K | Active (Project Glasswing only) |
|
||||
| Claude Opus 4.8 | `claude-opus-4-8` | — | 1M | 128K | Active |
|
||||
| Claude Opus 4.7 | `claude-opus-4-7` | — | 1M | 128K | Active |
|
||||
| Claude Opus 4.6 | `claude-opus-4-6` | — | 1M | 128K | Active |
|
||||
| Claude Sonnet 4.6 | `claude-sonnet-4-6` | - | 1M | 64K | Active |
|
||||
| Claude Haiku 4.5 | `claude-haiku-4-5` | `claude-haiku-4-5-20251001` | 200K | 64K | Active |
|
||||
|
||||
### Model Descriptions
|
||||
- **Claude Fable 5** — Anthropic's most capable widely released model, for the most demanding reasoning and long-horizon agentic work. Same API surface as Opus 4.7/4.8 with one new breaking change: an explicit `thinking: {type: "disabled"}` returns a 400 — omit the `thinking` parameter instead (thinking is always on, returned in protected/encrypted form). New tokenizer (~30% more tokens than Opus-tier for the same content). Safety classifiers may return `stop_reason: "refusal"`. No assistant prefill. Requires 30-day data retention (not available under ZDR). $10/$50 per MTok; 1M context window (default), 128K max output. See `shared/model-migration.md` → Migrating to Claude Fable 5.
|
||||
- **Claude Mythos 5** — Same capabilities, pricing, limits, and API behavior as Claude Fable 5; only the model ID differs. Available exclusively through Project Glasswing, where it joins (and succeeds) the invitation-only Claude Mythos Preview (`claude-mythos-preview`). Use it only when the org participates in Project Glasswing; otherwise use claude-fable-5.
|
||||
- **Claude Opus 4.8** — The most capable Opus-tier model — highly autonomous, state-of-the-art on long-horizon agentic work, knowledge work, and memory; clearer, warmer writing. Same API surface as Opus 4.7 (adaptive thinking only; sampling parameters and `budget_tokens` removed). 1M context window at standard API pricing (no long-context premium). See `shared/model-migration.md` → Migrating to Opus 4.8 — a 4.7 → 4.8 move is a model-ID swap plus prompt re-tuning, no new breaking changes.
|
||||
- **Claude Opus 4.7** — Previous-generation Opus. Highly autonomous; strong on long-horizon agentic work, knowledge work, vision, and memory. Adaptive thinking only; sampling parameters and `budget_tokens` removed. 1M context window. See `shared/model-migration.md` → Migrating to Opus 4.7.
|
||||
- **Claude Opus 4.6** — Older Opus. Supports adaptive thinking (recommended), 128K max output tokens (requires streaming for large outputs). 1M context window.
|
||||
- **Claude Sonnet 4.6** — Our best combination of speed and intelligence. Supports adaptive thinking (recommended). 1M context window. 64K max output tokens.
|
||||
- **Claude Haiku 4.5** — Fastest and most cost-effective model for simple tasks.
|
||||
|
||||
## Legacy Models (still active)
|
||||
|
||||
| Friendly Name | Alias (use this) | Full ID | Status |
|
||||
|-------------------|---------------------|-------------------------------|--------|
|
||||
| Claude Opus 4.5 | `claude-opus-4-5` | `claude-opus-4-5-20251101` | Active |
|
||||
| Claude Opus 4.1 | `claude-opus-4-1` | `claude-opus-4-1-20250805` | Deprecated (retires 2026-08-05 — migrate to `claude-opus-4-8`) |
|
||||
| Claude Sonnet 4.5 | `claude-sonnet-4-5` | `claude-sonnet-4-5-20250929` | Active |
|
||||
|
||||
## Deprecated Models (retiring soon)
|
||||
|
||||
| Friendly Name | Alias (use this) | Full ID | Status | Retires |
|
||||
|-------------------|---------------------|-------------------------------|------------|--------------|
|
||||
| Claude Sonnet 4 | `claude-sonnet-4-0` | `claude-sonnet-4-20250514` | Deprecated | TBD |
|
||||
| Claude Opus 4 | `claude-opus-4-0` | `claude-opus-4-20250514` | Deprecated | TBD |
|
||||
| Claude Haiku 3 | — | `claude-3-haiku-20240307` | Deprecated | Apr 19, 2026 |
|
||||
|
||||
## Retired Models (no longer available)
|
||||
|
||||
| Friendly Name | Full ID | Retired |
|
||||
|-------------------|-------------------------------|-------------|
|
||||
| Claude Sonnet 3.7 | `claude-3-7-sonnet-20250219` | Feb 19, 2026 |
|
||||
| Claude Haiku 3.5 | `claude-3-5-haiku-20241022` | Feb 19, 2026 |
|
||||
| Claude Opus 3 | `claude-3-opus-20240229` | Jan 5, 2026 |
|
||||
| Claude Sonnet 3.5 | `claude-3-5-sonnet-20241022` | Oct 28, 2025 |
|
||||
| Claude Sonnet 3.5 | `claude-3-5-sonnet-20240620` | Oct 28, 2025 |
|
||||
| Claude Sonnet 3 | `claude-3-sonnet-20240229` | Jul 21, 2025 |
|
||||
| Claude 2.1 | `claude-2.1` | Jul 21, 2025 |
|
||||
| Claude 2.0 | `claude-2.0` | Jul 21, 2025 |
|
||||
|
||||
## Resolving User Requests
|
||||
|
||||
When a user asks for a model by name, use this table to find the correct model ID:
|
||||
|
||||
| User says... | Use this model ID |
|
||||
|-------------------------------------------|--------------------------------|
|
||||
| "fable", "most capable model" | `claude-fable-5` |
|
||||
| "most powerful" | `claude-fable-5` |
|
||||
| "mythos", "mythos 5" | `claude-mythos-5` (Project Glasswing participants only; otherwise use `claude-fable-5`) |
|
||||
| "mythos preview" | `claude-mythos-5` (successor to `claude-mythos-preview` — see migration guide) |
|
||||
| "opus" | `claude-opus-4-8` |
|
||||
| "opus 4.8" | `claude-opus-4-8` |
|
||||
| "opus 4.7" | `claude-opus-4-7` |
|
||||
| "opus 4.6" | `claude-opus-4-6` |
|
||||
| "opus 4.5" | `claude-opus-4-5` |
|
||||
| "opus 4.1" | `claude-opus-4-1` (deprecated, retires 2026-08-05 — suggest `claude-opus-4-8`) |
|
||||
| "opus 4", "opus 4.0" | `claude-opus-4-0` (deprecated — suggest `claude-opus-4-8`) |
|
||||
| "sonnet", "balanced" | `claude-sonnet-4-6` |
|
||||
| "sonnet 4.6" | `claude-sonnet-4-6` |
|
||||
| "sonnet 4.5" | `claude-sonnet-4-5` |
|
||||
| "sonnet 4", "sonnet 4.0" | `claude-sonnet-4-0` (deprecated — suggest `claude-sonnet-4-6`) |
|
||||
| "sonnet 3.7" | Retired — suggest `claude-sonnet-4-6` |
|
||||
| "sonnet 3.5" | Retired — suggest `claude-sonnet-4-6` |
|
||||
| "haiku", "fast", "cheap" | `claude-haiku-4-5` |
|
||||
| "haiku 4.5" | `claude-haiku-4-5` |
|
||||
| "haiku 3.5" | Retired — suggest `claude-haiku-4-5` |
|
||||
| "haiku 3" | Deprecated — suggest `claude-haiku-4-5` |
|
||||
@@ -1,223 +0,0 @@
|
||||
# Prompt Caching — Design & Optimization
|
||||
|
||||
This file covers how to design prompt-building code for effective caching. For language-specific syntax, see the `## Prompt Caching` section in each language's README or single-file doc.
|
||||
|
||||
## The one invariant everything follows from
|
||||
|
||||
**Prompt caching is a prefix match. Any change anywhere in the prefix invalidates everything after it.**
|
||||
|
||||
The cache key is derived from the exact bytes of the rendered prompt up to each `cache_control` breakpoint. A single byte difference at position N — a timestamp, a reordered JSON key, a different tool in the list — invalidates the cache for all breakpoints at positions ≥ N.
|
||||
|
||||
Render order is: `tools` → `system` → `messages`. A breakpoint on the last system block caches both tools and system together.
|
||||
|
||||
Design the prompt-building path around this constraint. Get the ordering right and most caching works for free. Get it wrong and no amount of `cache_control` markers will help.
|
||||
|
||||
---
|
||||
|
||||
## Workflow for optimizing existing code
|
||||
|
||||
When asked to add or optimize caching:
|
||||
|
||||
1. **Trace the prompt assembly path.** Find where `system`, `tools`, and `messages` are constructed. Identify every input that flows into them.
|
||||
2. **Classify each input by stability:**
|
||||
- Never changes → belongs early in the prompt, before any breakpoint
|
||||
- Changes per-session → belongs after the global prefix, cache per-session
|
||||
- Changes per-turn → belongs at the end, after the last breakpoint
|
||||
- Changes per-request (timestamps, UUIDs, random IDs) → **eliminate or move to the very end**
|
||||
3. **Check rendered order matches stability order.** Stable content must physically precede volatile content. If a timestamp is interpolated into the system prompt header, everything after it is uncacheable regardless of markers.
|
||||
4. **Place breakpoints at stability boundaries.** See placement patterns below.
|
||||
5. **Audit for silent invalidators.** See anti-patterns table.
|
||||
|
||||
---
|
||||
|
||||
## Placement patterns
|
||||
|
||||
### Large system prompt shared across many requests
|
||||
|
||||
Put a breakpoint on the last system text block. If there are tools, they render before system — the marker on the last system block caches tools + system together.
|
||||
|
||||
```json
|
||||
"system": [
|
||||
{"type": "text", "text": "<large shared prompt>", "cache_control": {"type": "ephemeral"}}
|
||||
]
|
||||
```
|
||||
|
||||
### Multi-turn conversations
|
||||
|
||||
Put a breakpoint on the last content block of the most-recently-appended turn. Each subsequent request reuses the entire prior conversation prefix. Earlier breakpoints remain valid read points, so hits accrue incrementally as the conversation grows.
|
||||
|
||||
```json
|
||||
// Last content block of the last user turn
|
||||
messages[-1].content[-1].cache_control = {"type": "ephemeral"}
|
||||
```
|
||||
|
||||
### Shared prefix, varying suffix
|
||||
|
||||
Many requests share a large fixed preamble (few-shot examples, retrieved docs, instructions) but differ in the final question. Put the breakpoint at the end of the **shared** portion, not at the end of the whole prompt — otherwise every request writes a distinct cache entry and nothing is ever read.
|
||||
|
||||
```json
|
||||
"messages": [{"role": "user", "content": [
|
||||
{"type": "text", "text": "<shared context>", "cache_control": {"type": "ephemeral"}},
|
||||
{"type": "text", "text": "<varying question>"} // no marker — differs every time
|
||||
]}]
|
||||
```
|
||||
|
||||
### Mid-conversation system messages
|
||||
|
||||
**Beta, model-gated.** When an operator instruction arrives mid-conversation — a mode switch, updated context, dynamically injected state — send it as `{"role": "system", "content": "..."}` appended to `messages[]`, rather than editing top-level `system`. Editing top-level `system` changes the prefix ahead of the entire conversation history, so every cached turn is re-processed uncached; a `role: "system"` message sits after the history and leaves the cached prefix intact.
|
||||
|
||||
```json
|
||||
// Top-level system stays byte-identical; new instruction goes after the cached history
|
||||
"system": [{"type": "text", "text": "<stable core>", "cache_control": {"type": "ephemeral"}}],
|
||||
"messages": [
|
||||
...history,
|
||||
{"role": "user", "content": "..."},
|
||||
{"role": "system", "content": "Terse mode enabled — keep responses under 40 words."}
|
||||
]
|
||||
```
|
||||
|
||||
This is also the prompt-injection-safe replacement for embedding operator instructions as text inside a user turn (the `<system-reminder>` pattern): both have the same caching profile, but `role: "system"` is the non-spoofable operator channel, whereas text inside user/tool content can be forged by anything that writes to user-visible input.
|
||||
|
||||
Requires `anthropic-beta: mid-conversation-system-2026-04-07`. Must follow a `role: "user"` message (or an assistant message ending in a server tool result); cannot be `messages[0]` — use top-level `system` for the initial prompt. Content is text-only. Model-gated — unsupported models return a 400 (`BadRequestError`: `role 'system' is not supported on this model`); catch that error and fall back to putting the instruction in a user-turn `<system-reminder>` block.
|
||||
|
||||
### Prompts that change from the beginning every time
|
||||
|
||||
Don't cache. If the first 1K tokens differ per request, there is no reusable prefix. Adding `cache_control` only pays the cache-write premium with zero reads. Leave it off.
|
||||
|
||||
---
|
||||
|
||||
## Architectural guidance
|
||||
|
||||
These are the decisions that matter more than marker placement. Fix these first.
|
||||
|
||||
**Keep the system prompt frozen.** Don't interpolate "current date: X", "mode: Y", "user name: Z" into the system prompt — those sit at the front of the prefix and invalidate everything downstream. Inject dynamic context later in `messages` instead — as a `{"role": "system", ...}` message where supported (see § Mid-conversation system messages above), or as text in a user message otherwise. A message at turn 5 invalidates nothing before turn 5.
|
||||
|
||||
**Don't change tools or model mid-conversation.** Tools render at position 0; adding, removing, or reordering a tool invalidates the entire cache. Same for switching models (caches are model-scoped). If you need "modes", don't swap the tool set — give Claude a tool that records the mode transition, or pass the mode as message content. Serialize tools deterministically (sort by name).
|
||||
|
||||
**Fork operations must reuse the parent's exact prefix.** Side computations (summarization, compaction, sub-agents) often spin up a separate API call. If the fork rebuilds `system` / `tools` / `model` with any difference, it misses the parent's cache entirely. Copy the parent's `system`, `tools`, and `model` verbatim, then append fork-specific content at the end.
|
||||
|
||||
---
|
||||
|
||||
## Silent invalidators
|
||||
|
||||
When reviewing code, grep for these inside anything that feeds the prompt prefix:
|
||||
|
||||
| Pattern | Why it breaks caching |
|
||||
|---|---|
|
||||
| `datetime.now()` / `Date.now()` / `time.time()` in system prompt | Prefix changes every request |
|
||||
| `uuid4()` / `crypto.randomUUID()` / request IDs early in content | Same — every request is unique |
|
||||
| `json.dumps(d)` without `sort_keys=True` / iterating a `set` | Non-deterministic serialization → prefix bytes differ |
|
||||
| f-string interpolating session/user ID into system prompt | Per-user prefix; no cross-user sharing |
|
||||
| Conditional system sections (`if flag: system += ...`) | Every flag combination is a distinct prefix |
|
||||
| `tools=build_tools(user)` where set varies per user | Tools render at position 0; nothing caches across users |
|
||||
|
||||
Fix by moving the dynamic piece after the last breakpoint, making it deterministic, or deleting it if it's not load-bearing.
|
||||
|
||||
---
|
||||
|
||||
## API reference
|
||||
|
||||
```json
|
||||
"cache_control": {"type": "ephemeral"} // 5-minute TTL (default)
|
||||
"cache_control": {"type": "ephemeral", "ttl": "1h"} // 1-hour TTL
|
||||
```
|
||||
|
||||
- Max **4** `cache_control` breakpoints per request.
|
||||
- Goes on any content block: system text blocks, tool definitions, message content blocks (`text`, `image`, `tool_use`, `tool_result`, `document`).
|
||||
- Top-level `cache_control` on `messages.create()` auto-places on the last cacheable block — simplest option when you don't need fine-grained placement.
|
||||
- Minimum cacheable prefix is model-dependent. Shorter prefixes silently won't cache even with a marker — no error, just `cache_creation_input_tokens: 0`:
|
||||
|
||||
| Model | Minimum |
|
||||
|---|---:|
|
||||
| Opus 4.8, Opus 4.7, Opus 4.6, Opus 4.5, Haiku 4.5 | 4096 tokens |
|
||||
| Fable 5, Sonnet 4.6, Haiku 3.5, Haiku 3 | 2048 tokens |
|
||||
| Sonnet 4.5, Sonnet 4.1, Sonnet 4, Sonnet 3.7 | 1024 tokens |
|
||||
|
||||
A 3K-token prompt caches on Sonnet 4.5 and Fable 5 but silently won't on Opus 4.8.
|
||||
|
||||
**Economics:** Cache reads cost ~0.1× base input price. Cache writes cost **1.25× for 5-minute TTL, 2× for 1-hour TTL**. Break-even depends on TTL: with 5-minute TTL, two requests break even (1.25× + 0.1× = 1.35× vs 2× uncached); with 1-hour TTL, you need at least three requests (2× + 0.2× = 2.2× vs 3× uncached). The 1-hour TTL keeps entries alive across gaps in bursty traffic, but the doubled write cost means it needs more reads to pay off.
|
||||
|
||||
---
|
||||
|
||||
## Verifying cache hits
|
||||
|
||||
The response `usage` object reports cache activity:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `cache_creation_input_tokens` | Tokens written to cache this request (you paid the ~1.25× write premium) |
|
||||
| `cache_read_input_tokens` | Tokens served from cache this request (you paid ~0.1×) |
|
||||
| `input_tokens` | Tokens processed at full price (not cached) |
|
||||
|
||||
If `cache_read_input_tokens` is zero across repeated requests with identical prefixes, a silent invalidator is at work — diff the rendered prompt bytes between two requests to find it.
|
||||
|
||||
**`input_tokens` is the uncached remainder only.** Total prompt size = `input_tokens + cache_creation_input_tokens + cache_read_input_tokens`. If your agent ran for hours but `input_tokens` shows 4K, the rest was served from cache — check the sum, not the single field.
|
||||
|
||||
Language-specific access: `response.usage.cache_read_input_tokens` (Python/TS/Ruby), `$message->usage->cacheReadInputTokens` (PHP), `resp.Usage.CacheReadInputTokens` (Go/C#), `.usage().cacheReadInputTokens()` (Java).
|
||||
|
||||
---
|
||||
|
||||
## Invalidation hierarchy
|
||||
|
||||
Not every parameter change invalidates everything. The API has three cache tiers, and changes only invalidate their own tier and below:
|
||||
|
||||
| Change | Tools cache | System cache | Messages cache |
|
||||
|---|:---:|:---:|:---:|
|
||||
| Tool definitions (add/remove/reorder) | ❌ | ❌ | ❌ |
|
||||
| Model switch | ❌ | ❌ | ❌ |
|
||||
| `speed`, web-search, citations toggle | ✅ | ❌ | ❌ |
|
||||
| System prompt content | ✅ | ❌ | ❌ |
|
||||
| `tool_choice`, images, `thinking` enable/disable | ✅ | ✅ | ❌ |
|
||||
| Message content | ✅ | ✅ | ❌ |
|
||||
|
||||
Implication: you can change `tool_choice` per-request or toggle `thinking` without losing the tools+system cache. Don't over-worry about these — only tool-definition and model changes force a full rebuild.
|
||||
|
||||
---
|
||||
|
||||
## 20-block lookback window
|
||||
|
||||
Each breakpoint walks backward **at most 20 content blocks** to find a prior cache entry. If a single turn adds more than 20 blocks (common in agentic loops with many tool_use/tool_result pairs), the next request's breakpoint won't find the previous cache and silently misses.
|
||||
|
||||
Fix: place an intermediate breakpoint every ~15 blocks in long turns, or put the marker on a block that's within 20 of the previous turn's last cached block.
|
||||
|
||||
---
|
||||
|
||||
## Concurrent-request timing
|
||||
|
||||
A cache entry becomes readable only after the first response **begins streaming**. N parallel requests with identical prefixes all pay full price — none can read what the others are still writing.
|
||||
|
||||
For fan-out patterns: send 1 request, await the first streamed token (not the full response), then fire the remaining N−1. They'll read the cache the first one just wrote.
|
||||
|
||||
## Pre-warming the cache
|
||||
|
||||
To eliminate the cache-miss latency on the *first* real request, send a **`max_tokens: 0`** request at startup (or on an interval). The API runs prefill — writing the cache at your `cache_control` breakpoint — and returns immediately with `content: []`, `stop_reason: "max_tokens"`, and a populated `usage` block (zero output tokens billed; normal cache-write charge on `cache_creation_input_tokens`).
|
||||
|
||||
**When to pre-warm** — pre-warming trades a cache-write charge *now* for lower TTFT on the *next* real request. It's worth it when all three hold: (a) first-request latency is user-visible (chat/voice/interactive — not background jobs), (b) the shared prefix is large enough that a cold write is noticeably slow, and (c) there's a moment *before* traffic to fire it — app startup, worker boot, post-deploy, start of a scheduled window.
|
||||
|
||||
| Skip pre-warming when… | Because |
|
||||
|---|---|
|
||||
| Traffic is continuous (requests ≤ TTL apart) | The first real request warms the cache and every subsequent one hits it; a separate warm call is a pure extra write |
|
||||
| The prefix is small or below the cacheable minimum | The cold-write penalty is negligible |
|
||||
| The prefix varies per request/user | Nothing shared to pre-warm |
|
||||
| You'd pre-warm many distinct prefixes speculatively | Each is a ~1.25× write; cost can exceed the latency you save |
|
||||
|
||||
**Scheduled re-warms:** only needed when traffic has gaps longer than the TTL. If real requests arrive more often than every 5 minutes, they keep the cache warm on their own — don't add an interval re-warm. For bursty traffic with long idle gaps, either re-warm just under the TTL or switch to `ttl: "1h"` and re-warm less often.
|
||||
|
||||
```python
|
||||
client.messages.create(
|
||||
model="claude-opus-4-8",
|
||||
max_tokens=0,
|
||||
system=[{
|
||||
"type": "text",
|
||||
"text": SYSTEM_PROMPT,
|
||||
"cache_control": {"type": "ephemeral"},
|
||||
}],
|
||||
messages=[{"role": "user", "content": "warmup"}],
|
||||
)
|
||||
```
|
||||
|
||||
**Breakpoint placement:** put `cache_control` on the **last block shared with the real request** (the system prompt or tool definitions) — **not** on the placeholder user message, and **not** via top-level automatic caching (which would key the cache to the placeholder). The placeholder can be any non-whitespace string; it's read during prefill but never answered.
|
||||
|
||||
**Rejected combinations:** `max_tokens: 0` is an `invalid_request_error` with `stream: true`, `thinking.type: "enabled"`, `output_config.format`, `tool_choice` of `{"type":"tool"}` or `{"type":"any"}`, or inside a Message Batches request.
|
||||
|
||||
**TTL still applies** — re-warm at least every 5 minutes for the default cache, or use the 1-hour TTL. This replaces the older `max_tokens: 1` workaround (no single-token reply to discard, no output tokens billed, intent is unambiguous).
|
||||
@@ -1,56 +0,0 @@
|
||||
# Token Counting
|
||||
|
||||
Use the `count_tokens` endpoint (`POST /v1/messages/count_tokens`) for accurate
|
||||
token counts against Claude models. Token counts are **model-specific** — pass
|
||||
the same model ID you'll use for inference.
|
||||
|
||||
**Do not use `tiktoken`.** It's OpenAI's tokenizer. It undercounts Claude
|
||||
tokens by ~15–20% on typical text, and by much more on code or non-English
|
||||
input. Any estimate from `tiktoken`, `gpt-tokenizer`, or similar is wrong for
|
||||
Claude.
|
||||
|
||||
## Count a file or string
|
||||
|
||||
```python
|
||||
from anthropic import Anthropic
|
||||
|
||||
client = Anthropic()
|
||||
resp = client.messages.count_tokens(
|
||||
model="claude-opus-4-8",
|
||||
messages=[{"role": "user", "content": open("CLAUDE.md").read()}],
|
||||
)
|
||||
print(resp.input_tokens)
|
||||
```
|
||||
|
||||
TypeScript: `await client.messages.countTokens({model, messages})` →
|
||||
`.input_tokens`. See `{lang}/claude-api/README.md` for other SDKs.
|
||||
|
||||
## CLI
|
||||
|
||||
```sh
|
||||
ant messages count-tokens --model claude-opus-4-8 \
|
||||
--message '{role: user, content: "@./CLAUDE.md"}' \
|
||||
--transform input_tokens -r
|
||||
```
|
||||
|
||||
## Diffing a file across two versions
|
||||
|
||||
The endpoint is stateless — count each version separately and subtract:
|
||||
|
||||
```python
|
||||
from anthropic import Anthropic
|
||||
import subprocess
|
||||
|
||||
client = Anthropic()
|
||||
def count(text: str) -> int:
|
||||
return client.messages.count_tokens(
|
||||
model="claude-opus-4-8",
|
||||
messages=[{"role": "user", "content": text}],
|
||||
).input_tokens
|
||||
|
||||
before = subprocess.check_output(["git", "show", "HEAD:CLAUDE.md"], text=True)
|
||||
after = open("CLAUDE.md").read()
|
||||
print(count(after) - count(before))
|
||||
```
|
||||
|
||||
Full docs: see the Token Counting entry in `shared/live-sources.md`.
|
||||
@@ -1,347 +0,0 @@
|
||||
# Tool Use Concepts
|
||||
|
||||
This file covers the conceptual foundations of tool use with the Claude API. For language-specific code examples, see the `python/`, `typescript/`, or other language folders. For decision heuristics on which tools to expose, how to manage context in long-running agents, and caching strategy, see `agent-design.md`.
|
||||
|
||||
## User-Defined Tools
|
||||
|
||||
### Tool Definition Structure
|
||||
|
||||
> **Note:** When using the Tool Runner (beta), tool schemas are generated automatically from your function signatures (Python), Zod schemas (TypeScript), annotated classes (Java), `jsonschema` struct tags (Go), or `BaseTool` subclasses (Ruby). The raw JSON schema format below is for the manual approach — including PHP's `BetaRunnableTool`, which wraps a run closure around a hand-written schema — or SDKs without tool runner support.
|
||||
|
||||
Each tool requires a name, description, and JSON Schema for its inputs:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "City and state, e.g., San Francisco, CA"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Best practices for tool definitions:**
|
||||
|
||||
- Use clear, descriptive names (e.g., `get_weather`, `search_database`, `send_email`)
|
||||
- Write detailed descriptions — Claude uses these to decide when to use the tool. Be **prescriptive about *when* to call it**, not just what it does (e.g. "Call this when the user asks about current prices or recent events"). On recent Opus models, which reach for tools more conservatively, trigger conditions in the description give measurable lift in should-call rate.
|
||||
- Include descriptions for each property
|
||||
- Use `enum` for parameters with a fixed set of values
|
||||
- Mark truly required parameters in `required`; make others optional with defaults
|
||||
|
||||
---
|
||||
|
||||
### Tool Choice Options
|
||||
|
||||
Control when Claude uses tools:
|
||||
|
||||
| Value | Behavior |
|
||||
| --------------------------------- | --------------------------------------------- |
|
||||
| `{"type": "auto"}` | Claude decides whether to use tools (default) |
|
||||
| `{"type": "any"}` | Claude must use at least one tool |
|
||||
| `{"type": "tool", "name": "..."}` | Claude must use the specified tool |
|
||||
| `{"type": "none"}` | Claude cannot use tools |
|
||||
|
||||
Any `tool_choice` value can also include `"disable_parallel_tool_use": true` to force Claude to use at most one tool per response. By default, Claude may request multiple tool calls in a single response.
|
||||
|
||||
---
|
||||
|
||||
### Tool Runner vs Manual Loop
|
||||
|
||||
**Tool Runner (Recommended):** The SDK's tool runner handles the agentic loop automatically — it calls the API, detects tool use requests, executes your tool functions, feeds results back to Claude, and repeats until Claude stops calling tools. Available in Python, TypeScript, Java, Go, Ruby, and PHP SDKs (beta). The Python SDK also provides MCP conversion helpers (`anthropic.lib.tools.mcp`) to convert MCP tools, prompts, and resources for use with the tool runner — see `python/claude-api/tool-use.md` for details.
|
||||
|
||||
**Manual Agentic Loop:** Use when you need fine-grained control over the loop (e.g., custom logging, conditional tool execution, human-in-the-loop approval). Loop until `stop_reason == "end_turn"`, always append the full `response.content` to preserve tool_use blocks, and ensure each `tool_result` includes the matching `tool_use_id`.
|
||||
|
||||
**Stop reasons for server-side tools:** When using server-side tools (code execution, web search, etc.), the API runs a server-side sampling loop. If this loop reaches its default limit of 10 iterations, the response will have `stop_reason: "pause_turn"`. To continue, re-send the user message and assistant response and make another API request — the server will resume where it left off. Do NOT add an extra user message like "Continue." — the API detects the trailing `server_tool_use` block and knows to resume automatically.
|
||||
|
||||
```python
|
||||
# Handle pause_turn in your agentic loop
|
||||
if response.stop_reason == "pause_turn":
|
||||
messages = [
|
||||
{"role": "user", "content": user_query},
|
||||
{"role": "assistant", "content": response.content},
|
||||
]
|
||||
# Make another API request — server resumes automatically
|
||||
response = client.messages.create(
|
||||
model="claude-opus-4-8", messages=messages, tools=tools
|
||||
)
|
||||
```
|
||||
|
||||
Set a `max_continuations` limit (e.g., 5) to prevent infinite loops. For the full guide, see: `https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons`
|
||||
|
||||
> **Security:** The tool runner executes your tool functions automatically whenever Claude requests them. For tools with side effects (sending emails, modifying databases, financial transactions), validate inputs within your tool functions and consider requiring confirmation for destructive operations. Use the manual agentic loop if you need human-in-the-loop approval before each tool execution.
|
||||
|
||||
---
|
||||
|
||||
### Handling Tool Results
|
||||
|
||||
When Claude uses a tool, the response contains a `tool_use` block. You must:
|
||||
|
||||
1. Execute the tool with the provided input
|
||||
2. Send the result back in a `tool_result` message
|
||||
3. Continue the conversation
|
||||
|
||||
**Error handling in tool results:** When a tool execution fails, set `"is_error": true` and provide an informative error message. Claude will typically acknowledge the error and either try a different approach or ask for clarification.
|
||||
|
||||
**Multiple tool calls:** Claude can request multiple tools in a single response. Handle them all before continuing — send all results back in a single `user` message.
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools: Code Execution
|
||||
|
||||
The code execution tool lets Claude run code in a secure, sandboxed container. Unlike user-defined tools, server-side tools run on Anthropic's infrastructure — you don't execute anything client-side. Just include the tool definition and Claude handles the rest.
|
||||
|
||||
### Key Facts
|
||||
|
||||
- Runs in an isolated container (1 CPU, 5 GiB RAM, 5 GiB disk)
|
||||
- No internet access (fully sandboxed)
|
||||
- Python 3.11 with data science libraries pre-installed
|
||||
- Containers persist for 30 days and can be reused across requests
|
||||
- Free when used with web search/web fetch tools; otherwise $0.05/hour after 1,550 free hours/month per organization
|
||||
|
||||
### Tool Definition
|
||||
|
||||
The tool requires no schema — just declare it in the `tools` array:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "code_execution_20260120",
|
||||
"name": "code_execution"
|
||||
}
|
||||
```
|
||||
|
||||
Claude automatically gains access to `bash_code_execution` (run shell commands) and `text_editor_code_execution` (create/view/edit files).
|
||||
|
||||
### Pre-installed Python Libraries
|
||||
|
||||
- **Data science**: pandas, numpy, scipy, scikit-learn, statsmodels
|
||||
- **Visualization**: matplotlib, seaborn
|
||||
- **File processing**: openpyxl, xlsxwriter, pillow, pypdf, pdfplumber, python-docx, python-pptx
|
||||
- **Math**: sympy, mpmath
|
||||
- **Utilities**: tqdm, python-dateutil, pytz, sqlite3
|
||||
|
||||
Additional packages can be installed at runtime via `pip install`.
|
||||
|
||||
### Supported File Types for Upload
|
||||
|
||||
| Type | Extensions |
|
||||
| ------ | ---------------------------------- |
|
||||
| Data | CSV, Excel (.xlsx/.xls), JSON, XML |
|
||||
| Images | JPEG, PNG, GIF, WebP |
|
||||
| Text | .txt, .md, .py, .js, etc. |
|
||||
|
||||
### Container Reuse
|
||||
|
||||
Reuse containers across requests to maintain state (files, installed packages, variables). Extract the `container_id` from the first response and pass it to subsequent requests.
|
||||
|
||||
### Response Structure
|
||||
|
||||
The response contains interleaved text and tool result blocks:
|
||||
|
||||
- `text` — Claude's explanation
|
||||
- `server_tool_use` — What Claude is doing
|
||||
- `bash_code_execution_tool_result` — Code execution output (check `return_code` for success/failure)
|
||||
- `text_editor_code_execution_tool_result` — File operation results
|
||||
|
||||
> **Security:** Always sanitize filenames with `os.path.basename()` / `path.basename()` before writing downloaded files to disk to prevent path traversal attacks. Write files to a dedicated output directory.
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools: Web Search and Web Fetch
|
||||
|
||||
Web search and web fetch let Claude search the web and retrieve page content. They run server-side — just include the tool definitions and Claude handles queries, fetching, and result processing automatically.
|
||||
|
||||
### Tool Definitions
|
||||
|
||||
```json
|
||||
[
|
||||
{ "type": "web_search_20260209", "name": "web_search" },
|
||||
{ "type": "web_fetch_20260209", "name": "web_fetch" }
|
||||
]
|
||||
```
|
||||
|
||||
### Dynamic Filtering (Fable 5 / Opus 4.8 / Opus 4.7 / Opus 4.6 / Sonnet 4.6)
|
||||
|
||||
The `web_search_20260209` and `web_fetch_20260209` versions support **dynamic filtering** — Claude writes and executes code to filter search results before they reach the context window, improving accuracy and token efficiency. Dynamic filtering is built into these tool versions and activates automatically; you do not need to separately declare the `code_execution` tool or pass any beta header.
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
{ "type": "web_search_20260209", "name": "web_search" },
|
||||
{ "type": "web_fetch_20260209", "name": "web_fetch" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Without dynamic filtering, the previous `web_search_20250305` version is also available.
|
||||
|
||||
> **Note:** Only include the standalone `code_execution` tool when your application needs code execution for its own purposes (data analysis, file processing, visualization) independent of web search. Including it alongside `_20260209` web tools creates a second execution environment that can confuse the model.
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools: Programmatic Tool Calling
|
||||
|
||||
With standard tool use, each tool call is a round trip: Claude calls, the result enters Claude's context, Claude reasons, then calls the next tool. Chained calls accumulate latency and tokens — most of that intermediate data is never needed again.
|
||||
|
||||
Programmatic tool calling lets Claude compose those calls into a script. The script runs in the code execution container; when it invokes a tool, the container pauses, the call executes, and the result returns to the running code (not to Claude's context). The script processes it with normal control flow. Only the final output returns to Claude. Use it when chaining many tool calls or when intermediate results are large and should be filtered before reaching the context window.
|
||||
|
||||
For full documentation, use WebFetch:
|
||||
|
||||
- URL: `https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling`
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools: Tool Search
|
||||
|
||||
The tool search tool lets Claude dynamically discover tools from large libraries without loading all definitions into the context window. Use it when you have many tools but only a few are relevant to any given request. Discovered tool schemas are appended to the request, not swapped in — this preserves the prompt cache (see `agent-design.md` §Caching for Agents).
|
||||
|
||||
For full documentation, use WebFetch:
|
||||
|
||||
- URL: `https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool`
|
||||
|
||||
---
|
||||
|
||||
## Skills
|
||||
|
||||
Skills package task-specific instructions that Claude loads only when relevant. Each skill is a folder containing a `SKILL.md` file. The skill's short description sits in context by default; Claude reads the full file when the current task calls for it. Use skills to keep specialized instructions out of the base system prompt without losing discoverability.
|
||||
|
||||
For full documentation, use WebFetch:
|
||||
|
||||
- URL: `https://platform.claude.com/docs/en/agents-and-tools/skills`
|
||||
|
||||
---
|
||||
|
||||
## Tool Use Examples
|
||||
|
||||
You can provide sample tool calls directly in your tool definitions to demonstrate usage patterns and reduce parameter errors. This helps Claude understand how to correctly format tool inputs, especially for tools with complex schemas.
|
||||
|
||||
For full documentation, use WebFetch:
|
||||
|
||||
- URL: `https://platform.claude.com/docs/en/agents-and-tools/tool-use/implement-tool-use`
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools: Computer Use
|
||||
|
||||
Computer use lets Claude interact with a desktop environment (screenshots, mouse, keyboard). It can be Anthropic-hosted (server-side, like code execution) or self-hosted (you provide the environment and execute actions client-side).
|
||||
|
||||
For full documentation, use WebFetch:
|
||||
|
||||
- URL: `https://platform.claude.com/docs/en/agents-and-tools/computer-use/overview`
|
||||
|
||||
---
|
||||
|
||||
## Context Editing
|
||||
|
||||
Context editing clears stale tool results and thinking blocks from the transcript as a long-running agent accumulates turns. Unlike compaction (which summarizes), context editing prunes — the cleared content is removed, not replaced. Use it when old tool outputs are no longer relevant and you want to keep the transcript lean without losing the conversation structure. Thresholds for what to clear are configurable.
|
||||
|
||||
For full documentation, use WebFetch:
|
||||
|
||||
- URL: `https://platform.claude.com/docs/en/build-with-claude/context-editing`
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools: Advisor (Beta)
|
||||
|
||||
The advisor tool lets Claude consult a secondary model during a conversation. The advisor runs its own API call with a model you specify and returns its analysis to the primary model. Use it when you want a second opinion, specialized expertise, or cross-model verification without managing the orchestration yourself.
|
||||
|
||||
### Tool Definition
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "claude-sonnet-4-6"
|
||||
}
|
||||
```
|
||||
|
||||
The `model` parameter is required — it specifies which model the advisor uses for its own inference. Optional fields: `caching`, `max_uses`, `allowed_callers`, `defer_loading`, `strict`.
|
||||
|
||||
**Beta header required:** `advisor-tool-2026-03-01`. The SDK sets this automatically when using `client.beta.messages.create()` with advisor tools.
|
||||
|
||||
---
|
||||
|
||||
## Client-Side Tools: Memory
|
||||
|
||||
The memory tool enables Claude to store and retrieve information across conversations through a memory file directory. Claude can create, read, update, and delete files that persist between sessions.
|
||||
|
||||
### Key Facts
|
||||
|
||||
- Client-side tool — you control storage via your implementation
|
||||
- Supports commands: `view`, `create`, `str_replace`, `insert`, `delete`, `rename`
|
||||
- Operates on files in a `/memories` directory
|
||||
- The Python, TypeScript, and Java SDKs provide helper classes/functions for implementing the memory backend
|
||||
|
||||
> **Security:** Never store API keys, passwords, tokens, or other secrets in memory files. Be cautious with personally identifiable information (PII) — check data privacy regulations (GDPR, CCPA) before persisting user data. The reference implementations have no built-in access control; in multi-user systems, implement per-user memory directories and authentication in your tool handlers.
|
||||
|
||||
For full implementation examples, use WebFetch:
|
||||
|
||||
- Docs: `https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool.md`
|
||||
|
||||
---
|
||||
|
||||
## Structured Outputs
|
||||
|
||||
Structured outputs constrain Claude's responses to follow a specific JSON schema, guaranteeing valid, parseable output. This is not a separate tool — it enhances the Messages API response format and/or tool parameter validation.
|
||||
|
||||
Two features are available:
|
||||
|
||||
- **JSON outputs** (`output_config.format`): Control Claude's response format
|
||||
- **Strict tool use** (`strict: true`): Guarantee valid tool parameter schemas
|
||||
|
||||
**Supported models:** Claude Fable 5, Claude Opus 4.8, Claude Sonnet 4.6, and Claude Haiku 4.5. Legacy models (Claude Opus 4.5, Claude Opus 4.1) also support structured outputs.
|
||||
|
||||
> **Recommended:** Use `client.messages.parse()` which automatically validates responses against your schema. When using `messages.create()` directly, use `output_config: {format: {...}}`. The `output_format` convenience parameter is also accepted by some SDK methods (e.g., `.parse()`), but `output_config.format` is the canonical API-level parameter.
|
||||
|
||||
### JSON Schema Limitations
|
||||
|
||||
**Supported:**
|
||||
|
||||
- Basic types: object, array, string, integer, number, boolean, null
|
||||
- `enum`, `const`, `anyOf`, `allOf`, `$ref`/`$def`
|
||||
- String formats: `date-time`, `time`, `date`, `duration`, `email`, `hostname`, `uri`, `ipv4`, `ipv6`, `uuid`
|
||||
- `additionalProperties: false` (required for all objects)
|
||||
|
||||
**Not supported:**
|
||||
|
||||
- Recursive schemas
|
||||
- Numerical constraints (`minimum`, `maximum`, `multipleOf`)
|
||||
- String constraints (`minLength`, `maxLength`)
|
||||
- Complex array constraints
|
||||
- `additionalProperties` set to anything other than `false`
|
||||
|
||||
The Python and TypeScript SDKs automatically handle unsupported constraints by removing them from the schema sent to the API and validating them client-side.
|
||||
|
||||
### Important Notes
|
||||
|
||||
- **First request latency**: New schemas incur a one-time compilation cost. Subsequent requests with the same schema use a 24-hour cache.
|
||||
- **Refusals**: If Claude refuses for safety reasons (`stop_reason: "refusal"`), the output may not match your schema.
|
||||
- **Token limits**: If `stop_reason: "max_tokens"`, output may be incomplete. Increase `max_tokens`.
|
||||
- **Incompatible with**: Citations (returns 400 error), message prefilling.
|
||||
- **Works with**: Batches API, streaming, token counting, extended thinking.
|
||||
|
||||
---
|
||||
|
||||
## Tips for Effective Tool Use
|
||||
|
||||
1. **Provide detailed descriptions**: Claude relies heavily on descriptions to understand when and how to use tools
|
||||
2. **Use specific tool names**: `get_current_weather` is better than `weather`
|
||||
3. **Validate inputs**: Always validate tool inputs before execution
|
||||
4. **Handle errors gracefully**: Return informative error messages so Claude can adapt
|
||||
5. **Limit tool count**: Too many tools can confuse the model — keep the set focused
|
||||
6. **Test tool interactions**: Verify Claude uses tools correctly in various scenarios
|
||||
|
||||
For detailed tool use documentation, use WebFetch:
|
||||
|
||||
- URL: `https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview`
|
||||
@@ -1,372 +0,0 @@
|
||||
# Claude API — TypeScript
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install @anthropic-ai/sdk
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
// Default — resolves credentials from the environment:
|
||||
// ANTHROPIC_API_KEY, or ANTHROPIC_AUTH_TOKEN, or an `ant auth login` profile.
|
||||
// Prefer this for local dev; don't hardcode a key.
|
||||
const client = new Anthropic();
|
||||
|
||||
// Explicit API key (only when you must inject a specific key)
|
||||
const client = new Anthropic({ apiKey: "your-api-key" });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Message Request
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [{ role: "user", content: "What is the capital of France?" }],
|
||||
});
|
||||
// response.content is ContentBlock[] — a discriminated union. Narrow by .type
|
||||
// before accessing .text (TypeScript will error on content[0].text without this).
|
||||
for (const block of response.content) {
|
||||
if (block.type === "text") {
|
||||
console.log(block.text);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## System Prompts
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
system:
|
||||
"You are a helpful coding assistant. Always provide examples in Python.",
|
||||
messages: [{ role: "user", content: "How do I read a JSON file?" }],
|
||||
});
|
||||
```
|
||||
|
||||
### Mid-conversation system messages (beta, model-gated)
|
||||
|
||||
For operator instructions that arrive mid-conversation (mode switches, injected state), append `{role: "system", ...}` to `messages` instead of editing top-level `system` — this preserves the cached prefix and carries operator authority. Must follow a user message; cannot be `messages[0]`. Unsupported models return a 400 (`role 'system' is not supported on this model`). See `shared/prompt-caching.md` for when to use this vs. top-level `system`.
|
||||
|
||||
```typescript
|
||||
// SDK types for role:"system" in messages are pending — pass the beta header
|
||||
// directly until the SDK updates, then switch to client.beta.messages.create
|
||||
// with betas: ["mid-conversation-system-2026-04-07"].
|
||||
const response = await client.messages.create(
|
||||
{
|
||||
model: MODEL_ID, // must support mid-conversation system messages
|
||||
max_tokens: 16000,
|
||||
system: [
|
||||
{ type: "text", text: STABLE_SYSTEM, cache_control: { type: "ephemeral" } },
|
||||
],
|
||||
messages: [
|
||||
...history,
|
||||
{ role: "user", content: userMessage },
|
||||
// @ts-expect-error — role:"system" pending SDK types
|
||||
{ role: "system", content: "Terse mode enabled — keep responses under 40 words." },
|
||||
],
|
||||
},
|
||||
{ headers: { "anthropic-beta": "mid-conversation-system-2026-04-07" } },
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Vision (Images)
|
||||
|
||||
### URL
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{
|
||||
type: "image",
|
||||
source: { type: "url", url: "https://example.com/image.png" },
|
||||
},
|
||||
{ type: "text", text: "Describe this image" },
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
```
|
||||
|
||||
### Base64
|
||||
|
||||
```typescript
|
||||
import fs from "fs";
|
||||
|
||||
const imageData = fs.readFileSync("image.png").toString("base64");
|
||||
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{
|
||||
type: "image",
|
||||
source: { type: "base64", media_type: "image/png", data: imageData },
|
||||
},
|
||||
{ type: "text", text: "What's in this image?" },
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prompt Caching
|
||||
|
||||
**Caching is a prefix match** — any byte change anywhere in the prefix invalidates everything after it. For placement patterns, architectural guidance (frozen system prompt, deterministic tool order, where to put volatile content), and the silent-invalidator audit checklist, read `shared/prompt-caching.md`.
|
||||
|
||||
### Automatic Caching (Recommended)
|
||||
|
||||
Use top-level `cache_control` to automatically cache the last cacheable block in the request:
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
cache_control: { type: "ephemeral" }, // auto-caches the last cacheable block
|
||||
system: "You are an expert on this large document...",
|
||||
messages: [{ role: "user", content: "Summarize the key points" }],
|
||||
});
|
||||
```
|
||||
|
||||
### Manual Cache Control
|
||||
|
||||
For fine-grained control, add `cache_control` to specific content blocks:
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
system: [
|
||||
{
|
||||
type: "text",
|
||||
text: "You are an expert on this large document...",
|
||||
cache_control: { type: "ephemeral" }, // default TTL is 5 minutes
|
||||
},
|
||||
],
|
||||
messages: [{ role: "user", content: "Summarize the key points" }],
|
||||
});
|
||||
|
||||
// With explicit TTL (time-to-live)
|
||||
const response2 = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
system: [
|
||||
{
|
||||
type: "text",
|
||||
text: "You are an expert on this large document...",
|
||||
cache_control: { type: "ephemeral", ttl: "1h" }, // 1 hour TTL
|
||||
},
|
||||
],
|
||||
messages: [{ role: "user", content: "Summarize the key points" }],
|
||||
});
|
||||
```
|
||||
|
||||
### Verifying Cache Hits
|
||||
|
||||
```typescript
|
||||
console.log(response.usage.cache_creation_input_tokens); // tokens written to cache (~1.25x cost)
|
||||
console.log(response.usage.cache_read_input_tokens); // tokens served from cache (~0.1x cost)
|
||||
console.log(response.usage.input_tokens); // uncached tokens (full cost)
|
||||
```
|
||||
|
||||
If `cache_read_input_tokens` is zero across repeated identical-prefix requests, a silent invalidator is at work — `Date.now()` or a UUID in the system prompt, non-deterministic key ordering, or a varying tool set. See `shared/prompt-caching.md` for the full audit table.
|
||||
|
||||
---
|
||||
|
||||
## Extended Thinking
|
||||
|
||||
> **Fable 5, Opus 4.8, Opus 4.7, Opus 4.6, and Sonnet 4.6:** Use adaptive thinking. `budget_tokens` is removed on Fable 5, Opus 4.8, and 4.7 (400 if sent); deprecated on Opus 4.6 and Sonnet 4.6.
|
||||
> **Older models:** Use `thinking: {type: "enabled", budget_tokens: N}` (must be < `max_tokens`, min 1024).
|
||||
|
||||
```typescript
|
||||
// Fable 5 / Opus 4.8 / 4.7 / 4.6: adaptive thinking (recommended)
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
thinking: { type: "adaptive" },
|
||||
output_config: { effort: "high" }, // low | medium | high | max
|
||||
messages: [
|
||||
{ role: "user", content: "Solve this math problem step by step..." },
|
||||
],
|
||||
});
|
||||
|
||||
for (const block of response.content) {
|
||||
if (block.type === "thinking") {
|
||||
console.log("Thinking:", block.thinking);
|
||||
} else if (block.type === "text") {
|
||||
console.log("Response:", block.text);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
Use the SDK's typed exception classes — never check error messages with string matching:
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
try {
|
||||
const response = await client.messages.create({...});
|
||||
} catch (error) {
|
||||
if (error instanceof Anthropic.BadRequestError) {
|
||||
console.error("Bad request:", error.message);
|
||||
} else if (error instanceof Anthropic.AuthenticationError) {
|
||||
console.error("Invalid API key");
|
||||
} else if (error instanceof Anthropic.RateLimitError) {
|
||||
console.error("Rate limited - retry later");
|
||||
} else if (error instanceof Anthropic.APIError) {
|
||||
console.error(`API error ${error.status}:`, error.message);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
All classes extend `Anthropic.APIError` with a typed `status` field. Check from most specific to least specific. See [shared/error-codes.md](../../shared/error-codes.md) for the full error code reference.
|
||||
|
||||
---
|
||||
|
||||
## Multi-Turn Conversations
|
||||
|
||||
The API is stateless — send the full conversation history each time. Use `Anthropic.MessageParam[]` to type the messages array:
|
||||
|
||||
```typescript
|
||||
const messages: Anthropic.MessageParam[] = [
|
||||
{ role: "user", content: "My name is Alice." },
|
||||
{ role: "assistant", content: "Hello Alice! Nice to meet you." },
|
||||
{ role: "user", content: "What's my name?" },
|
||||
];
|
||||
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: messages,
|
||||
});
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
|
||||
- Consecutive same-role messages are allowed — the API combines them into a single turn
|
||||
- First message must be `user`
|
||||
- Use SDK types (`Anthropic.MessageParam`, `Anthropic.Message`, `Anthropic.Tool`, etc.) for all API data structures — don't redefine equivalent interfaces
|
||||
|
||||
---
|
||||
|
||||
### Compaction (long conversations)
|
||||
|
||||
> **Beta, Fable 5, Opus 4.8, Opus 4.7, Opus 4.6, and Sonnet 4.6.** When conversations approach the 200K context window, compaction automatically summarizes earlier context server-side. The API returns a `compaction` block; you must pass it back on subsequent requests — append `response.content`, not just the text.
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
const client = new Anthropic();
|
||||
const messages: Anthropic.Beta.BetaMessageParam[] = [];
|
||||
|
||||
async function chat(userMessage: string): Promise<string> {
|
||||
messages.push({ role: "user", content: userMessage });
|
||||
|
||||
const response = await client.beta.messages.create({
|
||||
betas: ["compact-2026-01-12"],
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages,
|
||||
context_management: {
|
||||
edits: [{ type: "compact_20260112" }],
|
||||
},
|
||||
});
|
||||
|
||||
// Append full content — compaction blocks must be preserved
|
||||
messages.push({ role: "assistant", content: response.content });
|
||||
|
||||
const textBlock = response.content.find(
|
||||
(b): b is Anthropic.Beta.BetaTextBlock => b.type === "text",
|
||||
);
|
||||
return textBlock?.text ?? "";
|
||||
}
|
||||
|
||||
// Compaction triggers automatically when context grows large
|
||||
console.log(await chat("Help me build a Python web scraper"));
|
||||
console.log(await chat("Add support for JavaScript-rendered pages"));
|
||||
console.log(await chat("Now add rate limiting and error handling"));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stop Reasons
|
||||
|
||||
The `stop_reason` field in the response indicates why the model stopped generating:
|
||||
|
||||
| Value | Meaning |
|
||||
| --------------- | --------------------------------------------------------------- |
|
||||
| `end_turn` | Claude finished its response naturally |
|
||||
| `max_tokens` | Hit the `max_tokens` limit — increase it or use streaming |
|
||||
| `stop_sequence` | Hit a custom stop sequence |
|
||||
| `tool_use` | Claude wants to call a tool — execute it and continue |
|
||||
| `pause_turn` | Model paused and can be resumed (agentic flows) |
|
||||
| `refusal` | Claude refused for safety reasons — check `stop_details` |
|
||||
|
||||
### Structured Stop Details
|
||||
|
||||
When `stop_reason` is `"refusal"`, the response includes a `stop_details` object with structured information about the refusal:
|
||||
|
||||
```typescript
|
||||
if (response.stop_reason === "refusal" && response.stop_details) {
|
||||
console.log(`Category: ${response.stop_details.category}`); // "cyber" | "bio" | null
|
||||
console.log(`Explanation: ${response.stop_details.explanation}`);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost Optimization Strategies
|
||||
|
||||
### 1. Use Prompt Caching for Repeated Context
|
||||
|
||||
```typescript
|
||||
// Automatic caching (simplest — caches the last cacheable block)
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
cache_control: { type: "ephemeral" },
|
||||
system: largeDocumentText, // e.g., 50KB of context
|
||||
messages: [{ role: "user", content: "Summarize the key points" }],
|
||||
});
|
||||
|
||||
// First request: full cost
|
||||
// Subsequent requests: ~90% cheaper for cached portion
|
||||
```
|
||||
|
||||
### 2. Use Token Counting Before Requests
|
||||
|
||||
```typescript
|
||||
const countResponse = await client.messages.countTokens({
|
||||
model: "claude-opus-4-8",
|
||||
messages: messages,
|
||||
system: system,
|
||||
});
|
||||
|
||||
const estimatedInputCost = countResponse.input_tokens * 0.000005; // $5/1M tokens
|
||||
console.log(`Estimated input cost: $${estimatedInputCost.toFixed(4)}`);
|
||||
```
|
||||
@@ -1,106 +0,0 @@
|
||||
# Message Batches API — TypeScript
|
||||
|
||||
The Batches API (`POST /v1/messages/batches`) processes Messages API requests asynchronously at 50% of standard prices.
|
||||
|
||||
## Key Facts
|
||||
|
||||
- Up to 100,000 requests or 256 MB per batch
|
||||
- Most batches complete within 1 hour; maximum 24 hours
|
||||
- Results available for 29 days after creation
|
||||
- 50% cost reduction on all token usage
|
||||
- All Messages API features supported (vision, tools, caching, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Create a Batch
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
const client = new Anthropic();
|
||||
|
||||
const messageBatch = await client.messages.batches.create({
|
||||
requests: [
|
||||
{
|
||||
custom_id: "request-1",
|
||||
params: {
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{ role: "user", content: "Summarize climate change impacts" },
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
custom_id: "request-2",
|
||||
params: {
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{ role: "user", content: "Explain quantum computing basics" },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
console.log(`Batch ID: ${messageBatch.id}`);
|
||||
console.log(`Status: ${messageBatch.processing_status}`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Poll for Completion
|
||||
|
||||
```typescript
|
||||
let batch;
|
||||
while (true) {
|
||||
batch = await client.messages.batches.retrieve(messageBatch.id);
|
||||
if (batch.processing_status === "ended") break;
|
||||
console.log(
|
||||
`Status: ${batch.processing_status}, processing: ${batch.request_counts.processing}`,
|
||||
);
|
||||
await new Promise((resolve) => setTimeout(resolve, 60_000));
|
||||
}
|
||||
|
||||
console.log("Batch complete!");
|
||||
console.log(`Succeeded: ${batch.request_counts.succeeded}`);
|
||||
console.log(`Errored: ${batch.request_counts.errored}`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Retrieve Results
|
||||
|
||||
```typescript
|
||||
for await (const result of await client.messages.batches.results(
|
||||
messageBatch.id,
|
||||
)) {
|
||||
switch (result.result.type) {
|
||||
case "succeeded":
|
||||
console.log(
|
||||
`[${result.custom_id}] ${result.result.message.content[0].text.slice(0, 100)}`,
|
||||
);
|
||||
break;
|
||||
case "errored":
|
||||
if (result.result.error.type === "invalid_request") {
|
||||
console.log(`[${result.custom_id}] Validation error - fix and retry`);
|
||||
} else {
|
||||
console.log(`[${result.custom_id}] Server error - safe to retry`);
|
||||
}
|
||||
break;
|
||||
case "expired":
|
||||
console.log(`[${result.custom_id}] Expired - resubmit`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cancel a Batch
|
||||
|
||||
```typescript
|
||||
const cancelled = await client.messages.batches.cancel(messageBatch.id);
|
||||
console.log(`Status: ${cancelled.processing_status}`); // "canceling"
|
||||
```
|
||||
@@ -1,98 +0,0 @@
|
||||
# Files API — TypeScript
|
||||
|
||||
The Files API uploads files for use in Messages API requests. Reference files via `file_id` in content blocks, avoiding re-uploads across multiple API calls.
|
||||
|
||||
**Beta:** Pass `betas: ["files-api-2025-04-14"]` in your API calls (the SDK sets the required header automatically).
|
||||
|
||||
## Key Facts
|
||||
|
||||
- Maximum file size: 500 MB
|
||||
- Total storage: 100 GB per organization
|
||||
- Files persist until deleted
|
||||
- File operations (upload, list, delete) are free; content used in messages is billed as input tokens
|
||||
- Not available on Amazon Bedrock or Google Vertex AI
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
```typescript
|
||||
import Anthropic, { toFile } from "@anthropic-ai/sdk";
|
||||
import fs from "fs";
|
||||
|
||||
const client = new Anthropic();
|
||||
|
||||
const uploaded = await client.beta.files.upload({
|
||||
file: await toFile(fs.createReadStream("report.pdf"), undefined, {
|
||||
type: "application/pdf",
|
||||
}),
|
||||
betas: ["files-api-2025-04-14"],
|
||||
});
|
||||
|
||||
console.log(`File ID: ${uploaded.id}`);
|
||||
console.log(`Size: ${uploaded.size_bytes} bytes`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Use a File in Messages
|
||||
|
||||
### PDF / Text Document
|
||||
|
||||
```typescript
|
||||
const response = await client.beta.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "Summarize the key findings in this report." },
|
||||
{
|
||||
type: "document",
|
||||
source: { type: "file", file_id: uploaded.id },
|
||||
title: "Q4 Report",
|
||||
citations: { enabled: true },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
betas: ["files-api-2025-04-14"],
|
||||
});
|
||||
|
||||
console.log(response.content[0].text);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manage Files
|
||||
|
||||
### List Files
|
||||
|
||||
```typescript
|
||||
const files = await client.beta.files.list({
|
||||
betas: ["files-api-2025-04-14"],
|
||||
});
|
||||
for (const f of files.data) {
|
||||
console.log(`${f.id}: ${f.filename} (${f.size_bytes} bytes)`);
|
||||
}
|
||||
```
|
||||
|
||||
### Delete a File
|
||||
|
||||
```typescript
|
||||
await client.beta.files.delete("file_011CNha8iCJcU1wXNR6q4V8w", {
|
||||
betas: ["files-api-2025-04-14"],
|
||||
});
|
||||
```
|
||||
|
||||
### Download a File
|
||||
|
||||
```typescript
|
||||
const response = await client.beta.files.download(
|
||||
"file_011CNha8iCJcU1wXNR6q4V8w",
|
||||
{ betas: ["files-api-2025-04-14"] },
|
||||
);
|
||||
const content = Buffer.from(await response.arrayBuffer());
|
||||
await fs.promises.writeFile("output.txt", content);
|
||||
```
|
||||
@@ -1,178 +0,0 @@
|
||||
# Streaming — TypeScript
|
||||
|
||||
## Quick Start
|
||||
|
||||
```typescript
|
||||
const stream = client.messages.stream({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 64000,
|
||||
messages: [{ role: "user", content: "Write a story" }],
|
||||
});
|
||||
|
||||
for await (const event of stream) {
|
||||
if (
|
||||
event.type === "content_block_delta" &&
|
||||
event.delta.type === "text_delta"
|
||||
) {
|
||||
process.stdout.write(event.delta.text);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Handling Different Content Types
|
||||
|
||||
> **Fable 5 / Opus 4.8 / Opus 4.7 / Opus 4.6:** Use `thinking: {type: "adaptive"}`. On older models, use `thinking: {type: "enabled", budget_tokens: N}` instead.
|
||||
|
||||
```typescript
|
||||
const stream = client.messages.stream({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 64000,
|
||||
thinking: { type: "adaptive" },
|
||||
messages: [{ role: "user", content: "Analyze this problem" }],
|
||||
});
|
||||
|
||||
for await (const event of stream) {
|
||||
switch (event.type) {
|
||||
case "content_block_start":
|
||||
switch (event.content_block.type) {
|
||||
case "thinking":
|
||||
console.log("\n[Thinking...]");
|
||||
break;
|
||||
case "text":
|
||||
console.log("\n[Response:]");
|
||||
break;
|
||||
}
|
||||
break;
|
||||
case "content_block_delta":
|
||||
switch (event.delta.type) {
|
||||
case "thinking_delta":
|
||||
process.stdout.write(event.delta.thinking);
|
||||
break;
|
||||
case "text_delta":
|
||||
process.stdout.write(event.delta.text);
|
||||
break;
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Streaming with Tool Use (Tool Runner)
|
||||
|
||||
Use the tool runner with `stream: true`. The outer loop iterates over tool runner iterations (messages), the inner loop processes stream events:
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
import { betaZodTool } from "@anthropic-ai/sdk/helpers/beta/zod";
|
||||
import { z } from "zod";
|
||||
|
||||
const client = new Anthropic();
|
||||
|
||||
const getWeather = betaZodTool({
|
||||
name: "get_weather",
|
||||
description: "Get current weather for a location",
|
||||
inputSchema: z.object({
|
||||
location: z.string().describe("City and state, e.g., San Francisco, CA"),
|
||||
}),
|
||||
run: async ({ location }) => `72°F and sunny in ${location}`,
|
||||
});
|
||||
|
||||
const runner = client.beta.messages.toolRunner({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 64000,
|
||||
tools: [getWeather],
|
||||
messages: [
|
||||
{ role: "user", content: "What's the weather in Paris and London?" },
|
||||
],
|
||||
stream: true,
|
||||
});
|
||||
|
||||
// Outer loop: each tool runner iteration
|
||||
for await (const messageStream of runner) {
|
||||
// Inner loop: stream events for this iteration
|
||||
for await (const event of messageStream) {
|
||||
switch (event.type) {
|
||||
case "content_block_delta":
|
||||
switch (event.delta.type) {
|
||||
case "text_delta":
|
||||
process.stdout.write(event.delta.text);
|
||||
break;
|
||||
case "input_json_delta":
|
||||
// Tool input being streamed
|
||||
break;
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting the Final Message
|
||||
|
||||
```typescript
|
||||
const stream = client.messages.stream({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 64000,
|
||||
messages: [{ role: "user", content: "Hello" }],
|
||||
});
|
||||
|
||||
for await (const event of stream) {
|
||||
// Process events...
|
||||
}
|
||||
|
||||
const finalMessage = await stream.finalMessage();
|
||||
console.log(`Tokens used: ${finalMessage.usage.output_tokens}`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stream Event Types
|
||||
|
||||
| Event Type | Description | When it fires |
|
||||
| --------------------- | --------------------------- | --------------------------------- |
|
||||
| `message_start` | Contains message metadata | Once at the beginning |
|
||||
| `content_block_start` | New content block beginning | When a text/tool_use block starts |
|
||||
| `content_block_delta` | Incremental content update | For each token/chunk |
|
||||
| `content_block_stop` | Content block complete | When a block finishes |
|
||||
| `message_delta` | Message-level updates | Contains `stop_reason`, usage |
|
||||
| `message_stop` | Message complete | Once at the end |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always flush output** — Use `process.stdout.write()` for immediate display
|
||||
2. **Handle partial responses** — If the stream is interrupted, you may have incomplete content
|
||||
3. **Track token usage** — The `message_delta` event contains usage information
|
||||
4. **Use `finalMessage()`** — Get the complete `Anthropic.Message` object even when streaming. Don't wrap `.on()` events in `new Promise()` — `finalMessage()` handles all completion/error/abort states internally
|
||||
5. **Buffer for web UIs** — Consider buffering a few tokens before rendering to avoid excessive DOM updates
|
||||
6. **Use `stream.on("text", ...)` for deltas** — The `text` event provides just the delta string, simpler than manually filtering `content_block_delta` events
|
||||
7. **For agentic loops with streaming** — See the [Streaming Manual Loop](./tool-use.md#streaming-manual-loop) section in tool-use.md for combining `stream()` + `finalMessage()` with a tool-use loop
|
||||
|
||||
## Raw SSE Format
|
||||
|
||||
If using raw HTTP (not SDKs), the stream returns Server-Sent Events:
|
||||
|
||||
```
|
||||
event: message_start
|
||||
data: {"type":"message_start","message":{"id":"msg_...","type":"message",...}}
|
||||
|
||||
event: content_block_start
|
||||
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}
|
||||
|
||||
event: content_block_stop
|
||||
data: {"type":"content_block_stop","index":0}
|
||||
|
||||
event: message_delta
|
||||
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":12}}
|
||||
|
||||
event: message_stop
|
||||
data: {"type":"message_stop"}
|
||||
```
|
||||
@@ -1,527 +0,0 @@
|
||||
# Tool Use — TypeScript
|
||||
|
||||
For conceptual overview (tool definitions, tool choice, tips), see [shared/tool-use-concepts.md](../../shared/tool-use-concepts.md).
|
||||
|
||||
## Tool Runner (Recommended)
|
||||
|
||||
**Beta:** The tool runner is in beta in the TypeScript SDK.
|
||||
|
||||
Use `betaZodTool` with Zod schemas to define tools with a `run` function, then pass them to `client.beta.messages.toolRunner()`:
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
import { betaZodTool } from "@anthropic-ai/sdk/helpers/beta/zod";
|
||||
import { z } from "zod";
|
||||
|
||||
const client = new Anthropic();
|
||||
|
||||
const getWeather = betaZodTool({
|
||||
name: "get_weather",
|
||||
description: "Get current weather for a location",
|
||||
inputSchema: z.object({
|
||||
location: z.string().describe("City and state, e.g., San Francisco, CA"),
|
||||
unit: z.enum(["celsius", "fahrenheit"]).optional(),
|
||||
}),
|
||||
run: async (input) => {
|
||||
// Your implementation here
|
||||
return `72°F and sunny in ${input.location}`;
|
||||
},
|
||||
});
|
||||
|
||||
// The tool runner handles the agentic loop and returns the final message
|
||||
const finalMessage = await client.beta.messages.toolRunner({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: [getWeather],
|
||||
messages: [{ role: "user", content: "What's the weather in Paris?" }],
|
||||
});
|
||||
|
||||
console.log(finalMessage.content);
|
||||
```
|
||||
|
||||
**Key benefits of the tool runner:**
|
||||
|
||||
- No manual loop — the SDK handles calling tools and feeding results back
|
||||
- Type-safe tool inputs via Zod schemas
|
||||
- Tool schemas are generated automatically from Zod definitions
|
||||
- Iteration stops automatically when Claude has no more tool calls
|
||||
|
||||
---
|
||||
|
||||
## Manual Agentic Loop
|
||||
|
||||
Use this when you need fine-grained control (custom logging, conditional tool execution, streaming individual iterations, human-in-the-loop approval):
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
const client = new Anthropic();
|
||||
const tools: Anthropic.Tool[] = [...]; // Your tool definitions
|
||||
let messages: Anthropic.MessageParam[] = [{ role: "user", content: userInput }];
|
||||
|
||||
while (true) {
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: tools,
|
||||
messages: messages,
|
||||
});
|
||||
|
||||
if (response.stop_reason === "end_turn") break;
|
||||
|
||||
// Server-side tool hit iteration limit; append assistant turn and re-send to continue
|
||||
if (response.stop_reason === "pause_turn") {
|
||||
messages.push({ role: "assistant", content: response.content });
|
||||
continue;
|
||||
}
|
||||
|
||||
const toolUseBlocks = response.content.filter(
|
||||
(b): b is Anthropic.ToolUseBlock => b.type === "tool_use",
|
||||
);
|
||||
|
||||
messages.push({ role: "assistant", content: response.content });
|
||||
|
||||
const toolResults: Anthropic.ToolResultBlockParam[] = [];
|
||||
for (const tool of toolUseBlocks) {
|
||||
const result = await executeTool(tool.name, tool.input);
|
||||
toolResults.push({
|
||||
type: "tool_result",
|
||||
tool_use_id: tool.id,
|
||||
content: result,
|
||||
});
|
||||
}
|
||||
|
||||
messages.push({ role: "user", content: toolResults });
|
||||
}
|
||||
```
|
||||
|
||||
### Streaming Manual Loop
|
||||
|
||||
Use `client.messages.stream()` + `finalMessage()` instead of `.create()` when you need streaming within a manual loop. Text deltas are streamed on each iteration; `finalMessage()` collects the complete `Message` so you can inspect `stop_reason` and extract tool-use blocks:
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
const client = new Anthropic();
|
||||
const tools: Anthropic.Tool[] = [...];
|
||||
let messages: Anthropic.MessageParam[] = [{ role: "user", content: userInput }];
|
||||
|
||||
while (true) {
|
||||
const stream = client.messages.stream({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 64000,
|
||||
tools,
|
||||
messages,
|
||||
});
|
||||
|
||||
// Stream text deltas on each iteration
|
||||
stream.on("text", (delta) => {
|
||||
process.stdout.write(delta);
|
||||
});
|
||||
|
||||
// finalMessage() resolves with the complete Message — no need to
|
||||
// manually wire up .on("message") / .on("error") / .on("abort")
|
||||
const message = await stream.finalMessage();
|
||||
|
||||
if (message.stop_reason === "end_turn") break;
|
||||
|
||||
// Server-side tool hit iteration limit; append assistant turn and re-send to continue
|
||||
if (message.stop_reason === "pause_turn") {
|
||||
messages.push({ role: "assistant", content: message.content });
|
||||
continue;
|
||||
}
|
||||
|
||||
const toolUseBlocks = message.content.filter(
|
||||
(b): b is Anthropic.ToolUseBlock => b.type === "tool_use",
|
||||
);
|
||||
|
||||
messages.push({ role: "assistant", content: message.content });
|
||||
|
||||
const toolResults: Anthropic.ToolResultBlockParam[] = [];
|
||||
for (const tool of toolUseBlocks) {
|
||||
const result = await executeTool(tool.name, tool.input);
|
||||
toolResults.push({
|
||||
type: "tool_result",
|
||||
tool_use_id: tool.id,
|
||||
content: result,
|
||||
});
|
||||
}
|
||||
|
||||
messages.push({ role: "user", content: toolResults });
|
||||
}
|
||||
```
|
||||
|
||||
> **Important:** Don't wrap `.on()` events in `new Promise()` to collect the final message — use `stream.finalMessage()` instead. The SDK handles all error/abort/completion states internally.
|
||||
|
||||
> **Error handling in the loop:** Use the SDK's typed exceptions (e.g., `Anthropic.RateLimitError`, `Anthropic.APIError`) — see [Error Handling](./README.md#error-handling) for examples. Don't check error messages with string matching.
|
||||
|
||||
> **SDK types:** Use `Anthropic.MessageParam`, `Anthropic.Tool`, `Anthropic.ToolUseBlock`, `Anthropic.ToolResultBlockParam`, `Anthropic.Message`, etc. for all API-related data structures. Don't redefine equivalent interfaces.
|
||||
|
||||
---
|
||||
|
||||
## Handling Tool Results
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: tools,
|
||||
messages: [{ role: "user", content: "What's the weather in Paris?" }],
|
||||
});
|
||||
|
||||
for (const block of response.content) {
|
||||
if (block.type === "tool_use") {
|
||||
const result = await executeTool(block.name, block.input);
|
||||
|
||||
const followup = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: tools,
|
||||
messages: [
|
||||
{ role: "user", content: "What's the weather in Paris?" },
|
||||
{ role: "assistant", content: response.content },
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "tool_result", tool_use_id: block.id, content: result },
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tool Choice
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: tools,
|
||||
tool_choice: { type: "tool", name: "get_weather" },
|
||||
messages: [{ role: "user", content: "What's the weather in Paris?" }],
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Server-Side Tools
|
||||
|
||||
Version-suffixed `type` literals; `name` is fixed per interface. Pass plain object literals — the `ToolUnion` type is satisfied structurally. **The `name`/`type` pair must match the interface**: mixing `str_replace_based_edit_tool` (20250728 name) with `text_editor_20250124` (which expects `str_replace_editor`) is a TS2322.
|
||||
|
||||
**Don't type-annotate as `Tool[]`** — `Tool` is just the custom-tool variant. Let structural typing infer from the `tools` param, or annotate as `Anthropic.Messages.ToolUnion[]` if you must:
|
||||
|
||||
```typescript
|
||||
// ✓ let inference work — no annotation
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: [
|
||||
{ type: "text_editor_20250728", name: "str_replace_based_edit_tool" },
|
||||
{ type: "bash_20250124", name: "bash" },
|
||||
{ type: "web_search_20260209", name: "web_search" },
|
||||
{ type: "code_execution_20260120", name: "code_execution" },
|
||||
],
|
||||
messages: [{ role: "user", content: "..." }],
|
||||
});
|
||||
|
||||
// ✗ this is a TS2352 — Tool is the CUSTOM tool variant only
|
||||
// const tools: Anthropic.Tool[] = [{ type: "text_editor_20250728", ... }]
|
||||
```
|
||||
|
||||
| Interface | `name` | `type` |
|
||||
|---|---|---|
|
||||
| `ToolTextEditor20250124` | `str_replace_editor` | `text_editor_20250124` |
|
||||
| `ToolTextEditor20250429` | `str_replace_based_edit_tool` | `text_editor_20250429` |
|
||||
| `ToolTextEditor20250728` | `str_replace_based_edit_tool` | `text_editor_20250728` |
|
||||
| `ToolBash20250124` | `bash` | `bash_20250124` |
|
||||
| `WebSearchTool20260209` | `web_search` | `web_search_20260209` |
|
||||
| `WebFetchTool20260209` | `web_fetch` | `web_fetch_20260209` |
|
||||
| `CodeExecutionTool20260120` | `code_execution` | `code_execution_20260120` |
|
||||
|
||||
**Don't mix beta and non-beta types**: if you call `client.beta.messages.create()`, the response `content` is `BetaContentBlock[]` — you cannot pass that to a non-beta `ContentBlockParam[]` without narrowing each element.
|
||||
|
||||
---
|
||||
|
||||
|
||||
## Code Execution
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
const client = new Anthropic();
|
||||
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content:
|
||||
"Calculate the mean and standard deviation of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]",
|
||||
},
|
||||
],
|
||||
tools: [{ type: "code_execution_20260120", name: "code_execution" }],
|
||||
});
|
||||
```
|
||||
|
||||
### Reading Local Files (ESM note)
|
||||
|
||||
`__dirname` doesn't exist in ES modules. For script-relative paths use `import.meta.url`:
|
||||
|
||||
```typescript
|
||||
import { readFileSync } from "fs";
|
||||
import { fileURLToPath } from "url";
|
||||
import { dirname, join } from "path";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const pdfBytes = readFileSync(join(__dirname, "sample.pdf"));
|
||||
```
|
||||
|
||||
Or use a CWD-relative path if the script runs from a known directory: `readFileSync("./sample.pdf")`.
|
||||
|
||||
### Upload Files for Analysis
|
||||
|
||||
```typescript
|
||||
import Anthropic, { toFile } from "@anthropic-ai/sdk";
|
||||
import { createReadStream } from "fs";
|
||||
|
||||
const client = new Anthropic();
|
||||
|
||||
// 1. Upload a file
|
||||
const uploaded = await client.beta.files.upload({
|
||||
file: await toFile(createReadStream("sales_data.csv"), undefined, {
|
||||
type: "text/csv",
|
||||
}),
|
||||
betas: ["files-api-2025-04-14"],
|
||||
});
|
||||
|
||||
// 2. Pass to code execution
|
||||
// Code execution is GA; Files API is still beta (pass via RequestOptions)
|
||||
const response = await client.messages.create(
|
||||
{
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{
|
||||
type: "text",
|
||||
text: "Analyze this sales data. Show trends and create a visualization.",
|
||||
},
|
||||
{ type: "container_upload", file_id: uploaded.id },
|
||||
],
|
||||
},
|
||||
],
|
||||
tools: [{ type: "code_execution_20260120", name: "code_execution" }],
|
||||
},
|
||||
{ headers: { "anthropic-beta": "files-api-2025-04-14" } },
|
||||
);
|
||||
```
|
||||
|
||||
### Retrieve Generated Files
|
||||
|
||||
```typescript
|
||||
import path from "path";
|
||||
import fs from "fs";
|
||||
|
||||
const OUTPUT_DIR = "./claude_outputs";
|
||||
await fs.promises.mkdir(OUTPUT_DIR, { recursive: true });
|
||||
|
||||
for (const block of response.content) {
|
||||
if (block.type === "bash_code_execution_tool_result") {
|
||||
const result = block.content;
|
||||
if (result.type === "bash_code_execution_result" && result.content) {
|
||||
for (const fileRef of result.content) {
|
||||
if (fileRef.type === "bash_code_execution_output") {
|
||||
const metadata = await client.beta.files.retrieveMetadata(
|
||||
fileRef.file_id,
|
||||
);
|
||||
const downloadResponse = await client.beta.files.download(fileRef.file_id);
|
||||
const fileBytes = Buffer.from(await downloadResponse.arrayBuffer());
|
||||
const safeName = path.basename(metadata.filename);
|
||||
if (!safeName || safeName === "." || safeName === "..") {
|
||||
console.warn(`Skipping invalid filename: ${metadata.filename}`);
|
||||
continue;
|
||||
}
|
||||
const outputPath = path.join(OUTPUT_DIR, safeName);
|
||||
await fs.promises.writeFile(outputPath, fileBytes);
|
||||
console.log(`Saved: ${outputPath}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Container Reuse
|
||||
|
||||
```typescript
|
||||
// First request: set up environment
|
||||
const response1 = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: "Install tabulate and create data.json with sample user data",
|
||||
},
|
||||
],
|
||||
tools: [{ type: "code_execution_20260120", name: "code_execution" }],
|
||||
});
|
||||
|
||||
// Reuse container
|
||||
// container is nullable — set only when using server-side code execution
|
||||
const containerId = response1.container!.id;
|
||||
|
||||
const response2 = await client.messages.create({
|
||||
container: containerId,
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: "Read data.json and display as a formatted table",
|
||||
},
|
||||
],
|
||||
tools: [{ type: "code_execution_20260120", name: "code_execution" }],
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Memory Tool
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: "Remember that my preferred language is TypeScript.",
|
||||
},
|
||||
],
|
||||
tools: [{ type: "memory_20250818", name: "memory" }],
|
||||
});
|
||||
```
|
||||
|
||||
### SDK Memory Helper
|
||||
|
||||
Use `betaMemoryTool` with a `MemoryToolHandlers` implementation:
|
||||
|
||||
```typescript
|
||||
import {
|
||||
betaMemoryTool,
|
||||
type MemoryToolHandlers,
|
||||
} from "@anthropic-ai/sdk/helpers/beta/memory";
|
||||
|
||||
const handlers: MemoryToolHandlers = {
|
||||
async view(command) { ... },
|
||||
async create(command) { ... },
|
||||
async str_replace(command) { ... },
|
||||
async insert(command) { ... },
|
||||
async delete(command) { ... },
|
||||
async rename(command) { ... },
|
||||
};
|
||||
|
||||
const memory = betaMemoryTool(handlers);
|
||||
|
||||
const runner = client.beta.messages.toolRunner({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
tools: [memory],
|
||||
messages: [{ role: "user", content: "Remember my preferences" }],
|
||||
});
|
||||
|
||||
for await (const message of runner) {
|
||||
console.log(message);
|
||||
}
|
||||
```
|
||||
|
||||
For full implementation examples, use WebFetch:
|
||||
|
||||
- `https://github.com/anthropics/anthropic-sdk-typescript/blob/main/examples/tools-helpers-memory.ts`
|
||||
|
||||
---
|
||||
|
||||
## Structured Outputs
|
||||
|
||||
### JSON Outputs (Zod — Recommended)
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
import { z } from "zod";
|
||||
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
|
||||
|
||||
const ContactInfoSchema = z.object({
|
||||
name: z.string(),
|
||||
email: z.string(),
|
||||
plan: z.string(),
|
||||
interests: z.array(z.string()),
|
||||
demo_requested: z.boolean(),
|
||||
});
|
||||
|
||||
const client = new Anthropic();
|
||||
|
||||
const response = await client.messages.parse({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content:
|
||||
"Extract: Jane Doe (jane@co.com) wants Enterprise, interested in API and SDKs, wants a demo.",
|
||||
},
|
||||
],
|
||||
output_config: {
|
||||
format: zodOutputFormat(ContactInfoSchema),
|
||||
},
|
||||
});
|
||||
|
||||
// parsed_output is null if parsing failed — assert or guard
|
||||
console.log(response.parsed_output!.name); // "Jane Doe"
|
||||
```
|
||||
|
||||
### Strict Tool Use
|
||||
|
||||
```typescript
|
||||
const response = await client.messages.create({
|
||||
model: "claude-opus-4-8",
|
||||
max_tokens: 16000,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: "Book a flight to Tokyo for 2 passengers on March 15",
|
||||
},
|
||||
],
|
||||
tools: [
|
||||
{
|
||||
name: "book_flight",
|
||||
description: "Book a flight to a destination",
|
||||
strict: true,
|
||||
input_schema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
destination: { type: "string" },
|
||||
date: { type: "string", format: "date" },
|
||||
passengers: {
|
||||
type: "integer",
|
||||
enum: [1, 2, 3, 4, 5, 6, 7, 8],
|
||||
},
|
||||
},
|
||||
required: ["destination", "date", "passengers"],
|
||||
additionalProperties: false,
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
```
|
||||
@@ -1,357 +0,0 @@
|
||||
# Managed Agents — TypeScript
|
||||
|
||||
> **Bindings not shown here:** This README covers the most common managed-agents flows for TypeScript. If you need a class, method, namespace, field, or behavior that isn't shown, WebFetch the TypeScript SDK repo **or the relevant docs page** from `shared/live-sources.md` rather than guess. Do not extrapolate from cURL shapes or another language's SDK.
|
||||
|
||||
> **Agents are persistent — create once, reference by ID.** Store the agent ID returned by `agents.create` and pass it to every subsequent `sessions.create`; do not call `agents.create` in the request path. The Anthropic CLI is one convenient way to create agents and environments from version-controlled YAML — its URL is in `shared/live-sources.md`. The examples below show in-code creation for completeness; in production the create call belongs in setup, not in the request path.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install @anthropic-ai/sdk
|
||||
```
|
||||
|
||||
## Client Initialization
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
// Default — resolves credentials from the environment:
|
||||
// ANTHROPIC_API_KEY, or ANTHROPIC_AUTH_TOKEN, or an `ant auth login` profile.
|
||||
// Prefer this for local dev; don't hardcode a key.
|
||||
const client = new Anthropic();
|
||||
|
||||
// Explicit API key (only when you must inject a specific key)
|
||||
const client = new Anthropic({ apiKey: "your-api-key" });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Environment
|
||||
|
||||
```typescript
|
||||
const environment = await client.beta.environments.create(
|
||||
{
|
||||
name: "my-dev-env",
|
||||
config: {
|
||||
type: "cloud",
|
||||
networking: { type: "unrestricted" },
|
||||
},
|
||||
},
|
||||
);
|
||||
console.log(environment.id); // env_...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Create an Agent (required first step)
|
||||
|
||||
> ⚠️ **There is no inline agent config.** `model`/`system`/`tools` live on the agent object, not the session. Always start with `agents.create()` — the session only takes `agent: { type: "agent", id: agent.id }`.
|
||||
|
||||
### Minimal
|
||||
|
||||
```typescript
|
||||
// 1. Create the agent (reusable, versioned)
|
||||
const agent = await client.beta.agents.create(
|
||||
{
|
||||
name: "Coding Assistant",
|
||||
model: "claude-opus-4-8",
|
||||
tools: [{ type: "agent_toolset_20260401", default_config: { enabled: true } }],
|
||||
},
|
||||
);
|
||||
|
||||
// 2. Start a session
|
||||
const session = await client.beta.sessions.create(
|
||||
{
|
||||
agent: { type: "agent", id: agent.id, version: agent.version },
|
||||
environment_id: environment.id,
|
||||
},
|
||||
);
|
||||
console.log(session.id, session.status);
|
||||
```
|
||||
|
||||
### With system prompt and custom tools
|
||||
|
||||
```typescript
|
||||
const agent = await client.beta.agents.create(
|
||||
{
|
||||
name: "Code Reviewer",
|
||||
model: "claude-opus-4-8",
|
||||
system: "You are a senior code reviewer.",
|
||||
tools: [
|
||||
{ type: "agent_toolset_20260401", default_config: { enabled: true } },
|
||||
{
|
||||
type: "custom",
|
||||
name: "run_tests",
|
||||
description: "Run the test suite",
|
||||
input_schema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
test_path: { type: "string", description: "Path to test file" },
|
||||
},
|
||||
required: ["test_path"],
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
|
||||
const session = await client.beta.sessions.create(
|
||||
{
|
||||
agent: { type: "agent", id: agent.id, version: agent.version },
|
||||
environment_id: environment.id,
|
||||
title: "Code review session",
|
||||
resources: [
|
||||
{
|
||||
type: "github_repository",
|
||||
url: "https://github.com/owner/repo",
|
||||
mount_path: "/workspace/repo",
|
||||
authorization_token: process.env.GITHUB_TOKEN,
|
||||
branch: "main",
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Send a User Message
|
||||
|
||||
```typescript
|
||||
await client.beta.sessions.events.send(
|
||||
session.id,
|
||||
{
|
||||
events: [
|
||||
{
|
||||
type: "user.message",
|
||||
content: [{ type: "text", text: "Review the auth module" }],
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
> 💡 **Stream-first:** Open the stream *before* (or concurrently with) sending the message. The stream only delivers events that occur after it opens — stream-after-send means early events arrive buffered in one batch. See [Steering Patterns](../../shared/managed-agents-events.md#steering-patterns).
|
||||
|
||||
---
|
||||
|
||||
## Stream Events (SSE)
|
||||
|
||||
```typescript
|
||||
// Stream-first: open stream and send concurrently
|
||||
const [events] = await Promise.all([
|
||||
collectStream(session.id),
|
||||
client.beta.sessions.events.send(
|
||||
session.id,
|
||||
{ events: [{ type: "user.message", content: [{ type: "text", text: "..." }] }] },
|
||||
),
|
||||
]);
|
||||
|
||||
// Standalone stream iteration:
|
||||
const stream = await client.beta.sessions.events.stream(
|
||||
session.id,
|
||||
);
|
||||
|
||||
for await (const event of stream) {
|
||||
switch (event.type) {
|
||||
case "agent.message":
|
||||
for (const block of event.content) {
|
||||
if (block.type === "text") {
|
||||
process.stdout.write(block.text);
|
||||
}
|
||||
}
|
||||
break;
|
||||
case "agent.custom_tool_use":
|
||||
// Custom tool invocation — session is now idle
|
||||
console.log(`\nCustom tool call: ${event.name}`);
|
||||
console.log(`Input: ${JSON.stringify(event.input)}`);
|
||||
break;
|
||||
case "session.status_idle":
|
||||
console.log("\n--- Agent idle ---");
|
||||
break;
|
||||
case "session.status_terminated":
|
||||
console.log("\n--- Session terminated ---");
|
||||
break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provide Custom Tool Result
|
||||
|
||||
```typescript
|
||||
await client.beta.sessions.events.send(
|
||||
session.id,
|
||||
{
|
||||
events: [
|
||||
{
|
||||
type: "user.custom_tool_result",
|
||||
custom_tool_use_id: "sevt_abc123",
|
||||
content: [{ type: "text", text: "All 42 tests passed." }],
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Poll Events
|
||||
|
||||
```typescript
|
||||
const events = await client.beta.sessions.events.list(
|
||||
session.id,
|
||||
);
|
||||
for (const event of events.data) {
|
||||
console.log(`${event.type}: ${event.id}`);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full Streaming Loop with Custom Tools
|
||||
|
||||
```typescript
|
||||
function runCustomTool(toolName: string, toolInput: unknown): string {
|
||||
if (toolName === "run_tests") {
|
||||
// Your tool implementation here
|
||||
return "All tests passed.";
|
||||
}
|
||||
return `Unknown tool: ${toolName}`;
|
||||
}
|
||||
|
||||
async function runSession(client: Anthropic, sessionId: string) {
|
||||
while (true) {
|
||||
const stream = await client.beta.sessions.events.stream(
|
||||
sessionId,
|
||||
);
|
||||
|
||||
const toolCalls: Anthropic.Beta.Sessions.BetaManagedAgentsAgentCustomToolUseEvent[] = [];
|
||||
|
||||
for await (const event of stream) {
|
||||
if (event.type === "agent.message") {
|
||||
for (const block of event.content) {
|
||||
if (block.type === "text") {
|
||||
process.stdout.write(block.text);
|
||||
}
|
||||
}
|
||||
} else if (event.type === "agent.custom_tool_use") {
|
||||
toolCalls.push(event);
|
||||
} else if (event.type === "session.status_idle") {
|
||||
break;
|
||||
} else if (event.type === "session.status_terminated") {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if (toolCalls.length === 0) break;
|
||||
|
||||
// Process custom tool calls
|
||||
const results = toolCalls.map((call) => ({
|
||||
type: "user.custom_tool_result" as const,
|
||||
custom_tool_use_id: call.id,
|
||||
content: [{ type: "text" as const, text: runCustomTool(call.name, call.input) }],
|
||||
}));
|
||||
|
||||
await client.beta.sessions.events.send(
|
||||
sessionId,
|
||||
{ events: results },
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload a File
|
||||
|
||||
```typescript
|
||||
import fs from "fs";
|
||||
|
||||
const file = await client.beta.files.upload({
|
||||
file: fs.createReadStream("data.csv"),
|
||||
});
|
||||
|
||||
// Use in a session
|
||||
const session = await client.beta.sessions.create(
|
||||
{
|
||||
agent: { type: "agent", id: agent.id, version: agent.version },
|
||||
environment_id: environment.id,
|
||||
resources: [{ type: "file", file_id: file.id, mount_path: "/workspace/data.csv" }],
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## List and Download Session Files
|
||||
|
||||
List files the agent wrote to `/mnt/session/outputs/` during a session, then download them.
|
||||
|
||||
```typescript
|
||||
import fs from "fs";
|
||||
|
||||
// List files associated with a session
|
||||
const files = await client.beta.files.list({
|
||||
scope_id: session.id,
|
||||
betas: ["managed-agents-2026-04-01"],
|
||||
});
|
||||
for (const f of files.data) {
|
||||
console.log(f.filename, f.size_bytes);
|
||||
|
||||
// Download and save to disk
|
||||
const resp = await client.beta.files.download(f.id);
|
||||
const buffer = Buffer.from(await resp.arrayBuffer());
|
||||
fs.writeFileSync(f.filename, buffer);
|
||||
}
|
||||
```
|
||||
|
||||
> 💡 There's a brief indexing lag (~1–3s) between `session.status_idle` and output files appearing in `files.list`. Retry once or twice if the list is empty.
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
```typescript
|
||||
// Get session details
|
||||
const session = await client.beta.sessions.retrieve("sesn_011CZxAbc123Def456");
|
||||
console.log(session.status, session.usage);
|
||||
|
||||
// List sessions
|
||||
const sessions = await client.beta.sessions.list();
|
||||
|
||||
// Delete a session
|
||||
await client.beta.sessions.delete("sesn_011CZxAbc123Def456");
|
||||
|
||||
// Archive a session
|
||||
await client.beta.sessions.archive("sesn_011CZxAbc123Def456");
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Integration
|
||||
|
||||
```typescript
|
||||
// Agent declares MCP server (no auth here — auth goes in a vault)
|
||||
const agent = await client.beta.agents.create({
|
||||
name: "MCP Agent",
|
||||
model: "claude-opus-4-8",
|
||||
mcp_servers: [
|
||||
{ type: "url", name: "my-tools", url: "https://my-mcp-server.example.com/sse" },
|
||||
],
|
||||
tools: [
|
||||
{ type: "agent_toolset_20260401", default_config: { enabled: true } },
|
||||
{ type: "mcp_toolset", mcp_server_name: "my-tools" },
|
||||
],
|
||||
});
|
||||
|
||||
// Session attaches vault(s) containing credentials for those MCP server URLs
|
||||
const session = await client.beta.sessions.create({
|
||||
agent: agent.id,
|
||||
environment_id: environment.id,
|
||||
vault_ids: [vault.id],
|
||||
});
|
||||
```
|
||||
|
||||
See `shared/managed-agents-tools.md` §Vaults for creating vaults and adding credentials.
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: docx
|
||||
description: "Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files). Triggers include: any mention of 'Word doc', 'word document', '.docx', or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting or reorganizing content from .docx files, inserting or replacing images in documents, performing find-and-replace in Word files, working with tracked changes or comments, or converting content into a polished Word document. If the user asks for a 'report', 'memo', 'letter', 'template', or similar deliverable as a Word or .docx file, use this skill. Do NOT use for PDFs, spreadsheets, Google Docs, or general coding tasks unrelated to document generation."
|
||||
description: "Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files). Triggers include: any mention of \"Word doc\", \"word document\", \".docx\", or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting or reorganizing content from .docx files, inserting or replacing images in documents, performing find-and-replace in Word files, working with tracked changes or comments, or converting content into a polished Word document. If the user asks for a \"report\", \"memo\", \"letter\", \"template\", or similar deliverable as a Word or .docx file, use this skill. Do NOT use for PDFs, spreadsheets, Google Docs, or general coding tasks unrelated to document generation."
|
||||
license: Proprietary. LICENSE.txt has complete terms
|
||||
---
|
||||
|
||||
@@ -61,9 +61,6 @@ Generate .docx files with JavaScript, then validate. Install: `npm install -g do
|
||||
```javascript
|
||||
const { Document, Packer, Paragraph, TextRun, Table, TableRow, TableCell, ImageRun,
|
||||
Header, Footer, AlignmentType, PageOrientation, LevelFormat, ExternalHyperlink,
|
||||
InternalHyperlink, Bookmark, FootnoteReferenceRun, PositionalTab,
|
||||
PositionalTabAlignment, PositionalTabRelativeTo, PositionalTabLeader,
|
||||
TabStopType, TabStopPosition, Column, SectionType,
|
||||
TableOfContents, HeadingLevel, BorderStyle, WidthType, ShadingType,
|
||||
VerticalAlign, PageNumber, PageBreak } = require('docx');
|
||||
|
||||
@@ -244,111 +241,6 @@ new Paragraph({ children: [new PageBreak()] })
|
||||
new Paragraph({ pageBreakBefore: true, children: [new TextRun("New page")] })
|
||||
```
|
||||
|
||||
### Hyperlinks
|
||||
|
||||
```javascript
|
||||
// External link
|
||||
new Paragraph({
|
||||
children: [new ExternalHyperlink({
|
||||
children: [new TextRun({ text: "Click here", style: "Hyperlink" })],
|
||||
link: "https://example.com",
|
||||
})]
|
||||
})
|
||||
|
||||
// Internal link (bookmark + reference)
|
||||
// 1. Create bookmark at destination
|
||||
new Paragraph({ heading: HeadingLevel.HEADING_1, children: [
|
||||
new Bookmark({ id: "chapter1", children: [new TextRun("Chapter 1")] }),
|
||||
]})
|
||||
// 2. Link to it
|
||||
new Paragraph({ children: [new InternalHyperlink({
|
||||
children: [new TextRun({ text: "See Chapter 1", style: "Hyperlink" })],
|
||||
anchor: "chapter1",
|
||||
})]})
|
||||
```
|
||||
|
||||
### Footnotes
|
||||
|
||||
```javascript
|
||||
const doc = new Document({
|
||||
footnotes: {
|
||||
1: { children: [new Paragraph("Source: Annual Report 2024")] },
|
||||
2: { children: [new Paragraph("See appendix for methodology")] },
|
||||
},
|
||||
sections: [{
|
||||
children: [new Paragraph({
|
||||
children: [
|
||||
new TextRun("Revenue grew 15%"),
|
||||
new FootnoteReferenceRun(1),
|
||||
new TextRun(" using adjusted metrics"),
|
||||
new FootnoteReferenceRun(2),
|
||||
],
|
||||
})]
|
||||
}]
|
||||
});
|
||||
```
|
||||
|
||||
### Tab Stops
|
||||
|
||||
```javascript
|
||||
// Right-align text on same line (e.g., date opposite a title)
|
||||
new Paragraph({
|
||||
children: [
|
||||
new TextRun("Company Name"),
|
||||
new TextRun("\tJanuary 2025"),
|
||||
],
|
||||
tabStops: [{ type: TabStopType.RIGHT, position: TabStopPosition.MAX }],
|
||||
})
|
||||
|
||||
// Dot leader (e.g., TOC-style)
|
||||
new Paragraph({
|
||||
children: [
|
||||
new TextRun("Introduction"),
|
||||
new TextRun({ children: [
|
||||
new PositionalTab({
|
||||
alignment: PositionalTabAlignment.RIGHT,
|
||||
relativeTo: PositionalTabRelativeTo.MARGIN,
|
||||
leader: PositionalTabLeader.DOT,
|
||||
}),
|
||||
"3",
|
||||
]}),
|
||||
],
|
||||
})
|
||||
```
|
||||
|
||||
### Multi-Column Layouts
|
||||
|
||||
```javascript
|
||||
// Equal-width columns
|
||||
sections: [{
|
||||
properties: {
|
||||
column: {
|
||||
count: 2, // number of columns
|
||||
space: 720, // gap between columns in DXA (720 = 0.5 inch)
|
||||
equalWidth: true,
|
||||
separate: true, // vertical line between columns
|
||||
},
|
||||
},
|
||||
children: [/* content flows naturally across columns */]
|
||||
}]
|
||||
|
||||
// Custom-width columns (equalWidth must be false)
|
||||
sections: [{
|
||||
properties: {
|
||||
column: {
|
||||
equalWidth: false,
|
||||
children: [
|
||||
new Column({ width: 5400, space: 720 }),
|
||||
new Column({ width: 3240 }),
|
||||
],
|
||||
},
|
||||
},
|
||||
children: [/* content */]
|
||||
}]
|
||||
```
|
||||
|
||||
Force a column break with a new section using `type: SectionType.NEXT_COLUMN`.
|
||||
|
||||
### Table of Contents
|
||||
|
||||
```javascript
|
||||
@@ -388,7 +280,6 @@ sections: [{
|
||||
- **Table width = sum of columnWidths** - for DXA, ensure they add up exactly
|
||||
- **Always add cell margins** - use `margins: { top: 80, bottom: 80, left: 120, right: 120 }` for readable padding
|
||||
- **Use `ShadingType.CLEAR`** - never SOLID for table shading
|
||||
- **Never use tables as dividers/rules** - cells have minimum height and render as empty boxes (including in headers/footers); use `border: { bottom: { style: BorderStyle.SINGLE, size: 6, color: "2E75B6", space: 1 } }` on a Paragraph instead. For two-column footers, use tab stops (see Tab Stops section), not tables
|
||||
- **TOC requires HeadingLevel only** - no custom styles on heading paragraphs
|
||||
- **Override built-in styles** - use exact IDs: "Heading1", "Heading2", etc.
|
||||
- **Include `outlineLevel`** - required for TOC (0 for H1, 1 for H2, etc.)
|
||||
|
||||
0
skills/docx/scripts/accept_changes.py
Executable file → Normal file
0
skills/docx/scripts/accept_changes.py
Executable file → Normal file
0
skills/docx/scripts/comment.py
Executable file → Normal file
0
skills/docx/scripts/comment.py
Executable file → Normal file
@@ -1,55 +1,42 @@
|
||||
---
|
||||
name: frontend-design
|
||||
description: Guidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.
|
||||
description: Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML/CSS layouts, or when styling/beautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.
|
||||
license: Complete terms in LICENSE.txt
|
||||
---
|
||||
|
||||
# Frontend Design
|
||||
This skill guides creation of distinctive, production-grade frontend interfaces that avoid generic "AI slop" aesthetics. Implement real working code with exceptional attention to aesthetic details and creative choices.
|
||||
|
||||
Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's. This client has already rejected proposals that felt templated, and is paying for a distinctive point of view: make deliberate, opinionated choices about palette, typography, and layout that are specific to this brief, and take one real aesthetic risk you can justify.
|
||||
The user provides frontend requirements: a component, page, application, or interface to build. They may include context about the purpose, audience, or technical constraints.
|
||||
|
||||
## Ground it in the subject
|
||||
## Design Thinking
|
||||
|
||||
If the brief does not pin down what the product or subject is, pin it yourself before designing: name one concrete subject, its audience, and the page's single job, and state your choice. If there's any information in your memory about the human's preferences, context about what they're building, or designs you've made before – use that as a hint. The subject's own world, its materials, instruments, artifacts, and vernacular, is where distinctive choices come from. Build with the brief's real content and subject matter throughout.
|
||||
Before coding, understand the context and commit to a BOLD aesthetic direction:
|
||||
- **Purpose**: What problem does this interface solve? Who uses it?
|
||||
- **Tone**: Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined, playful/toy-like, editorial/magazine, brutalist/raw, art deco/geometric, soft/pastel, industrial/utilitarian, etc. There are so many flavors to choose from. Use these for inspiration but design one that is true to the aesthetic direction.
|
||||
- **Constraints**: Technical requirements (framework, performance, accessibility).
|
||||
- **Differentiation**: What makes this UNFORGETTABLE? What's the one thing someone will remember?
|
||||
|
||||
## Design principles
|
||||
**CRITICAL**: Choose a clear conceptual direction and execute it with precision. Bold maximalism and refined minimalism both work - the key is intentionality, not intensity.
|
||||
|
||||
For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment. Be deliberate with your choice: a big number with a small label, supporting stats, and a gradient accent is the template answer, only use if that's truly the best option.
|
||||
Then implement working code (HTML/CSS/JS, React, Vue, etc.) that is:
|
||||
- Production-grade and functional
|
||||
- Visually striking and memorable
|
||||
- Cohesive with a clear aesthetic point-of-view
|
||||
- Meticulously refined in every detail
|
||||
|
||||
Typography carries the personality of the page. Pair the display and body faces deliberately, not the same families you would reach for on any other project, and set a clear type scale with intentional weights, widths, and spacing. Make the type treatment itself a memorable part of the design, not a neutral delivery vehicle for the content.
|
||||
## Frontend Aesthetics Guidelines
|
||||
|
||||
Structure is information. Structural devices, numbering, eyebrows, dividers, labels, should encode something true about the content, not decorate it. Many generic designs use numbered markers (01 / 02 / 03), but that's only appropriate if the content actually is a sequence - like a real process or a typed timeline where order carries information the reader needs. Question if choices like numbered markers actually make sense before incorporating them.
|
||||
Focus on:
|
||||
- **Typography**: Choose fonts that are beautiful, unique, and interesting. Avoid generic fonts like Arial and Inter; opt instead for distinctive choices that elevate the frontend's aesthetics; unexpected, characterful font choices. Pair a distinctive display font with a refined body font.
|
||||
- **Color & Theme**: Commit to a cohesive aesthetic. Use CSS variables for consistency. Dominant colors with sharp accents outperform timid, evenly-distributed palettes.
|
||||
- **Motion**: Use animations for effects and micro-interactions. Prioritize CSS-only solutions for HTML. Use Motion library for React when available. Focus on high-impact moments: one well-orchestrated page load with staggered reveals (animation-delay) creates more delight than scattered micro-interactions. Use scroll-triggering and hover states that surprise.
|
||||
- **Spatial Composition**: Unexpected layouts. Asymmetry. Overlap. Diagonal flow. Grid-breaking elements. Generous negative space OR controlled density.
|
||||
- **Backgrounds & Visual Details**: Create atmosphere and depth rather than defaulting to solid colors. Add contextual effects and textures that match the overall aesthetic. Apply creative forms like gradient meshes, noise textures, geometric patterns, layered transparencies, dramatic shadows, decorative borders, custom cursors, and grain overlays.
|
||||
|
||||
Leverage motion deliberately. Think about where and if animation can serve the subject: a page-load sequence, a scroll-triggered reveal, hover micro-interactions, ambient atmosphere. An orchestrated moment usually lands harder than scattered effects; choose what the direction calls for. However, sometimes less is more, and extra animation contributes to the feeling that the design is AI-generated.
|
||||
NEVER use generic AI-generated aesthetics like overused font families (Inter, Roboto, Arial, system fonts), cliched color schemes (particularly purple gradients on white backgrounds), predictable layouts and component patterns, and cookie-cutter design that lacks context-specific character.
|
||||
|
||||
Match complexity to the vision. Maximalist directions need elaborate execution; minimal directions need precision in spacing, type, and detail. Elegance is executing the chosen vision well.
|
||||
Interpret creatively and make unexpected choices that feel genuinely designed for the context. No design should be the same. Vary between light and dark themes, different fonts, different aesthetics. NEVER converge on common choices (Space Grotesk, for example) across generations.
|
||||
|
||||
Consider written content carefully. Often a design brief may not contain real content, and it's up to you to come up with copy. Copy can make a design feel as templated as the design itself. See the below section on writing for more guidance.
|
||||
**IMPORTANT**: Match implementation complexity to the aesthetic vision. Maximalist designs need elaborate code with extensive animations and effects. Minimalist or refined designs need restraint, precision, and careful attention to spacing, typography, and subtle details. Elegance comes from executing the vision well.
|
||||
|
||||
## Process: brainstorm, explore, plan, critique, build, critique again
|
||||
|
||||
For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns. All three are legitimate for some briefs, but they are defaults rather than choices, and they appear regardless of subject. Where the brief pins down a visual direction, follow it exactly — the brief's own words always win, including when it asks for one of these looks. Where it leaves an axis free, don't spend that freedom on one of these defaults. Just like a human designer who's hired, there's often a careful balance between doing what you're good at and taking each project as a chance to experiment and learn.
|
||||
|
||||
Work in two passes. First, brainstorm a short design plan based on the human's design brief: create a compact token system with color, type, layout, and signature. Color: describe the palette as 4–6 named hex values. Type: the typefaces for 2+ roles (a characterful display face that's used with restraint, a complementary body face, and a utility face for captions or data if needed). Layout: a layout concept, using one-sentence prose descriptions and ASCII wireframes to ideate and compare. Signature: the single unique element this page will be remembered by that embodies the brief in an appropriate way.
|
||||
|
||||
Then review that plan against the brief before building: if any part of it reads like the generic default you would produce for any similar page (work through a similar prompt to see if you arrive somewhere similar) rather than a choice made for this specific brief — revise that part, say what you changed and why. Only after you've confirmed the relative uniqueness of your design plan should you start to write the code, following the revised plan exactly and deriving every color and type decision from it.
|
||||
|
||||
When writing the code, be careful of structuring your CSS selector specificities. It's easy to generate CSS classes that cancel each other out (especially with a type-based selector like .section and a element-based selector like .cta). This can happen often with paddings/margins between sections.
|
||||
|
||||
Try to do a lot of this planning and iteration in your thinking, and only show ideas to the user when you have higher confidence it'll delight them.
|
||||
|
||||
## Restraint and self-critique
|
||||
|
||||
Spend your boldness in one place. Let the signature element be the one memorable thing, keep everything around it quiet and disciplined, and cut any decoration that does not serve the brief. Not taking a risk can be a risk itself! Build to a quality floor without announcing it: responsive down to mobile, visible keyboard focus, reduced motion respected. Critique your own work as you build, taking screenshots if your environment supports it – a picture is worth 1000 tokens. Consider Chanel's advice: before leaving the house, take a look in the mirror and remove one accessory. Human creators have memory and always try to do something new, so if you have a space to quickly jot down notes about what you've tried, it can help you in future passes.
|
||||
|
||||
## More on writing in design
|
||||
|
||||
Words appear in a design for one reason: to make it easier to understand, and therefore easier to use. They are design material, not decoration. Bring the same intentionality to copy that you would bring to spacing and color. Before writing anything, ask what the design needs to say, and how it can best be said to help the person navigate the experience.
|
||||
|
||||
Write from the end user's side of the screen. Name things by what people control and recognize, never by how the system is built. A person manages notifications, not webhook config. Describe what something does in plain terms rather than selling it. Being specific is always better than being clever.
|
||||
|
||||
Use active voice as default. A control should say exactly what happens when it's used: "Save changes," not "Submit." An action keeps the same name through the whole flow, so the button that says "Publish" produces a toast that says "Published." The vocabulary of an interface is the signposting for someone navigating the product. Cohesion and consistency are how people learn their way around.
|
||||
|
||||
Treat failure and emptiness as moments for direction, not mood. Explain what went wrong and how to fix it, in the interface's voice rather than a person's. Errors don't apologize, and they are never vague about what happened. An empty screen is an invitation to act.
|
||||
|
||||
Keep the register conversational and tuned: plain verbs, sentence case, no filler, with tone matched to the brand and the audience. Let each element do exactly one job. A label labels, an example demonstrates, and nothing quietly does double duty.
|
||||
Remember: Claude is capable of extraordinary creative work. Don't hold back, show what can truly be created when thinking outside the box and committing fully to a distinctive vision.
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
0
skills/pdf/scripts/extract_form_structure.py
Executable file → Normal file
0
skills/pdf/scripts/extract_form_structure.py
Executable file → Normal file
0
skills/pptx/scripts/add_slide.py
Executable file → Normal file
0
skills/pptx/scripts/add_slide.py
Executable file → Normal file
0
skills/pptx/scripts/thumbnail.py
Executable file → Normal file
0
skills/pptx/scripts/thumbnail.py
Executable file → Normal file
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -1,485 +1,356 @@
|
||||
---
|
||||
name: skill-creator
|
||||
description: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
|
||||
description: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.
|
||||
license: Complete terms in LICENSE.txt
|
||||
---
|
||||
|
||||
# Skill Creator
|
||||
|
||||
A skill for creating new skills and iteratively improving them.
|
||||
This skill provides guidance for creating effective skills.
|
||||
|
||||
At a high level, the process of creating a skill goes like this:
|
||||
## About Skills
|
||||
|
||||
- Decide what you want the skill to do and roughly how it should do it
|
||||
- Write a draft of the skill
|
||||
- Create a few test prompts and run claude-with-access-to-the-skill on them
|
||||
- Help the user evaluate the results both qualitatively and quantitatively
|
||||
- While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)
|
||||
- Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics
|
||||
- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)
|
||||
- Repeat until you're satisfied
|
||||
- Expand the test set and try again at larger scale
|
||||
Skills are modular, self-contained packages that extend Claude's capabilities by providing
|
||||
specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific
|
||||
domains or tasks—they transform Claude from a general-purpose agent into a specialized agent
|
||||
equipped with procedural knowledge that no model can fully possess.
|
||||
|
||||
Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like "I want to make a skill for X". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.
|
||||
### What Skills Provide
|
||||
|
||||
On the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.
|
||||
1. Specialized workflows - Multi-step procedures for specific domains
|
||||
2. Tool integrations - Instructions for working with specific file formats or APIs
|
||||
3. Domain expertise - Company-specific knowledge, schemas, business logic
|
||||
4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks
|
||||
|
||||
Of course, you should always be flexible and if the user is like "I don't need to run a bunch of evaluations, just vibe with me", you can do that instead.
|
||||
## Core Principles
|
||||
|
||||
Then after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.
|
||||
### Concise is Key
|
||||
|
||||
Cool? Cool.
|
||||
The context window is a public good. Skills share the context window with everything else Claude needs: system prompt, conversation history, other Skills' metadata, and the actual user request.
|
||||
|
||||
## Communicating with the user
|
||||
**Default assumption: Claude is already very smart.** Only add context Claude doesn't already have. Challenge each piece of information: "Does Claude really need this explanation?" and "Does this paragraph justify its token cost?"
|
||||
|
||||
The skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google "how to install npm". On the other hand, the bulk of users are probably fairly computer-literate.
|
||||
Prefer concise examples over verbose explanations.
|
||||
|
||||
So please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:
|
||||
### Set Appropriate Degrees of Freedom
|
||||
|
||||
- "evaluation" and "benchmark" are borderline, but OK
|
||||
- for "JSON" and "assertion" you want to see serious cues from the user that they know what those things are before using them without explaining them
|
||||
Match the level of specificity to the task's fragility and variability:
|
||||
|
||||
It's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.
|
||||
**High freedom (text-based instructions)**: Use when multiple approaches are valid, decisions depend on context, or heuristics guide the approach.
|
||||
|
||||
---
|
||||
**Medium freedom (pseudocode or scripts with parameters)**: Use when a preferred pattern exists, some variation is acceptable, or configuration affects behavior.
|
||||
|
||||
## Creating a skill
|
||||
**Low freedom (specific scripts, few parameters)**: Use when operations are fragile and error-prone, consistency is critical, or a specific sequence must be followed.
|
||||
|
||||
### Capture Intent
|
||||
Think of Claude as exploring a path: a narrow bridge with cliffs needs specific guardrails (low freedom), while an open field allows many routes (high freedom).
|
||||
|
||||
Start by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say "turn this into a skill"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.
|
||||
### Anatomy of a Skill
|
||||
|
||||
1. What should this skill enable Claude to do?
|
||||
2. When should this skill trigger? (what user phrases/contexts)
|
||||
3. What's the expected output format?
|
||||
4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.
|
||||
|
||||
### Interview and Research
|
||||
|
||||
Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.
|
||||
|
||||
Check available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.
|
||||
|
||||
### Write the SKILL.md
|
||||
|
||||
Based on the user interview, fill in these components:
|
||||
|
||||
- **name**: Skill identifier
|
||||
- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All "when to use" info goes here, not in the body. Note: currently Claude has a tendency to "undertrigger" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit "pushy". So for instance, instead of "How to build a simple fast dashboard to display internal Anthropic data.", you might write "How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'"
|
||||
- **compatibility**: Required tools, dependencies (optional, rarely needed)
|
||||
- **the rest of the skill :)**
|
||||
|
||||
### Skill Writing Guide
|
||||
|
||||
#### Anatomy of a Skill
|
||||
Every skill consists of a required SKILL.md file and optional bundled resources:
|
||||
|
||||
```
|
||||
skill-name/
|
||||
├── SKILL.md (required)
|
||||
│ ├── YAML frontmatter (name, description required)
|
||||
│ └── Markdown instructions
|
||||
│ ├── YAML frontmatter metadata (required)
|
||||
│ │ ├── name: (required)
|
||||
│ │ └── description: (required)
|
||||
│ └── Markdown instructions (required)
|
||||
└── Bundled Resources (optional)
|
||||
├── scripts/ - Executable code for deterministic/repetitive tasks
|
||||
├── references/ - Docs loaded into context as needed
|
||||
└── assets/ - Files used in output (templates, icons, fonts)
|
||||
├── scripts/ - Executable code (Python/Bash/etc.)
|
||||
├── references/ - Documentation intended to be loaded into context as needed
|
||||
└── assets/ - Files used in output (templates, icons, fonts, etc.)
|
||||
```
|
||||
|
||||
#### Progressive Disclosure
|
||||
#### SKILL.md (required)
|
||||
|
||||
Skills use a three-level loading system:
|
||||
1. **Metadata** (name + description) - Always in context (~100 words)
|
||||
2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)
|
||||
3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)
|
||||
Every SKILL.md consists of:
|
||||
|
||||
These word counts are approximate and you can feel free to go longer if needed.
|
||||
- **Frontmatter** (YAML): Contains `name` and `description` fields. These are the only fields that Claude reads to determine when the skill gets used, thus it is very important to be clear and comprehensive in describing what the skill is, and when it should be used.
|
||||
- **Body** (Markdown): Instructions and guidance for using the skill. Only loaded AFTER the skill triggers (if at all).
|
||||
|
||||
**Key patterns:**
|
||||
- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.
|
||||
- Reference files clearly from SKILL.md with guidance on when to read them
|
||||
- For large reference files (>300 lines), include a table of contents
|
||||
#### Bundled Resources (optional)
|
||||
|
||||
##### Scripts (`scripts/`)
|
||||
|
||||
Executable code (Python/Bash/etc.) for tasks that require deterministic reliability or are repeatedly rewritten.
|
||||
|
||||
- **When to include**: When the same code is being rewritten repeatedly or deterministic reliability is needed
|
||||
- **Example**: `scripts/rotate_pdf.py` for PDF rotation tasks
|
||||
- **Benefits**: Token efficient, deterministic, may be executed without loading into context
|
||||
- **Note**: Scripts may still need to be read by Claude for patching or environment-specific adjustments
|
||||
|
||||
##### References (`references/`)
|
||||
|
||||
Documentation and reference material intended to be loaded as needed into context to inform Claude's process and thinking.
|
||||
|
||||
- **When to include**: For documentation that Claude should reference while working
|
||||
- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template, `references/policies.md` for company policies, `references/api_docs.md` for API specifications
|
||||
- **Use cases**: Database schemas, API documentation, domain knowledge, company policies, detailed workflow guides
|
||||
- **Benefits**: Keeps SKILL.md lean, loaded only when Claude determines it's needed
|
||||
- **Best practice**: If files are large (>10k words), include grep search patterns in SKILL.md
|
||||
- **Avoid duplication**: Information should live in either SKILL.md or references files, not both. Prefer references files for detailed information unless it's truly core to the skill—this keeps SKILL.md lean while making information discoverable without hogging the context window. Keep only essential procedural instructions and workflow guidance in SKILL.md; move detailed reference material, schemas, and examples to references files.
|
||||
|
||||
##### Assets (`assets/`)
|
||||
|
||||
Files not intended to be loaded into context, but rather used within the output Claude produces.
|
||||
|
||||
- **When to include**: When the skill needs files that will be used in the final output
|
||||
- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate, `assets/font.ttf` for typography
|
||||
- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified
|
||||
- **Benefits**: Separates output resources from documentation, enables Claude to use files without loading them into context
|
||||
|
||||
#### What to Not Include in a Skill
|
||||
|
||||
A skill should only contain essential files that directly support its functionality. Do NOT create extraneous documentation or auxiliary files, including:
|
||||
|
||||
- README.md
|
||||
- INSTALLATION_GUIDE.md
|
||||
- QUICK_REFERENCE.md
|
||||
- CHANGELOG.md
|
||||
- etc.
|
||||
|
||||
The skill should only contain the information needed for an AI agent to do the job at hand. It should not contain auxilary context about the process that went into creating it, setup and testing procedures, user-facing documentation, etc. Creating additional documentation files just adds clutter and confusion.
|
||||
|
||||
### Progressive Disclosure Design Principle
|
||||
|
||||
Skills use a three-level loading system to manage context efficiently:
|
||||
|
||||
1. **Metadata (name + description)** - Always in context (~100 words)
|
||||
2. **SKILL.md body** - When skill triggers (<5k words)
|
||||
3. **Bundled resources** - As needed by Claude (Unlimited because scripts can be executed without reading into context window)
|
||||
|
||||
#### Progressive Disclosure Patterns
|
||||
|
||||
Keep SKILL.md body to the essentials and under 500 lines to minimize context bloat. Split content into separate files when approaching this limit. When splitting out content into other files, it is very important to reference them from SKILL.md and describe clearly when to read them, to ensure the reader of the skill knows they exist and when to use them.
|
||||
|
||||
**Key principle:** When a skill supports multiple variations, frameworks, or options, keep only the core workflow and selection guidance in SKILL.md. Move variant-specific details (patterns, examples, configuration) into separate reference files.
|
||||
|
||||
**Pattern 1: High-level guide with references**
|
||||
|
||||
```markdown
|
||||
# PDF Processing
|
||||
|
||||
## Quick start
|
||||
|
||||
Extract text with pdfplumber:
|
||||
[code example]
|
||||
|
||||
## Advanced features
|
||||
|
||||
- **Form filling**: See [FORMS.md](FORMS.md) for complete guide
|
||||
- **API reference**: See [REFERENCE.md](REFERENCE.md) for all methods
|
||||
- **Examples**: See [EXAMPLES.md](EXAMPLES.md) for common patterns
|
||||
```
|
||||
|
||||
Claude loads FORMS.md, REFERENCE.md, or EXAMPLES.md only when needed.
|
||||
|
||||
**Pattern 2: Domain-specific organization**
|
||||
|
||||
For Skills with multiple domains, organize content by domain to avoid loading irrelevant context:
|
||||
|
||||
```
|
||||
bigquery-skill/
|
||||
├── SKILL.md (overview and navigation)
|
||||
└── reference/
|
||||
├── finance.md (revenue, billing metrics)
|
||||
├── sales.md (opportunities, pipeline)
|
||||
├── product.md (API usage, features)
|
||||
└── marketing.md (campaigns, attribution)
|
||||
```
|
||||
|
||||
When a user asks about sales metrics, Claude only reads sales.md.
|
||||
|
||||
Similarly, for skills supporting multiple frameworks or variants, organize by variant:
|
||||
|
||||
**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:
|
||||
```
|
||||
cloud-deploy/
|
||||
├── SKILL.md (workflow + selection)
|
||||
├── SKILL.md (workflow + provider selection)
|
||||
└── references/
|
||||
├── aws.md
|
||||
├── gcp.md
|
||||
└── azure.md
|
||||
├── aws.md (AWS deployment patterns)
|
||||
├── gcp.md (GCP deployment patterns)
|
||||
└── azure.md (Azure deployment patterns)
|
||||
```
|
||||
Claude reads only the relevant reference file.
|
||||
|
||||
#### Principle of Lack of Surprise
|
||||
When the user chooses AWS, Claude only reads aws.md.
|
||||
|
||||
This goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a "roleplay as an XYZ" are OK though.
|
||||
**Pattern 3: Conditional details**
|
||||
|
||||
#### Writing Patterns
|
||||
Show basic content, link to advanced content:
|
||||
|
||||
Prefer using the imperative form in instructions.
|
||||
|
||||
**Defining output formats** - You can do it like this:
|
||||
```markdown
|
||||
## Report structure
|
||||
ALWAYS use this exact template:
|
||||
# [Title]
|
||||
## Executive summary
|
||||
## Key findings
|
||||
## Recommendations
|
||||
# DOCX Processing
|
||||
|
||||
## Creating documents
|
||||
|
||||
Use docx-js for new documents. See [DOCX-JS.md](DOCX-JS.md).
|
||||
|
||||
## Editing documents
|
||||
|
||||
For simple edits, modify the XML directly.
|
||||
|
||||
**For tracked changes**: See [REDLINING.md](REDLINING.md)
|
||||
**For OOXML details**: See [OOXML.md](OOXML.md)
|
||||
```
|
||||
|
||||
**Examples pattern** - It's useful to include examples. You can format them like this (but if "Input" and "Output" are in the examples you might want to deviate a little):
|
||||
```markdown
|
||||
## Commit message format
|
||||
**Example 1:**
|
||||
Input: Added user authentication with JWT tokens
|
||||
Output: feat(auth): implement JWT-based authentication
|
||||
```
|
||||
Claude reads REDLINING.md or OOXML.md only when the user needs those features.
|
||||
|
||||
### Writing Style
|
||||
**Important guidelines:**
|
||||
|
||||
Try to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.
|
||||
- **Avoid deeply nested references** - Keep references one level deep from SKILL.md. All reference files should link directly from SKILL.md.
|
||||
- **Structure longer reference files** - For files longer than 100 lines, include a table of contents at the top so Claude can see the full scope when previewing.
|
||||
|
||||
### Test Cases
|
||||
## Skill Creation Process
|
||||
|
||||
After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" Then run them.
|
||||
Skill creation involves these steps:
|
||||
|
||||
Save test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.
|
||||
1. Understand the skill with concrete examples
|
||||
2. Plan reusable skill contents (scripts, references, assets)
|
||||
3. Initialize the skill (run init_skill.py)
|
||||
4. Edit the skill (implement resources and write SKILL.md)
|
||||
5. Package the skill (run package_skill.py)
|
||||
6. Iterate based on real usage
|
||||
|
||||
```json
|
||||
{
|
||||
"skill_name": "example-skill",
|
||||
"evals": [
|
||||
{
|
||||
"id": 1,
|
||||
"prompt": "User's task prompt",
|
||||
"expected_output": "Description of expected result",
|
||||
"files": []
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
Follow these steps in order, skipping only if there is a clear reason why they are not applicable.
|
||||
|
||||
See `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).
|
||||
### Step 1: Understanding the Skill with Concrete Examples
|
||||
|
||||
## Running and evaluating test cases
|
||||
Skip this step only when the skill's usage patterns are already clearly understood. It remains valuable even when working with an existing skill.
|
||||
|
||||
This section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.
|
||||
To create an effective skill, clearly understand concrete examples of how the skill will be used. This understanding can come from either direct user examples or generated examples that are validated with user feedback.
|
||||
|
||||
Put results in `<skill-name>-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.
|
||||
For example, when building an image-editor skill, relevant questions include:
|
||||
|
||||
### Step 1: Spawn all runs (with-skill AND baseline) in the same turn
|
||||
- "What functionality should the image-editor skill support? Editing, rotating, anything else?"
|
||||
- "Can you give some examples of how this skill would be used?"
|
||||
- "I can imagine users asking for things like 'Remove the red-eye from this image' or 'Rotate this image'. Are there other ways you imagine this skill being used?"
|
||||
- "What would a user say that should trigger this skill?"
|
||||
|
||||
For each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.
|
||||
To avoid overwhelming users, avoid asking too many questions in a single message. Start with the most important questions and follow up as needed for better effectiveness.
|
||||
|
||||
**With-skill run:**
|
||||
Conclude this step when there is a clear sense of the functionality the skill should support.
|
||||
|
||||
```
|
||||
Execute this task:
|
||||
- Skill path: <path-to-skill>
|
||||
- Task: <eval prompt>
|
||||
- Input files: <eval files if any, or "none">
|
||||
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
|
||||
- Outputs to save: <what the user cares about — e.g., "the .docx file", "the final CSV">
|
||||
```
|
||||
### Step 2: Planning the Reusable Skill Contents
|
||||
|
||||
**Baseline run** (same prompt, but the baseline depends on context):
|
||||
- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.
|
||||
- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r <skill-path> <workspace>/skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.
|
||||
To turn concrete examples into an effective skill, analyze each example by:
|
||||
|
||||
Write an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just "eval-0". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.
|
||||
1. Considering how to execute on the example from scratch
|
||||
2. Identifying what scripts, references, and assets would be helpful when executing these workflows repeatedly
|
||||
|
||||
```json
|
||||
{
|
||||
"eval_id": 0,
|
||||
"eval_name": "descriptive-name-here",
|
||||
"prompt": "The user's task prompt",
|
||||
"assertions": []
|
||||
}
|
||||
```
|
||||
Example: When building a `pdf-editor` skill to handle queries like "Help me rotate this PDF," the analysis shows:
|
||||
|
||||
### Step 2: While runs are in progress, draft assertions
|
||||
1. Rotating a PDF requires re-writing the same code each time
|
||||
2. A `scripts/rotate_pdf.py` script would be helpful to store in the skill
|
||||
|
||||
Don't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.
|
||||
Example: When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app" or "Build me a dashboard to track my steps," the analysis shows:
|
||||
|
||||
Good assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.
|
||||
1. Writing a frontend webapp requires the same boilerplate HTML/React each time
|
||||
2. An `assets/hello-world/` template containing the boilerplate HTML/React project files would be helpful to store in the skill
|
||||
|
||||
Update the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.
|
||||
Example: When building a `big-query` skill to handle queries like "How many users have logged in today?" the analysis shows:
|
||||
|
||||
### Step 3: As runs complete, capture timing data
|
||||
1. Querying BigQuery requires re-discovering the table schemas and relationships each time
|
||||
2. A `references/schema.md` file documenting the table schemas would be helpful to store in the skill
|
||||
|
||||
When each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:
|
||||
To establish the skill's contents, analyze each concrete example to create a list of the reusable resources to include: scripts, references, and assets.
|
||||
|
||||
```json
|
||||
{
|
||||
"total_tokens": 84852,
|
||||
"duration_ms": 23332,
|
||||
"total_duration_seconds": 23.3
|
||||
}
|
||||
```
|
||||
### Step 3: Initializing the Skill
|
||||
|
||||
This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.
|
||||
At this point, it is time to actually create the skill.
|
||||
|
||||
### Step 4: Grade, aggregate, and launch the viewer
|
||||
Skip this step only if the skill being developed already exists, and iteration or packaging is needed. In this case, continue to the next step.
|
||||
|
||||
Once all runs are done:
|
||||
When creating a new skill from scratch, always run the `init_skill.py` script. The script conveniently generates a new template skill directory that automatically includes everything a skill requires, making the skill creation process much more efficient and reliable.
|
||||
|
||||
1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.
|
||||
|
||||
2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:
|
||||
```bash
|
||||
python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
|
||||
```
|
||||
This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.
|
||||
Put each with_skill version before its baseline counterpart.
|
||||
|
||||
3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the "Analyzing Benchmark Results" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.
|
||||
|
||||
4. **Launch the viewer** with both qualitative outputs and quantitative data:
|
||||
```bash
|
||||
nohup python <skill-creator-path>/eval-viewer/generate_review.py \
|
||||
<workspace>/iteration-N \
|
||||
--skill-name "my-skill" \
|
||||
--benchmark <workspace>/iteration-N/benchmark.json \
|
||||
> /dev/null 2>&1 &
|
||||
VIEWER_PID=$!
|
||||
```
|
||||
For iteration 2+, also pass `--previous-workspace <workspace>/iteration-<N-1>`.
|
||||
|
||||
**Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks "Submit All Reviews". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.
|
||||
|
||||
Note: please use generate_review.py to create the viewer; there's no need to write custom HTML.
|
||||
|
||||
5. **Tell the user** something like: "I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know."
|
||||
|
||||
### What the user sees in the viewer
|
||||
|
||||
The "Outputs" tab shows one test case at a time:
|
||||
- **Prompt**: the task that was given
|
||||
- **Output**: the files the skill produced, rendered inline where possible
|
||||
- **Previous Output** (iteration 2+): collapsed section showing last iteration's output
|
||||
- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail
|
||||
- **Feedback**: a textbox that auto-saves as they type
|
||||
- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox
|
||||
|
||||
The "Benchmark" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.
|
||||
|
||||
Navigation is via prev/next buttons or arrow keys. When done, they click "Submit All Reviews" which saves all feedback to `feedback.json`.
|
||||
|
||||
### Step 5: Read the feedback
|
||||
|
||||
When the user tells you they're done, read `feedback.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"reviews": [
|
||||
{"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."},
|
||||
{"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."},
|
||||
{"run_id": "eval-2-with_skill", "feedback": "perfect, love this", "timestamp": "..."}
|
||||
],
|
||||
"status": "complete"
|
||||
}
|
||||
```
|
||||
|
||||
Empty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.
|
||||
|
||||
Kill the viewer server when you're done with it:
|
||||
Usage:
|
||||
|
||||
```bash
|
||||
kill $VIEWER_PID 2>/dev/null
|
||||
scripts/init_skill.py <skill-name> --path <output-directory>
|
||||
```
|
||||
|
||||
---
|
||||
The script:
|
||||
|
||||
## Improving the skill
|
||||
- Creates the skill directory at the specified path
|
||||
- Generates a SKILL.md template with proper frontmatter and TODO placeholders
|
||||
- Creates example resource directories: `scripts/`, `references/`, and `assets/`
|
||||
- Adds example files in each directory that can be customized or deleted
|
||||
|
||||
This is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.
|
||||
After initialization, customize or remove the generated SKILL.md and example files as needed.
|
||||
|
||||
### How to think about improvements
|
||||
### Step 4: Edit the Skill
|
||||
|
||||
1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.
|
||||
When editing the (newly-generated or existing) skill, remember that the skill is being created for another instance of Claude to use. Include information that would be beneficial and non-obvious to Claude. Consider what procedural knowledge, domain-specific details, or reusable assets would help another Claude instance execute these tasks more effectively.
|
||||
|
||||
2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.
|
||||
#### Learn Proven Design Patterns
|
||||
|
||||
3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.
|
||||
Consult these helpful guides based on your skill's needs:
|
||||
|
||||
4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.
|
||||
- **Multi-step processes**: See references/workflows.md for sequential workflows and conditional logic
|
||||
- **Specific output formats or quality standards**: See references/output-patterns.md for template and example patterns
|
||||
|
||||
This task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.
|
||||
These files contain established best practices for effective skill design.
|
||||
|
||||
### The iteration loop
|
||||
#### Start with Reusable Skill Contents
|
||||
|
||||
After improving the skill:
|
||||
To begin implementation, start with the reusable resources identified above: `scripts/`, `references/`, and `assets/` files. Note that this step may require user input. For example, when implementing a `brand-guidelines` skill, the user may need to provide brand assets or templates to store in `assets/`, or documentation to store in `references/`.
|
||||
|
||||
1. Apply your improvements to the skill
|
||||
2. Rerun all test cases into a new `iteration-<N+1>/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.
|
||||
3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration
|
||||
4. Wait for the user to review and tell you they're done
|
||||
5. Read the new feedback, improve again, repeat
|
||||
Added scripts must be tested by actually running them to ensure there are no bugs and that the output matches what is expected. If there are many similar scripts, only a representative sample needs to be tested to ensure confidence that they all work while balancing time to completion.
|
||||
|
||||
Keep going until:
|
||||
- The user says they're happy
|
||||
- The feedback is all empty (everything looks good)
|
||||
- You're not making meaningful progress
|
||||
Any example files and directories not needed for the skill should be deleted. The initialization script creates example files in `scripts/`, `references/`, and `assets/` to demonstrate structure, but most skills won't need all of them.
|
||||
|
||||
---
|
||||
#### Update SKILL.md
|
||||
|
||||
## Advanced: Blind comparison
|
||||
**Writing Guidelines:** Always use imperative/infinitive form.
|
||||
|
||||
For situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks "is the new version actually better?"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.
|
||||
##### Frontmatter
|
||||
|
||||
This is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.
|
||||
Write the YAML frontmatter with `name` and `description`:
|
||||
|
||||
---
|
||||
- `name`: The skill name
|
||||
- `description`: This is the primary triggering mechanism for your skill, and helps Claude understand when to use the skill.
|
||||
- Include both what the Skill does and specific triggers/contexts for when to use it.
|
||||
- Include all "when to use" information here - Not in the body. The body is only loaded after triggering, so "When to Use This Skill" sections in the body are not helpful to Claude.
|
||||
- Example description for a `docx` skill: "Comprehensive document creation, editing, and analysis with support for tracked changes, comments, formatting preservation, and text extraction. Use when Claude needs to work with professional documents (.docx files) for: (1) Creating new documents, (2) Modifying or editing content, (3) Working with tracked changes, (4) Adding comments, or any other document tasks"
|
||||
|
||||
## Description Optimization
|
||||
Do not include any other fields in YAML frontmatter.
|
||||
|
||||
The description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.
|
||||
##### Body
|
||||
|
||||
### Step 1: Generate trigger eval queries
|
||||
Write instructions for using the skill and its bundled resources.
|
||||
|
||||
Create 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:
|
||||
### Step 5: Packaging a Skill
|
||||
|
||||
```json
|
||||
[
|
||||
{"query": "the user prompt", "should_trigger": true},
|
||||
{"query": "another prompt", "should_trigger": false}
|
||||
]
|
||||
```
|
||||
|
||||
The queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).
|
||||
|
||||
Bad: `"Format this data"`, `"Extract text from PDF"`, `"Create a chart"`
|
||||
|
||||
Good: `"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"`
|
||||
|
||||
For the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.
|
||||
|
||||
For the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.
|
||||
|
||||
The key thing to avoid: don't make should-not-trigger queries obviously irrelevant. "Write a fibonacci function" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.
|
||||
|
||||
### Step 2: Review with user
|
||||
|
||||
Present the eval set to the user for review using the HTML template:
|
||||
|
||||
1. Read the template from `assets/eval_review.html`
|
||||
2. Replace the placeholders:
|
||||
- `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)
|
||||
- `__SKILL_NAME_PLACEHOLDER__` → the skill's name
|
||||
- `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description
|
||||
3. Write to a temp file (e.g., `/tmp/eval_review_<skill-name>.html`) and open it: `open /tmp/eval_review_<skill-name>.html`
|
||||
4. The user can edit queries, toggle should-trigger, add/remove entries, then click "Export Eval Set"
|
||||
5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)
|
||||
|
||||
This step matters — bad eval queries lead to bad descriptions.
|
||||
|
||||
### Step 3: Run the optimization loop
|
||||
|
||||
Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."
|
||||
|
||||
Save the eval set to the workspace, then run in the background:
|
||||
Once development of the skill is complete, it must be packaged into a distributable .skill file that gets shared with the user. The packaging process automatically validates the skill first to ensure it meets all requirements:
|
||||
|
||||
```bash
|
||||
python -m scripts.run_loop \
|
||||
--eval-set <path-to-trigger-eval.json> \
|
||||
--skill-path <path-to-skill> \
|
||||
--model <model-id-powering-this-session> \
|
||||
--max-iterations 5 \
|
||||
--verbose
|
||||
scripts/package_skill.py <path/to/skill-folder>
|
||||
```
|
||||
|
||||
Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.
|
||||
|
||||
While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.
|
||||
|
||||
This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.
|
||||
|
||||
### How skill triggering works
|
||||
|
||||
Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.
|
||||
|
||||
This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality.
|
||||
|
||||
### Step 4: Apply the result
|
||||
|
||||
Take `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.
|
||||
|
||||
---
|
||||
|
||||
### Package and Present (only if `present_files` tool is available)
|
||||
|
||||
Check whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:
|
||||
Optional output directory specification:
|
||||
|
||||
```bash
|
||||
python -m scripts.package_skill <path/to/skill-folder>
|
||||
scripts/package_skill.py <path/to/skill-folder> ./dist
|
||||
```
|
||||
|
||||
After packaging, direct the user to the resulting `.skill` file path so they can install it.
|
||||
The packaging script will:
|
||||
|
||||
---
|
||||
1. **Validate** the skill automatically, checking:
|
||||
|
||||
## Claude.ai-specific instructions
|
||||
- YAML frontmatter format and required fields
|
||||
- Skill naming conventions and directory structure
|
||||
- Description completeness and quality
|
||||
- File organization and resource references
|
||||
|
||||
In Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:
|
||||
2. **Package** the skill if validation passes, creating a .skill file named after the skill (e.g., `my-skill.skill`) that includes all files and maintains the proper directory structure for distribution. The .skill file is a zip file with a .skill extension.
|
||||
|
||||
**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.
|
||||
If validation fails, the script will report the errors and exit without creating a package. Fix any validation errors and run the packaging command again.
|
||||
|
||||
**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: "How does this look? Anything you'd change?"
|
||||
### Step 6: Iterate
|
||||
|
||||
**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.
|
||||
After testing the skill, users may request improvements. Often this happens right after using the skill, with fresh context of how the skill performed.
|
||||
|
||||
**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.
|
||||
**Iteration workflow:**
|
||||
|
||||
**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.
|
||||
|
||||
**Blind comparison**: Requires subagents. Skip it.
|
||||
|
||||
**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.
|
||||
|
||||
**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:
|
||||
- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).
|
||||
- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.
|
||||
- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.
|
||||
|
||||
---
|
||||
|
||||
## Cowork-Specific Instructions
|
||||
|
||||
If you're in Cowork, the main things to know are:
|
||||
|
||||
- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)
|
||||
- You don't have a browser or display, so when generating the eval viewer, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.
|
||||
- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!
|
||||
- Feedback works differently: since there's no running server, the viewer's "Submit All Reviews" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).
|
||||
- Packaging works — `package_skill.py` just needs Python and a filesystem.
|
||||
- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.
|
||||
- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.
|
||||
|
||||
---
|
||||
|
||||
## Reference files
|
||||
|
||||
The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.
|
||||
|
||||
- `agents/grader.md` — How to evaluate assertions against outputs
|
||||
- `agents/comparator.md` — How to do blind A/B comparison between two outputs
|
||||
- `agents/analyzer.md` — How to analyze why one version beat another
|
||||
|
||||
The references/ directory has additional documentation:
|
||||
- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.
|
||||
|
||||
---
|
||||
|
||||
Repeating one more time the core loop here for emphasis:
|
||||
|
||||
- Figure out what the skill is about
|
||||
- Draft or edit the skill
|
||||
- Run claude-with-access-to-the-skill on test prompts
|
||||
- With the user, evaluate the outputs:
|
||||
- Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them
|
||||
- Run quantitative evals
|
||||
- Repeat until you and the user are satisfied
|
||||
- Package the final skill and return it to the user.
|
||||
|
||||
Please add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put "Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases" in your TodoList to make sure it happens.
|
||||
|
||||
Good luck!
|
||||
1. Use the skill on real tasks
|
||||
2. Notice struggles or inefficiencies
|
||||
3. Identify how SKILL.md or bundled resources should be updated
|
||||
4. Implement changes and test again
|
||||
|
||||
@@ -1,274 +0,0 @@
|
||||
# Post-hoc Analyzer Agent
|
||||
|
||||
Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.
|
||||
|
||||
## Role
|
||||
|
||||
After the blind comparator determines a winner, the Post-hoc Analyzer "unblids" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?
|
||||
|
||||
## Inputs
|
||||
|
||||
You receive these parameters in your prompt:
|
||||
|
||||
- **winner**: "A" or "B" (from blind comparison)
|
||||
- **winner_skill_path**: Path to the skill that produced the winning output
|
||||
- **winner_transcript_path**: Path to the execution transcript for the winner
|
||||
- **loser_skill_path**: Path to the skill that produced the losing output
|
||||
- **loser_transcript_path**: Path to the execution transcript for the loser
|
||||
- **comparison_result_path**: Path to the blind comparator's output JSON
|
||||
- **output_path**: Where to save the analysis results
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Read Comparison Result
|
||||
|
||||
1. Read the blind comparator's output at comparison_result_path
|
||||
2. Note the winning side (A or B), the reasoning, and any scores
|
||||
3. Understand what the comparator valued in the winning output
|
||||
|
||||
### Step 2: Read Both Skills
|
||||
|
||||
1. Read the winner skill's SKILL.md and key referenced files
|
||||
2. Read the loser skill's SKILL.md and key referenced files
|
||||
3. Identify structural differences:
|
||||
- Instructions clarity and specificity
|
||||
- Script/tool usage patterns
|
||||
- Example coverage
|
||||
- Edge case handling
|
||||
|
||||
### Step 3: Read Both Transcripts
|
||||
|
||||
1. Read the winner's transcript
|
||||
2. Read the loser's transcript
|
||||
3. Compare execution patterns:
|
||||
- How closely did each follow their skill's instructions?
|
||||
- What tools were used differently?
|
||||
- Where did the loser diverge from optimal behavior?
|
||||
- Did either encounter errors or make recovery attempts?
|
||||
|
||||
### Step 4: Analyze Instruction Following
|
||||
|
||||
For each transcript, evaluate:
|
||||
- Did the agent follow the skill's explicit instructions?
|
||||
- Did the agent use the skill's provided tools/scripts?
|
||||
- Were there missed opportunities to leverage skill content?
|
||||
- Did the agent add unnecessary steps not in the skill?
|
||||
|
||||
Score instruction following 1-10 and note specific issues.
|
||||
|
||||
### Step 5: Identify Winner Strengths
|
||||
|
||||
Determine what made the winner better:
|
||||
- Clearer instructions that led to better behavior?
|
||||
- Better scripts/tools that produced better output?
|
||||
- More comprehensive examples that guided edge cases?
|
||||
- Better error handling guidance?
|
||||
|
||||
Be specific. Quote from skills/transcripts where relevant.
|
||||
|
||||
### Step 6: Identify Loser Weaknesses
|
||||
|
||||
Determine what held the loser back:
|
||||
- Ambiguous instructions that led to suboptimal choices?
|
||||
- Missing tools/scripts that forced workarounds?
|
||||
- Gaps in edge case coverage?
|
||||
- Poor error handling that caused failures?
|
||||
|
||||
### Step 7: Generate Improvement Suggestions
|
||||
|
||||
Based on the analysis, produce actionable suggestions for improving the loser skill:
|
||||
- Specific instruction changes to make
|
||||
- Tools/scripts to add or modify
|
||||
- Examples to include
|
||||
- Edge cases to address
|
||||
|
||||
Prioritize by impact. Focus on changes that would have changed the outcome.
|
||||
|
||||
### Step 8: Write Analysis Results
|
||||
|
||||
Save structured analysis to `{output_path}`.
|
||||
|
||||
## Output Format
|
||||
|
||||
Write a JSON file with this structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"comparison_summary": {
|
||||
"winner": "A",
|
||||
"winner_skill": "path/to/winner/skill",
|
||||
"loser_skill": "path/to/loser/skill",
|
||||
"comparator_reasoning": "Brief summary of why comparator chose winner"
|
||||
},
|
||||
"winner_strengths": [
|
||||
"Clear step-by-step instructions for handling multi-page documents",
|
||||
"Included validation script that caught formatting errors",
|
||||
"Explicit guidance on fallback behavior when OCR fails"
|
||||
],
|
||||
"loser_weaknesses": [
|
||||
"Vague instruction 'process the document appropriately' led to inconsistent behavior",
|
||||
"No script for validation, agent had to improvise and made errors",
|
||||
"No guidance on OCR failure, agent gave up instead of trying alternatives"
|
||||
],
|
||||
"instruction_following": {
|
||||
"winner": {
|
||||
"score": 9,
|
||||
"issues": [
|
||||
"Minor: skipped optional logging step"
|
||||
]
|
||||
},
|
||||
"loser": {
|
||||
"score": 6,
|
||||
"issues": [
|
||||
"Did not use the skill's formatting template",
|
||||
"Invented own approach instead of following step 3",
|
||||
"Missed the 'always validate output' instruction"
|
||||
]
|
||||
}
|
||||
},
|
||||
"improvement_suggestions": [
|
||||
{
|
||||
"priority": "high",
|
||||
"category": "instructions",
|
||||
"suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",
|
||||
"expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
|
||||
},
|
||||
{
|
||||
"priority": "high",
|
||||
"category": "tools",
|
||||
"suggestion": "Add validate_output.py script similar to winner skill's validation approach",
|
||||
"expected_impact": "Would catch formatting errors before final output"
|
||||
},
|
||||
{
|
||||
"priority": "medium",
|
||||
"category": "error_handling",
|
||||
"suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",
|
||||
"expected_impact": "Would prevent early failure on difficult documents"
|
||||
}
|
||||
],
|
||||
"transcript_insights": {
|
||||
"winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",
|
||||
"loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **Be specific**: Quote from skills and transcripts, don't just say "instructions were unclear"
|
||||
- **Be actionable**: Suggestions should be concrete changes, not vague advice
|
||||
- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent
|
||||
- **Prioritize by impact**: Which changes would most likely have changed the outcome?
|
||||
- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?
|
||||
- **Stay objective**: Analyze what happened, don't editorialize
|
||||
- **Think about generalization**: Would this improvement help on other evals too?
|
||||
|
||||
## Categories for Suggestions
|
||||
|
||||
Use these categories to organize improvement suggestions:
|
||||
|
||||
| Category | Description |
|
||||
|----------|-------------|
|
||||
| `instructions` | Changes to the skill's prose instructions |
|
||||
| `tools` | Scripts, templates, or utilities to add/modify |
|
||||
| `examples` | Example inputs/outputs to include |
|
||||
| `error_handling` | Guidance for handling failures |
|
||||
| `structure` | Reorganization of skill content |
|
||||
| `references` | External docs or resources to add |
|
||||
|
||||
## Priority Levels
|
||||
|
||||
- **high**: Would likely change the outcome of this comparison
|
||||
- **medium**: Would improve quality but may not change win/loss
|
||||
- **low**: Nice to have, marginal improvement
|
||||
|
||||
---
|
||||
|
||||
# Analyzing Benchmark Results
|
||||
|
||||
When analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.
|
||||
|
||||
## Role
|
||||
|
||||
Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.
|
||||
|
||||
## Inputs
|
||||
|
||||
You receive these parameters in your prompt:
|
||||
|
||||
- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results
|
||||
- **skill_path**: Path to the skill being benchmarked
|
||||
- **output_path**: Where to save the notes (as JSON array of strings)
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Read Benchmark Data
|
||||
|
||||
1. Read the benchmark.json containing all run results
|
||||
2. Note the configurations tested (with_skill, without_skill)
|
||||
3. Understand the run_summary aggregates already calculated
|
||||
|
||||
### Step 2: Analyze Per-Assertion Patterns
|
||||
|
||||
For each expectation across all runs:
|
||||
- Does it **always pass** in both configurations? (may not differentiate skill value)
|
||||
- Does it **always fail** in both configurations? (may be broken or beyond capability)
|
||||
- Does it **always pass with skill but fail without**? (skill clearly adds value here)
|
||||
- Does it **always fail with skill but pass without**? (skill may be hurting)
|
||||
- Is it **highly variable**? (flaky expectation or non-deterministic behavior)
|
||||
|
||||
### Step 3: Analyze Cross-Eval Patterns
|
||||
|
||||
Look for patterns across evals:
|
||||
- Are certain eval types consistently harder/easier?
|
||||
- Do some evals show high variance while others are stable?
|
||||
- Are there surprising results that contradict expectations?
|
||||
|
||||
### Step 4: Analyze Metrics Patterns
|
||||
|
||||
Look at time_seconds, tokens, tool_calls:
|
||||
- Does the skill significantly increase execution time?
|
||||
- Is there high variance in resource usage?
|
||||
- Are there outlier runs that skew the aggregates?
|
||||
|
||||
### Step 5: Generate Notes
|
||||
|
||||
Write freeform observations as a list of strings. Each note should:
|
||||
- State a specific observation
|
||||
- Be grounded in the data (not speculation)
|
||||
- Help the user understand something the aggregate metrics don't show
|
||||
|
||||
Examples:
|
||||
- "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value"
|
||||
- "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky"
|
||||
- "Without-skill runs consistently fail on table extraction expectations (0% pass rate)"
|
||||
- "Skill adds 13s average execution time but improves pass rate by 50%"
|
||||
- "Token usage is 80% higher with skill, primarily due to script output parsing"
|
||||
- "All 3 without-skill runs for eval 1 produced empty output"
|
||||
|
||||
### Step 6: Write Notes
|
||||
|
||||
Save notes to `{output_path}` as a JSON array of strings:
|
||||
|
||||
```json
|
||||
[
|
||||
"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
|
||||
"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure",
|
||||
"Without-skill runs consistently fail on table extraction expectations",
|
||||
"Skill adds 13s average execution time but improves pass rate by 50%"
|
||||
]
|
||||
```
|
||||
|
||||
## Guidelines
|
||||
|
||||
**DO:**
|
||||
- Report what you observe in the data
|
||||
- Be specific about which evals, expectations, or runs you're referring to
|
||||
- Note patterns that aggregate metrics would hide
|
||||
- Provide context that helps interpret the numbers
|
||||
|
||||
**DO NOT:**
|
||||
- Suggest improvements to the skill (that's for the improvement step, not benchmarking)
|
||||
- Make subjective quality judgments ("the output was good/bad")
|
||||
- Speculate about causes without evidence
|
||||
- Repeat information already in the run_summary aggregates
|
||||
@@ -1,202 +0,0 @@
|
||||
# Blind Comparator Agent
|
||||
|
||||
Compare two outputs WITHOUT knowing which skill produced them.
|
||||
|
||||
## Role
|
||||
|
||||
The Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.
|
||||
|
||||
Your judgment is based purely on output quality and task completion.
|
||||
|
||||
## Inputs
|
||||
|
||||
You receive these parameters in your prompt:
|
||||
|
||||
- **output_a_path**: Path to the first output file or directory
|
||||
- **output_b_path**: Path to the second output file or directory
|
||||
- **eval_prompt**: The original task/prompt that was executed
|
||||
- **expectations**: List of expectations to check (optional - may be empty)
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Read Both Outputs
|
||||
|
||||
1. Examine output A (file or directory)
|
||||
2. Examine output B (file or directory)
|
||||
3. Note the type, structure, and content of each
|
||||
4. If outputs are directories, examine all relevant files inside
|
||||
|
||||
### Step 2: Understand the Task
|
||||
|
||||
1. Read the eval_prompt carefully
|
||||
2. Identify what the task requires:
|
||||
- What should be produced?
|
||||
- What qualities matter (accuracy, completeness, format)?
|
||||
- What would distinguish a good output from a poor one?
|
||||
|
||||
### Step 3: Generate Evaluation Rubric
|
||||
|
||||
Based on the task, generate a rubric with two dimensions:
|
||||
|
||||
**Content Rubric** (what the output contains):
|
||||
| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |
|
||||
|-----------|----------|----------------|---------------|
|
||||
| Correctness | Major errors | Minor errors | Fully correct |
|
||||
| Completeness | Missing key elements | Mostly complete | All elements present |
|
||||
| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |
|
||||
|
||||
**Structure Rubric** (how the output is organized):
|
||||
| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |
|
||||
|-----------|----------|----------------|---------------|
|
||||
| Organization | Disorganized | Reasonably organized | Clear, logical structure |
|
||||
| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |
|
||||
| Usability | Difficult to use | Usable with effort | Easy to use |
|
||||
|
||||
Adapt criteria to the specific task. For example:
|
||||
- PDF form → "Field alignment", "Text readability", "Data placement"
|
||||
- Document → "Section structure", "Heading hierarchy", "Paragraph flow"
|
||||
- Data output → "Schema correctness", "Data types", "Completeness"
|
||||
|
||||
### Step 4: Evaluate Each Output Against the Rubric
|
||||
|
||||
For each output (A and B):
|
||||
|
||||
1. **Score each criterion** on the rubric (1-5 scale)
|
||||
2. **Calculate dimension totals**: Content score, Structure score
|
||||
3. **Calculate overall score**: Average of dimension scores, scaled to 1-10
|
||||
|
||||
### Step 5: Check Assertions (if provided)
|
||||
|
||||
If expectations are provided:
|
||||
|
||||
1. Check each expectation against output A
|
||||
2. Check each expectation against output B
|
||||
3. Count pass rates for each output
|
||||
4. Use expectation scores as secondary evidence (not the primary decision factor)
|
||||
|
||||
### Step 6: Determine the Winner
|
||||
|
||||
Compare A and B based on (in priority order):
|
||||
|
||||
1. **Primary**: Overall rubric score (content + structure)
|
||||
2. **Secondary**: Assertion pass rates (if applicable)
|
||||
3. **Tiebreaker**: If truly equal, declare a TIE
|
||||
|
||||
Be decisive - ties should be rare. One output is usually better, even if marginally.
|
||||
|
||||
### Step 7: Write Comparison Results
|
||||
|
||||
Save results to a JSON file at the path specified (or `comparison.json` if not specified).
|
||||
|
||||
## Output Format
|
||||
|
||||
Write a JSON file with this structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"winner": "A",
|
||||
"reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
|
||||
"rubric": {
|
||||
"A": {
|
||||
"content": {
|
||||
"correctness": 5,
|
||||
"completeness": 5,
|
||||
"accuracy": 4
|
||||
},
|
||||
"structure": {
|
||||
"organization": 4,
|
||||
"formatting": 5,
|
||||
"usability": 4
|
||||
},
|
||||
"content_score": 4.7,
|
||||
"structure_score": 4.3,
|
||||
"overall_score": 9.0
|
||||
},
|
||||
"B": {
|
||||
"content": {
|
||||
"correctness": 3,
|
||||
"completeness": 2,
|
||||
"accuracy": 3
|
||||
},
|
||||
"structure": {
|
||||
"organization": 3,
|
||||
"formatting": 2,
|
||||
"usability": 3
|
||||
},
|
||||
"content_score": 2.7,
|
||||
"structure_score": 2.7,
|
||||
"overall_score": 5.4
|
||||
}
|
||||
},
|
||||
"output_quality": {
|
||||
"A": {
|
||||
"score": 9,
|
||||
"strengths": ["Complete solution", "Well-formatted", "All fields present"],
|
||||
"weaknesses": ["Minor style inconsistency in header"]
|
||||
},
|
||||
"B": {
|
||||
"score": 5,
|
||||
"strengths": ["Readable output", "Correct basic structure"],
|
||||
"weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]
|
||||
}
|
||||
},
|
||||
"expectation_results": {
|
||||
"A": {
|
||||
"passed": 4,
|
||||
"total": 5,
|
||||
"pass_rate": 0.80,
|
||||
"details": [
|
||||
{"text": "Output includes name", "passed": true},
|
||||
{"text": "Output includes date", "passed": true},
|
||||
{"text": "Format is PDF", "passed": true},
|
||||
{"text": "Contains signature", "passed": false},
|
||||
{"text": "Readable text", "passed": true}
|
||||
]
|
||||
},
|
||||
"B": {
|
||||
"passed": 3,
|
||||
"total": 5,
|
||||
"pass_rate": 0.60,
|
||||
"details": [
|
||||
{"text": "Output includes name", "passed": true},
|
||||
{"text": "Output includes date", "passed": false},
|
||||
{"text": "Format is PDF", "passed": true},
|
||||
{"text": "Contains signature", "passed": false},
|
||||
{"text": "Readable text", "passed": true}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If no expectations were provided, omit the `expectation_results` field entirely.
|
||||
|
||||
## Field Descriptions
|
||||
|
||||
- **winner**: "A", "B", or "TIE"
|
||||
- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)
|
||||
- **rubric**: Structured rubric evaluation for each output
|
||||
- **content**: Scores for content criteria (correctness, completeness, accuracy)
|
||||
- **structure**: Scores for structure criteria (organization, formatting, usability)
|
||||
- **content_score**: Average of content criteria (1-5)
|
||||
- **structure_score**: Average of structure criteria (1-5)
|
||||
- **overall_score**: Combined score scaled to 1-10
|
||||
- **output_quality**: Summary quality assessment
|
||||
- **score**: 1-10 rating (should match rubric overall_score)
|
||||
- **strengths**: List of positive aspects
|
||||
- **weaknesses**: List of issues or shortcomings
|
||||
- **expectation_results**: (Only if expectations provided)
|
||||
- **passed**: Number of expectations that passed
|
||||
- **total**: Total number of expectations
|
||||
- **pass_rate**: Fraction passed (0.0 to 1.0)
|
||||
- **details**: Individual expectation results
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.
|
||||
- **Be specific**: Cite specific examples when explaining strengths and weaknesses.
|
||||
- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.
|
||||
- **Output quality first**: Assertion scores are secondary to overall task completion.
|
||||
- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.
|
||||
- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.
|
||||
- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.
|
||||
@@ -1,223 +0,0 @@
|
||||
# Grader Agent
|
||||
|
||||
Evaluate expectations against an execution transcript and outputs.
|
||||
|
||||
## Role
|
||||
|
||||
The Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.
|
||||
|
||||
You have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.
|
||||
|
||||
## Inputs
|
||||
|
||||
You receive these parameters in your prompt:
|
||||
|
||||
- **expectations**: List of expectations to evaluate (strings)
|
||||
- **transcript_path**: Path to the execution transcript (markdown file)
|
||||
- **outputs_dir**: Directory containing output files from execution
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Read the Transcript
|
||||
|
||||
1. Read the transcript file completely
|
||||
2. Note the eval prompt, execution steps, and final result
|
||||
3. Identify any issues or errors documented
|
||||
|
||||
### Step 2: Examine Output Files
|
||||
|
||||
1. List files in outputs_dir
|
||||
2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.
|
||||
3. Note contents, structure, and quality
|
||||
|
||||
### Step 3: Evaluate Each Assertion
|
||||
|
||||
For each expectation:
|
||||
|
||||
1. **Search for evidence** in the transcript and outputs
|
||||
2. **Determine verdict**:
|
||||
- **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance
|
||||
- **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)
|
||||
3. **Cite the evidence**: Quote the specific text or describe what you found
|
||||
|
||||
### Step 4: Extract and Verify Claims
|
||||
|
||||
Beyond the predefined expectations, extract implicit claims from the outputs and verify them:
|
||||
|
||||
1. **Extract claims** from the transcript and outputs:
|
||||
- Factual statements ("The form has 12 fields")
|
||||
- Process claims ("Used pypdf to fill the form")
|
||||
- Quality claims ("All fields were filled correctly")
|
||||
|
||||
2. **Verify each claim**:
|
||||
- **Factual claims**: Can be checked against the outputs or external sources
|
||||
- **Process claims**: Can be verified from the transcript
|
||||
- **Quality claims**: Evaluate whether the claim is justified
|
||||
|
||||
3. **Flag unverifiable claims**: Note claims that cannot be verified with available information
|
||||
|
||||
This catches issues that predefined expectations might miss.
|
||||
|
||||
### Step 5: Read User Notes
|
||||
|
||||
If `{outputs_dir}/user_notes.md` exists:
|
||||
1. Read it and note any uncertainties or issues flagged by the executor
|
||||
2. Include relevant concerns in the grading output
|
||||
3. These may reveal problems even when expectations pass
|
||||
|
||||
### Step 6: Critique the Evals
|
||||
|
||||
After grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.
|
||||
|
||||
Good suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't.
|
||||
|
||||
Suggestions worth raising:
|
||||
- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)
|
||||
- An important outcome you observed — good or bad — that no assertion covers at all
|
||||
- An assertion that can't actually be verified from the available outputs
|
||||
|
||||
Keep the bar high. The goal is to flag things the eval author would say "good catch" about, not to nitpick every assertion.
|
||||
|
||||
### Step 7: Write Grading Results
|
||||
|
||||
Save results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).
|
||||
|
||||
## Grading Criteria
|
||||
|
||||
**PASS when**:
|
||||
- The transcript or outputs clearly demonstrate the expectation is true
|
||||
- Specific evidence can be cited
|
||||
- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)
|
||||
|
||||
**FAIL when**:
|
||||
- No evidence found for the expectation
|
||||
- Evidence contradicts the expectation
|
||||
- The expectation cannot be verified from available information
|
||||
- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete
|
||||
- The output appears to meet the assertion by coincidence rather than by actually doing the work
|
||||
|
||||
**When uncertain**: The burden of proof to pass is on the expectation.
|
||||
|
||||
### Step 8: Read Executor Metrics and Timing
|
||||
|
||||
1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output
|
||||
2. If `{outputs_dir}/../timing.json` exists, read it and include timing data
|
||||
|
||||
## Output Format
|
||||
|
||||
Write a JSON file with this structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"expectations": [
|
||||
{
|
||||
"text": "The output includes the name 'John Smith'",
|
||||
"passed": true,
|
||||
"evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
|
||||
},
|
||||
{
|
||||
"text": "The spreadsheet has a SUM formula in cell B10",
|
||||
"passed": false,
|
||||
"evidence": "No spreadsheet was created. The output was a text file."
|
||||
},
|
||||
{
|
||||
"text": "The assistant used the skill's OCR script",
|
||||
"passed": true,
|
||||
"evidence": "Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"passed": 2,
|
||||
"failed": 1,
|
||||
"total": 3,
|
||||
"pass_rate": 0.67
|
||||
},
|
||||
"execution_metrics": {
|
||||
"tool_calls": {
|
||||
"Read": 5,
|
||||
"Write": 2,
|
||||
"Bash": 8
|
||||
},
|
||||
"total_tool_calls": 15,
|
||||
"total_steps": 6,
|
||||
"errors_encountered": 0,
|
||||
"output_chars": 12450,
|
||||
"transcript_chars": 3200
|
||||
},
|
||||
"timing": {
|
||||
"executor_duration_seconds": 165.0,
|
||||
"grader_duration_seconds": 26.0,
|
||||
"total_duration_seconds": 191.0
|
||||
},
|
||||
"claims": [
|
||||
{
|
||||
"claim": "The form has 12 fillable fields",
|
||||
"type": "factual",
|
||||
"verified": true,
|
||||
"evidence": "Counted 12 fields in field_info.json"
|
||||
},
|
||||
{
|
||||
"claim": "All required fields were populated",
|
||||
"type": "quality",
|
||||
"verified": false,
|
||||
"evidence": "Reference section was left blank despite data being available"
|
||||
}
|
||||
],
|
||||
"user_notes_summary": {
|
||||
"uncertainties": ["Used 2023 data, may be stale"],
|
||||
"needs_review": [],
|
||||
"workarounds": ["Fell back to text overlay for non-fillable fields"]
|
||||
},
|
||||
"eval_feedback": {
|
||||
"suggestions": [
|
||||
{
|
||||
"assertion": "The output includes the name 'John Smith'",
|
||||
"reason": "A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input"
|
||||
},
|
||||
{
|
||||
"reason": "No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught"
|
||||
}
|
||||
],
|
||||
"overall": "Assertions check presence but not correctness. Consider adding content verification."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Field Descriptions
|
||||
|
||||
- **expectations**: Array of graded expectations
|
||||
- **text**: The original expectation text
|
||||
- **passed**: Boolean - true if expectation passes
|
||||
- **evidence**: Specific quote or description supporting the verdict
|
||||
- **summary**: Aggregate statistics
|
||||
- **passed**: Count of passed expectations
|
||||
- **failed**: Count of failed expectations
|
||||
- **total**: Total expectations evaluated
|
||||
- **pass_rate**: Fraction passed (0.0 to 1.0)
|
||||
- **execution_metrics**: Copied from executor's metrics.json (if available)
|
||||
- **output_chars**: Total character count of output files (proxy for tokens)
|
||||
- **transcript_chars**: Character count of transcript
|
||||
- **timing**: Wall clock timing from timing.json (if available)
|
||||
- **executor_duration_seconds**: Time spent in executor subagent
|
||||
- **total_duration_seconds**: Total elapsed time for the run
|
||||
- **claims**: Extracted and verified claims from the output
|
||||
- **claim**: The statement being verified
|
||||
- **type**: "factual", "process", or "quality"
|
||||
- **verified**: Boolean - whether the claim holds
|
||||
- **evidence**: Supporting or contradicting evidence
|
||||
- **user_notes_summary**: Issues flagged by the executor
|
||||
- **uncertainties**: Things the executor wasn't sure about
|
||||
- **needs_review**: Items requiring human attention
|
||||
- **workarounds**: Places where the skill didn't work as expected
|
||||
- **eval_feedback**: Improvement suggestions for the evals (only when warranted)
|
||||
- **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to
|
||||
- **overall**: Brief assessment — can be "No suggestions, evals look solid" if nothing to flag
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **Be objective**: Base verdicts on evidence, not assumptions
|
||||
- **Be specific**: Quote the exact text that supports your verdict
|
||||
- **Be thorough**: Check both transcript and output files
|
||||
- **Be consistent**: Apply the same standard to each expectation
|
||||
- **Explain failures**: Make it clear why evidence was insufficient
|
||||
- **No partial credit**: Each expectation is pass or fail, not partial
|
||||
@@ -1,146 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Eval Set Review - __SKILL_NAME_PLACEHOLDER__</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
* { box-sizing: border-box; margin: 0; padding: 0; }
|
||||
body { font-family: 'Lora', Georgia, serif; background: #faf9f5; padding: 2rem; color: #141413; }
|
||||
h1 { font-family: 'Poppins', sans-serif; margin-bottom: 0.5rem; font-size: 1.5rem; }
|
||||
.description { color: #b0aea5; margin-bottom: 1.5rem; font-style: italic; max-width: 900px; }
|
||||
.controls { margin-bottom: 1rem; display: flex; gap: 0.5rem; }
|
||||
.btn { font-family: 'Poppins', sans-serif; padding: 0.5rem 1rem; border: none; border-radius: 6px; cursor: pointer; font-size: 0.875rem; font-weight: 500; }
|
||||
.btn-add { background: #6a9bcc; color: white; }
|
||||
.btn-add:hover { background: #5889b8; }
|
||||
.btn-export { background: #d97757; color: white; }
|
||||
.btn-export:hover { background: #c4613f; }
|
||||
table { width: 100%; max-width: 1100px; border-collapse: collapse; background: white; border-radius: 6px; overflow: hidden; box-shadow: 0 1px 3px rgba(0,0,0,0.08); }
|
||||
th { font-family: 'Poppins', sans-serif; background: #141413; color: #faf9f5; padding: 0.75rem 1rem; text-align: left; font-size: 0.875rem; }
|
||||
td { padding: 0.75rem 1rem; border-bottom: 1px solid #e8e6dc; vertical-align: top; }
|
||||
tr:nth-child(even) td { background: #faf9f5; }
|
||||
tr:hover td { background: #f3f1ea; }
|
||||
.section-header td { background: #e8e6dc; font-family: 'Poppins', sans-serif; font-weight: 500; font-size: 0.8rem; color: #141413; text-transform: uppercase; letter-spacing: 0.05em; }
|
||||
.query-input { width: 100%; padding: 0.4rem; border: 1px solid #e8e6dc; border-radius: 4px; font-size: 0.875rem; font-family: 'Lora', Georgia, serif; resize: vertical; min-height: 60px; }
|
||||
.query-input:focus { outline: none; border-color: #d97757; box-shadow: 0 0 0 2px rgba(217,119,87,0.15); }
|
||||
.toggle { position: relative; display: inline-block; width: 44px; height: 24px; }
|
||||
.toggle input { opacity: 0; width: 0; height: 0; }
|
||||
.toggle .slider { position: absolute; inset: 0; background: #b0aea5; border-radius: 24px; cursor: pointer; transition: 0.2s; }
|
||||
.toggle .slider::before { content: ""; position: absolute; width: 18px; height: 18px; left: 3px; bottom: 3px; background: white; border-radius: 50%; transition: 0.2s; }
|
||||
.toggle input:checked + .slider { background: #d97757; }
|
||||
.toggle input:checked + .slider::before { transform: translateX(20px); }
|
||||
.btn-delete { background: #c44; color: white; padding: 0.3rem 0.6rem; border: none; border-radius: 4px; cursor: pointer; font-size: 0.75rem; font-family: 'Poppins', sans-serif; }
|
||||
.btn-delete:hover { background: #a33; }
|
||||
.summary { margin-top: 1rem; color: #b0aea5; font-size: 0.875rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Eval Set Review: <span id="skill-name">__SKILL_NAME_PLACEHOLDER__</span></h1>
|
||||
<p class="description">Current description: <span id="skill-desc">__SKILL_DESCRIPTION_PLACEHOLDER__</span></p>
|
||||
|
||||
<div class="controls">
|
||||
<button class="btn btn-add" onclick="addRow()">+ Add Query</button>
|
||||
<button class="btn btn-export" onclick="exportEvalSet()">Export Eval Set</button>
|
||||
</div>
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="width:65%">Query</th>
|
||||
<th style="width:18%">Should Trigger</th>
|
||||
<th style="width:10%">Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="eval-body"></tbody>
|
||||
</table>
|
||||
|
||||
<p class="summary" id="summary"></p>
|
||||
|
||||
<script>
|
||||
const EVAL_DATA = __EVAL_DATA_PLACEHOLDER__;
|
||||
|
||||
let evalItems = [...EVAL_DATA];
|
||||
|
||||
function render() {
|
||||
const tbody = document.getElementById('eval-body');
|
||||
tbody.innerHTML = '';
|
||||
|
||||
// Sort: should-trigger first, then should-not-trigger
|
||||
const sorted = evalItems
|
||||
.map((item, origIdx) => ({ ...item, origIdx }))
|
||||
.sort((a, b) => (b.should_trigger ? 1 : 0) - (a.should_trigger ? 1 : 0));
|
||||
|
||||
let lastGroup = null;
|
||||
sorted.forEach(item => {
|
||||
const group = item.should_trigger ? 'trigger' : 'no-trigger';
|
||||
if (group !== lastGroup) {
|
||||
const headerRow = document.createElement('tr');
|
||||
headerRow.className = 'section-header';
|
||||
headerRow.innerHTML = `<td colspan="3">${item.should_trigger ? 'Should Trigger' : 'Should NOT Trigger'}</td>`;
|
||||
tbody.appendChild(headerRow);
|
||||
lastGroup = group;
|
||||
}
|
||||
|
||||
const idx = item.origIdx;
|
||||
const tr = document.createElement('tr');
|
||||
tr.innerHTML = `
|
||||
<td><textarea class="query-input" onchange="updateQuery(${idx}, this.value)">${escapeHtml(item.query)}</textarea></td>
|
||||
<td>
|
||||
<label class="toggle">
|
||||
<input type="checkbox" ${item.should_trigger ? 'checked' : ''} onchange="updateTrigger(${idx}, this.checked)">
|
||||
<span class="slider"></span>
|
||||
</label>
|
||||
<span style="margin-left:8px;font-size:0.8rem;color:#b0aea5">${item.should_trigger ? 'Yes' : 'No'}</span>
|
||||
</td>
|
||||
<td><button class="btn-delete" onclick="deleteRow(${idx})">Delete</button></td>
|
||||
`;
|
||||
tbody.appendChild(tr);
|
||||
});
|
||||
updateSummary();
|
||||
}
|
||||
|
||||
function escapeHtml(text) {
|
||||
const div = document.createElement('div');
|
||||
div.textContent = text;
|
||||
return div.innerHTML;
|
||||
}
|
||||
|
||||
function updateQuery(idx, value) { evalItems[idx].query = value; updateSummary(); }
|
||||
function updateTrigger(idx, value) { evalItems[idx].should_trigger = value; render(); }
|
||||
function deleteRow(idx) { evalItems.splice(idx, 1); render(); }
|
||||
|
||||
function addRow() {
|
||||
evalItems.push({ query: '', should_trigger: true });
|
||||
render();
|
||||
const inputs = document.querySelectorAll('.query-input');
|
||||
inputs[inputs.length - 1].focus();
|
||||
}
|
||||
|
||||
function updateSummary() {
|
||||
const trigger = evalItems.filter(i => i.should_trigger).length;
|
||||
const noTrigger = evalItems.filter(i => !i.should_trigger).length;
|
||||
document.getElementById('summary').textContent =
|
||||
`${evalItems.length} queries total: ${trigger} should trigger, ${noTrigger} should not trigger`;
|
||||
}
|
||||
|
||||
function exportEvalSet() {
|
||||
const valid = evalItems.filter(i => i.query.trim() !== '');
|
||||
const data = valid.map(i => ({ query: i.query.trim(), should_trigger: i.should_trigger }));
|
||||
const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = 'eval_set.json';
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
|
||||
render();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,471 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate and serve a review page for eval results.
|
||||
|
||||
Reads the workspace directory, discovers runs (directories with outputs/),
|
||||
embeds all output data into a self-contained HTML page, and serves it via
|
||||
a tiny HTTP server. Feedback auto-saves to feedback.json in the workspace.
|
||||
|
||||
Usage:
|
||||
python generate_review.py <workspace-path> [--port PORT] [--skill-name NAME]
|
||||
python generate_review.py <workspace-path> --previous-feedback /path/to/old/feedback.json
|
||||
|
||||
No dependencies beyond the Python stdlib are required.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import webbrowser
|
||||
from functools import partial
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
from pathlib import Path
|
||||
|
||||
# Files to exclude from output listings
|
||||
METADATA_FILES = {"transcript.md", "user_notes.md", "metrics.json"}
|
||||
|
||||
# Extensions we render as inline text
|
||||
TEXT_EXTENSIONS = {
|
||||
".txt", ".md", ".json", ".csv", ".py", ".js", ".ts", ".tsx", ".jsx",
|
||||
".yaml", ".yml", ".xml", ".html", ".css", ".sh", ".rb", ".go", ".rs",
|
||||
".java", ".c", ".cpp", ".h", ".hpp", ".sql", ".r", ".toml",
|
||||
}
|
||||
|
||||
# Extensions we render as inline images
|
||||
IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp"}
|
||||
|
||||
# MIME type overrides for common types
|
||||
MIME_OVERRIDES = {
|
||||
".svg": "image/svg+xml",
|
||||
".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
||||
}
|
||||
|
||||
|
||||
def get_mime_type(path: Path) -> str:
|
||||
ext = path.suffix.lower()
|
||||
if ext in MIME_OVERRIDES:
|
||||
return MIME_OVERRIDES[ext]
|
||||
mime, _ = mimetypes.guess_type(str(path))
|
||||
return mime or "application/octet-stream"
|
||||
|
||||
|
||||
def find_runs(workspace: Path) -> list[dict]:
|
||||
"""Recursively find directories that contain an outputs/ subdirectory."""
|
||||
runs: list[dict] = []
|
||||
_find_runs_recursive(workspace, workspace, runs)
|
||||
runs.sort(key=lambda r: (r.get("eval_id", float("inf")), r["id"]))
|
||||
return runs
|
||||
|
||||
|
||||
def _find_runs_recursive(root: Path, current: Path, runs: list[dict]) -> None:
|
||||
if not current.is_dir():
|
||||
return
|
||||
|
||||
outputs_dir = current / "outputs"
|
||||
if outputs_dir.is_dir():
|
||||
run = build_run(root, current)
|
||||
if run:
|
||||
runs.append(run)
|
||||
return
|
||||
|
||||
skip = {"node_modules", ".git", "__pycache__", "skill", "inputs"}
|
||||
for child in sorted(current.iterdir()):
|
||||
if child.is_dir() and child.name not in skip:
|
||||
_find_runs_recursive(root, child, runs)
|
||||
|
||||
|
||||
def build_run(root: Path, run_dir: Path) -> dict | None:
|
||||
"""Build a run dict with prompt, outputs, and grading data."""
|
||||
prompt = ""
|
||||
eval_id = None
|
||||
|
||||
# Try eval_metadata.json
|
||||
for candidate in [run_dir / "eval_metadata.json", run_dir.parent / "eval_metadata.json"]:
|
||||
if candidate.exists():
|
||||
try:
|
||||
metadata = json.loads(candidate.read_text())
|
||||
prompt = metadata.get("prompt", "")
|
||||
eval_id = metadata.get("eval_id")
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
if prompt:
|
||||
break
|
||||
|
||||
# Fall back to transcript.md
|
||||
if not prompt:
|
||||
for candidate in [run_dir / "transcript.md", run_dir / "outputs" / "transcript.md"]:
|
||||
if candidate.exists():
|
||||
try:
|
||||
text = candidate.read_text()
|
||||
match = re.search(r"## Eval Prompt\n\n([\s\S]*?)(?=\n##|$)", text)
|
||||
if match:
|
||||
prompt = match.group(1).strip()
|
||||
except OSError:
|
||||
pass
|
||||
if prompt:
|
||||
break
|
||||
|
||||
if not prompt:
|
||||
prompt = "(No prompt found)"
|
||||
|
||||
run_id = str(run_dir.relative_to(root)).replace("/", "-").replace("\\", "-")
|
||||
|
||||
# Collect output files
|
||||
outputs_dir = run_dir / "outputs"
|
||||
output_files: list[dict] = []
|
||||
if outputs_dir.is_dir():
|
||||
for f in sorted(outputs_dir.iterdir()):
|
||||
if f.is_file() and f.name not in METADATA_FILES:
|
||||
output_files.append(embed_file(f))
|
||||
|
||||
# Load grading if present
|
||||
grading = None
|
||||
for candidate in [run_dir / "grading.json", run_dir.parent / "grading.json"]:
|
||||
if candidate.exists():
|
||||
try:
|
||||
grading = json.loads(candidate.read_text())
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
if grading:
|
||||
break
|
||||
|
||||
return {
|
||||
"id": run_id,
|
||||
"prompt": prompt,
|
||||
"eval_id": eval_id,
|
||||
"outputs": output_files,
|
||||
"grading": grading,
|
||||
}
|
||||
|
||||
|
||||
def embed_file(path: Path) -> dict:
|
||||
"""Read a file and return an embedded representation."""
|
||||
ext = path.suffix.lower()
|
||||
mime = get_mime_type(path)
|
||||
|
||||
if ext in TEXT_EXTENSIONS:
|
||||
try:
|
||||
content = path.read_text(errors="replace")
|
||||
except OSError:
|
||||
content = "(Error reading file)"
|
||||
return {
|
||||
"name": path.name,
|
||||
"type": "text",
|
||||
"content": content,
|
||||
}
|
||||
elif ext in IMAGE_EXTENSIONS:
|
||||
try:
|
||||
raw = path.read_bytes()
|
||||
b64 = base64.b64encode(raw).decode("ascii")
|
||||
except OSError:
|
||||
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
|
||||
return {
|
||||
"name": path.name,
|
||||
"type": "image",
|
||||
"mime": mime,
|
||||
"data_uri": f"data:{mime};base64,{b64}",
|
||||
}
|
||||
elif ext == ".pdf":
|
||||
try:
|
||||
raw = path.read_bytes()
|
||||
b64 = base64.b64encode(raw).decode("ascii")
|
||||
except OSError:
|
||||
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
|
||||
return {
|
||||
"name": path.name,
|
||||
"type": "pdf",
|
||||
"data_uri": f"data:{mime};base64,{b64}",
|
||||
}
|
||||
elif ext == ".xlsx":
|
||||
try:
|
||||
raw = path.read_bytes()
|
||||
b64 = base64.b64encode(raw).decode("ascii")
|
||||
except OSError:
|
||||
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
|
||||
return {
|
||||
"name": path.name,
|
||||
"type": "xlsx",
|
||||
"data_b64": b64,
|
||||
}
|
||||
else:
|
||||
# Binary / unknown — base64 download link
|
||||
try:
|
||||
raw = path.read_bytes()
|
||||
b64 = base64.b64encode(raw).decode("ascii")
|
||||
except OSError:
|
||||
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
|
||||
return {
|
||||
"name": path.name,
|
||||
"type": "binary",
|
||||
"mime": mime,
|
||||
"data_uri": f"data:{mime};base64,{b64}",
|
||||
}
|
||||
|
||||
|
||||
def load_previous_iteration(workspace: Path) -> dict[str, dict]:
|
||||
"""Load previous iteration's feedback and outputs.
|
||||
|
||||
Returns a map of run_id -> {"feedback": str, "outputs": list[dict]}.
|
||||
"""
|
||||
result: dict[str, dict] = {}
|
||||
|
||||
# Load feedback
|
||||
feedback_map: dict[str, str] = {}
|
||||
feedback_path = workspace / "feedback.json"
|
||||
if feedback_path.exists():
|
||||
try:
|
||||
data = json.loads(feedback_path.read_text())
|
||||
feedback_map = {
|
||||
r["run_id"]: r["feedback"]
|
||||
for r in data.get("reviews", [])
|
||||
if r.get("feedback", "").strip()
|
||||
}
|
||||
except (json.JSONDecodeError, OSError, KeyError):
|
||||
pass
|
||||
|
||||
# Load runs (to get outputs)
|
||||
prev_runs = find_runs(workspace)
|
||||
for run in prev_runs:
|
||||
result[run["id"]] = {
|
||||
"feedback": feedback_map.get(run["id"], ""),
|
||||
"outputs": run.get("outputs", []),
|
||||
}
|
||||
|
||||
# Also add feedback for run_ids that had feedback but no matching run
|
||||
for run_id, fb in feedback_map.items():
|
||||
if run_id not in result:
|
||||
result[run_id] = {"feedback": fb, "outputs": []}
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def generate_html(
|
||||
runs: list[dict],
|
||||
skill_name: str,
|
||||
previous: dict[str, dict] | None = None,
|
||||
benchmark: dict | None = None,
|
||||
) -> str:
|
||||
"""Generate the complete standalone HTML page with embedded data."""
|
||||
template_path = Path(__file__).parent / "viewer.html"
|
||||
template = template_path.read_text()
|
||||
|
||||
# Build previous_feedback and previous_outputs maps for the template
|
||||
previous_feedback: dict[str, str] = {}
|
||||
previous_outputs: dict[str, list[dict]] = {}
|
||||
if previous:
|
||||
for run_id, data in previous.items():
|
||||
if data.get("feedback"):
|
||||
previous_feedback[run_id] = data["feedback"]
|
||||
if data.get("outputs"):
|
||||
previous_outputs[run_id] = data["outputs"]
|
||||
|
||||
embedded = {
|
||||
"skill_name": skill_name,
|
||||
"runs": runs,
|
||||
"previous_feedback": previous_feedback,
|
||||
"previous_outputs": previous_outputs,
|
||||
}
|
||||
if benchmark:
|
||||
embedded["benchmark"] = benchmark
|
||||
|
||||
data_json = json.dumps(embedded)
|
||||
|
||||
return template.replace("/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HTTP server (stdlib only, zero dependencies)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _kill_port(port: int) -> None:
|
||||
"""Kill any process listening on the given port."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["lsof", "-ti", f":{port}"],
|
||||
capture_output=True, text=True, timeout=5,
|
||||
)
|
||||
for pid_str in result.stdout.strip().split("\n"):
|
||||
if pid_str.strip():
|
||||
try:
|
||||
os.kill(int(pid_str.strip()), signal.SIGTERM)
|
||||
except (ProcessLookupError, ValueError):
|
||||
pass
|
||||
if result.stdout.strip():
|
||||
time.sleep(0.5)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
except FileNotFoundError:
|
||||
print("Note: lsof not found, cannot check if port is in use", file=sys.stderr)
|
||||
|
||||
class ReviewHandler(BaseHTTPRequestHandler):
|
||||
"""Serves the review HTML and handles feedback saves.
|
||||
|
||||
Regenerates the HTML on each page load so that refreshing the browser
|
||||
picks up new eval outputs without restarting the server.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
workspace: Path,
|
||||
skill_name: str,
|
||||
feedback_path: Path,
|
||||
previous: dict[str, dict],
|
||||
benchmark_path: Path | None,
|
||||
*args,
|
||||
**kwargs,
|
||||
):
|
||||
self.workspace = workspace
|
||||
self.skill_name = skill_name
|
||||
self.feedback_path = feedback_path
|
||||
self.previous = previous
|
||||
self.benchmark_path = benchmark_path
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
def do_GET(self) -> None:
|
||||
if self.path == "/" or self.path == "/index.html":
|
||||
# Regenerate HTML on each request (re-scans workspace for new outputs)
|
||||
runs = find_runs(self.workspace)
|
||||
benchmark = None
|
||||
if self.benchmark_path and self.benchmark_path.exists():
|
||||
try:
|
||||
benchmark = json.loads(self.benchmark_path.read_text())
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
html = generate_html(runs, self.skill_name, self.previous, benchmark)
|
||||
content = html.encode("utf-8")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(content)))
|
||||
self.end_headers()
|
||||
self.wfile.write(content)
|
||||
elif self.path == "/api/feedback":
|
||||
data = b"{}"
|
||||
if self.feedback_path.exists():
|
||||
data = self.feedback_path.read_bytes()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
else:
|
||||
self.send_error(404)
|
||||
|
||||
def do_POST(self) -> None:
|
||||
if self.path == "/api/feedback":
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
body = self.rfile.read(length)
|
||||
try:
|
||||
data = json.loads(body)
|
||||
if not isinstance(data, dict) or "reviews" not in data:
|
||||
raise ValueError("Expected JSON object with 'reviews' key")
|
||||
self.feedback_path.write_text(json.dumps(data, indent=2) + "\n")
|
||||
resp = b'{"ok":true}'
|
||||
self.send_response(200)
|
||||
except (json.JSONDecodeError, OSError, ValueError) as e:
|
||||
resp = json.dumps({"error": str(e)}).encode()
|
||||
self.send_response(500)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(resp)))
|
||||
self.end_headers()
|
||||
self.wfile.write(resp)
|
||||
else:
|
||||
self.send_error(404)
|
||||
|
||||
def log_message(self, format: str, *args: object) -> None:
|
||||
# Suppress request logging to keep terminal clean
|
||||
pass
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Generate and serve eval review")
|
||||
parser.add_argument("workspace", type=Path, help="Path to workspace directory")
|
||||
parser.add_argument("--port", "-p", type=int, default=3117, help="Server port (default: 3117)")
|
||||
parser.add_argument("--skill-name", "-n", type=str, default=None, help="Skill name for header")
|
||||
parser.add_argument(
|
||||
"--previous-workspace", type=Path, default=None,
|
||||
help="Path to previous iteration's workspace (shows old outputs and feedback as context)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--benchmark", type=Path, default=None,
|
||||
help="Path to benchmark.json to show in the Benchmark tab",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--static", "-s", type=Path, default=None,
|
||||
help="Write standalone HTML to this path instead of starting a server",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
workspace = args.workspace.resolve()
|
||||
if not workspace.is_dir():
|
||||
print(f"Error: {workspace} is not a directory", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
runs = find_runs(workspace)
|
||||
if not runs:
|
||||
print(f"No runs found in {workspace}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
skill_name = args.skill_name or workspace.name.replace("-workspace", "")
|
||||
feedback_path = workspace / "feedback.json"
|
||||
|
||||
previous: dict[str, dict] = {}
|
||||
if args.previous_workspace:
|
||||
previous = load_previous_iteration(args.previous_workspace.resolve())
|
||||
|
||||
benchmark_path = args.benchmark.resolve() if args.benchmark else None
|
||||
benchmark = None
|
||||
if benchmark_path and benchmark_path.exists():
|
||||
try:
|
||||
benchmark = json.loads(benchmark_path.read_text())
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
|
||||
if args.static:
|
||||
html = generate_html(runs, skill_name, previous, benchmark)
|
||||
args.static.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.static.write_text(html)
|
||||
print(f"\n Static viewer written to: {args.static}\n")
|
||||
sys.exit(0)
|
||||
|
||||
# Kill any existing process on the target port
|
||||
port = args.port
|
||||
_kill_port(port)
|
||||
handler = partial(ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path)
|
||||
try:
|
||||
server = HTTPServer(("127.0.0.1", port), handler)
|
||||
except OSError:
|
||||
# Port still in use after kill attempt — find a free one
|
||||
server = HTTPServer(("127.0.0.1", 0), handler)
|
||||
port = server.server_address[1]
|
||||
|
||||
url = f"http://localhost:{port}"
|
||||
print(f"\n Eval Viewer")
|
||||
print(f" ─────────────────────────────────")
|
||||
print(f" URL: {url}")
|
||||
print(f" Workspace: {workspace}")
|
||||
print(f" Feedback: {feedback_path}")
|
||||
if previous:
|
||||
print(f" Previous: {args.previous_workspace} ({len(previous)} runs)")
|
||||
if benchmark_path:
|
||||
print(f" Benchmark: {benchmark_path}")
|
||||
print(f"\n Press Ctrl+C to stop.\n")
|
||||
|
||||
webbrowser.open(url)
|
||||
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
print("\nStopped.")
|
||||
server.server_close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
82
skills/skill-creator/references/output-patterns.md
Normal file
82
skills/skill-creator/references/output-patterns.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Output Patterns
|
||||
|
||||
Use these patterns when skills need to produce consistent, high-quality output.
|
||||
|
||||
## Template Pattern
|
||||
|
||||
Provide templates for output format. Match the level of strictness to your needs.
|
||||
|
||||
**For strict requirements (like API responses or data formats):**
|
||||
|
||||
```markdown
|
||||
## Report structure
|
||||
|
||||
ALWAYS use this exact template structure:
|
||||
|
||||
# [Analysis Title]
|
||||
|
||||
## Executive summary
|
||||
[One-paragraph overview of key findings]
|
||||
|
||||
## Key findings
|
||||
- Finding 1 with supporting data
|
||||
- Finding 2 with supporting data
|
||||
- Finding 3 with supporting data
|
||||
|
||||
## Recommendations
|
||||
1. Specific actionable recommendation
|
||||
2. Specific actionable recommendation
|
||||
```
|
||||
|
||||
**For flexible guidance (when adaptation is useful):**
|
||||
|
||||
```markdown
|
||||
## Report structure
|
||||
|
||||
Here is a sensible default format, but use your best judgment:
|
||||
|
||||
# [Analysis Title]
|
||||
|
||||
## Executive summary
|
||||
[Overview]
|
||||
|
||||
## Key findings
|
||||
[Adapt sections based on what you discover]
|
||||
|
||||
## Recommendations
|
||||
[Tailor to the specific context]
|
||||
|
||||
Adjust sections as needed for the specific analysis type.
|
||||
```
|
||||
|
||||
## Examples Pattern
|
||||
|
||||
For skills where output quality depends on seeing examples, provide input/output pairs:
|
||||
|
||||
```markdown
|
||||
## Commit message format
|
||||
|
||||
Generate commit messages following these examples:
|
||||
|
||||
**Example 1:**
|
||||
Input: Added user authentication with JWT tokens
|
||||
Output:
|
||||
```
|
||||
feat(auth): implement JWT-based authentication
|
||||
|
||||
Add login endpoint and token validation middleware
|
||||
```
|
||||
|
||||
**Example 2:**
|
||||
Input: Fixed bug where dates displayed incorrectly in reports
|
||||
Output:
|
||||
```
|
||||
fix(reports): correct date formatting in timezone conversion
|
||||
|
||||
Use UTC timestamps consistently across report generation
|
||||
```
|
||||
|
||||
Follow this style: type(scope): brief description, then detailed explanation.
|
||||
```
|
||||
|
||||
Examples help Claude understand the desired style and level of detail more clearly than descriptions alone.
|
||||
@@ -1,430 +0,0 @@
|
||||
# JSON Schemas
|
||||
|
||||
This document defines the JSON schemas used by skill-creator.
|
||||
|
||||
---
|
||||
|
||||
## evals.json
|
||||
|
||||
Defines the evals for a skill. Located at `evals/evals.json` within the skill directory.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill_name": "example-skill",
|
||||
"evals": [
|
||||
{
|
||||
"id": 1,
|
||||
"prompt": "User's example prompt",
|
||||
"expected_output": "Description of expected result",
|
||||
"files": ["evals/files/sample1.pdf"],
|
||||
"expectations": [
|
||||
"The output includes X",
|
||||
"The skill used script Y"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `skill_name`: Name matching the skill's frontmatter
|
||||
- `evals[].id`: Unique integer identifier
|
||||
- `evals[].prompt`: The task to execute
|
||||
- `evals[].expected_output`: Human-readable description of success
|
||||
- `evals[].files`: Optional list of input file paths (relative to skill root)
|
||||
- `evals[].expectations`: List of verifiable statements
|
||||
|
||||
---
|
||||
|
||||
## history.json
|
||||
|
||||
Tracks version progression in Improve mode. Located at workspace root.
|
||||
|
||||
```json
|
||||
{
|
||||
"started_at": "2026-01-15T10:30:00Z",
|
||||
"skill_name": "pdf",
|
||||
"current_best": "v2",
|
||||
"iterations": [
|
||||
{
|
||||
"version": "v0",
|
||||
"parent": null,
|
||||
"expectation_pass_rate": 0.65,
|
||||
"grading_result": "baseline",
|
||||
"is_current_best": false
|
||||
},
|
||||
{
|
||||
"version": "v1",
|
||||
"parent": "v0",
|
||||
"expectation_pass_rate": 0.75,
|
||||
"grading_result": "won",
|
||||
"is_current_best": false
|
||||
},
|
||||
{
|
||||
"version": "v2",
|
||||
"parent": "v1",
|
||||
"expectation_pass_rate": 0.85,
|
||||
"grading_result": "won",
|
||||
"is_current_best": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `started_at`: ISO timestamp of when improvement started
|
||||
- `skill_name`: Name of the skill being improved
|
||||
- `current_best`: Version identifier of the best performer
|
||||
- `iterations[].version`: Version identifier (v0, v1, ...)
|
||||
- `iterations[].parent`: Parent version this was derived from
|
||||
- `iterations[].expectation_pass_rate`: Pass rate from grading
|
||||
- `iterations[].grading_result`: "baseline", "won", "lost", or "tie"
|
||||
- `iterations[].is_current_best`: Whether this is the current best version
|
||||
|
||||
---
|
||||
|
||||
## grading.json
|
||||
|
||||
Output from the grader agent. Located at `<run-dir>/grading.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"expectations": [
|
||||
{
|
||||
"text": "The output includes the name 'John Smith'",
|
||||
"passed": true,
|
||||
"evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
|
||||
},
|
||||
{
|
||||
"text": "The spreadsheet has a SUM formula in cell B10",
|
||||
"passed": false,
|
||||
"evidence": "No spreadsheet was created. The output was a text file."
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"passed": 2,
|
||||
"failed": 1,
|
||||
"total": 3,
|
||||
"pass_rate": 0.67
|
||||
},
|
||||
"execution_metrics": {
|
||||
"tool_calls": {
|
||||
"Read": 5,
|
||||
"Write": 2,
|
||||
"Bash": 8
|
||||
},
|
||||
"total_tool_calls": 15,
|
||||
"total_steps": 6,
|
||||
"errors_encountered": 0,
|
||||
"output_chars": 12450,
|
||||
"transcript_chars": 3200
|
||||
},
|
||||
"timing": {
|
||||
"executor_duration_seconds": 165.0,
|
||||
"grader_duration_seconds": 26.0,
|
||||
"total_duration_seconds": 191.0
|
||||
},
|
||||
"claims": [
|
||||
{
|
||||
"claim": "The form has 12 fillable fields",
|
||||
"type": "factual",
|
||||
"verified": true,
|
||||
"evidence": "Counted 12 fields in field_info.json"
|
||||
}
|
||||
],
|
||||
"user_notes_summary": {
|
||||
"uncertainties": ["Used 2023 data, may be stale"],
|
||||
"needs_review": [],
|
||||
"workarounds": ["Fell back to text overlay for non-fillable fields"]
|
||||
},
|
||||
"eval_feedback": {
|
||||
"suggestions": [
|
||||
{
|
||||
"assertion": "The output includes the name 'John Smith'",
|
||||
"reason": "A hallucinated document that mentions the name would also pass"
|
||||
}
|
||||
],
|
||||
"overall": "Assertions check presence but not correctness."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `expectations[]`: Graded expectations with evidence
|
||||
- `summary`: Aggregate pass/fail counts
|
||||
- `execution_metrics`: Tool usage and output size (from executor's metrics.json)
|
||||
- `timing`: Wall clock timing (from timing.json)
|
||||
- `claims`: Extracted and verified claims from the output
|
||||
- `user_notes_summary`: Issues flagged by the executor
|
||||
- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising
|
||||
|
||||
---
|
||||
|
||||
## metrics.json
|
||||
|
||||
Output from the executor agent. Located at `<run-dir>/outputs/metrics.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"tool_calls": {
|
||||
"Read": 5,
|
||||
"Write": 2,
|
||||
"Bash": 8,
|
||||
"Edit": 1,
|
||||
"Glob": 2,
|
||||
"Grep": 0
|
||||
},
|
||||
"total_tool_calls": 18,
|
||||
"total_steps": 6,
|
||||
"files_created": ["filled_form.pdf", "field_values.json"],
|
||||
"errors_encountered": 0,
|
||||
"output_chars": 12450,
|
||||
"transcript_chars": 3200
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `tool_calls`: Count per tool type
|
||||
- `total_tool_calls`: Sum of all tool calls
|
||||
- `total_steps`: Number of major execution steps
|
||||
- `files_created`: List of output files created
|
||||
- `errors_encountered`: Number of errors during execution
|
||||
- `output_chars`: Total character count of output files
|
||||
- `transcript_chars`: Character count of transcript
|
||||
|
||||
---
|
||||
|
||||
## timing.json
|
||||
|
||||
Wall clock timing for a run. Located at `<run-dir>/timing.json`.
|
||||
|
||||
**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.
|
||||
|
||||
```json
|
||||
{
|
||||
"total_tokens": 84852,
|
||||
"duration_ms": 23332,
|
||||
"total_duration_seconds": 23.3,
|
||||
"executor_start": "2026-01-15T10:30:00Z",
|
||||
"executor_end": "2026-01-15T10:32:45Z",
|
||||
"executor_duration_seconds": 165.0,
|
||||
"grader_start": "2026-01-15T10:32:46Z",
|
||||
"grader_end": "2026-01-15T10:33:12Z",
|
||||
"grader_duration_seconds": 26.0
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## benchmark.json
|
||||
|
||||
Output from Benchmark mode. Located at `benchmarks/<timestamp>/benchmark.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"skill_name": "pdf",
|
||||
"skill_path": "/path/to/pdf",
|
||||
"executor_model": "claude-sonnet-4-20250514",
|
||||
"analyzer_model": "most-capable-model",
|
||||
"timestamp": "2026-01-15T10:30:00Z",
|
||||
"evals_run": [1, 2, 3],
|
||||
"runs_per_configuration": 3
|
||||
},
|
||||
|
||||
"runs": [
|
||||
{
|
||||
"eval_id": 1,
|
||||
"eval_name": "Ocean",
|
||||
"configuration": "with_skill",
|
||||
"run_number": 1,
|
||||
"result": {
|
||||
"pass_rate": 0.85,
|
||||
"passed": 6,
|
||||
"failed": 1,
|
||||
"total": 7,
|
||||
"time_seconds": 42.5,
|
||||
"tokens": 3800,
|
||||
"tool_calls": 18,
|
||||
"errors": 0
|
||||
},
|
||||
"expectations": [
|
||||
{"text": "...", "passed": true, "evidence": "..."}
|
||||
],
|
||||
"notes": [
|
||||
"Used 2023 data, may be stale",
|
||||
"Fell back to text overlay for non-fillable fields"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
"run_summary": {
|
||||
"with_skill": {
|
||||
"pass_rate": {"mean": 0.85, "stddev": 0.05, "min": 0.80, "max": 0.90},
|
||||
"time_seconds": {"mean": 45.0, "stddev": 12.0, "min": 32.0, "max": 58.0},
|
||||
"tokens": {"mean": 3800, "stddev": 400, "min": 3200, "max": 4100}
|
||||
},
|
||||
"without_skill": {
|
||||
"pass_rate": {"mean": 0.35, "stddev": 0.08, "min": 0.28, "max": 0.45},
|
||||
"time_seconds": {"mean": 32.0, "stddev": 8.0, "min": 24.0, "max": 42.0},
|
||||
"tokens": {"mean": 2100, "stddev": 300, "min": 1800, "max": 2500}
|
||||
},
|
||||
"delta": {
|
||||
"pass_rate": "+0.50",
|
||||
"time_seconds": "+13.0",
|
||||
"tokens": "+1700"
|
||||
}
|
||||
},
|
||||
|
||||
"notes": [
|
||||
"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
|
||||
"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent",
|
||||
"Without-skill runs consistently fail on table extraction expectations",
|
||||
"Skill adds 13s average execution time but improves pass rate by 50%"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `metadata`: Information about the benchmark run
|
||||
- `skill_name`: Name of the skill
|
||||
- `timestamp`: When the benchmark was run
|
||||
- `evals_run`: List of eval names or IDs
|
||||
- `runs_per_configuration`: Number of runs per config (e.g. 3)
|
||||
- `runs[]`: Individual run results
|
||||
- `eval_id`: Numeric eval identifier
|
||||
- `eval_name`: Human-readable eval name (used as section header in the viewer)
|
||||
- `configuration`: Must be `"with_skill"` or `"without_skill"` (the viewer uses this exact string for grouping and color coding)
|
||||
- `run_number`: Integer run number (1, 2, 3...)
|
||||
- `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`
|
||||
- `run_summary`: Statistical aggregates per configuration
|
||||
- `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields
|
||||
- `delta`: Difference strings like `"+0.50"`, `"+13.0"`, `"+1700"`
|
||||
- `notes`: Freeform observations from the analyzer
|
||||
|
||||
**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.
|
||||
|
||||
---
|
||||
|
||||
## comparison.json
|
||||
|
||||
Output from blind comparator. Located at `<grading-dir>/comparison-N.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"winner": "A",
|
||||
"reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
|
||||
"rubric": {
|
||||
"A": {
|
||||
"content": {
|
||||
"correctness": 5,
|
||||
"completeness": 5,
|
||||
"accuracy": 4
|
||||
},
|
||||
"structure": {
|
||||
"organization": 4,
|
||||
"formatting": 5,
|
||||
"usability": 4
|
||||
},
|
||||
"content_score": 4.7,
|
||||
"structure_score": 4.3,
|
||||
"overall_score": 9.0
|
||||
},
|
||||
"B": {
|
||||
"content": {
|
||||
"correctness": 3,
|
||||
"completeness": 2,
|
||||
"accuracy": 3
|
||||
},
|
||||
"structure": {
|
||||
"organization": 3,
|
||||
"formatting": 2,
|
||||
"usability": 3
|
||||
},
|
||||
"content_score": 2.7,
|
||||
"structure_score": 2.7,
|
||||
"overall_score": 5.4
|
||||
}
|
||||
},
|
||||
"output_quality": {
|
||||
"A": {
|
||||
"score": 9,
|
||||
"strengths": ["Complete solution", "Well-formatted", "All fields present"],
|
||||
"weaknesses": ["Minor style inconsistency in header"]
|
||||
},
|
||||
"B": {
|
||||
"score": 5,
|
||||
"strengths": ["Readable output", "Correct basic structure"],
|
||||
"weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]
|
||||
}
|
||||
},
|
||||
"expectation_results": {
|
||||
"A": {
|
||||
"passed": 4,
|
||||
"total": 5,
|
||||
"pass_rate": 0.80,
|
||||
"details": [
|
||||
{"text": "Output includes name", "passed": true}
|
||||
]
|
||||
},
|
||||
"B": {
|
||||
"passed": 3,
|
||||
"total": 5,
|
||||
"pass_rate": 0.60,
|
||||
"details": [
|
||||
{"text": "Output includes name", "passed": true}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## analysis.json
|
||||
|
||||
Output from post-hoc analyzer. Located at `<grading-dir>/analysis.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"comparison_summary": {
|
||||
"winner": "A",
|
||||
"winner_skill": "path/to/winner/skill",
|
||||
"loser_skill": "path/to/loser/skill",
|
||||
"comparator_reasoning": "Brief summary of why comparator chose winner"
|
||||
},
|
||||
"winner_strengths": [
|
||||
"Clear step-by-step instructions for handling multi-page documents",
|
||||
"Included validation script that caught formatting errors"
|
||||
],
|
||||
"loser_weaknesses": [
|
||||
"Vague instruction 'process the document appropriately' led to inconsistent behavior",
|
||||
"No script for validation, agent had to improvise"
|
||||
],
|
||||
"instruction_following": {
|
||||
"winner": {
|
||||
"score": 9,
|
||||
"issues": ["Minor: skipped optional logging step"]
|
||||
},
|
||||
"loser": {
|
||||
"score": 6,
|
||||
"issues": [
|
||||
"Did not use the skill's formatting template",
|
||||
"Invented own approach instead of following step 3"
|
||||
]
|
||||
}
|
||||
},
|
||||
"improvement_suggestions": [
|
||||
{
|
||||
"priority": "high",
|
||||
"category": "instructions",
|
||||
"suggestion": "Replace 'process the document appropriately' with explicit steps",
|
||||
"expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
|
||||
}
|
||||
],
|
||||
"transcript_insights": {
|
||||
"winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script",
|
||||
"loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods"
|
||||
}
|
||||
}
|
||||
```
|
||||
28
skills/skill-creator/references/workflows.md
Normal file
28
skills/skill-creator/references/workflows.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Workflow Patterns
|
||||
|
||||
## Sequential Workflows
|
||||
|
||||
For complex tasks, break operations into clear, sequential steps. It is often helpful to give Claude an overview of the process towards the beginning of SKILL.md:
|
||||
|
||||
```markdown
|
||||
Filling a PDF form involves these steps:
|
||||
|
||||
1. Analyze the form (run analyze_form.py)
|
||||
2. Create field mapping (edit fields.json)
|
||||
3. Validate mapping (run validate_fields.py)
|
||||
4. Fill the form (run fill_form.py)
|
||||
5. Verify output (run verify_output.py)
|
||||
```
|
||||
|
||||
## Conditional Workflows
|
||||
|
||||
For tasks with branching logic, guide Claude through decision points:
|
||||
|
||||
```markdown
|
||||
1. Determine the modification type:
|
||||
**Creating new content?** → Follow "Creation workflow" below
|
||||
**Editing existing content?** → Follow "Editing workflow" below
|
||||
|
||||
2. Creation workflow: [steps]
|
||||
3. Editing workflow: [steps]
|
||||
```
|
||||
@@ -1,401 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Aggregate individual run results into benchmark summary statistics.
|
||||
|
||||
Reads grading.json files from run directories and produces:
|
||||
- run_summary with mean, stddev, min, max for each metric
|
||||
- delta between with_skill and without_skill configurations
|
||||
|
||||
Usage:
|
||||
python aggregate_benchmark.py <benchmark_dir>
|
||||
|
||||
Example:
|
||||
python aggregate_benchmark.py benchmarks/2026-01-15T10-30-00/
|
||||
|
||||
The script supports two directory layouts:
|
||||
|
||||
Workspace layout (from skill-creator iterations):
|
||||
<benchmark_dir>/
|
||||
└── eval-N/
|
||||
├── with_skill/
|
||||
│ ├── run-1/grading.json
|
||||
│ └── run-2/grading.json
|
||||
└── without_skill/
|
||||
├── run-1/grading.json
|
||||
└── run-2/grading.json
|
||||
|
||||
Legacy layout (with runs/ subdirectory):
|
||||
<benchmark_dir>/
|
||||
└── runs/
|
||||
└── eval-N/
|
||||
├── with_skill/
|
||||
│ └── run-1/grading.json
|
||||
└── without_skill/
|
||||
└── run-1/grading.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def calculate_stats(values: list[float]) -> dict:
|
||||
"""Calculate mean, stddev, min, max for a list of values."""
|
||||
if not values:
|
||||
return {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0}
|
||||
|
||||
n = len(values)
|
||||
mean = sum(values) / n
|
||||
|
||||
if n > 1:
|
||||
variance = sum((x - mean) ** 2 for x in values) / (n - 1)
|
||||
stddev = math.sqrt(variance)
|
||||
else:
|
||||
stddev = 0.0
|
||||
|
||||
return {
|
||||
"mean": round(mean, 4),
|
||||
"stddev": round(stddev, 4),
|
||||
"min": round(min(values), 4),
|
||||
"max": round(max(values), 4)
|
||||
}
|
||||
|
||||
|
||||
def load_run_results(benchmark_dir: Path) -> dict:
|
||||
"""
|
||||
Load all run results from a benchmark directory.
|
||||
|
||||
Returns dict keyed by config name (e.g. "with_skill"/"without_skill",
|
||||
or "new_skill"/"old_skill"), each containing a list of run results.
|
||||
"""
|
||||
# Support both layouts: eval dirs directly under benchmark_dir, or under runs/
|
||||
runs_dir = benchmark_dir / "runs"
|
||||
if runs_dir.exists():
|
||||
search_dir = runs_dir
|
||||
elif list(benchmark_dir.glob("eval-*")):
|
||||
search_dir = benchmark_dir
|
||||
else:
|
||||
print(f"No eval directories found in {benchmark_dir} or {benchmark_dir / 'runs'}")
|
||||
return {}
|
||||
|
||||
results: dict[str, list] = {}
|
||||
|
||||
for eval_idx, eval_dir in enumerate(sorted(search_dir.glob("eval-*"))):
|
||||
metadata_path = eval_dir / "eval_metadata.json"
|
||||
if metadata_path.exists():
|
||||
try:
|
||||
with open(metadata_path) as mf:
|
||||
eval_id = json.load(mf).get("eval_id", eval_idx)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
eval_id = eval_idx
|
||||
else:
|
||||
try:
|
||||
eval_id = int(eval_dir.name.split("-")[1])
|
||||
except ValueError:
|
||||
eval_id = eval_idx
|
||||
|
||||
# Discover config directories dynamically rather than hardcoding names
|
||||
for config_dir in sorted(eval_dir.iterdir()):
|
||||
if not config_dir.is_dir():
|
||||
continue
|
||||
# Skip non-config directories (inputs, outputs, etc.)
|
||||
if not list(config_dir.glob("run-*")):
|
||||
continue
|
||||
config = config_dir.name
|
||||
if config not in results:
|
||||
results[config] = []
|
||||
|
||||
for run_dir in sorted(config_dir.glob("run-*")):
|
||||
run_number = int(run_dir.name.split("-")[1])
|
||||
grading_file = run_dir / "grading.json"
|
||||
|
||||
if not grading_file.exists():
|
||||
print(f"Warning: grading.json not found in {run_dir}")
|
||||
continue
|
||||
|
||||
try:
|
||||
with open(grading_file) as f:
|
||||
grading = json.load(f)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Warning: Invalid JSON in {grading_file}: {e}")
|
||||
continue
|
||||
|
||||
# Extract metrics
|
||||
result = {
|
||||
"eval_id": eval_id,
|
||||
"run_number": run_number,
|
||||
"pass_rate": grading.get("summary", {}).get("pass_rate", 0.0),
|
||||
"passed": grading.get("summary", {}).get("passed", 0),
|
||||
"failed": grading.get("summary", {}).get("failed", 0),
|
||||
"total": grading.get("summary", {}).get("total", 0),
|
||||
}
|
||||
|
||||
# Extract timing — check grading.json first, then sibling timing.json
|
||||
timing = grading.get("timing", {})
|
||||
result["time_seconds"] = timing.get("total_duration_seconds", 0.0)
|
||||
timing_file = run_dir / "timing.json"
|
||||
if result["time_seconds"] == 0.0 and timing_file.exists():
|
||||
try:
|
||||
with open(timing_file) as tf:
|
||||
timing_data = json.load(tf)
|
||||
result["time_seconds"] = timing_data.get("total_duration_seconds", 0.0)
|
||||
result["tokens"] = timing_data.get("total_tokens", 0)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Extract metrics if available
|
||||
metrics = grading.get("execution_metrics", {})
|
||||
result["tool_calls"] = metrics.get("total_tool_calls", 0)
|
||||
if not result.get("tokens"):
|
||||
result["tokens"] = metrics.get("output_chars", 0)
|
||||
result["errors"] = metrics.get("errors_encountered", 0)
|
||||
|
||||
# Extract expectations — viewer requires fields: text, passed, evidence
|
||||
raw_expectations = grading.get("expectations", [])
|
||||
for exp in raw_expectations:
|
||||
if "text" not in exp or "passed" not in exp:
|
||||
print(f"Warning: expectation in {grading_file} missing required fields (text, passed, evidence): {exp}")
|
||||
result["expectations"] = raw_expectations
|
||||
|
||||
# Extract notes from user_notes_summary
|
||||
notes_summary = grading.get("user_notes_summary", {})
|
||||
notes = []
|
||||
notes.extend(notes_summary.get("uncertainties", []))
|
||||
notes.extend(notes_summary.get("needs_review", []))
|
||||
notes.extend(notes_summary.get("workarounds", []))
|
||||
result["notes"] = notes
|
||||
|
||||
results[config].append(result)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def aggregate_results(results: dict) -> dict:
|
||||
"""
|
||||
Aggregate run results into summary statistics.
|
||||
|
||||
Returns run_summary with stats for each configuration and delta.
|
||||
"""
|
||||
run_summary = {}
|
||||
configs = list(results.keys())
|
||||
|
||||
for config in configs:
|
||||
runs = results.get(config, [])
|
||||
|
||||
if not runs:
|
||||
run_summary[config] = {
|
||||
"pass_rate": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0},
|
||||
"time_seconds": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0},
|
||||
"tokens": {"mean": 0, "stddev": 0, "min": 0, "max": 0}
|
||||
}
|
||||
continue
|
||||
|
||||
pass_rates = [r["pass_rate"] for r in runs]
|
||||
times = [r["time_seconds"] for r in runs]
|
||||
tokens = [r.get("tokens", 0) for r in runs]
|
||||
|
||||
run_summary[config] = {
|
||||
"pass_rate": calculate_stats(pass_rates),
|
||||
"time_seconds": calculate_stats(times),
|
||||
"tokens": calculate_stats(tokens)
|
||||
}
|
||||
|
||||
# Calculate delta between the first two configs (if two exist)
|
||||
if len(configs) >= 2:
|
||||
primary = run_summary.get(configs[0], {})
|
||||
baseline = run_summary.get(configs[1], {})
|
||||
else:
|
||||
primary = run_summary.get(configs[0], {}) if configs else {}
|
||||
baseline = {}
|
||||
|
||||
delta_pass_rate = primary.get("pass_rate", {}).get("mean", 0) - baseline.get("pass_rate", {}).get("mean", 0)
|
||||
delta_time = primary.get("time_seconds", {}).get("mean", 0) - baseline.get("time_seconds", {}).get("mean", 0)
|
||||
delta_tokens = primary.get("tokens", {}).get("mean", 0) - baseline.get("tokens", {}).get("mean", 0)
|
||||
|
||||
run_summary["delta"] = {
|
||||
"pass_rate": f"{delta_pass_rate:+.2f}",
|
||||
"time_seconds": f"{delta_time:+.1f}",
|
||||
"tokens": f"{delta_tokens:+.0f}"
|
||||
}
|
||||
|
||||
return run_summary
|
||||
|
||||
|
||||
def generate_benchmark(benchmark_dir: Path, skill_name: str = "", skill_path: str = "") -> dict:
|
||||
"""
|
||||
Generate complete benchmark.json from run results.
|
||||
"""
|
||||
results = load_run_results(benchmark_dir)
|
||||
run_summary = aggregate_results(results)
|
||||
|
||||
# Build runs array for benchmark.json
|
||||
runs = []
|
||||
for config in results:
|
||||
for result in results[config]:
|
||||
runs.append({
|
||||
"eval_id": result["eval_id"],
|
||||
"configuration": config,
|
||||
"run_number": result["run_number"],
|
||||
"result": {
|
||||
"pass_rate": result["pass_rate"],
|
||||
"passed": result["passed"],
|
||||
"failed": result["failed"],
|
||||
"total": result["total"],
|
||||
"time_seconds": result["time_seconds"],
|
||||
"tokens": result.get("tokens", 0),
|
||||
"tool_calls": result.get("tool_calls", 0),
|
||||
"errors": result.get("errors", 0)
|
||||
},
|
||||
"expectations": result["expectations"],
|
||||
"notes": result["notes"]
|
||||
})
|
||||
|
||||
# Determine eval IDs from results
|
||||
eval_ids = sorted(set(
|
||||
r["eval_id"]
|
||||
for config in results.values()
|
||||
for r in config
|
||||
))
|
||||
|
||||
benchmark = {
|
||||
"metadata": {
|
||||
"skill_name": skill_name or "<skill-name>",
|
||||
"skill_path": skill_path or "<path/to/skill>",
|
||||
"executor_model": "<model-name>",
|
||||
"analyzer_model": "<model-name>",
|
||||
"timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"evals_run": eval_ids,
|
||||
"runs_per_configuration": 3
|
||||
},
|
||||
"runs": runs,
|
||||
"run_summary": run_summary,
|
||||
"notes": [] # To be filled by analyzer
|
||||
}
|
||||
|
||||
return benchmark
|
||||
|
||||
|
||||
def generate_markdown(benchmark: dict) -> str:
|
||||
"""Generate human-readable benchmark.md from benchmark data."""
|
||||
metadata = benchmark["metadata"]
|
||||
run_summary = benchmark["run_summary"]
|
||||
|
||||
# Determine config names (excluding "delta")
|
||||
configs = [k for k in run_summary if k != "delta"]
|
||||
config_a = configs[0] if len(configs) >= 1 else "config_a"
|
||||
config_b = configs[1] if len(configs) >= 2 else "config_b"
|
||||
label_a = config_a.replace("_", " ").title()
|
||||
label_b = config_b.replace("_", " ").title()
|
||||
|
||||
lines = [
|
||||
f"# Skill Benchmark: {metadata['skill_name']}",
|
||||
"",
|
||||
f"**Model**: {metadata['executor_model']}",
|
||||
f"**Date**: {metadata['timestamp']}",
|
||||
f"**Evals**: {', '.join(map(str, metadata['evals_run']))} ({metadata['runs_per_configuration']} runs each per configuration)",
|
||||
"",
|
||||
"## Summary",
|
||||
"",
|
||||
f"| Metric | {label_a} | {label_b} | Delta |",
|
||||
"|--------|------------|---------------|-------|",
|
||||
]
|
||||
|
||||
a_summary = run_summary.get(config_a, {})
|
||||
b_summary = run_summary.get(config_b, {})
|
||||
delta = run_summary.get("delta", {})
|
||||
|
||||
# Format pass rate
|
||||
a_pr = a_summary.get("pass_rate", {})
|
||||
b_pr = b_summary.get("pass_rate", {})
|
||||
lines.append(f"| Pass Rate | {a_pr.get('mean', 0)*100:.0f}% ± {a_pr.get('stddev', 0)*100:.0f}% | {b_pr.get('mean', 0)*100:.0f}% ± {b_pr.get('stddev', 0)*100:.0f}% | {delta.get('pass_rate', '—')} |")
|
||||
|
||||
# Format time
|
||||
a_time = a_summary.get("time_seconds", {})
|
||||
b_time = b_summary.get("time_seconds", {})
|
||||
lines.append(f"| Time | {a_time.get('mean', 0):.1f}s ± {a_time.get('stddev', 0):.1f}s | {b_time.get('mean', 0):.1f}s ± {b_time.get('stddev', 0):.1f}s | {delta.get('time_seconds', '—')}s |")
|
||||
|
||||
# Format tokens
|
||||
a_tokens = a_summary.get("tokens", {})
|
||||
b_tokens = b_summary.get("tokens", {})
|
||||
lines.append(f"| Tokens | {a_tokens.get('mean', 0):.0f} ± {a_tokens.get('stddev', 0):.0f} | {b_tokens.get('mean', 0):.0f} ± {b_tokens.get('stddev', 0):.0f} | {delta.get('tokens', '—')} |")
|
||||
|
||||
# Notes section
|
||||
if benchmark.get("notes"):
|
||||
lines.extend([
|
||||
"",
|
||||
"## Notes",
|
||||
""
|
||||
])
|
||||
for note in benchmark["notes"]:
|
||||
lines.append(f"- {note}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Aggregate benchmark run results into summary statistics"
|
||||
)
|
||||
parser.add_argument(
|
||||
"benchmark_dir",
|
||||
type=Path,
|
||||
help="Path to the benchmark directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skill-name",
|
||||
default="",
|
||||
help="Name of the skill being benchmarked"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skill-path",
|
||||
default="",
|
||||
help="Path to the skill being benchmarked"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output", "-o",
|
||||
type=Path,
|
||||
help="Output path for benchmark.json (default: <benchmark_dir>/benchmark.json)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.benchmark_dir.exists():
|
||||
print(f"Directory not found: {args.benchmark_dir}")
|
||||
sys.exit(1)
|
||||
|
||||
# Generate benchmark
|
||||
benchmark = generate_benchmark(args.benchmark_dir, args.skill_name, args.skill_path)
|
||||
|
||||
# Determine output paths
|
||||
output_json = args.output or (args.benchmark_dir / "benchmark.json")
|
||||
output_md = output_json.with_suffix(".md")
|
||||
|
||||
# Write benchmark.json
|
||||
with open(output_json, "w") as f:
|
||||
json.dump(benchmark, f, indent=2)
|
||||
print(f"Generated: {output_json}")
|
||||
|
||||
# Write benchmark.md
|
||||
markdown = generate_markdown(benchmark)
|
||||
with open(output_md, "w") as f:
|
||||
f.write(markdown)
|
||||
print(f"Generated: {output_md}")
|
||||
|
||||
# Print summary
|
||||
run_summary = benchmark["run_summary"]
|
||||
configs = [k for k in run_summary if k != "delta"]
|
||||
delta = run_summary.get("delta", {})
|
||||
|
||||
print(f"\nSummary:")
|
||||
for config in configs:
|
||||
pr = run_summary[config]["pass_rate"]["mean"]
|
||||
label = config.replace("_", " ").title()
|
||||
print(f" {label}: {pr*100:.1f}% pass rate")
|
||||
print(f" Delta: {delta.get('pass_rate', '—')}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,326 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate an HTML report from run_loop.py output.
|
||||
|
||||
Takes the JSON output from run_loop.py and generates a visual HTML report
|
||||
showing each description attempt with check/x for each test case.
|
||||
Distinguishes between train and test queries.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import html
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def generate_html(data: dict, auto_refresh: bool = False, skill_name: str = "") -> str:
|
||||
"""Generate HTML report from loop output data. If auto_refresh is True, adds a meta refresh tag."""
|
||||
history = data.get("history", [])
|
||||
holdout = data.get("holdout", 0)
|
||||
title_prefix = html.escape(skill_name + " \u2014 ") if skill_name else ""
|
||||
|
||||
# Get all unique queries from train and test sets, with should_trigger info
|
||||
train_queries: list[dict] = []
|
||||
test_queries: list[dict] = []
|
||||
if history:
|
||||
for r in history[0].get("train_results", history[0].get("results", [])):
|
||||
train_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)})
|
||||
if history[0].get("test_results"):
|
||||
for r in history[0].get("test_results", []):
|
||||
test_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)})
|
||||
|
||||
refresh_tag = ' <meta http-equiv="refresh" content="5">\n' if auto_refresh else ""
|
||||
|
||||
html_parts = ["""<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
""" + refresh_tag + """ <title>""" + title_prefix + """Skill Description Optimization</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
body {
|
||||
font-family: 'Lora', Georgia, serif;
|
||||
max-width: 100%;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
background: #faf9f5;
|
||||
color: #141413;
|
||||
}
|
||||
h1 { font-family: 'Poppins', sans-serif; color: #141413; }
|
||||
.explainer {
|
||||
background: white;
|
||||
padding: 15px;
|
||||
border-radius: 6px;
|
||||
margin-bottom: 20px;
|
||||
border: 1px solid #e8e6dc;
|
||||
color: #b0aea5;
|
||||
font-size: 0.875rem;
|
||||
line-height: 1.6;
|
||||
}
|
||||
.summary {
|
||||
background: white;
|
||||
padding: 15px;
|
||||
border-radius: 6px;
|
||||
margin-bottom: 20px;
|
||||
border: 1px solid #e8e6dc;
|
||||
}
|
||||
.summary p { margin: 5px 0; }
|
||||
.best { color: #788c5d; font-weight: bold; }
|
||||
.table-container {
|
||||
overflow-x: auto;
|
||||
width: 100%;
|
||||
}
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
background: white;
|
||||
border: 1px solid #e8e6dc;
|
||||
border-radius: 6px;
|
||||
font-size: 12px;
|
||||
min-width: 100%;
|
||||
}
|
||||
th, td {
|
||||
padding: 8px;
|
||||
text-align: left;
|
||||
border: 1px solid #e8e6dc;
|
||||
white-space: normal;
|
||||
word-wrap: break-word;
|
||||
}
|
||||
th {
|
||||
font-family: 'Poppins', sans-serif;
|
||||
background: #141413;
|
||||
color: #faf9f5;
|
||||
font-weight: 500;
|
||||
}
|
||||
th.test-col {
|
||||
background: #6a9bcc;
|
||||
}
|
||||
th.query-col { min-width: 200px; }
|
||||
td.description {
|
||||
font-family: monospace;
|
||||
font-size: 11px;
|
||||
word-wrap: break-word;
|
||||
max-width: 400px;
|
||||
}
|
||||
td.result {
|
||||
text-align: center;
|
||||
font-size: 16px;
|
||||
min-width: 40px;
|
||||
}
|
||||
td.test-result {
|
||||
background: #f0f6fc;
|
||||
}
|
||||
.pass { color: #788c5d; }
|
||||
.fail { color: #c44; }
|
||||
.rate {
|
||||
font-size: 9px;
|
||||
color: #b0aea5;
|
||||
display: block;
|
||||
}
|
||||
tr:hover { background: #faf9f5; }
|
||||
.score {
|
||||
display: inline-block;
|
||||
padding: 2px 6px;
|
||||
border-radius: 4px;
|
||||
font-weight: bold;
|
||||
font-size: 11px;
|
||||
}
|
||||
.score-good { background: #eef2e8; color: #788c5d; }
|
||||
.score-ok { background: #fef3c7; color: #d97706; }
|
||||
.score-bad { background: #fceaea; color: #c44; }
|
||||
.train-label { color: #b0aea5; font-size: 10px; }
|
||||
.test-label { color: #6a9bcc; font-size: 10px; font-weight: bold; }
|
||||
.best-row { background: #f5f8f2; }
|
||||
th.positive-col { border-bottom: 3px solid #788c5d; }
|
||||
th.negative-col { border-bottom: 3px solid #c44; }
|
||||
th.test-col.positive-col { border-bottom: 3px solid #788c5d; }
|
||||
th.test-col.negative-col { border-bottom: 3px solid #c44; }
|
||||
.legend { font-family: 'Poppins', sans-serif; display: flex; gap: 20px; margin-bottom: 10px; font-size: 13px; align-items: center; }
|
||||
.legend-item { display: flex; align-items: center; gap: 6px; }
|
||||
.legend-swatch { width: 16px; height: 16px; border-radius: 3px; display: inline-block; }
|
||||
.swatch-positive { background: #141413; border-bottom: 3px solid #788c5d; }
|
||||
.swatch-negative { background: #141413; border-bottom: 3px solid #c44; }
|
||||
.swatch-test { background: #6a9bcc; }
|
||||
.swatch-train { background: #141413; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>""" + title_prefix + """Skill Description Optimization</h1>
|
||||
<div class="explainer">
|
||||
<strong>Optimizing your skill's description.</strong> This page updates automatically as Claude tests different versions of your skill's description. Each row is an iteration — a new description attempt. The columns show test queries: green checkmarks mean the skill triggered correctly (or correctly didn't trigger), red crosses mean it got it wrong. The "Train" score shows performance on queries used to improve the description; the "Test" score shows performance on held-out queries the optimizer hasn't seen. When it's done, Claude will apply the best-performing description to your skill.
|
||||
</div>
|
||||
"""]
|
||||
|
||||
# Summary section
|
||||
best_test_score = data.get('best_test_score')
|
||||
best_train_score = data.get('best_train_score')
|
||||
html_parts.append(f"""
|
||||
<div class="summary">
|
||||
<p><strong>Original:</strong> {html.escape(data.get('original_description', 'N/A'))}</p>
|
||||
<p class="best"><strong>Best:</strong> {html.escape(data.get('best_description', 'N/A'))}</p>
|
||||
<p><strong>Best Score:</strong> {data.get('best_score', 'N/A')} {'(test)' if best_test_score else '(train)'}</p>
|
||||
<p><strong>Iterations:</strong> {data.get('iterations_run', 0)} | <strong>Train:</strong> {data.get('train_size', '?')} | <strong>Test:</strong> {data.get('test_size', '?')}</p>
|
||||
</div>
|
||||
""")
|
||||
|
||||
# Legend
|
||||
html_parts.append("""
|
||||
<div class="legend">
|
||||
<span style="font-weight:600">Query columns:</span>
|
||||
<span class="legend-item"><span class="legend-swatch swatch-positive"></span> Should trigger</span>
|
||||
<span class="legend-item"><span class="legend-swatch swatch-negative"></span> Should NOT trigger</span>
|
||||
<span class="legend-item"><span class="legend-swatch swatch-train"></span> Train</span>
|
||||
<span class="legend-item"><span class="legend-swatch swatch-test"></span> Test</span>
|
||||
</div>
|
||||
""")
|
||||
|
||||
# Table header
|
||||
html_parts.append("""
|
||||
<div class="table-container">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Iter</th>
|
||||
<th>Train</th>
|
||||
<th>Test</th>
|
||||
<th class="query-col">Description</th>
|
||||
""")
|
||||
|
||||
# Add column headers for train queries
|
||||
for qinfo in train_queries:
|
||||
polarity = "positive-col" if qinfo["should_trigger"] else "negative-col"
|
||||
html_parts.append(f' <th class="{polarity}">{html.escape(qinfo["query"])}</th>\n')
|
||||
|
||||
# Add column headers for test queries (different color)
|
||||
for qinfo in test_queries:
|
||||
polarity = "positive-col" if qinfo["should_trigger"] else "negative-col"
|
||||
html_parts.append(f' <th class="test-col {polarity}">{html.escape(qinfo["query"])}</th>\n')
|
||||
|
||||
html_parts.append(""" </tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
""")
|
||||
|
||||
# Find best iteration for highlighting
|
||||
if test_queries:
|
||||
best_iter = max(history, key=lambda h: h.get("test_passed") or 0).get("iteration")
|
||||
else:
|
||||
best_iter = max(history, key=lambda h: h.get("train_passed", h.get("passed", 0))).get("iteration")
|
||||
|
||||
# Add rows for each iteration
|
||||
for h in history:
|
||||
iteration = h.get("iteration", "?")
|
||||
train_passed = h.get("train_passed", h.get("passed", 0))
|
||||
train_total = h.get("train_total", h.get("total", 0))
|
||||
test_passed = h.get("test_passed")
|
||||
test_total = h.get("test_total")
|
||||
description = h.get("description", "")
|
||||
train_results = h.get("train_results", h.get("results", []))
|
||||
test_results = h.get("test_results", [])
|
||||
|
||||
# Create lookups for results by query
|
||||
train_by_query = {r["query"]: r for r in train_results}
|
||||
test_by_query = {r["query"]: r for r in test_results} if test_results else {}
|
||||
|
||||
# Compute aggregate correct/total runs across all retries
|
||||
def aggregate_runs(results: list[dict]) -> tuple[int, int]:
|
||||
correct = 0
|
||||
total = 0
|
||||
for r in results:
|
||||
runs = r.get("runs", 0)
|
||||
triggers = r.get("triggers", 0)
|
||||
total += runs
|
||||
if r.get("should_trigger", True):
|
||||
correct += triggers
|
||||
else:
|
||||
correct += runs - triggers
|
||||
return correct, total
|
||||
|
||||
train_correct, train_runs = aggregate_runs(train_results)
|
||||
test_correct, test_runs = aggregate_runs(test_results)
|
||||
|
||||
# Determine score classes
|
||||
def score_class(correct: int, total: int) -> str:
|
||||
if total > 0:
|
||||
ratio = correct / total
|
||||
if ratio >= 0.8:
|
||||
return "score-good"
|
||||
elif ratio >= 0.5:
|
||||
return "score-ok"
|
||||
return "score-bad"
|
||||
|
||||
train_class = score_class(train_correct, train_runs)
|
||||
test_class = score_class(test_correct, test_runs)
|
||||
|
||||
row_class = "best-row" if iteration == best_iter else ""
|
||||
|
||||
html_parts.append(f""" <tr class="{row_class}">
|
||||
<td>{iteration}</td>
|
||||
<td><span class="score {train_class}">{train_correct}/{train_runs}</span></td>
|
||||
<td><span class="score {test_class}">{test_correct}/{test_runs}</span></td>
|
||||
<td class="description">{html.escape(description)}</td>
|
||||
""")
|
||||
|
||||
# Add result for each train query
|
||||
for qinfo in train_queries:
|
||||
r = train_by_query.get(qinfo["query"], {})
|
||||
did_pass = r.get("pass", False)
|
||||
triggers = r.get("triggers", 0)
|
||||
runs = r.get("runs", 0)
|
||||
|
||||
icon = "✓" if did_pass else "✗"
|
||||
css_class = "pass" if did_pass else "fail"
|
||||
|
||||
html_parts.append(f' <td class="result {css_class}">{icon}<span class="rate">{triggers}/{runs}</span></td>\n')
|
||||
|
||||
# Add result for each test query (with different background)
|
||||
for qinfo in test_queries:
|
||||
r = test_by_query.get(qinfo["query"], {})
|
||||
did_pass = r.get("pass", False)
|
||||
triggers = r.get("triggers", 0)
|
||||
runs = r.get("runs", 0)
|
||||
|
||||
icon = "✓" if did_pass else "✗"
|
||||
css_class = "pass" if did_pass else "fail"
|
||||
|
||||
html_parts.append(f' <td class="result test-result {css_class}">{icon}<span class="rate">{triggers}/{runs}</span></td>\n')
|
||||
|
||||
html_parts.append(" </tr>\n")
|
||||
|
||||
html_parts.append(""" </tbody>
|
||||
</table>
|
||||
</div>
|
||||
""")
|
||||
|
||||
html_parts.append("""
|
||||
</body>
|
||||
</html>
|
||||
""")
|
||||
|
||||
return "".join(html_parts)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Generate HTML report from run_loop output")
|
||||
parser.add_argument("input", help="Path to JSON output from run_loop.py (or - for stdin)")
|
||||
parser.add_argument("-o", "--output", default=None, help="Output HTML file (default: stdout)")
|
||||
parser.add_argument("--skill-name", default="", help="Skill name to include in the report title")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.input == "-":
|
||||
data = json.load(sys.stdin)
|
||||
else:
|
||||
data = json.loads(Path(args.input).read_text())
|
||||
|
||||
html_output = generate_html(data, skill_name=args.skill_name)
|
||||
|
||||
if args.output:
|
||||
Path(args.output).write_text(html_output)
|
||||
print(f"Report written to {args.output}", file=sys.stderr)
|
||||
else:
|
||||
print(html_output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,247 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Improve a skill description based on eval results.
|
||||
|
||||
Takes eval results (from run_eval.py) and generates an improved description
|
||||
by calling `claude -p` as a subprocess (same auth pattern as run_eval.py —
|
||||
uses the session's Claude Code auth, no separate ANTHROPIC_API_KEY needed).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.utils import parse_skill_md
|
||||
|
||||
|
||||
def _call_claude(prompt: str, model: str | None, timeout: int = 300) -> str:
|
||||
"""Run `claude -p` with the prompt on stdin and return the text response.
|
||||
|
||||
Prompt goes over stdin (not argv) because it embeds the full SKILL.md
|
||||
body and can easily exceed comfortable argv length.
|
||||
"""
|
||||
cmd = ["claude", "-p", "--output-format", "text"]
|
||||
if model:
|
||||
cmd.extend(["--model", model])
|
||||
|
||||
# Remove CLAUDECODE env var to allow nesting claude -p inside a
|
||||
# Claude Code session. The guard is for interactive terminal conflicts;
|
||||
# programmatic subprocess usage is safe. Same pattern as run_eval.py.
|
||||
env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}
|
||||
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
input=prompt,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
timeout=timeout,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"claude -p exited {result.returncode}\nstderr: {result.stderr}"
|
||||
)
|
||||
return result.stdout
|
||||
|
||||
|
||||
def improve_description(
|
||||
skill_name: str,
|
||||
skill_content: str,
|
||||
current_description: str,
|
||||
eval_results: dict,
|
||||
history: list[dict],
|
||||
model: str,
|
||||
test_results: dict | None = None,
|
||||
log_dir: Path | None = None,
|
||||
iteration: int | None = None,
|
||||
) -> str:
|
||||
"""Call Claude to improve the description based on eval results."""
|
||||
failed_triggers = [
|
||||
r for r in eval_results["results"]
|
||||
if r["should_trigger"] and not r["pass"]
|
||||
]
|
||||
false_triggers = [
|
||||
r for r in eval_results["results"]
|
||||
if not r["should_trigger"] and not r["pass"]
|
||||
]
|
||||
|
||||
# Build scores summary
|
||||
train_score = f"{eval_results['summary']['passed']}/{eval_results['summary']['total']}"
|
||||
if test_results:
|
||||
test_score = f"{test_results['summary']['passed']}/{test_results['summary']['total']}"
|
||||
scores_summary = f"Train: {train_score}, Test: {test_score}"
|
||||
else:
|
||||
scores_summary = f"Train: {train_score}"
|
||||
|
||||
prompt = f"""You are optimizing a skill description for a Claude Code skill called "{skill_name}". A "skill" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.
|
||||
|
||||
The description appears in Claude's "available_skills" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.
|
||||
|
||||
Here's the current description:
|
||||
<current_description>
|
||||
"{current_description}"
|
||||
</current_description>
|
||||
|
||||
Current scores ({scores_summary}):
|
||||
<scores_summary>
|
||||
"""
|
||||
if failed_triggers:
|
||||
prompt += "FAILED TO TRIGGER (should have triggered but didn't):\n"
|
||||
for r in failed_triggers:
|
||||
prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n'
|
||||
prompt += "\n"
|
||||
|
||||
if false_triggers:
|
||||
prompt += "FALSE TRIGGERS (triggered but shouldn't have):\n"
|
||||
for r in false_triggers:
|
||||
prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n'
|
||||
prompt += "\n"
|
||||
|
||||
if history:
|
||||
prompt += "PREVIOUS ATTEMPTS (do NOT repeat these — try something structurally different):\n\n"
|
||||
for h in history:
|
||||
train_s = f"{h.get('train_passed', h.get('passed', 0))}/{h.get('train_total', h.get('total', 0))}"
|
||||
test_s = f"{h.get('test_passed', '?')}/{h.get('test_total', '?')}" if h.get('test_passed') is not None else None
|
||||
score_str = f"train={train_s}" + (f", test={test_s}" if test_s else "")
|
||||
prompt += f'<attempt {score_str}>\n'
|
||||
prompt += f'Description: "{h["description"]}"\n'
|
||||
if "results" in h:
|
||||
prompt += "Train results:\n"
|
||||
for r in h["results"]:
|
||||
status = "PASS" if r["pass"] else "FAIL"
|
||||
prompt += f' [{status}] "{r["query"][:80]}" (triggered {r["triggers"]}/{r["runs"]})\n'
|
||||
if h.get("note"):
|
||||
prompt += f'Note: {h["note"]}\n'
|
||||
prompt += "</attempt>\n\n"
|
||||
|
||||
prompt += f"""</scores_summary>
|
||||
|
||||
Skill content (for context on what the skill does):
|
||||
<skill_content>
|
||||
{skill_content}
|
||||
</skill_content>
|
||||
|
||||
Based on the failures, write a new and improved description that is more likely to trigger correctly. When I say "based on the failures", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:
|
||||
|
||||
1. Avoid overfitting
|
||||
2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.
|
||||
|
||||
Concretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters — descriptions over that will be truncated, so stay comfortably under it.
|
||||
|
||||
Here are some tips that we've found to work well in writing these descriptions:
|
||||
- The skill should be phrased in the imperative -- "Use this skill for" rather than "this skill does"
|
||||
- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.
|
||||
- The description competes with other skills for Claude's attention — make it distinctive and immediately recognizable.
|
||||
- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.
|
||||
|
||||
I'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end.
|
||||
|
||||
Please respond with only the new description text in <new_description> tags, nothing else."""
|
||||
|
||||
text = _call_claude(prompt, model)
|
||||
|
||||
match = re.search(r"<new_description>(.*?)</new_description>", text, re.DOTALL)
|
||||
description = match.group(1).strip().strip('"') if match else text.strip().strip('"')
|
||||
|
||||
transcript: dict = {
|
||||
"iteration": iteration,
|
||||
"prompt": prompt,
|
||||
"response": text,
|
||||
"parsed_description": description,
|
||||
"char_count": len(description),
|
||||
"over_limit": len(description) > 1024,
|
||||
}
|
||||
|
||||
# Safety net: the prompt already states the 1024-char hard limit, but if
|
||||
# the model blew past it anyway, make one fresh single-turn call that
|
||||
# quotes the too-long version and asks for a shorter rewrite. (The old
|
||||
# SDK path did this as a true multi-turn; `claude -p` is one-shot, so we
|
||||
# inline the prior output into the new prompt instead.)
|
||||
if len(description) > 1024:
|
||||
shorten_prompt = (
|
||||
f"{prompt}\n\n"
|
||||
f"---\n\n"
|
||||
f"A previous attempt produced this description, which at "
|
||||
f"{len(description)} characters is over the 1024-character hard limit:\n\n"
|
||||
f'"{description}"\n\n'
|
||||
f"Rewrite it to be under 1024 characters while keeping the most "
|
||||
f"important trigger words and intent coverage. Respond with only "
|
||||
f"the new description in <new_description> tags."
|
||||
)
|
||||
shorten_text = _call_claude(shorten_prompt, model)
|
||||
match = re.search(r"<new_description>(.*?)</new_description>", shorten_text, re.DOTALL)
|
||||
shortened = match.group(1).strip().strip('"') if match else shorten_text.strip().strip('"')
|
||||
|
||||
transcript["rewrite_prompt"] = shorten_prompt
|
||||
transcript["rewrite_response"] = shorten_text
|
||||
transcript["rewrite_description"] = shortened
|
||||
transcript["rewrite_char_count"] = len(shortened)
|
||||
description = shortened
|
||||
|
||||
transcript["final_description"] = description
|
||||
|
||||
if log_dir:
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_file = log_dir / f"improve_iter_{iteration or 'unknown'}.json"
|
||||
log_file.write_text(json.dumps(transcript, indent=2))
|
||||
|
||||
return description
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Improve a skill description based on eval results")
|
||||
parser.add_argument("--eval-results", required=True, help="Path to eval results JSON (from run_eval.py)")
|
||||
parser.add_argument("--skill-path", required=True, help="Path to skill directory")
|
||||
parser.add_argument("--history", default=None, help="Path to history JSON (previous attempts)")
|
||||
parser.add_argument("--model", required=True, help="Model for improvement")
|
||||
parser.add_argument("--verbose", action="store_true", help="Print thinking to stderr")
|
||||
args = parser.parse_args()
|
||||
|
||||
skill_path = Path(args.skill_path)
|
||||
if not (skill_path / "SKILL.md").exists():
|
||||
print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
eval_results = json.loads(Path(args.eval_results).read_text())
|
||||
history = []
|
||||
if args.history:
|
||||
history = json.loads(Path(args.history).read_text())
|
||||
|
||||
name, _, content = parse_skill_md(skill_path)
|
||||
current_description = eval_results["description"]
|
||||
|
||||
if args.verbose:
|
||||
print(f"Current: {current_description}", file=sys.stderr)
|
||||
print(f"Score: {eval_results['summary']['passed']}/{eval_results['summary']['total']}", file=sys.stderr)
|
||||
|
||||
new_description = improve_description(
|
||||
skill_name=name,
|
||||
skill_content=content,
|
||||
current_description=current_description,
|
||||
eval_results=eval_results,
|
||||
history=history,
|
||||
model=args.model,
|
||||
)
|
||||
|
||||
if args.verbose:
|
||||
print(f"Improved: {new_description}", file=sys.stderr)
|
||||
|
||||
# Output as JSON with both the new description and updated history
|
||||
output = {
|
||||
"description": new_description,
|
||||
"history": history + [{
|
||||
"description": current_description,
|
||||
"passed": eval_results["summary"]["passed"],
|
||||
"failed": eval_results["summary"]["failed"],
|
||||
"total": eval_results["summary"]["total"],
|
||||
"results": eval_results["results"],
|
||||
}],
|
||||
}
|
||||
print(json.dumps(output, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
303
skills/skill-creator/scripts/init_skill.py
Executable file
303
skills/skill-creator/scripts/init_skill.py
Executable file
@@ -0,0 +1,303 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Skill Initializer - Creates a new skill from template
|
||||
|
||||
Usage:
|
||||
init_skill.py <skill-name> --path <path>
|
||||
|
||||
Examples:
|
||||
init_skill.py my-new-skill --path skills/public
|
||||
init_skill.py my-api-helper --path skills/private
|
||||
init_skill.py custom-skill --path /custom/location
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SKILL_TEMPLATE = """---
|
||||
name: {skill_name}
|
||||
description: [TODO: Complete and informative explanation of what the skill does and when to use it. Include WHEN to use this skill - specific scenarios, file types, or tasks that trigger it.]
|
||||
---
|
||||
|
||||
# {skill_title}
|
||||
|
||||
## Overview
|
||||
|
||||
[TODO: 1-2 sentences explaining what this skill enables]
|
||||
|
||||
## Structuring This Skill
|
||||
|
||||
[TODO: Choose the structure that best fits this skill's purpose. Common patterns:
|
||||
|
||||
**1. Workflow-Based** (best for sequential processes)
|
||||
- Works well when there are clear step-by-step procedures
|
||||
- Example: DOCX skill with "Workflow Decision Tree" → "Reading" → "Creating" → "Editing"
|
||||
- Structure: ## Overview → ## Workflow Decision Tree → ## Step 1 → ## Step 2...
|
||||
|
||||
**2. Task-Based** (best for tool collections)
|
||||
- Works well when the skill offers different operations/capabilities
|
||||
- Example: PDF skill with "Quick Start" → "Merge PDFs" → "Split PDFs" → "Extract Text"
|
||||
- Structure: ## Overview → ## Quick Start → ## Task Category 1 → ## Task Category 2...
|
||||
|
||||
**3. Reference/Guidelines** (best for standards or specifications)
|
||||
- Works well for brand guidelines, coding standards, or requirements
|
||||
- Example: Brand styling with "Brand Guidelines" → "Colors" → "Typography" → "Features"
|
||||
- Structure: ## Overview → ## Guidelines → ## Specifications → ## Usage...
|
||||
|
||||
**4. Capabilities-Based** (best for integrated systems)
|
||||
- Works well when the skill provides multiple interrelated features
|
||||
- Example: Product Management with "Core Capabilities" → numbered capability list
|
||||
- Structure: ## Overview → ## Core Capabilities → ### 1. Feature → ### 2. Feature...
|
||||
|
||||
Patterns can be mixed and matched as needed. Most skills combine patterns (e.g., start with task-based, add workflow for complex operations).
|
||||
|
||||
Delete this entire "Structuring This Skill" section when done - it's just guidance.]
|
||||
|
||||
## [TODO: Replace with the first main section based on chosen structure]
|
||||
|
||||
[TODO: Add content here. See examples in existing skills:
|
||||
- Code samples for technical skills
|
||||
- Decision trees for complex workflows
|
||||
- Concrete examples with realistic user requests
|
||||
- References to scripts/templates/references as needed]
|
||||
|
||||
## Resources
|
||||
|
||||
This skill includes example resource directories that demonstrate how to organize different types of bundled resources:
|
||||
|
||||
### scripts/
|
||||
Executable code (Python/Bash/etc.) that can be run directly to perform specific operations.
|
||||
|
||||
**Examples from other skills:**
|
||||
- PDF skill: `fill_fillable_fields.py`, `extract_form_field_info.py` - utilities for PDF manipulation
|
||||
- DOCX skill: `document.py`, `utilities.py` - Python modules for document processing
|
||||
|
||||
**Appropriate for:** Python scripts, shell scripts, or any executable code that performs automation, data processing, or specific operations.
|
||||
|
||||
**Note:** Scripts may be executed without loading into context, but can still be read by Claude for patching or environment adjustments.
|
||||
|
||||
### references/
|
||||
Documentation and reference material intended to be loaded into context to inform Claude's process and thinking.
|
||||
|
||||
**Examples from other skills:**
|
||||
- Product management: `communication.md`, `context_building.md` - detailed workflow guides
|
||||
- BigQuery: API reference documentation and query examples
|
||||
- Finance: Schema documentation, company policies
|
||||
|
||||
**Appropriate for:** In-depth documentation, API references, database schemas, comprehensive guides, or any detailed information that Claude should reference while working.
|
||||
|
||||
### assets/
|
||||
Files not intended to be loaded into context, but rather used within the output Claude produces.
|
||||
|
||||
**Examples from other skills:**
|
||||
- Brand styling: PowerPoint template files (.pptx), logo files
|
||||
- Frontend builder: HTML/React boilerplate project directories
|
||||
- Typography: Font files (.ttf, .woff2)
|
||||
|
||||
**Appropriate for:** Templates, boilerplate code, document templates, images, icons, fonts, or any files meant to be copied or used in the final output.
|
||||
|
||||
---
|
||||
|
||||
**Any unneeded directories can be deleted.** Not every skill requires all three types of resources.
|
||||
"""
|
||||
|
||||
EXAMPLE_SCRIPT = '''#!/usr/bin/env python3
|
||||
"""
|
||||
Example helper script for {skill_name}
|
||||
|
||||
This is a placeholder script that can be executed directly.
|
||||
Replace with actual implementation or delete if not needed.
|
||||
|
||||
Example real scripts from other skills:
|
||||
- pdf/scripts/fill_fillable_fields.py - Fills PDF form fields
|
||||
- pdf/scripts/convert_pdf_to_images.py - Converts PDF pages to images
|
||||
"""
|
||||
|
||||
def main():
|
||||
print("This is an example script for {skill_name}")
|
||||
# TODO: Add actual script logic here
|
||||
# This could be data processing, file conversion, API calls, etc.
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
'''
|
||||
|
||||
EXAMPLE_REFERENCE = """# Reference Documentation for {skill_title}
|
||||
|
||||
This is a placeholder for detailed reference documentation.
|
||||
Replace with actual reference content or delete if not needed.
|
||||
|
||||
Example real reference docs from other skills:
|
||||
- product-management/references/communication.md - Comprehensive guide for status updates
|
||||
- product-management/references/context_building.md - Deep-dive on gathering context
|
||||
- bigquery/references/ - API references and query examples
|
||||
|
||||
## When Reference Docs Are Useful
|
||||
|
||||
Reference docs are ideal for:
|
||||
- Comprehensive API documentation
|
||||
- Detailed workflow guides
|
||||
- Complex multi-step processes
|
||||
- Information too lengthy for main SKILL.md
|
||||
- Content that's only needed for specific use cases
|
||||
|
||||
## Structure Suggestions
|
||||
|
||||
### API Reference Example
|
||||
- Overview
|
||||
- Authentication
|
||||
- Endpoints with examples
|
||||
- Error codes
|
||||
- Rate limits
|
||||
|
||||
### Workflow Guide Example
|
||||
- Prerequisites
|
||||
- Step-by-step instructions
|
||||
- Common patterns
|
||||
- Troubleshooting
|
||||
- Best practices
|
||||
"""
|
||||
|
||||
EXAMPLE_ASSET = """# Example Asset File
|
||||
|
||||
This placeholder represents where asset files would be stored.
|
||||
Replace with actual asset files (templates, images, fonts, etc.) or delete if not needed.
|
||||
|
||||
Asset files are NOT intended to be loaded into context, but rather used within
|
||||
the output Claude produces.
|
||||
|
||||
Example asset files from other skills:
|
||||
- Brand guidelines: logo.png, slides_template.pptx
|
||||
- Frontend builder: hello-world/ directory with HTML/React boilerplate
|
||||
- Typography: custom-font.ttf, font-family.woff2
|
||||
- Data: sample_data.csv, test_dataset.json
|
||||
|
||||
## Common Asset Types
|
||||
|
||||
- Templates: .pptx, .docx, boilerplate directories
|
||||
- Images: .png, .jpg, .svg, .gif
|
||||
- Fonts: .ttf, .otf, .woff, .woff2
|
||||
- Boilerplate code: Project directories, starter files
|
||||
- Icons: .ico, .svg
|
||||
- Data files: .csv, .json, .xml, .yaml
|
||||
|
||||
Note: This is a text placeholder. Actual assets can be any file type.
|
||||
"""
|
||||
|
||||
|
||||
def title_case_skill_name(skill_name):
|
||||
"""Convert hyphenated skill name to Title Case for display."""
|
||||
return ' '.join(word.capitalize() for word in skill_name.split('-'))
|
||||
|
||||
|
||||
def init_skill(skill_name, path):
|
||||
"""
|
||||
Initialize a new skill directory with template SKILL.md.
|
||||
|
||||
Args:
|
||||
skill_name: Name of the skill
|
||||
path: Path where the skill directory should be created
|
||||
|
||||
Returns:
|
||||
Path to created skill directory, or None if error
|
||||
"""
|
||||
# Determine skill directory path
|
||||
skill_dir = Path(path).resolve() / skill_name
|
||||
|
||||
# Check if directory already exists
|
||||
if skill_dir.exists():
|
||||
print(f"❌ Error: Skill directory already exists: {skill_dir}")
|
||||
return None
|
||||
|
||||
# Create skill directory
|
||||
try:
|
||||
skill_dir.mkdir(parents=True, exist_ok=False)
|
||||
print(f"✅ Created skill directory: {skill_dir}")
|
||||
except Exception as e:
|
||||
print(f"❌ Error creating directory: {e}")
|
||||
return None
|
||||
|
||||
# Create SKILL.md from template
|
||||
skill_title = title_case_skill_name(skill_name)
|
||||
skill_content = SKILL_TEMPLATE.format(
|
||||
skill_name=skill_name,
|
||||
skill_title=skill_title
|
||||
)
|
||||
|
||||
skill_md_path = skill_dir / 'SKILL.md'
|
||||
try:
|
||||
skill_md_path.write_text(skill_content)
|
||||
print("✅ Created SKILL.md")
|
||||
except Exception as e:
|
||||
print(f"❌ Error creating SKILL.md: {e}")
|
||||
return None
|
||||
|
||||
# Create resource directories with example files
|
||||
try:
|
||||
# Create scripts/ directory with example script
|
||||
scripts_dir = skill_dir / 'scripts'
|
||||
scripts_dir.mkdir(exist_ok=True)
|
||||
example_script = scripts_dir / 'example.py'
|
||||
example_script.write_text(EXAMPLE_SCRIPT.format(skill_name=skill_name))
|
||||
example_script.chmod(0o755)
|
||||
print("✅ Created scripts/example.py")
|
||||
|
||||
# Create references/ directory with example reference doc
|
||||
references_dir = skill_dir / 'references'
|
||||
references_dir.mkdir(exist_ok=True)
|
||||
example_reference = references_dir / 'api_reference.md'
|
||||
example_reference.write_text(EXAMPLE_REFERENCE.format(skill_title=skill_title))
|
||||
print("✅ Created references/api_reference.md")
|
||||
|
||||
# Create assets/ directory with example asset placeholder
|
||||
assets_dir = skill_dir / 'assets'
|
||||
assets_dir.mkdir(exist_ok=True)
|
||||
example_asset = assets_dir / 'example_asset.txt'
|
||||
example_asset.write_text(EXAMPLE_ASSET)
|
||||
print("✅ Created assets/example_asset.txt")
|
||||
except Exception as e:
|
||||
print(f"❌ Error creating resource directories: {e}")
|
||||
return None
|
||||
|
||||
# Print next steps
|
||||
print(f"\n✅ Skill '{skill_name}' initialized successfully at {skill_dir}")
|
||||
print("\nNext steps:")
|
||||
print("1. Edit SKILL.md to complete the TODO items and update the description")
|
||||
print("2. Customize or delete the example files in scripts/, references/, and assets/")
|
||||
print("3. Run the validator when ready to check the skill structure")
|
||||
|
||||
return skill_dir
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 4 or sys.argv[2] != '--path':
|
||||
print("Usage: init_skill.py <skill-name> --path <path>")
|
||||
print("\nSkill name requirements:")
|
||||
print(" - Hyphen-case identifier (e.g., 'data-analyzer')")
|
||||
print(" - Lowercase letters, digits, and hyphens only")
|
||||
print(" - Max 40 characters")
|
||||
print(" - Must match directory name exactly")
|
||||
print("\nExamples:")
|
||||
print(" init_skill.py my-new-skill --path skills/public")
|
||||
print(" init_skill.py my-api-helper --path skills/private")
|
||||
print(" init_skill.py custom-skill --path /custom/location")
|
||||
sys.exit(1)
|
||||
|
||||
skill_name = sys.argv[1]
|
||||
path = sys.argv[3]
|
||||
|
||||
print(f"🚀 Initializing skill: {skill_name}")
|
||||
print(f" Location: {path}")
|
||||
print()
|
||||
|
||||
result = init_skill(skill_name, path)
|
||||
|
||||
if result:
|
||||
sys.exit(0)
|
||||
else:
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -10,33 +10,10 @@ Example:
|
||||
python utils/package_skill.py skills/public/my-skill ./dist
|
||||
"""
|
||||
|
||||
import fnmatch
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from scripts.quick_validate import validate_skill
|
||||
|
||||
# Patterns to exclude when packaging skills.
|
||||
EXCLUDE_DIRS = {"__pycache__", "node_modules"}
|
||||
EXCLUDE_GLOBS = {"*.pyc"}
|
||||
EXCLUDE_FILES = {".DS_Store"}
|
||||
# Directories excluded only at the skill root (not when nested deeper).
|
||||
ROOT_EXCLUDE_DIRS = {"evals"}
|
||||
|
||||
|
||||
def should_exclude(rel_path: Path) -> bool:
|
||||
"""Check if a path should be excluded from packaging."""
|
||||
parts = rel_path.parts
|
||||
if any(part in EXCLUDE_DIRS for part in parts):
|
||||
return True
|
||||
# rel_path is relative to skill_path.parent, so parts[0] is the skill
|
||||
# folder name and parts[1] (if present) is the first subdir.
|
||||
if len(parts) > 1 and parts[1] in ROOT_EXCLUDE_DIRS:
|
||||
return True
|
||||
name = rel_path.name
|
||||
if name in EXCLUDE_FILES:
|
||||
return True
|
||||
return any(fnmatch.fnmatch(name, pat) for pat in EXCLUDE_GLOBS)
|
||||
from quick_validate import validate_skill
|
||||
|
||||
|
||||
def package_skill(skill_path, output_dir=None):
|
||||
@@ -89,16 +66,13 @@ def package_skill(skill_path, output_dir=None):
|
||||
# Create the .skill file (zip format)
|
||||
try:
|
||||
with zipfile.ZipFile(skill_filename, 'w', zipfile.ZIP_DEFLATED) as zipf:
|
||||
# Walk through the skill directory, excluding build artifacts
|
||||
# Walk through the skill directory
|
||||
for file_path in skill_path.rglob('*'):
|
||||
if not file_path.is_file():
|
||||
continue
|
||||
arcname = file_path.relative_to(skill_path.parent)
|
||||
if should_exclude(arcname):
|
||||
print(f" Skipped: {arcname}")
|
||||
continue
|
||||
zipf.write(file_path, arcname)
|
||||
print(f" Added: {arcname}")
|
||||
if file_path.is_file():
|
||||
# Calculate the relative path within the zip
|
||||
arcname = file_path.relative_to(skill_path.parent)
|
||||
zipf.write(file_path, arcname)
|
||||
print(f" Added: {arcname}")
|
||||
|
||||
print(f"\n✅ Successfully packaged skill to: {skill_filename}")
|
||||
return skill_filename
|
||||
|
||||
@@ -39,7 +39,7 @@ def validate_skill(skill_path):
|
||||
return False, f"Invalid YAML in frontmatter: {e}"
|
||||
|
||||
# Define allowed properties
|
||||
ALLOWED_PROPERTIES = {'name', 'description', 'license', 'allowed-tools', 'metadata', 'compatibility'}
|
||||
ALLOWED_PROPERTIES = {'name', 'description', 'license', 'allowed-tools', 'metadata'}
|
||||
|
||||
# Check for unexpected properties (excluding nested keys under metadata)
|
||||
unexpected_keys = set(frontmatter.keys()) - ALLOWED_PROPERTIES
|
||||
@@ -61,9 +61,9 @@ def validate_skill(skill_path):
|
||||
return False, f"Name must be a string, got {type(name).__name__}"
|
||||
name = name.strip()
|
||||
if name:
|
||||
# Check naming convention (kebab-case: lowercase with hyphens)
|
||||
# Check naming convention (hyphen-case: lowercase with hyphens)
|
||||
if not re.match(r'^[a-z0-9-]+$', name):
|
||||
return False, f"Name '{name}' should be kebab-case (lowercase letters, digits, and hyphens only)"
|
||||
return False, f"Name '{name}' should be hyphen-case (lowercase letters, digits, and hyphens only)"
|
||||
if name.startswith('-') or name.endswith('-') or '--' in name:
|
||||
return False, f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens"
|
||||
# Check name length (max 64 characters per spec)
|
||||
@@ -83,14 +83,6 @@ def validate_skill(skill_path):
|
||||
if len(description) > 1024:
|
||||
return False, f"Description is too long ({len(description)} characters). Maximum is 1024 characters."
|
||||
|
||||
# Validate compatibility field if present (optional)
|
||||
compatibility = frontmatter.get('compatibility', '')
|
||||
if compatibility:
|
||||
if not isinstance(compatibility, str):
|
||||
return False, f"Compatibility must be a string, got {type(compatibility).__name__}"
|
||||
if len(compatibility) > 500:
|
||||
return False, f"Compatibility is too long ({len(compatibility)} characters). Maximum is 500 characters."
|
||||
|
||||
return True, "Skill is valid!"
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,310 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run trigger evaluation for a skill description.
|
||||
|
||||
Tests whether a skill's description causes Claude to trigger (read the skill)
|
||||
for a set of queries. Outputs results as JSON.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import select
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.utils import parse_skill_md
|
||||
|
||||
|
||||
def find_project_root() -> Path:
|
||||
"""Find the project root by walking up from cwd looking for .claude/.
|
||||
|
||||
Mimics how Claude Code discovers its project root, so the command file
|
||||
we create ends up where claude -p will look for it.
|
||||
"""
|
||||
current = Path.cwd()
|
||||
for parent in [current, *current.parents]:
|
||||
if (parent / ".claude").is_dir():
|
||||
return parent
|
||||
return current
|
||||
|
||||
|
||||
def run_single_query(
|
||||
query: str,
|
||||
skill_name: str,
|
||||
skill_description: str,
|
||||
timeout: int,
|
||||
project_root: str,
|
||||
model: str | None = None,
|
||||
) -> bool:
|
||||
"""Run a single query and return whether the skill was triggered.
|
||||
|
||||
Creates a command file in .claude/commands/ so it appears in Claude's
|
||||
available_skills list, then runs `claude -p` with the raw query.
|
||||
Uses --include-partial-messages to detect triggering early from
|
||||
stream events (content_block_start) rather than waiting for the
|
||||
full assistant message, which only arrives after tool execution.
|
||||
"""
|
||||
unique_id = uuid.uuid4().hex[:8]
|
||||
clean_name = f"{skill_name}-skill-{unique_id}"
|
||||
project_commands_dir = Path(project_root) / ".claude" / "commands"
|
||||
command_file = project_commands_dir / f"{clean_name}.md"
|
||||
|
||||
try:
|
||||
project_commands_dir.mkdir(parents=True, exist_ok=True)
|
||||
# Use YAML block scalar to avoid breaking on quotes in description
|
||||
indented_desc = "\n ".join(skill_description.split("\n"))
|
||||
command_content = (
|
||||
f"---\n"
|
||||
f"description: |\n"
|
||||
f" {indented_desc}\n"
|
||||
f"---\n\n"
|
||||
f"# {skill_name}\n\n"
|
||||
f"This skill handles: {skill_description}\n"
|
||||
)
|
||||
command_file.write_text(command_content)
|
||||
|
||||
cmd = [
|
||||
"claude",
|
||||
"-p", query,
|
||||
"--output-format", "stream-json",
|
||||
"--verbose",
|
||||
"--include-partial-messages",
|
||||
]
|
||||
if model:
|
||||
cmd.extend(["--model", model])
|
||||
|
||||
# Remove CLAUDECODE env var to allow nesting claude -p inside a
|
||||
# Claude Code session. The guard is for interactive terminal conflicts;
|
||||
# programmatic subprocess usage is safe.
|
||||
env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}
|
||||
|
||||
process = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
cwd=project_root,
|
||||
env=env,
|
||||
)
|
||||
|
||||
triggered = False
|
||||
start_time = time.time()
|
||||
buffer = ""
|
||||
# Track state for stream event detection
|
||||
pending_tool_name = None
|
||||
accumulated_json = ""
|
||||
|
||||
try:
|
||||
while time.time() - start_time < timeout:
|
||||
if process.poll() is not None:
|
||||
remaining = process.stdout.read()
|
||||
if remaining:
|
||||
buffer += remaining.decode("utf-8", errors="replace")
|
||||
break
|
||||
|
||||
ready, _, _ = select.select([process.stdout], [], [], 1.0)
|
||||
if not ready:
|
||||
continue
|
||||
|
||||
chunk = os.read(process.stdout.fileno(), 8192)
|
||||
if not chunk:
|
||||
break
|
||||
buffer += chunk.decode("utf-8", errors="replace")
|
||||
|
||||
while "\n" in buffer:
|
||||
line, buffer = buffer.split("\n", 1)
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
# Early detection via stream events
|
||||
if event.get("type") == "stream_event":
|
||||
se = event.get("event", {})
|
||||
se_type = se.get("type", "")
|
||||
|
||||
if se_type == "content_block_start":
|
||||
cb = se.get("content_block", {})
|
||||
if cb.get("type") == "tool_use":
|
||||
tool_name = cb.get("name", "")
|
||||
if tool_name in ("Skill", "Read"):
|
||||
pending_tool_name = tool_name
|
||||
accumulated_json = ""
|
||||
else:
|
||||
return False
|
||||
|
||||
elif se_type == "content_block_delta" and pending_tool_name:
|
||||
delta = se.get("delta", {})
|
||||
if delta.get("type") == "input_json_delta":
|
||||
accumulated_json += delta.get("partial_json", "")
|
||||
if clean_name in accumulated_json:
|
||||
return True
|
||||
|
||||
elif se_type in ("content_block_stop", "message_stop"):
|
||||
if pending_tool_name:
|
||||
return clean_name in accumulated_json
|
||||
if se_type == "message_stop":
|
||||
return False
|
||||
|
||||
# Fallback: full assistant message
|
||||
elif event.get("type") == "assistant":
|
||||
message = event.get("message", {})
|
||||
for content_item in message.get("content", []):
|
||||
if content_item.get("type") != "tool_use":
|
||||
continue
|
||||
tool_name = content_item.get("name", "")
|
||||
tool_input = content_item.get("input", {})
|
||||
if tool_name == "Skill" and clean_name in tool_input.get("skill", ""):
|
||||
triggered = True
|
||||
elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""):
|
||||
triggered = True
|
||||
return triggered
|
||||
|
||||
elif event.get("type") == "result":
|
||||
return triggered
|
||||
finally:
|
||||
# Clean up process on any exit path (return, exception, timeout)
|
||||
if process.poll() is None:
|
||||
process.kill()
|
||||
process.wait()
|
||||
|
||||
return triggered
|
||||
finally:
|
||||
if command_file.exists():
|
||||
command_file.unlink()
|
||||
|
||||
|
||||
def run_eval(
|
||||
eval_set: list[dict],
|
||||
skill_name: str,
|
||||
description: str,
|
||||
num_workers: int,
|
||||
timeout: int,
|
||||
project_root: Path,
|
||||
runs_per_query: int = 1,
|
||||
trigger_threshold: float = 0.5,
|
||||
model: str | None = None,
|
||||
) -> dict:
|
||||
"""Run the full eval set and return results."""
|
||||
results = []
|
||||
|
||||
with ProcessPoolExecutor(max_workers=num_workers) as executor:
|
||||
future_to_info = {}
|
||||
for item in eval_set:
|
||||
for run_idx in range(runs_per_query):
|
||||
future = executor.submit(
|
||||
run_single_query,
|
||||
item["query"],
|
||||
skill_name,
|
||||
description,
|
||||
timeout,
|
||||
str(project_root),
|
||||
model,
|
||||
)
|
||||
future_to_info[future] = (item, run_idx)
|
||||
|
||||
query_triggers: dict[str, list[bool]] = {}
|
||||
query_items: dict[str, dict] = {}
|
||||
for future in as_completed(future_to_info):
|
||||
item, _ = future_to_info[future]
|
||||
query = item["query"]
|
||||
query_items[query] = item
|
||||
if query not in query_triggers:
|
||||
query_triggers[query] = []
|
||||
try:
|
||||
query_triggers[query].append(future.result())
|
||||
except Exception as e:
|
||||
print(f"Warning: query failed: {e}", file=sys.stderr)
|
||||
query_triggers[query].append(False)
|
||||
|
||||
for query, triggers in query_triggers.items():
|
||||
item = query_items[query]
|
||||
trigger_rate = sum(triggers) / len(triggers)
|
||||
should_trigger = item["should_trigger"]
|
||||
if should_trigger:
|
||||
did_pass = trigger_rate >= trigger_threshold
|
||||
else:
|
||||
did_pass = trigger_rate < trigger_threshold
|
||||
results.append({
|
||||
"query": query,
|
||||
"should_trigger": should_trigger,
|
||||
"trigger_rate": trigger_rate,
|
||||
"triggers": sum(triggers),
|
||||
"runs": len(triggers),
|
||||
"pass": did_pass,
|
||||
})
|
||||
|
||||
passed = sum(1 for r in results if r["pass"])
|
||||
total = len(results)
|
||||
|
||||
return {
|
||||
"skill_name": skill_name,
|
||||
"description": description,
|
||||
"results": results,
|
||||
"summary": {
|
||||
"total": total,
|
||||
"passed": passed,
|
||||
"failed": total - passed,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Run trigger evaluation for a skill description")
|
||||
parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file")
|
||||
parser.add_argument("--skill-path", required=True, help="Path to skill directory")
|
||||
parser.add_argument("--description", default=None, help="Override description to test")
|
||||
parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers")
|
||||
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
|
||||
parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query")
|
||||
parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold")
|
||||
parser.add_argument("--model", default=None, help="Model to use for claude -p (default: user's configured model)")
|
||||
parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")
|
||||
args = parser.parse_args()
|
||||
|
||||
eval_set = json.loads(Path(args.eval_set).read_text())
|
||||
skill_path = Path(args.skill_path)
|
||||
|
||||
if not (skill_path / "SKILL.md").exists():
|
||||
print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
name, original_description, content = parse_skill_md(skill_path)
|
||||
description = args.description or original_description
|
||||
project_root = find_project_root()
|
||||
|
||||
if args.verbose:
|
||||
print(f"Evaluating: {description}", file=sys.stderr)
|
||||
|
||||
output = run_eval(
|
||||
eval_set=eval_set,
|
||||
skill_name=name,
|
||||
description=description,
|
||||
num_workers=args.num_workers,
|
||||
timeout=args.timeout,
|
||||
project_root=project_root,
|
||||
runs_per_query=args.runs_per_query,
|
||||
trigger_threshold=args.trigger_threshold,
|
||||
model=args.model,
|
||||
)
|
||||
|
||||
if args.verbose:
|
||||
summary = output["summary"]
|
||||
print(f"Results: {summary['passed']}/{summary['total']} passed", file=sys.stderr)
|
||||
for r in output["results"]:
|
||||
status = "PASS" if r["pass"] else "FAIL"
|
||||
rate_str = f"{r['triggers']}/{r['runs']}"
|
||||
print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:70]}", file=sys.stderr)
|
||||
|
||||
print(json.dumps(output, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,328 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the eval + improve loop until all pass or max iterations reached.
|
||||
|
||||
Combines run_eval.py and improve_description.py in a loop, tracking history
|
||||
and returning the best description found. Supports train/test split to prevent
|
||||
overfitting.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import webbrowser
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.generate_report import generate_html
|
||||
from scripts.improve_description import improve_description
|
||||
from scripts.run_eval import find_project_root, run_eval
|
||||
from scripts.utils import parse_skill_md
|
||||
|
||||
|
||||
def split_eval_set(eval_set: list[dict], holdout: float, seed: int = 42) -> tuple[list[dict], list[dict]]:
|
||||
"""Split eval set into train and test sets, stratified by should_trigger."""
|
||||
random.seed(seed)
|
||||
|
||||
# Separate by should_trigger
|
||||
trigger = [e for e in eval_set if e["should_trigger"]]
|
||||
no_trigger = [e for e in eval_set if not e["should_trigger"]]
|
||||
|
||||
# Shuffle each group
|
||||
random.shuffle(trigger)
|
||||
random.shuffle(no_trigger)
|
||||
|
||||
# Calculate split points
|
||||
n_trigger_test = max(1, int(len(trigger) * holdout))
|
||||
n_no_trigger_test = max(1, int(len(no_trigger) * holdout))
|
||||
|
||||
# Split
|
||||
test_set = trigger[:n_trigger_test] + no_trigger[:n_no_trigger_test]
|
||||
train_set = trigger[n_trigger_test:] + no_trigger[n_no_trigger_test:]
|
||||
|
||||
return train_set, test_set
|
||||
|
||||
|
||||
def run_loop(
|
||||
eval_set: list[dict],
|
||||
skill_path: Path,
|
||||
description_override: str | None,
|
||||
num_workers: int,
|
||||
timeout: int,
|
||||
max_iterations: int,
|
||||
runs_per_query: int,
|
||||
trigger_threshold: float,
|
||||
holdout: float,
|
||||
model: str,
|
||||
verbose: bool,
|
||||
live_report_path: Path | None = None,
|
||||
log_dir: Path | None = None,
|
||||
) -> dict:
|
||||
"""Run the eval + improvement loop."""
|
||||
project_root = find_project_root()
|
||||
name, original_description, content = parse_skill_md(skill_path)
|
||||
current_description = description_override or original_description
|
||||
|
||||
# Split into train/test if holdout > 0
|
||||
if holdout > 0:
|
||||
train_set, test_set = split_eval_set(eval_set, holdout)
|
||||
if verbose:
|
||||
print(f"Split: {len(train_set)} train, {len(test_set)} test (holdout={holdout})", file=sys.stderr)
|
||||
else:
|
||||
train_set = eval_set
|
||||
test_set = []
|
||||
|
||||
history = []
|
||||
exit_reason = "unknown"
|
||||
|
||||
for iteration in range(1, max_iterations + 1):
|
||||
if verbose:
|
||||
print(f"\n{'='*60}", file=sys.stderr)
|
||||
print(f"Iteration {iteration}/{max_iterations}", file=sys.stderr)
|
||||
print(f"Description: {current_description}", file=sys.stderr)
|
||||
print(f"{'='*60}", file=sys.stderr)
|
||||
|
||||
# Evaluate train + test together in one batch for parallelism
|
||||
all_queries = train_set + test_set
|
||||
t0 = time.time()
|
||||
all_results = run_eval(
|
||||
eval_set=all_queries,
|
||||
skill_name=name,
|
||||
description=current_description,
|
||||
num_workers=num_workers,
|
||||
timeout=timeout,
|
||||
project_root=project_root,
|
||||
runs_per_query=runs_per_query,
|
||||
trigger_threshold=trigger_threshold,
|
||||
model=model,
|
||||
)
|
||||
eval_elapsed = time.time() - t0
|
||||
|
||||
# Split results back into train/test by matching queries
|
||||
train_queries_set = {q["query"] for q in train_set}
|
||||
train_result_list = [r for r in all_results["results"] if r["query"] in train_queries_set]
|
||||
test_result_list = [r for r in all_results["results"] if r["query"] not in train_queries_set]
|
||||
|
||||
train_passed = sum(1 for r in train_result_list if r["pass"])
|
||||
train_total = len(train_result_list)
|
||||
train_summary = {"passed": train_passed, "failed": train_total - train_passed, "total": train_total}
|
||||
train_results = {"results": train_result_list, "summary": train_summary}
|
||||
|
||||
if test_set:
|
||||
test_passed = sum(1 for r in test_result_list if r["pass"])
|
||||
test_total = len(test_result_list)
|
||||
test_summary = {"passed": test_passed, "failed": test_total - test_passed, "total": test_total}
|
||||
test_results = {"results": test_result_list, "summary": test_summary}
|
||||
else:
|
||||
test_results = None
|
||||
test_summary = None
|
||||
|
||||
history.append({
|
||||
"iteration": iteration,
|
||||
"description": current_description,
|
||||
"train_passed": train_summary["passed"],
|
||||
"train_failed": train_summary["failed"],
|
||||
"train_total": train_summary["total"],
|
||||
"train_results": train_results["results"],
|
||||
"test_passed": test_summary["passed"] if test_summary else None,
|
||||
"test_failed": test_summary["failed"] if test_summary else None,
|
||||
"test_total": test_summary["total"] if test_summary else None,
|
||||
"test_results": test_results["results"] if test_results else None,
|
||||
# For backward compat with report generator
|
||||
"passed": train_summary["passed"],
|
||||
"failed": train_summary["failed"],
|
||||
"total": train_summary["total"],
|
||||
"results": train_results["results"],
|
||||
})
|
||||
|
||||
# Write live report if path provided
|
||||
if live_report_path:
|
||||
partial_output = {
|
||||
"original_description": original_description,
|
||||
"best_description": current_description,
|
||||
"best_score": "in progress",
|
||||
"iterations_run": len(history),
|
||||
"holdout": holdout,
|
||||
"train_size": len(train_set),
|
||||
"test_size": len(test_set),
|
||||
"history": history,
|
||||
}
|
||||
live_report_path.write_text(generate_html(partial_output, auto_refresh=True, skill_name=name))
|
||||
|
||||
if verbose:
|
||||
def print_eval_stats(label, results, elapsed):
|
||||
pos = [r for r in results if r["should_trigger"]]
|
||||
neg = [r for r in results if not r["should_trigger"]]
|
||||
tp = sum(r["triggers"] for r in pos)
|
||||
pos_runs = sum(r["runs"] for r in pos)
|
||||
fn = pos_runs - tp
|
||||
fp = sum(r["triggers"] for r in neg)
|
||||
neg_runs = sum(r["runs"] for r in neg)
|
||||
tn = neg_runs - fp
|
||||
total = tp + tn + fp + fn
|
||||
precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0
|
||||
recall = tp / (tp + fn) if (tp + fn) > 0 else 1.0
|
||||
accuracy = (tp + tn) / total if total > 0 else 0.0
|
||||
print(f"{label}: {tp+tn}/{total} correct, precision={precision:.0%} recall={recall:.0%} accuracy={accuracy:.0%} ({elapsed:.1f}s)", file=sys.stderr)
|
||||
for r in results:
|
||||
status = "PASS" if r["pass"] else "FAIL"
|
||||
rate_str = f"{r['triggers']}/{r['runs']}"
|
||||
print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:60]}", file=sys.stderr)
|
||||
|
||||
print_eval_stats("Train", train_results["results"], eval_elapsed)
|
||||
if test_summary:
|
||||
print_eval_stats("Test ", test_results["results"], 0)
|
||||
|
||||
if train_summary["failed"] == 0:
|
||||
exit_reason = f"all_passed (iteration {iteration})"
|
||||
if verbose:
|
||||
print(f"\nAll train queries passed on iteration {iteration}!", file=sys.stderr)
|
||||
break
|
||||
|
||||
if iteration == max_iterations:
|
||||
exit_reason = f"max_iterations ({max_iterations})"
|
||||
if verbose:
|
||||
print(f"\nMax iterations reached ({max_iterations}).", file=sys.stderr)
|
||||
break
|
||||
|
||||
# Improve the description based on train results
|
||||
if verbose:
|
||||
print(f"\nImproving description...", file=sys.stderr)
|
||||
|
||||
t0 = time.time()
|
||||
# Strip test scores from history so improvement model can't see them
|
||||
blinded_history = [
|
||||
{k: v for k, v in h.items() if not k.startswith("test_")}
|
||||
for h in history
|
||||
]
|
||||
new_description = improve_description(
|
||||
skill_name=name,
|
||||
skill_content=content,
|
||||
current_description=current_description,
|
||||
eval_results=train_results,
|
||||
history=blinded_history,
|
||||
model=model,
|
||||
log_dir=log_dir,
|
||||
iteration=iteration,
|
||||
)
|
||||
improve_elapsed = time.time() - t0
|
||||
|
||||
if verbose:
|
||||
print(f"Proposed ({improve_elapsed:.1f}s): {new_description}", file=sys.stderr)
|
||||
|
||||
current_description = new_description
|
||||
|
||||
# Find the best iteration by TEST score (or train if no test set)
|
||||
if test_set:
|
||||
best = max(history, key=lambda h: h["test_passed"] or 0)
|
||||
best_score = f"{best['test_passed']}/{best['test_total']}"
|
||||
else:
|
||||
best = max(history, key=lambda h: h["train_passed"])
|
||||
best_score = f"{best['train_passed']}/{best['train_total']}"
|
||||
|
||||
if verbose:
|
||||
print(f"\nExit reason: {exit_reason}", file=sys.stderr)
|
||||
print(f"Best score: {best_score} (iteration {best['iteration']})", file=sys.stderr)
|
||||
|
||||
return {
|
||||
"exit_reason": exit_reason,
|
||||
"original_description": original_description,
|
||||
"best_description": best["description"],
|
||||
"best_score": best_score,
|
||||
"best_train_score": f"{best['train_passed']}/{best['train_total']}",
|
||||
"best_test_score": f"{best['test_passed']}/{best['test_total']}" if test_set else None,
|
||||
"final_description": current_description,
|
||||
"iterations_run": len(history),
|
||||
"holdout": holdout,
|
||||
"train_size": len(train_set),
|
||||
"test_size": len(test_set),
|
||||
"history": history,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Run eval + improve loop")
|
||||
parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file")
|
||||
parser.add_argument("--skill-path", required=True, help="Path to skill directory")
|
||||
parser.add_argument("--description", default=None, help="Override starting description")
|
||||
parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers")
|
||||
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
|
||||
parser.add_argument("--max-iterations", type=int, default=5, help="Max improvement iterations")
|
||||
parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query")
|
||||
parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold")
|
||||
parser.add_argument("--holdout", type=float, default=0.4, help="Fraction of eval set to hold out for testing (0 to disable)")
|
||||
parser.add_argument("--model", required=True, help="Model for improvement")
|
||||
parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")
|
||||
parser.add_argument("--report", default="auto", help="Generate HTML report at this path (default: 'auto' for temp file, 'none' to disable)")
|
||||
parser.add_argument("--results-dir", default=None, help="Save all outputs (results.json, report.html, log.txt) to a timestamped subdirectory here")
|
||||
args = parser.parse_args()
|
||||
|
||||
eval_set = json.loads(Path(args.eval_set).read_text())
|
||||
skill_path = Path(args.skill_path)
|
||||
|
||||
if not (skill_path / "SKILL.md").exists():
|
||||
print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
name, _, _ = parse_skill_md(skill_path)
|
||||
|
||||
# Set up live report path
|
||||
if args.report != "none":
|
||||
if args.report == "auto":
|
||||
timestamp = time.strftime("%Y%m%d_%H%M%S")
|
||||
live_report_path = Path(tempfile.gettempdir()) / f"skill_description_report_{skill_path.name}_{timestamp}.html"
|
||||
else:
|
||||
live_report_path = Path(args.report)
|
||||
# Open the report immediately so the user can watch
|
||||
live_report_path.write_text("<html><body><h1>Starting optimization loop...</h1><meta http-equiv='refresh' content='5'></body></html>")
|
||||
webbrowser.open(str(live_report_path))
|
||||
else:
|
||||
live_report_path = None
|
||||
|
||||
# Determine output directory (create before run_loop so logs can be written)
|
||||
if args.results_dir:
|
||||
timestamp = time.strftime("%Y-%m-%d_%H%M%S")
|
||||
results_dir = Path(args.results_dir) / timestamp
|
||||
results_dir.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
results_dir = None
|
||||
|
||||
log_dir = results_dir / "logs" if results_dir else None
|
||||
|
||||
output = run_loop(
|
||||
eval_set=eval_set,
|
||||
skill_path=skill_path,
|
||||
description_override=args.description,
|
||||
num_workers=args.num_workers,
|
||||
timeout=args.timeout,
|
||||
max_iterations=args.max_iterations,
|
||||
runs_per_query=args.runs_per_query,
|
||||
trigger_threshold=args.trigger_threshold,
|
||||
holdout=args.holdout,
|
||||
model=args.model,
|
||||
verbose=args.verbose,
|
||||
live_report_path=live_report_path,
|
||||
log_dir=log_dir,
|
||||
)
|
||||
|
||||
# Save JSON output
|
||||
json_output = json.dumps(output, indent=2)
|
||||
print(json_output)
|
||||
if results_dir:
|
||||
(results_dir / "results.json").write_text(json_output)
|
||||
|
||||
# Write final HTML report (without auto-refresh)
|
||||
if live_report_path:
|
||||
live_report_path.write_text(generate_html(output, auto_refresh=False, skill_name=name))
|
||||
print(f"\nReport: {live_report_path}", file=sys.stderr)
|
||||
|
||||
if results_dir and live_report_path:
|
||||
(results_dir / "report.html").write_text(generate_html(output, auto_refresh=False, skill_name=name))
|
||||
|
||||
if results_dir:
|
||||
print(f"Results saved to: {results_dir}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,47 +0,0 @@
|
||||
"""Shared utilities for skill-creator scripts."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
|
||||
def parse_skill_md(skill_path: Path) -> tuple[str, str, str]:
|
||||
"""Parse a SKILL.md file, returning (name, description, full_content)."""
|
||||
content = (skill_path / "SKILL.md").read_text()
|
||||
lines = content.split("\n")
|
||||
|
||||
if lines[0].strip() != "---":
|
||||
raise ValueError("SKILL.md missing frontmatter (no opening ---)")
|
||||
|
||||
end_idx = None
|
||||
for i, line in enumerate(lines[1:], start=1):
|
||||
if line.strip() == "---":
|
||||
end_idx = i
|
||||
break
|
||||
|
||||
if end_idx is None:
|
||||
raise ValueError("SKILL.md missing frontmatter (no closing ---)")
|
||||
|
||||
name = ""
|
||||
description = ""
|
||||
frontmatter_lines = lines[1:end_idx]
|
||||
i = 0
|
||||
while i < len(frontmatter_lines):
|
||||
line = frontmatter_lines[i]
|
||||
if line.startswith("name:"):
|
||||
name = line[len("name:"):].strip().strip('"').strip("'")
|
||||
elif line.startswith("description:"):
|
||||
value = line[len("description:"):].strip()
|
||||
# Handle YAML multiline indicators (>, |, >-, |-)
|
||||
if value in (">", "|", ">-", "|-"):
|
||||
continuation_lines: list[str] = []
|
||||
i += 1
|
||||
while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith(" ") or frontmatter_lines[i].startswith("\t")):
|
||||
continuation_lines.append(frontmatter_lines[i].strip())
|
||||
i += 1
|
||||
description = " ".join(continuation_lines)
|
||||
continue
|
||||
else:
|
||||
description = value.strip('"').strip("'")
|
||||
i += 1
|
||||
|
||||
return name, description, content
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Anthropic, PBC.
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
0
skills/xlsx/scripts/recalc.py
Executable file → Normal file
0
skills/xlsx/scripts/recalc.py
Executable file → Normal file
Reference in New Issue
Block a user