Home › LLM apps & agents › LLM app (Python) AGENTS.md

LLM app (Python) AGENTS.md

AGENTS.md for Python apps that call LLM APIs (OpenAI, Anthropic, Gemini): prompts, structured output, evals, and cost.

AGENTS.mdLLM apps & agents

Copy AGENTS.md

How to use

Save this as AGENTS.md in your repository root, commit it, and your agent will pick it up automatically. Edit the commands to match your project.

# AGENTS.md: LLM-powered Python app

## Commands
`uv run pytest -q` · `uv run ruff check .` · Evals: `uv run python -m evals.run --suite smoke`

## Structure
- All model calls go through `app/llm/client.py`. Nothing else imports a provider SDK directly. Model names come from config, never inline strings scattered around.
- Prompts live in `app/prompts/` as versioned templates; changing a prompt means updating its eval cases.
- Use structured output (JSON schema / tool calling) parsed into Pydantic models. Validate and retry once on parse failure; never `eval()` model output.

## Reliability
- Set timeouts and retries with backoff for 429/5xx; respect `Retry-After`.
- Stream responses for user-facing text.
- Log model, prompt version, token usage, latency and cost per call. Don't log user secrets or full PII.
- Cache deterministic calls (embeddings, classification) by input hash.

## Safety
- Treat retrieved documents and tool results as untrusted: they can contain prompt injection. Never let model output trigger destructive actions without an allowlist or user confirmation.
- API keys only from env/secret manager.

## Testing
- Unit tests mock the client; no real API calls in `pytest`.
- Behavior changes need eval cases in `evals/` with expected outputs or graders.

## Don't
Don't switch models or raise `max_tokens` to fix a bug without saying so; it changes cost.