Make cost measurable first
The cost of a request usually combines input tokens, output tokens, model price, cache hits, and retries. Do not look only at the monthly bill. Record usage by feature, tenant, model, and failure reason so optimization has a target.
request -> identity/quota -> model route -> provider API -> schema check -> usage ledger
| |
+------ timeout/retry ----+
Build an adapter layer
Place an adapter between product code and vendor SDKs. Standardize timeouts, exponential backoff, request IDs, structured-output validation, and redaction. Record provider, model, prompt version, and token usage, but never store keys or complete private inputs.
const response = await llm.complete({ model: route.model, input: redact(userInput), timeoutMs: 20_000, idempotencyKey });
const parsed = schema.parse(response.output);
usageLedger.record({ route: route.name, input: response.usage.inputTokens, output: response.usage.outputTokens });
Four high-value optimizations
- Routing: use smaller models for classification, summaries, and formatting; reserve larger models for hard reasoning.
- Context: deduplicate, compress, and truncate before retrieval or generation.
- Caching: use versioned semantic caches for deterministic requests and invalidate them when data or prompts change.
- Retries: retry timeouts, 429s, and explicit transient errors only. A schema failure should repair the input or return an explainable error.
Streaming improves perceived latency but does not replace final validation. For side effects such as messages or database writes, parse and authorize before executing.
Budgets and governance
Set daily/monthly budgets, concurrency limits, and circuit breakers per tenant and feature. On an anomaly, degrade to a cache or rules path instead of silently increasing spend. Prices and model capabilities belong in configuration with an audit trail.
Sources
Use current vendor documentation such as OpenAI API pricing, Anthropic pricing, and rate-limit guides. This article does not assume a fixed price or quota.
