LLM Cost Optimization — Routing, Caching, Token Discipline, Batching

Step 1 — Establish the cost and quality baseline. Break cost down by use case, tokens, model, route and completed workflow. Create a quality set before changing the route.

Step 2 — Layer 0: avoid the model call. Use deterministic code for exact calculations, return approved static answers, stop duplicate retries, reject invalid requests.

Step 3 — Layer 1: exact response cache. Reuse an answer only when the normalized request is identical, user and permission scope match, source freshness remains valid, and the answer is safe to reuse.

Step 4 — Layer 2: semantic response cache. Use similarity to identify equivalent requests, then apply domain boundaries, confidence threshold, answer version, quality validation and exclusions for personal or high-risk questions.

Step 5 — Layer 3: retrieval cache. Reuse search results, embeddings or reranked passages when the question, access scope and corpus version permit it.

Step 6 — Layer 4: prompt-prefix and provider cache. Keep stable instructions, tool definitions and shared context at the beginning of the request so supported providers can reuse the prefix.

Step 7 — Layer 5: model routing. Use rules for obvious simple tasks, a small classifier for variable complexity, escalate uncertain or high-stakes work to a stronger model, fall back when a route misses the quality threshold.

Step 8 — Layer 6: token and context discipline. Retrieve only useful evidence, remove duplicated context, set output length by task, use structured outputs, load tools only when relevant.

Step 9 — Layer 7: batch and serving efficiency. Send non-urgent work through a batch path. For self-hosted models, use continuous batching and prefix caching.

Step 10 — Quality-controlled rollout. Shadow the optimized route, compare with the baseline, release by traffic segment and monitor quality, cost and latency together.