Benchmark: measured token reduction
Run it yourself, no API key required. The benchmark uses token estimation (chars ÷ 4), not Anthropic API calls.
Run the benchmark
$ npx tsx scripts/benchmark.tsSample output
[D] Standard session ............ 40% reduction (37k → 22k tokens)
[D] Heavy agentic session ....... 53% reduction (51k → 24k tokens)
[D] Monorepo exploration ........ 74% reduction (40k → 11k tokens)
[D] Long session trim ........... 85% reduction (26k → 3.9k tokens)
[D] Test-debug loop ............. 80% reduction (47k → 9.5k tokens)
Multi-provider support
Token counting works across every provider The Distillery proxies. Extraction is response-based — no parallel count-tokens calls, no API key shenanigans.
| Provider | Wire format | Token extraction | Sample savings |
|---|---|---|---|
| Anthropic | Anthropic native | SSE message_start + message_delta | 30-60% |
| OpenAI | OpenAI chat/completions | SSE stream_options.include_usage | 25-50% |
| Groq | OpenAI-compat | SSE stream_options.include_usage | 25-50% |
| Mistral | OpenAI-compat | SSE stream_options.include_usage | 25-50% |
| Ollama | OpenAI-compat | Response body usage object | Local (no $) |
| Gemini | Google contents/parts | SSE usageMetadata in final chunk | Passthrough (token-counted) |
| Bedrock | Bedrock Mantle (OpenAI-compat) | SSE stream_options.include_usage | Passthrough (token-counted) |
Sample savings ranges are illustrative for OpenAI-compatible providers — context patterns and the compression presets you select determine the realised saving. Run the benchmark against your own sessions for grounded numbers.
Methodology
Token counts use chars ÷ 4 estimation, the same method Anthropic uses for pricing previews. No Anthropic API calls are made during the benchmark. Model: claude-sonnet-4-5.
Raw data: /benchmark.json
Install and measure your own usage
Install in 30 seconds →