The API rate limit email arrived at 4 PM on a Friday. My agent had been happily processing requests all week, and somewhere between the morning coffee automation and the afternoon code review, it crossed the line.
Getting rate limited isn’t embarrassing — it happens to everyone. Getting rate limited without knowing you were close to the limit is embarrassing. It means you have no visibility into your API consumption, and that’s a problem I should have solved weeks earlier.
Where Rate Limits Bite
Most AI API providers enforce multiple limits, and the one that catches you is never the one you expected:
Requests per minute. The obvious one. Send too many requests in a short burst and you get throttled. Batch operations are the usual culprit — processing 50 items fires 50 requests in rapid succession.
Tokens per minute. Less obvious. Even if you’re sending few requests, each one might process a large context window. Three requests with 50K tokens each = 150K tokens per minute, which exceeds many standard tier limits.
Tokens per day. The sneaky one. You might be well within your per-minute limits but gradually accumulate throughout the day. Long conversations, heavy cron jobs, and background tasks all contribute.
Concurrent connections. The most frustrating one. Even if you have budget remaining, having too many simultaneous open connections gets you throttled.
My Rate Limit Strategy
After getting burned, I built a three-layer approach:
Layer 1: Awareness. A simple dashboard widget showing current usage as a percentage of each limit. Updated every 60 seconds. When usage exceeds 70%, the widget turns yellow. At 90%, it turns red. This takes 10 minutes to implement and saves hours of surprise.
Layer 2: Automatic throttling. When usage approaches 80% of any limit, the system automatically slows down non-critical requests. Interactive user messages still go through immediately. Background tasks (cron jobs, batch processing) get queued and spread over a longer time window.
The implementation: a token bucket rate limiter that sits between OpenClaw and the API. It tracks usage against all four limit types and gates requests accordingly.
Layer 3: Graceful degradation. When a limit is actually hit (429 response), the system:
1. Backs off with exponential delay (1s, 4s, 16s)
2. Switches non-critical tasks to a cheaper/slower model if available
3. Alerts me that a limit was hit (so I can investigate if unexpected)
4. Queues any requests that can wait
The key insight: not all requests are equal. A user waiting for a response in Slack is very different from a background analytics job. The rate limiter should prioritize accordingly.
Reducing API Consumption
The best rate limit strategy is consuming fewer API tokens:
Prompt caching. If the same system prompt is sent with every request (and it usually is), ask your provider about prompt caching. Anthropic caches the first part of the prompt and charges less for cached tokens. This can reduce costs by 30-50% for repetitive workloads.
Response caching. For questions your agent gets asked repeatedly, cache the response and serve it without making a new API call. “What’s our refund policy?” doesn’t need to be processed by the AI model every time.
Context trimming. The biggest single source of unnecessary token consumption is bloated conversation context. Old messages that aren’t relevant to the current question are still being sent to the API and consuming tokens. Enable compaction. Trim history. Be aggressive about removing irrelevant context.
Smart model routing. Simple tasks (classification, formatting, yes/no questions) don’t need your most expensive model. Route them to a cheaper model that’s adequate for the task. Save the premium model for complex reasoning.
Monitoring What Matters
The metrics I track daily:
– Total tokens consumed (input and output, separately)
– Tokens per interaction (average and p95)
– Retry rate (what percentage of requests needed retrying)
– Queue depth (how many background requests are waiting)
– Cost per interaction (for budgeting)
The metric that’s most useful for optimization: tokens per interaction. If this number creeps up over time, my context is growing or my prompts are getting bloated. If it spikes suddenly, something changed that I should investigate.
The Practical Outcome
After implementing all of this:
– Zero unexpected rate limit events in the past 4 months
– Token consumption reduced by about 35% (from context trimming and smart routing)
– API costs reduced by about 40% (from prompt caching and cheaper model routing)
– No impact on response quality for user-facing interactions
The rate limit email that started all this was actually a gift. It forced me to build visibility and control over my API consumption. Without it, I’d still be flying blind, paying more than necessary, and occasionally getting surprised.
🕒 Last updated: · Originally published: January 14, 2026