Six months of OpenClaw logs. That’s what I had when I finally sat down to figure out why some debugging sessions took 5 minutes and others took 2 hours. The answer was obvious in retrospect: logging.
Not whether I had logs — I always had logs. The problem was that half my logs were useless noise (“Process started… process running… process still running…”) and the other half were missing the information I actually needed when things broke.
Here’s what I changed, and how my debugging time dropped from an average of 45 minutes to about 12 minutes per incident.
The Problem With Default Logging
Default logging is designed by developers who know what everything means. When the developer sees “Context compaction triggered at 142K chars,” they know exactly what that means and what to check next. When I see it at 3 AM, I think “is that normal? Is 142K high? Was it supposed to compact at 142K or at 100K? Is this related to the error I’m investigating?”
Default logs assume you have perfect knowledge of the system. Production debugging happens when you have imperfect knowledge and are probably stressed.
What I Log Now
I restructured my logging around one principle: every log entry should help me answer “what happened and why?” without needing to look at any other system.
API calls: Model used, input token count, output token count, response time, status (success/error), error message if any. One line per call. This tells me immediately if the API is slow, failing, or expensive.
Tool executions: Tool name, input summary, output summary, duration, success/failure. When a tool fails, I can see exactly what was attempted and what went wrong without digging through raw output.
Session activity: Session start, significant events (new user message, tool call, context compaction), session end. This gives me a timeline of what happened in each session.
Errors: Full error message, stack trace, relevant context (session ID, user request, recent activity). The context is crucial — an error without context tells you something broke, but not why.
What I stopped logging: Routine heartbeats (“still alive” messages every 30 seconds), successful health checks, internal state transitions that are normal and expected. These added volume without adding information.
Log Levels That Make Sense
Most logging frameworks offer DEBUG, INFO, WARN, ERROR levels. I use them like this:
ERROR: Something failed and needs human attention. I read every ERROR log. If I’m getting more than 5 ERROR entries per day in normal operation, my thresholds are wrong.
WARN: Something unusual happened but the system handled it. Rate limit hit and backed off, context compaction triggered, retry succeeded after failure. I review WARN entries daily to spot patterns.
INFO: Normal operations I might want to trace. API calls, tool executions, session events. I only read these when debugging a specific issue.
DEBUG: Detailed internal state for deep debugging. Input/output of every function, memory allocation, connection pool status. Disabled in production unless I’m investigating a specific bug.
The key: in production, I run at INFO level. This gives me enough detail to diagnose most issues without the noise of DEBUG. I switch to DEBUG temporarily when investigating specific problems, then switch back.
Structured Logging
Plain text logs are hard to search and impossible to aggregate. I switched to JSON structured logging:
Instead of: 2024-03-15 14:23:45 ERROR API call failed: timeout after 30s
I log: a JSON object with timestamp, level, event type, model, error, duration, session ID, and request ID.
The JSON format lets me:
– Search by any field (all errors for session X, all timeouts for model Y)
– Aggregate metrics (average response time per model per hour)
– Build dashboards (Grafana can read JSON logs directly)
– Correlate events (follow a request from arrival through processing to response)
The tradeoff: JSON logs are less human-readable when you’re tailing the log file. I use a log viewer tool that formats JSON prettily for real-time monitoring.
Log Rotation and Retention
AI agent logs grow fast. A moderately active OpenClaw instance generates 50-200MB of logs per day at INFO level. Without rotation, your disk fills up in weeks.
My retention policy:
– Last 7 days: full logs (INFO level), uncompressed for quick access
– Days 8-30: compressed logs (gzipped, about 10x size reduction)
– Days 31-90: ERROR and WARN entries only (extracted from full logs before deletion)
– Beyond 90 days: monthly aggregate metrics only (no raw logs)
This keeps my total log storage under 5GB while maintaining enough history for trend analysis and incident investigation.
The Debugging Workflow
When something breaks, I follow this sequence:
1. Check the last 10 ERROR entries — usually reveals the immediate cause
2. Search for the same error type in the past week — is this a recurring issue or a one-off?
3. Look at the timeline around the error — what happened in the 60 seconds before the error?
4. Check for correlating events — did the error coincide with a deployment, a config change, or an external service outage?
This systematic approach, combined with good logging, resolves most issues in 10-15 minutes. Before structured logging, the same issues took 30-60 minutes because step 3 and 4 required manual log file archaeology.
🕒 Last updated: · Originally published: January 19, 2026