The pod died every few hours, always the same way: RSS crept up until the kernel’s OOM-killer reaped it. No crash, no stack trace — just a 137 exit code and a graph that looked like a staircase.
The symptom
- RSS grew ~50MB/hour under steady, unremarkable load
- Heap snapshot diffs showed no single obvious leaker
- GC pauses lengthened the longer the process lived
The hunt
Snapshots pointed at thousands of retained objects hanging off the logger. The culprit: a "helpful" config that serialised the entire request context on every log line — headers, the user object, the parsed body. Each line pinned a whole object graph until the transport flushed.
// Before — pins the whole request graph on every line
logger.info({ req }, 'settlement swept')
// After — log identifiers, not object graphs
logger.info({ orderId, attempt }, 'settlement swept')Structured logging is a gift until you hand it objects with circular references or huge nested payloads. The serializer keeps them alive far longer than you think.
The fix
Trim the bindings to scalars, add a redaction allow-list, and cap serialised depth. Memory flatlined; we reclaimed 53% of heap across this and four sibling leaks.
If it isn’t observable, it isn’t done — but observability you can’t afford is just another leak.
The lesson stuck: log what you’d grep for, not what’s convenient to pass.