ObservabilityMay 22, 20262 min read

The day a logging config ate half my heap

Memory climbed under load with no leak in sight — because the cause wasn't a bug, it was the logs. We capture full request and response bodies on critical flows; here is why that is worth doing, and what it quietly costs.

Memory crept up under load. Not a spike — a climb. The kind that ends in an OOM kill a few hours later. I went hunting for a leak and found something more embarrassing: there was no leak. The system was doing exactly what I had told it to. I had just told it to remember too much.

The good intention

On the flows that matter most — order creation, payments, refunds — we log the full request and response body. When money moves and a customer disputes a charge, "what exactly did the client send, and what did we return?" is the difference between a five-minute answer and a forensic afternoon. A single interceptor captures it:

logging.interceptor.tsTypeScript
@Injectable()
export class LoggingInterceptor implements NestInterceptor {
  intercept(ctx: ExecutionContext, next: CallHandler) {
    const req = ctx.switchToHttp().getRequest();
    return next.handle().pipe(
      tap((responseBody) => {
        logger.info({
          route: `${req.method} ${req.url}`,
          status: ctx.switchToHttp().getResponse().statusCode,
          // full payloads on critical routes ONLY — never globally
          ...(isCritical(req.url) && { requestBody: req.body, responseBody }),
        });
      }),
    );
  }
}

Note the discipline already baked in: full bodies on critical routes only. Everywhere else we log metadata — method, path, status, a correlation id — and nothing more. That restraint is the only reason this story is about memory and not about leaking customer data into log files.

Where the memory actually went

Here is the part I hadn't internalised: a logged object isn't free, and it isn't fire-and-forget. A structured logger serialises the whole object graph, and the transport buffers it before it ships. Until that buffer drains, every captured body — a cart with forty line items, a fat provider response — stays alive on the heap. Now multiply by request rate.

Under normal traffic the buffer drains faster than it fills and you never notice. Under load it inverts: bodies arrive faster than the transport flushes, the queue grows, and the queue is made of large objects that can't be collected. On a graph it looks exactly like a leak. It isn't. It's backpressure in your logger.

The trap

A structured logger doesn't log a string — it retains an object until it's serialised and flushed. A logged request body is memory you hold, not memory you spend.

The fix wasn't logging less

The reflex is "turn off the body logging." Wrong — that capability is genuinely valuable on the critical paths. The fix is logging deliberately:

The rule

Log what you'd grep for, not what's convenient to keep.

Observability has a price, and the bill comes due under exactly the load where you most want the logs. The cheapest log line is the one you decided, on purpose, not to keep. I didn't log less after this — I logged on purpose.

← All field notes