Postgres WAL, replication, and how Argus was born

Every row you write to PostgreSQL touches at least two places before it is safe: the Write-Ahead Log and the shared buffer pool. Not the table file on disk — that comes later, lazily, by the checkpointer. Understanding this write path is what led me to build Argus, a zero-trigger audit sidecar that reads change events straight off the WAL and routes them to Kafka. But before the architecture, the fundamentals.

ACID — the contract databases make with you

ACID is not a checklist. It is four properties that together make a database trustworthy, and durability is the one that explains everything about the WAL.

• Atomicity — a transaction is all-or-nothing. Either every statement in it commits, or none do. No half-written orders, no rows that saw only the first INSERT.
• Consistency — the database moves from one valid state to another. Constraints, foreign keys, and check predicates are enforced on commit.
• Isolation — concurrent transactions see a consistent view of each other. How much they bleed into each other is controlled by isolation levels (read committed, repeatable read, serializable).
• Durability — once a transaction commits, it survives. A crash, a power cut, a kernel panic — the data is still there when the process comes back. This is the hard one, and the WAL is how PostgreSQL delivers it.

Durability is expensive if done naively. Writing every changed byte to the table file and calling fsync on commit would serialise your write throughput to the speed of one disk seek per transaction. The WAL is the solution: write sequentially to one file, fsync that, and do everything else lazily.

What the WAL actually is

The Write-Ahead Log is an append-only file sequence in $PGDATA/pg_wal. Every change to the database — inserts, updates, deletes, even internal bookkeeping like hint bits — is recorded here before it is applied anywhere else. That is the "write-ahead" in the name: you write the intention before you write the fact.

Each record in the WAL is identified by a Log Sequence Number (LSN): a monotonically increasing byte offset into the WAL stream. LSNs are your coordinates in the history of the database. You will see them everywhere once you start working with replication.

-- Current WAL write position
SELECT pg_current_wal_lsn();
--  pg_current_wal_lsn
-- --------------------
--  1/3B4A2C80

-- How far a replica is behind
SELECT write_lag, flush_lag, replay_lag
FROM pg_stat_replication;

The PostgreSQL write path

When a transaction commits, here is what actually happens — the table file on disk is not touched yet.

• Shared buffer pool — dirty pages (modified data blocks) live here in memory. The table file on disk may be completely stale.
• WAL buffer → WAL file — the WAL record describing the change is written to a WAL buffer, then flushed to the WAL file with fsync on commit. This is what makes durability real.
• Checkpointer — periodically (by default every 5 minutes or 1 GB of WAL, whichever comes first) the checkpointer walks the dirty page list and writes modified blocks from shared buffers to the actual table files. Only then does the data reach the heap file on disk.
• WAL archiver / replication — separately, WAL segments can be shipped to replicas or archived for point-in-time recovery. Replicas replay the same WAL records to reconstruct the database state.

Transaction COMMIT
       │
       ▼
  WAL buffer ──fsync──▶ pg_wal/*.wal   ← durable HERE
       │
       ▼
  Shared buffer pool (dirty pages in RAM)
       │
       ▼  (async, later)
  Checkpointer writes dirty pages
       │
       ▼
  Table files on disk   ← eventually consistent

How other databases handle this

The WAL concept is universal. Every database that offers durability has one — they just call it different things.

• MySQL / InnoDB — has two logs. The redo log (InnoDB's WAL equivalent) handles crash recovery and write durability. The binary log (binlog) is a separate, higher-level log used for replication and point-in-time recovery. The binlog records statements or row images, not physical page changes. MySQL replication subscribes to the binlog — the direct analogue to PostgreSQL's logical replication.
• SQLite — uses a journal file by default (rollback journal). In WAL mode (PRAGMA journal_mode=WAL), SQLite writes changes to a separate .wal file instead of modifying the DB file directly. Readers can continue reading the original file while a writer appends to the WAL — concurrent reads without blocking writes. Much simpler than Postgres WAL but the same fundamental idea.
• MongoDB — the WT (WiredTiger) storage engine maintains a write-ahead journal at journal/WiredTigerLog.*. Durability semantics depend on the write concern; with j:true the operation is durable as soon as the journal is flushed, before the main data files are updated.
• CockroachDB / Pebble — uses a WAL based on LevelDB / RocksDB conventions. Each node writes a WAL; Raft consensus across nodes provides the distributed durability guarantee on top.

The WAL is the oldest trick in the database book: promise durability through a fast sequential write, and settle the expensive random I/O on your own schedule.

Streaming replication — sending WAL to replicas

The simplest form of Postgres replication is physical streaming replication: the primary sends WAL bytes to one or more standbys, which replay them to produce an identical copy of the database. The standby connects with the replication privilege, and the primary streams WAL in real time.

Physical replication is byte-for-byte identical. You cannot replicate a single table, filter rows, or transform events — you get everything or nothing. That is fine for high-availability standbys. It is useless for audit logging, change data capture, or event-driven architectures.

Logical replication — decoding WAL into row events

Logical replication, introduced in PostgreSQL 10, decodes the physical WAL into a stream of logical change records: INSERT, UPDATE, DELETE on named tables, with actual column values. This is the layer Debezium, pglogical, and Argus all sit on.

Two primitives make it work.

Publications

A publication is a named set of tables (or ALL TABLES) whose changes will be exported. It lives on the source database and declares what you are replicating.

-- Publish every table in the database
CREATE PUBLICATION audit_pub FOR ALL TABLES;

-- Or be selective
CREATE PUBLICATION orders_pub FOR TABLE orders, order_items;

-- See what is published
SELECT * FROM pg_publication;
SELECT * FROM pg_publication_tables WHERE pubname = 'audit_pub';

A publication is cheap — it is metadata, no data is copied. Multiple subscribers can consume the same publication independently. Argus looks for audit_pub during its pre-flight validation and suggests the CREATE PUBLICATION command if it does not exist.

Replication slots

A replication slot is the server-side cursor into the WAL stream. It tracks how far a consumer has read and, critically, prevents the server from discarding WAL segments the consumer has not yet read. The slot persists across restarts — if Argus goes offline for an hour, the WAL is held for it.

-- Create a logical slot using the pgoutput plugin
SELECT pg_create_logical_replication_slot('audit_stream_slot', 'pgoutput');

-- See active slots and their LSN positions
SELECT slot_name, plugin, confirmed_flush_lsn, wal_status
FROM pg_replication_slots;

The pgoutput plugin

Postgres ships with a logical decoding output plugin called pgoutput. It translates internal WAL records into a binary protocol that consumers can parse. You subscribe to a slot using this plugin, and it streams back messages tagged begin, relation, insert, update, delete, commit — one for every row changed inside every committed transaction.

A relation message describes the table schema (column names, types, keys). It is sent before the first change on a table, and again if the schema changes. This is how Argus knows the column names without querying pg_attribute itself.

Building Argus on top of all of this

Argus (the project I called AuditStream internally) is a NestJS service that taps a source Postgres database via logical replication and fans the resulting events out to pluggable sinks. No triggers, no application-level hooks, no changes to the source schema. The database itself is the emitter.

The pipeline has five stages, each with a specific durability guarantee.

1. Pre-flight validation

Before starting, Argus runs a probe that checks: can it connect, is wal_level = logical, and does the audit_pub publication exist? If not, it returns the exact SQL to fix each problem.

// probe.service.ts — simplified
const walResult = await client.query('SHOW wal_level;');
if (walResult.rows[0].wal_level !== 'logical') {
  result.suggestions.push(
    "ALTER SYSTEM SET wal_level = 'logical'; -- then restart PostgreSQL"
  );
}
const pubResult = await client.query(
  "SELECT 1 FROM pg_publication WHERE pubname = 'audit_pub'"
);
if (!pubResult.rowCount) {
  result.suggestions.push("CREATE PUBLICATION audit_pub FOR ALL TABLES;");
}

2. Subscribing to the WAL

The ReplicationService creates or verifies the replication slot, then subscribes via pg-logical-replication — a Node.js library that speaks the Postgres replication wire protocol. Events arrive as a stream of PgOutputMessage objects.

// replication.service.ts
const plugin = new PgoutputPlugin({
  protoVersion: 1,
  publicationNames: ['audit_pub'],
});

this.replicationService.on('data', (lsn, log) => {
  void this.handleWalEvent(lsn, log);
});

await this.replicationService.subscribe(plugin, 'audit_stream_slot');

3. Turning WAL messages into canonical events

Each insert, update, or delete message becomes an AuditEvent: a stable envelope carrying a UUID idempotency key, the action, schema, table name, old/new row data, a field-level diff for updates, the LSN, and a timestamp.

// replication.service.ts — handleWalEvent
const event: AuditEvent = {
  eventId: randomUUID(),          // idempotency key — sinks dedupe on this
  action: log.tag.toUpperCase(),  // INSERT | UPDATE | DELETE
  schema: log.relation?.schema ?? 'public',
  table:  log.relation?.name ?? 'unknown',
  oldData: log.old ?? null,       // full row before change (for UPDATE/DELETE)
  newData: log.new ?? null,       // full row after change  (for INSERT/UPDATE)
  diff: DiffCalculator.calculateDiff(log.old, log.new),  // changed fields only
  lsn,
  ts: new Date().toISOString(),
};

The diff is computed at read time, once, so every downstream sink gets it without recomputing. For an UPDATE that changed three columns out of thirty, the diff is a small object with exactly those three keys — old value and new value side by side.

4. Producing to Kafka — WAL position as backpressure

The event is produced to Kafka with acks=-1 (all in-sync replicas) and an idempotent producer. The produce call is synchronous from the WAL reader's perspective: if it throws, the WAL LSN is not advanced. Postgres holds that segment until Argus catches up. This is the front-door durability guarantee — an event can enter the pipeline or not, but it cannot be silently dropped.

// kafka-producer.service.ts
await this.producer.send({
  topic: 'audit-events',
  acks: -1,   // wait for all in-sync replicas to confirm
  messages: [{
    key: `${event.schema}.${event.table}`,  // per-table ordering
    value: JSON.stringify(event),
    headers: { eventId: event.eventId, action: event.action },
  }],
});
// If this throws → WAL LSN is NOT advanced → Postgres retains the segment

5. Consuming and fanning out to sinks

A separate Kafka consumer group (audit-sink) drains the topic with autoCommit=false. Offsets commit only after every enabled sink has acknowledged the batch. A sink failure means no commit, which means Kafka redelivers — the consumer lags but nothing is lost.

Sinks run concurrently via Promise.allSettled. Any failure throws a SinkWriteError that blocks the commit. Transient failures retry with exponential backoff (1 s → 30 s cap). Poison messages that cannot be parsed go to a dead-letter topic immediately so they do not block healthy events.

// kafka-consumer.service.ts — offset commit contract
await this.sinks.write(events);   // all sinks must ack
resolveOffset(batch.lastOffset());
await commitOffsetsIfNecessary(); // ONLY now does Kafka move forward

Four sinks ship with Argus, each idempotent on eventId:

• Postgres audit sink — createMany({ skipDuplicates: true }) on unique eventId. Also publishes to an in-process SSE bus for live-tail in the UI.
• External DB sink — pipes into your Postgres with ON CONFLICT (event_id) DO NOTHING. Auto-creates the table on first write.
• Webhook sink — POSTs the batch to any HTTP endpoint with HMAC-SHA256 signatures. Idempotent because the eventId travels in the payload.
• Object store sink — writes NDJSON to S3 or MinIO. The key is audit/{table}/{yyyy}/{mm}/{dd}/{firstEventId}.ndjson — deterministic, so re-delivery rewrites the same object instead of creating a duplicate.

The elegant part: zero impact on source

No triggers. No shadow tables. No application code changes. No extra queries on the source database. Argus attaches to the logical replication stream — a mechanism Postgres already maintains for physical replicas and logical subscribers. The source database does exactly the work it would do anyway, plus the marginal cost of decoding WAL records the checkpointer would have processed regardless.

The only configuration required on the source:

-- One-time setup on the source database
ALTER SYSTEM SET wal_level = 'logical';
-- restart PostgreSQL once

CREATE PUBLICATION audit_pub FOR ALL TABLES;
-- that's it. Argus handles the slot, the subscription, and everything downstream.

Debezium built an industry on this idea. Argus is a smaller, self-hosted version — but the principle is identical: the WAL is already the most accurate ledger of what happened to your data. Read it.

The WAL has been in PostgreSQL since version 7.1 in 2001. Twenty-five years later, it is still the cleanest foundation for building change-data capture, audit logs, and event-driven pipelines. You just have to know it is there.

Resources

• PostgreSQL: Write-Ahead Logging — official docs covering WAL configuration, checkpoints, and archiving. postgresql.org/docs/current/wal.html
• PostgreSQL: Logical Replication — publications, subscriptions, replication slots, and conflict handling. postgresql.org/docs/current/logical-replication.html
• PostgreSQL: Logical Decoding — the pgoutput plugin, output plugin API, and how to consume a replication slot programmatically. postgresql.org/docs/current/logicaldecoding.html
• pg-logical-replication (npm) — the Node.js library Argus uses to subscribe to a slot and receive decoded pgoutput messages. npmjs.com/package/pg-logical-replication
• Debezium — the production-grade CDC platform that inspired this approach. Its architecture docs are the best explanation of slot-based capture at scale. debezium.io/documentation
• MySQL: The Binary Log — MySQL's equivalent of logical decoding. Explains the difference between statement, row, and mixed formats — analogous to wal_level in Postgres. dev.mysql.com/doc/refman/8.0/en/binary-log.html
• SQLite WAL Mode — how SQLite implements WAL with a separate .wal file, its concurrent-reader model, and the automatic checkpointing that collapses it back to the main DB file. sqlite.org/wal.html
• Designing Data-Intensive Applications — Kleppmann — chapter 5 (Replication) and chapter 11 (Stream Processing) give the deepest mental model for why WAL-based CDC is the right foundation for event-driven systems.