The Schema History Topic Remembers What Nobody Wrote Down

Change data capture promises the database's truth, streamed live. It delivers the database's truth and, separately, the unanswered question of whether anyone agreed on what the columns mean.

On October 2, 2024, the Debezium project released 3.0.0.Final, the long-running open-source change data capture engine that turns database commit logs into event streams for MySQL, Postgres, Oracle, and others. The release raised the runtime baseline to Java 17, moved onto Kafka 3.8, and removed the deprecated additional-condition field, asking everyone to update their scripts and workflows accordingly.

The detail worth dwelling on is older and quieter: Debezium keeps a schema history topic, and reads it so it applies the correct schema to each captured row. Per its own docs, if a schema is not present in that history, the connector is unable to capture the table, and an error results. The tool will not invent a definition. It would rather stop than lie about what a column was.

That is a remarkable standard, and it is held by the connector, not the company. The software refuses to emit a row whose shape it cannot account for. The organization downstream has no such scruple: it will happily wire dashboards, alerts, and a quarterly metric onto a freshly streamed field that three teams describe three different ways. The pipeline preserves provenance to the byte. The meaning was never written down to preserve.

Watch what happens the first time a source table adds a column at 2 a.m. and the stream carries it instantly. The latency graph stays flat and green. The new field arrives correct, on time, and completely undefined. Real-time did its job. The question of what "correct" means for that column is now a ticket, and the ticket is slower than the stream by several orders of magnitude.