Change Data Capture (CDC)

Change data capture (CDC) is a technique for detecting and capturing insert, update, and delete events in a source system as they happen, so those changes can be propagated to other systems without reprocessing the entire dataset.

How Change Data Capture Works

Most CDC implementations read a database's transaction log directly, the same internal record the database uses for its own recovery and replication, rather than repeatedly querying tables to check what changed. Log based CDC is efficient because it captures every change exactly once, in order, with minimal load on the source system. Older approaches relied on comparing full snapshots or on triggers attached to each table, both of which scale poorly as data volume grows.

Once captured, changes are typically streamed through a message system such as Apache Kafka to one or more downstream consumers: a data warehouse, a search index, a cache, or another application. The result is that downstream systems stay close to real time with the source, instead of waiting for a nightly batch job to catch up.

CDC is now a standard building block for keeping a warehouse or lake synchronized with operational databases, for feeding streaming analytics, and for replicating data across regions or platforms without full reprocessing.

How Change Data Capture Relates to Adjacent Terms

CDC is often positioned against batch ETL. Batch ETL extracts, transforms, and loads data on a schedule, often hours apart, and typically reprocesses larger slices of data each run. CDC instead streams individual change events continuously, which is why it is the mechanism of choice when downstream systems need to reflect a source system within seconds or minutes.

CDC is also frequently confused with data lineage. CDC tells you that a row changed, and what its new values are. It does not tell you why the row changed, meaning which business logic, application code, or transformation produced that change. That explanatory layer comes from lineage, not from the change stream itself.

The Blind Spot CDC Leaves Behind

A CDC feed is excellent at telling a data engineering team that something changed. It is not designed to explain what caused it. When a downstream metric shifts unexpectedly, the CDC stream shows the row level change but not the SQL query, ORM call, or job that generated it. Closing that gap requires source code analysis: tracing the actual SQL, Python, Java, dbt, and Spark logic that produces and transforms each field connects a change captured by a CDC pipeline back to the exact piece of code responsible for it, turning row level change detection into a real root cause answer.

For a data engineering lead running CDC pipelines at scale, that means less time spent tracing a downstream anomaly backward through logs and more time spent looking directly at the code that caused it.

Related Glossary Terms

• ETL (Extract, Transform, Load)
• Data Lineage
• Deterministic Lineage
• Apache Spark

Frequently Asked Questions

What is change data capture used for?

Change data capture is used to keep downstream systems, such as a data warehouse, search index, or replica database, synchronized with a source system in near real time. Instead of reprocessing an entire dataset on a schedule, CDC captures individual insert, update, and delete events as they occur and streams them to wherever they are needed, which reduces load and latency compared to full batch reprocessing.

Is change data capture the same as ETL?

No. ETL typically runs on a schedule and processes data in scheduled batches, often extracting and transforming larger slices of a dataset at once. CDC continuously captures individual change events as they happen in the source system. Many pipelines use both together, with CDC feeding a stream of changes that a transformation layer then processes, often in smaller, more frequent increments than a traditional batch job.

Can change data capture tell me why a value changed?

Not on its own. CDC captures that a row changed and what its new values are, but not the business logic or code that produced the change. Answering why requires tracing the actual SQL, application, or pipeline code responsible, which is a lineage problem rather than a change capture problem. Source code based lineage is what connects a captured change back to its cause.