Catalog Lineage Limitations Start at the Application Layer

Data engineering teams trust their catalog's lineage graph until an audit or an incident proves it wrong. A catalog tool draws a clean line from source table to dashboard, but that line is inferred from query logs and warehouse metadata, not from the code that actually moved and transformed the data. Anything that happens before data lands in the warehouse, inside an ORM, a Python job, a Java service, or a Spark pipeline, is invisible to that graph.

The consequence shows up at the worst possible moment: an auditor asks where a regulated field originated and the answer stops at a staging table with no path back to the application that wrote it, or an AI agent reasons over a field whose true source was never mapped and produces a confidently wrong answer. Source code analysis closes this gap by tracing lineage from the line of code that created a value, not from the log that recorded its arrival.

This post covers how catalog based lineage is actually built, where the application layer breaks it, what deterministic lineage means in this context, and what full coverage looks like in practice.

How Catalog Tools Actually Build Lineage

Most catalog tools build their lineage graphs by parsing SQL query logs and warehouse metadata: which query read from which table, which job wrote to which destination. This method works well inside the warehouse, where every operation is a logged SQL statement.

It breaks down the moment a transformation happens outside SQL. A Python script that reshapes a dataframe before loading it, a Java service that calculates a derived field before an API call, or an ORM that silently renames and casts columns between an application database and a reporting layer: none of these generate a SQL log entry a catalog tool can parse. The lineage graph simply stops at the boundary where the data entered the warehouse, showing a source that is really just the most recent hop.

Where the Application Layer Breaks the Chain

The application layer is where most enterprise business logic actually lives, and it is exactly the layer catalog tools cannot see into. Here is where log based lineage typically loses the thread, layer by layer:

• SQL and dbt models: declarative transformations inside the warehouse. Catalog visibility here is full, since this is the layer catalogs were built for.
• ORMs such as Django, Hibernate, and SQLAlchemy: column mapping, renaming, and casting between application and database schemas. Catalog visibility here is minimal to none.
• Python and Java application code: business logic, derived fields, conditional transformations. Catalog visibility here is effectively none, since the code is never parsed.
• Spark and batch pipelines: joins, aggregations, and reshaping before warehouse load. Catalog visibility here is partial, depending on whether jobs emit logs.
• AI and ML feature pipelines: feature engineering from raw and derived fields. Catalog visibility here is effectively none, and this is a frequent source of untraceable model inputs.

Each layer with minimal or no catalog visibility is a place where a governance or compliance question cannot be answered from the catalog alone. An engineering lead who has debugged a lineage discrepancy already knows this: the catalog says the field came from a staging table, but the real answer is three services and an ORM mapping upstream, and no one wrote that mapping down.

What Is Deterministic Lineage

Deterministic lineage is lineage derived directly from the source code that performs a transformation, rather than inferred from logs, samples, or naming conventions. Because it is read from the actual instructions a program executes, it produces the same, exact path every time, with no probabilistic guessing about which column fed which output.

This distinction matters because inferred lineage is only ever an approximation of what code does. Deterministic lineage traces every assignment, join, and conditional in the code itself, across SQL, Python, Java, Scala, and ORM layers, so the graph reflects what the system actually does rather than what its logs suggest it might have done.

What Good Looks Like: Lineage Traced From the Source, Not the Logs

Complete lineage requires reading the code that moves data, not just the tables it eventually lands in. Foundational is the only data and AI governance platform that analyzes source code directly, including SQL, Python, Java, Scala, and the ORM layer, to build lineage from where a value originates rather than where it happens to land. Catalog tools see only where data lands. Foundational sees where it originates.

That difference is measurable. After adopting source code based lineage, Lightricks now prevents approximately 100 data issues every month across more than 150 pull requests, according to Eyal El-Bahar, VP of BI and Analytics at Lightricks. That is not a warehouse level metric. It is a direct result of catching problems in the application and pipeline code before they ever reach a table a catalog tool would monitor.

For a data engineering lead, the practical shift is this: instead of maintaining a separate mental map of "what the catalog shows" versus "what I know actually happens in the code," the lineage graph and the codebase tell the same story, because one is generated from the other.

Frequently Asked Questions

Does catalog-based lineage cover Python and Java transformations?

Generally, no. Catalog tools infer lineage from SQL logs and warehouse metadata, so transformations that happen inside Python scripts, Java services, or ORMs before data reaches the warehouse are typically invisible to the lineage graph. Teams often discover this gap during an audit or incident, when the graph cannot explain how a field was actually derived. Closing it requires analyzing the application source code directly rather than only the warehouse's query history.

What is the difference between catalog lineage and source code based lineage?

Catalog lineage is inferred from query logs and metadata after data has already moved. Source code based lineage, sometimes called deterministic lineage, is derived by reading the actual code (SQL, Python, Java, Scala, or ORM mappings) that performs a transformation. The result is a lineage graph that reflects exactly what the code does rather than an approximation built from what its logs suggest.

Why does this gap matter for AI governance specifically?

AI pipelines consume features and derived fields that are frequently built in Python or application code rather than SQL. If lineage stops at the warehouse boundary, no one can verify what raw data actually fed a model's inputs. That breaks the chain of trust auditors and AI governance frameworks require: data trust depends on complete, deterministic lineage all the way back to origin, not just visibility inside the warehouse.

Can this gap be fixed without replacing an existing catalog?

Yes. Source code analysis is complementary to a catalog rather than a replacement for one. It fills in the application layer coverage a catalog was never built to provide, so teams keep their existing catalog investment while adding the piece that traces lineage back through ORMs, Python, Java, and Spark to where data actually originates.

How much engineering effort does closing this gap typically require?

Deployment is designed to be lightweight since it reads existing source code and pull requests rather than requiring teams to instrument new logging. Some Foundational customers report deployment in under an hour, with lineage coverage extending immediately to the application layers a catalog could never reach.

What to Do With an Incomplete Lineage Graph

An incomplete lineage graph is not a documentation gap, it is a governance and audit risk that compounds every time a new pipeline ships without visibility into its application layer logic. The fix is to trace lineage from the source code itself rather than adding more manual documentation on top of a graph that was never built to see past the warehouse. See how Foundational's cross-platform lineage traces data from application code through to the warehouse, or request a demo to see source code based lineage mapped against your own pipelines.