The Lineage Graph Is Free Now, Right Up to Where It Hurts

In 2023 two things made automated lineage close to a commodity: OpenLineage graduated within the LF AI & Data Foundation in September, and Apache Airflow 2.7 made OpenLineage a built-in provider in August. The pipeline now narrates itself. The arrow still stops at the last hop, which is the only hop a regulator or an executive ever asks about.

OpenLineage's September 2023 graduation confirmed an open standard for collecting lineage as jobs run, with built-in integrations for Airflow, Spark, Flink, dbt, and warehouses. Weeks earlier, Airflow 2.7 had folded OpenLineage from a brittle external plugin into a native provider you enable by configuration. Capturing how a table was built is no longer the expensive part.

This matters because it relocates the hard part without removing it. The graph reliably covers the orchestrated, transform-shaped middle — the part written in code that already emits events. It thins out at the two ends humans actually argue over: the source system that emits nothing, and the dashboard, the exported spreadsheet, the metric in the board deck where the figure is finally believed and acted on.

What it reveals is which work the org was willing to automate and which it wasn't. Code-to-code lineage got standardized because engineers wanted it and machines could supply it. The last mile stayed manual because it requires a person to declare that this number on this slide is that column, and to keep saying so as both drift. That is stewardship, and stewardship resists being shipped as a provider.

Watch where the graph ends in your own tool, and whether anyone notices. A lineage map that terminates at the warehouse boundary is honest infrastructure; a lineage map sold as "end-to-end" that quietly terminates there is theater with good production values. The gap is not a missing connector. It is the hop nobody was assigned to walk.