Announcing Looker Support in Foundational

Our goal at Foundational is to make it easy, predictable, and straightforward to develop code for data – which is any piece of code that might affect data throughout the data stack.

We are proud to announce our full support for Looker, a prominent business intelligence tool. With this addition, companies running Looker can ensure that:

• Any pull request created within any repository will be analyzed pre merge, to determine its downstream impact on Looker dashboards, views, looks, and explores.
• Any LookML pull request created within Looker will be analyzed pre merge, to validate its code and identify issues before they affect the data.
• End-to-end, fully automated column-level lineage, that's always updated to the latest commit, to understand every form of dependency between Looker and all other parts of the stack.

There’s more, but before we dive further into specifics, let’s zoom out to understand better why something as basic as avoiding a broken dashboard is still a very common problem today.

Challenge with scaling business intelligence

Business Intelligence, or BI, is often one of the most fragile components of a data stack. There are many reasons for this, among them:

• The code that affects the data a dashboard is reporting is scattered across many different repositories.
• Many tools (and BI teams) are not managing BI changes in git and don’t have a pull request process in place - A dashboard change is rarely peer reviewed.
• BI tools were built for individual report optimization but are not optimized for the broader organization, for example, duplicate and almost-identical dashboards exist side-by-side, causing both confusion but also do not benefit from running against the same data set (This is often a cost problem as well, as more teams are realizing these days)

With Looker, the problem is even worse since the data or analytics engineers do not own LookML, and the analysts who do own LookML, do not “live” in the same repository and often do not even report to the same team. This creates a reality where the data engineering team is blind to what’s happening in LookML, and the analyst team is blind to everything that happens before Looker, whether it’s dbt, Airflow, Spark or even the ORM layer itself, which is the operational database where data is ingested. Any change throughout those code spaces can have a catastrophic impact on data and is often isolated.

Our approach at Foundational

At Foundational, we are pragmatists. We consider the starting point of existing teams and repositories as one that is hard to change – maybe even impossible to. Our goal is simple, we want to provide the context and validation to every code change that impacts data. How does it pertain to Looker? We can consider three main use cases:

1. Code changes that impact Looker entities - Here, there’s a PR outside of Looker but there are lineage-based dependencies that would impact the data in Looker, whether it’s a view or a dashboard. Our goal is to provide that context automatically every time such a PR is created, and to help the developer to enforce a “virtual” contract that is de-facto already in place, but was never defined. Simply put, if the developer is introducing a semantic issue, or is breaking a dashboard, we want them to know right away when the PR is created.

2. Code changes that are made within Looker - In this case, a code change is made within Looker and LookML and would cause an unintended data issue. These are unfortunately frequent since Looker is only doing basic syntax checks, so most SQL issues would pass unnoticed.

3. Column-level lineage between Looker and other platforms - While more tools today can provide some form of lineage that is based on warehouse data, Foundational is the only one that does this from the source code, which means that a developer can always understand what is the actual piece of code that is determining a certain dependency, know that lineage is updated to the latest commit, and trace it back to the exact location in the code. This is essential for common development workflows such as understanding who is the owner, when was something last changed and by whom, reference the actual pre-compiled source code to understand any nuances, and many other critical elements that are essential for large-scale development.

Chat with us 

At Foundational, we are solving extremely complex problems that data teams face on a day-to-day basis. Not breaking dashboards is only one aspect – Connect with us to learn more.