Sigma Integration with Databricks Unity Catalog

We recently announced our partnership and integration with Databricks at the 2022 Data + AI Summit. Sigma’s new Databricks integration enables business users to leverage data in their organizations’ Data Lakehouses to make better and faster decisions using our no-code spreadsheet interface.

At its core, Sigma’s Databricks integration is built on top of Databricks SQL, Databricks’ data warehouse for the Lakehouse platform. Sigma connects to your Databricks SQL endpoint and runs machine-generated SQL queries as you interact with and ask questions about your data using Sigma Workbooks.

A big part of data analytics is discovering what data is available for you to use. As such, we’re excited to announce that Sigma is one of the first partners of Databricks to integrate with Unity Catalog, Databricks’ new data governance and catalog solution. Unity Catalog provides a unified governance solution for all data and AI  assets including files, tables, ML models in your Lakehouse on any cloud. Data teams can centrally manage access permissions and audit controls using a single interface based on ANSI SQL. Unity Catalog also offers automated and real-time lineage for tables, columns, notebooks, workflows, and dashboards. Built-in data search experience allows data teams to quickly find, understand and reference relevant data.

Once you’ve connected your Lakehouse to Sigma, you can use Sigma to search, explore, and discover all of the data that’s available to you.

To provide an up-to-date view of your data, Sigma builds an index of all of the data in your Lakehouse. Sigma uses  Databricks’ Unity Catalog API to fetch metadata for all of the catalogs, schemas, and tables from your Lakehouse. This metadata then goes on to power data searches and exploration capabilities. Additionally, Sigma uses metadata obtained from Unity Catalog to understand your tables’ schemas, in order to generate SQL to query the data in your Lakehouse.

For users that don’t have Unity Catalog enabled, Sigma falls back to indexing data that’s in your legacy Hive Metastore. However, using Unity Catalog makes it easier to manage your data’s access permissions and track your data usage and lineage in a centralized manner.

With Sigma’s Unity Catalog integration, it’s easy to find and browse through the tables that you need, build a Workbook to do your analysis, and collaborate with the rest of your team to make important business decisions.

Want to read more? Check out our blog post, A Practical Guide to Using Sigma with Databricks.