Sigma for Data Teams: Part 1

Build Data Products Faster with Sigma

A series of blog posts and demos showcasing features that enable collaboration between data practitioners and business users to build, test, and productionalize data products fast.

There’s never been more excitement about the modern data stack. At the same time, there’s never been more discussion about its limitations. No, the modern data stack isn’t dead. But it is evolving, and it must evolve, because no tool alone can address the central problem organizations face when developing data products. And that is, the people who know how to transform data don’t understand it. And those who do understand the data can’t design and deploy analytics solutions. Despite the incredible advances in the scale, speed, and automation of cloud data, the process of data product development is still slow, cumbersome, and opaque. It’s no wonder that Gartner found that nearly 85% of data products fail to make it to production, and an abysmal 22% derive valuable insights for business users.

I witnessed this frustration firsthand as a Data Platform Architect managing cloud migrations and implementations for organizations large and small. Data teams can onboard and make data available in record time, but making sense of that data is a frustrating process that requires endless meetings, meetings about meetings, and often leaves data teams frustrated and business users in the dark. This is because creating data products isn’t just about the stack, but about the experience for end users. Creating data products is a conversation between business stakeholders and data practitioners, and organizations need a tool that enables, rather than hinders, that conversation. 

Human-Centric Development

Sigma was designed to bring data into everyday work and enable a conversation about data between consumers and providers. We know companies are making million-dollar decisions based on insights from their data. And they need to be able to make those decisions in the moment and answer questions as they arise. 

To understand why, consider an example of what a human-centric approach to data product development can look like, as we demonstrated during our Snowflake Summit demo last year. In short, human-centric data product development is an approach that centers the perspectives of individuals in the process of creating analytics that enable focused decision-making. 

Human-centric data product development:

• Brings data to the individuals who know how to utilize it
• Encourages continuous feedback and exploration of answers to novel questions
• Engenders trust in data consumers through transparency and cooperation, and
• Fosters collaboration as the central mechanism by which data products are created and distributed.

Human-Centric Data Product Development Must Scale

Legacy tools that cannot accommodate modern data at scale cannot form the basis for a human-centric design approach, because they necessarily limit the data set to a subset that prohibits true ad-hoc exploration. For end consumers to make actionable insights against their data, they first need to have access to it. Consider the limitations of some legacy tools.

The limitations of Excel are well known. As a legacy tool designed for on-premise data exploration, an Excel table can only process 1,048,576 rows of data. DirectyQuery, PowerBI’s connection to cloud data, is limited to one million rows. That might sound like a lot, but the average company maintains over 1.6TB of data, or roughly 160 billion rows. 

Tableau Online fares better, but recommends using extracts or aggregate tables after a data source exceeds 10GB. A workbook in Tableau can never exceed 15GB from all sources, with load times of five minutes, or more. 

The truth is, these tools were never designed for cloud scale. The modern data stack provides access to terabytes of data stored across billions of rows through cloud data warehouses, but many data consumers lack a platform that can handle such enormous workloads. By leveraging the virtual compute layer of your Cloud Data Warehouse, Sigma can process and surface data sets with no limitations, in lightning speed. Want a pivot table that can sit on top of a multi-billion row table? Sigma can handle that. 

Human-Centric Design Enables Exploration 

When a data consumer discovers an anomaly in a dashboard, what do they do? They turn to the data team to provide an answer. That is because data consumers lack the ability to write code and access row-level data at cloud scale.

Truly ad-hoc exploration requires enabling data consumers to perform root cause analysis at the row level. With Sigma, end users are not constrained by pre-defined drill-down paths. They can drill-down or up and perform calculations across groupings at multiple levels, with one-click access to the underlying data. 

Human-Centric Design Is Transparent 

Legacy BI tools rely on upstream models to apply business logic, often burying in complicated CASE switches that transform data as it flows between multiple layers. When upstream processes change and data outputs are no longer accurate, the data team–or consultants–are tasked with untangling a complicated web of logic and dependencies. Anyone who has done this knows how arduous this task is.

A human-centric approach exposes business logic to the data consumer in a platform they use and understand. Sigma allows data analysts to explore data directly from the warehouse, and apply their logic using familiar spreadsheet formulas and archetypes. They can apply business logic themselves, rapidly prototype a data product, and give the data engineers a working model of how to productionalize it upstream.

Human-Centric Design Encourages Collaboration 

It’s important to emphasize that no tool or platform can fully automate the human process of understanding and modeling data to derive business value.

‍There is no silver bullet to the churn of data product development. The modern data experience requires solutions that enable data teams and business users to collaborate together, on live data, and iterate toward a productionalized product that will be useful and actually adopted.  

Sigma centers this collaboration with a variety of features–live edit, versioning, sharing, commenting, and embedding. It is designed for teams to work together.

In this series, I will focus on how Sigma shortens the feedback loops between data teams and business users. My goal is to discuss how to build a truly collaborative approach to analytics.

Nathan Meinzer is a former Data Platform Architect, who has over 8 years experience designing and implementing cloud data platforms for organizations large and small at consultancies such as Thoughtworks & Slalom. Contact him at nate@sigmacomputing.com.

Ready to see how Sigma can help your organization build data products faster? Sign up for a free trial of Sigma.

Let's Sigma together! Schedule a demo today.