Why Sigma Exists: How The Modern Data Stack Is Changing the World of Work

The world of data analysis is fundamentally changing. Companies that don’t adapt will be left behind.

Leading researchers at Moffitt Cancer Center rely on data for everything. The Florida-based nonprofit hospital is one of the best in the U.S., identifying the most promising new treatments, finding patients for clinical trials, and studying which cancer treatments are working or not working.

That means timely and accurate data exploration is key. And because analyzing healthcare data is also incredibly sensitive – data must be de-identified for different parties, as well as be HIPAA-compliant – they turned to Sigma in early 2022.

The vast wealth of information stored in disparate, siloed datasets and databases and challenges with analyzing and sharing millions of rows of data have plagued the field of healthcare for decades, and Moffitt is no exception. Previously at Moffitt, the only data analysis method was teams using Excel and an older medical analytics tool that capped the number of rows it accepted. And Moffitt had 1.5 billion rows of data. That meant a lot of manual labor combining Excel sheets and prepping data, because the old tools couldn’t process that much data. 

Now, not only can Moffitt’s data teams manipulate and curate the data they need in half the time, analysts can do cohort analysis much faster, find the right patients for trials, and ultimately advance cancer research at a faster rate.  

Moffitt Cancer Center’s story of working with data before Sigma isn’t all that unique. In fact, it’s the way most data teams work today: Have a few analysts be the “data experts” who take the time to find the answers the higher-ups want, which typically requires a lot of manual work and time. But that old way of working has to change. It’s slowing businesses down, and ultimately impacting their ability to accomplish their missions. 

At Sigma, we believe that businesses that don’t adapt to the new world of instant data analysis will be left behind. And there’s a lot on the line. 

The Analytics Gap

Using data to make decisions is how the world works today: Retailers use data to monitor live inventory of warehouses. Financial institutions rely on data to stay in compliance – or face huge fines if they aren’t. Pharmaceutical companies decide which diseases to work on, all based on making sense of the data at hand. Analytics spending is literally expected to hit $274 billion in 2022, more than double the spend from 2015. 

The problem is that analyzing this data can take a really long time. It’s far too complicated. Data sizes are massive and ever-increasing. And the tools aren't getting any easier to use, at least not for non-data experts. 

A proliferation of analytics tools like Tableau, Looker, and Power BI, make it easier to build a simple dashboard, but harder for non-data pros to get deeper answers or ask new questions of the data. Designed in the 2000s and 2010s, these tools weren’t made for the new world of massive, cloud-based data, so they come with the many limitations of old-world computing. While easy to use for basic use cases, they ultimately are a band-aid approach to a deeper problem of the on-premises computing world. 

The New World of Work: Risks Companies Take By Not Adapting

The first risk is that rollups and aggregations of data are inherently risky. That’s because you’re not using all of the data in your warehouse, which means, at best, that you have a false sense of security; And at worst, you are missing serious patterns and trends. Take this simple example: If you look at the average of 100 and 0, it’s 50. But the number 50 may not actually be a descriptor for what’s happening in the data. Rolled-up data can hide the very trends that you need to understand in order to compete.

The second risk of the old way of analyzing data is that most businesses just can’t respond fast enough to the pace of change. If you run a business and recognize there’s been a business impact from a war overseas or a significant market dip, you need data to analyze it. And you need that data NOW. By the time you get data experts aligned and the data modeled, too much time has passed. The game has changed again, and now you’re even more behind. 

The final risk of the old way of analyzing data is that it’s just a really inefficient way to use company resources. If every time I ask a question about company data I need to get a data pro involved, there are a few likely results: One result is that I’ll slow down the pace of getting answers because I’m constantly waiting for the experts. A second result is that I’ll just stop asking questions. Because I feel bad bothering the data pros, I’ll save only my top questions for them. 

Or a third result: I’ll just default to whatever data I know I can get quickly. This last one is what happens the most often. You get the personal analysis you need for your one project, and now it’s sitting on your personal drive never to be collaborated on again. You try to make it useful by emailing it around, but since everyone can’t collaborate, it doesn't become the living document it needs to be. 

How Sigma Can Help

Sigma uses an easily understood interface that’s immediately familiar to more than 1 billion people in the world: the spreadsheet. If you’re one of those billions that use a spreadsheet on a monthly basis, you already know how Sigma works. Sigma is a spreadsheet-like interface on top of all your company’s cloud data warehouse, like AWS or Snowflake. But beyond that, it’s an interface that introduces a radically different way of analyzing data. Instead of relying on analytics teams who know programming languages to pull reports that are instantly outdated, business analysts can do it themselves. Really. 

With the necessary SQL automatically generated in the background, Sigma users quickly grab last month’s sales in the northwest region, this week’s rise in employee satisfaction, or today’s change in inventory. The program manager can drill down to any specific piece of data the company owns. Nothing is inaccessible, and they don’t need code to find it. Most importantly, Sigma doesn’t store or persist your company’s data – ever. It never even leaves the cloud data warehouse. 

The bottom line is that working with data in Sigma  makes things that are impossible with other tools  — like pivot tables and cohort analysis  — accessible to the business person.  It makes drilling down to transaction level detail simple. And it also allows easy, synchronous collaboration on building analytics and analyzing them with your team. If the median Fortune 1000 business increased the usability of its data by just 10%, it would translate to an increase in $2.01 billion in total revenue every year. That’s a lot of cash for an incremental change in work effort. 

Building the More Open World

In the on-premises, non-cloud world, data lives with the data people. It’s essentially a top down approach. 

Sigma works from the bottom up. Now business users are empowered to go look things up on their own. Instead of falling behind and having analytics teams pull data from last week or month, they’re working in a live edit space with unconstrained analysis. Ultimately, the goal of Sigma is fundamentally changing the way people work, and ushering in a more open world – a distinct difference from products that are more closed.

Here’s what we find the most exciting: businesses are finding answers in their data to questions they didn’t even know to ask. This becomes an entirely different process – and an entirely different end result – to go looking at data and being open to what you might find, vs. finding the solution you expected. That’s the difference between shallow dashboards and the scalability of the cloud, and it’s why Sigma exists.

‍