Unlocking Cost Savings and Performance Benefits with Sigma on Databricks Serverless

In the ever-evolving realm of data analytics and business intelligence (BI), adaptability and efficiency are paramount. To stay competitive in this data-driven era, organizations are continually seeking ways to optimize their operations while keeping costs in check. 

The synergy of serverless architecture and Sigma’s cloud-native analytics platform empowers analytics and BI workloads, fostering a cost-effective and agile environment, keeping you ahead in the data-driven competition.

Traditional Databricks SQL Warehouses

Before we dive into the world of Serverless architecture and Sigma, let's establish a clear understanding of what we refer to as "traditional" Databricks SQL warehouses. For the purpose of this blog, the term “traditional” encompasses both Classic and Pro SQL warehouses. These data warehousing solutions are distinct in their offerings but have the same fundamental infrastructure–their computing resources are hosted within the user's own cloud environment.

This conventional infrastructure has provided valuable insights to many organizations since its inception. Within this paradigm, Databricks users provision  virtual machines (VMs) within their chosen cloud before these VMs can be accessed and queried by the traditional SQL warehouse. Data scientists and engineers typically anticipate and find it acceptable to wait for the provisioning of compute resources, as these resources are often tailored to specific workloads. 

This customization comes at a price, though. This pattern necessitates careful configuration and monitoring that takes time to set up properly. Additionally, there are constraints concerning scalability, cost-effectiveness, and adaptability that arises as a consequence. As organizations expand and their data volumes grow, these traditional warehouses can turn into bottlenecks, resulting in query delays and escalating operational costs.

The Promise of Serverless

Serverless architecture represents a transformative shift in the way data processing and analytics are approached. But what is serverless architecture? In a serverless architecture,  Databricks host the “compute” within their environment then let you, the customer, use it in an instantaneous elastic way. This is different from the “compute” that is hosted within your instance of Databricks because this “compute” must be spun up. It offers several compelling benefits, including:

1. Scalability: Serverless SQL warehouses automatically scale resources up or down based on demand, ensuring that your analytics workloads can handle even the most substantial data volumes without any slowdown.
2. Cost Efficiency: With Serverless, you pay only for the resources you actually use, eliminating the need to provision and pay for excess capacity. This results in significant cost savings over time.
3. Flexibility: Serverless architecture is designed to be agile and adaptable, making it easier to accommodate evolving data requirements and new analytics workloads.
4. Improved Performance: Databricks Serverless includes features that are unavailable in traditional SQL warehouses, these features allow faster and more efficient  query execution. 

Empowering Your Analytics and BI Workloads with Sigma

Transitioning to a serverless architecture brings numerous benefits,especially when coupled with Sigma. Sigma, a self-service analytics and business intelligence platform, seamlessly integrates with your serverless infrastructure. It provides data professionals with a user-friendly experience that enhances efficiency and accessibility. Here's how Sigma can empower your analytics and BI workloads:

1. Self-Service Analytics: Sigma's intuitive interface empowers users to conduct complex analyses and build interactive dashboards without the need for advanced coding skills. This self-service capability accelerates data-driven decision-making.
2. Cost Savings: Sigma's interactive approach allows users to optimize queries and minimize resource consumption, resulting in direct cost savings. In the section below, we will highlight just how much you can save. 
3. Enhanced Collaboration: Sigma fosters cross-functional teamwork by enabling users to iteratively analyze data. It reduces dependency on specialized teams for reporting and analysis, enhancing overall efficiency.

Live Cost Savings Calculator with Sigma

One of the questions facing organizations migrating to serverless architecture is "What exactly is the cost savings of this migration?" In the workbook below, you can input your current traditional infrastructure configuration, and the Sigma workbook will calculate your cost savings for a variety of workloads and scenarios.

Example Scenario:

In the provided Sigma workbook, we demonstrate a scenario of the costs associated with usage for two Pro and one Classic Databricks SQL data warehouses hosted on AWS. These resources are set-up to be operational for 7 hours a day and run 5 days per week. 

• By utilizing Sigma's Compute Cost Calculator, you can conveniently determine the expenses associated with your existing Databricks infrastructure, as well as the potential cost reductions achievable when transitioning to a Serverless model.

This workbook is designed to enable you to input your specific data configurations, to receive an estimate of the financial benefits that can be realized by adopting a Databricks SQL serverless architecture.

Conclusion

The fusion of a serverless architecture and Sigma, not only empowers your analytics and BI workloads, but also creates a more cost-effective and agile environment. By harnessing the scalability and efficiency of serverless platforms and the user-friendly power of Sigma, you can future-proof your data analytics capabilities and stay ahead in the data-driven competition.

Get started now by connecting to Databricks.

Sigma is proud to be a Diamond Sponsor at this year's Databricks Data+AI World Tour in NYC! Stop-by our booth (booth # B21) or attend our Live session to learn how Sigma can empower you to drive insights and simplify data-driven decisions.

Ready to jump into Sigma and check it out for yourself? Start exploring your data today.

‍