What Is Analytics Engineering? (And What It’s Not)

Just as changes in macro-economic markets bring about new types of jobs and new roles, changes in the field of data and analytics have brought about new tools, methodologies, and roles. One of the roles that emerged, and popularized by dbt Labs, is the analytics engineer. While this role is not necessarily new, the requirements and responsibilities for this role have changed and evolved over the years. In this blog, we will discuss what an analytics engineer does (and doesn’t do) and how this role can be extremely valuable to a data and analytics organization.

Why are analytics engineers needed?

Analytics engineers have been called by many names - Business Analysts, Data Wranglers, or BI Developers, to name a few. But more importantly than their name, it is important to understand the role that analytics engineers play within an organization. In order to understand why analytics engineers are so important, we must first understand how the role emerged and how we got here — so first, a history lesson. 

Let’s rewind to the late 1990s and early 2000s when systems we now refer to as legacy BI were emerging in popularity. These legacy BI tools — such as Cognos, BusinessObjects, and Hyperion — were being owned and operated within central IT teams. This was standard operating procedure for the time, and it made sense because the technical knowledge and understanding of data systems were not as proliferated within the workforce as they are today. These tools were fairly clunky and designed with IT users in mind. They generally output what we would refer to today as paginated reports for end users to either print out or export to Excel to put into a Pivot Table for further analysis. 

Fast forward to 2010 - a new wave of BI tools hit the market that focused on being low/no code and marketed at the line of business (LOB) users - the most notable of them, Tableau. This wave of low/no code drag and drop BI tools proliferated the idea of self-service analytics that allowed line of business users to create their own reports and dashboards for their own business processes rather than having to rely on Central IT for analytics. While self-service analytics had many upsides, there were also a number of drawbacks that came with it including, massive training and upskilling budgets needed for users, loss of central IT governance and control over data models and usage, and duplicative work across LOBs that ultimately resulted in increased cost of ownership and time lost. 

This is where the analytics engineer was born. 

What is an analytics engineer?

In order to solve some of the problems introduced by the widespread adoption of low/no code BI tools such as Power BI and Tableau, organizations knew that they needed a way in for the LOBs and Central IT to work more efficiently together. This is where the analytics engineer comes in. 

Two things can be true at once — there is value to Central IT maintaining ownership of data, assets including governance and security of data models, and there is value to LOB users being able to access data and use said data for their real-time analytics and use cases. Because both of these statements are true, companies need people in their organizations who understand business use cases, speak the language of IT, and have technical chops to translate business use cases to IT while understanding the overarching IT data strategy and processes. The role being described is that of an analytics engineer. 

In almost every organization, there are gaps between central IT and the users working in the lines of business. These gaps are not a negative thing but rather a difference in understanding, skill sets, and context for the data being used by the organization. Central IT teams are very focused on the technical aspect of data - the ingestion, transformation, and modeling of stored data in a Central Data Warehouse (CDW). On the other hand, users in the lines of business have a much different focus. They are more focused on leveraging the data that IT has provided to gain value-added insights into how their business is doing or how they could be doing better. analytics engineers are the bridge that span these gaps. 

What do analytics engineers do?

Now that we’ve covered how we got here and the role of an analytics engineer, let’s dive into how this plays out in a real organization. In any given LOB, there are users who are extremely knowledgeable about the business solution but are less knowledgeable about data schemas, ETL processes, and data modeling. This is where analytics engineers step in. 

When a business need arises, it is up to analytics engineers to meet with the project’s key stakeholders to ascertain all of the user requirements. Because the analytics engineer also has knowledge of the CDW and the IT processes, they will be able to piece together a plan of action to get the business use case solution. The analytics engineer will then take these requirements and work to build out the necessary data models needed for the desired data products. Analytics Engineers will often use tools such as Alteryx, KNIME, or platforms such as Sigma to investigate existing data models and find what data is needed and how it needs to be aggregated to solve the business use case. 

Once the analytics engineers craft the solution, they will collaborate with IT to either 1) leverage existing data models in the CDW or 2) build a new data model that meets the needs of the business. The analytics engineer can operate in a more agile way than the line of business (LOB) user and at a lower cost than IT resources. This enables the rapid creation of data products at a reduced cost.

Another example of what analytics engineers do at organizations is to take already existing data products and create more sustainable solutions for businesses. In this example, the LOB user creates a self-service data product (or an array of data products), but the existing data process does not fall under Central IT’s purview. The analytics engineer is then able to work in collaboration with IT to either 1) move the existing data product’s model into an existing IT data model or 2) build a new data model with IT that fits the business solutions. This methodology ensures that the LOB still has the flexibility to create self-service analytics products while also ensuring that Central IT still owns the data and models needed for the LOB analytics. 

Why are analytics engineers so important?

Analytics engineers are important not only because they are able to relay business needs to IT personas, but also because they ultimately reduce costs and increase the efficiency of data and analytics operations at an organization. 

Generally, IT resources have higher cost rates than LOB analysts. This means that the more time that IT works on a LOB data project, the less ROI a LOB will see on said project. That logic is true for any resource, but given the high bill rate of IT resources, the impact is even greater. As mentioned in the previous section, because of the unique skill set that analytics engineers have, they are able to more rapidly prototype and productionalize data assets needed for data production creation. Not only does this have budget and cost implications for organizations, but it also has huge impacts on the timeliness in which data needs are fulfilled. Because of the scale that most large organizational IT departments are operating, the timeliness piece of the analytics engineer cannot be understated. 

In addition to the cost advantages of having an analytics engineer working in your scrum team, analytics engineers also provide an effective component to any data team. Inherently, IT teams and LOB teams think about data differently. As the bridge between the teams, analytics engineers can fully comprehend business processes and needs — but because of their technical acumen, they can better communicate those needs to IT personas than the LOB analysts. This results in less time lost in the process of development and UAT, as well as more accurate requirements given to the IT teams in a way that they can understand. 

Analytics engineers as part of fusion teams

Analytics engineers have become such a part of the modern data organizational structure that when Gartner formalized and popularized the concept of Fusion Teams around 2018, the analytics engineer played a central role in that operational structure. For those unfamiliar with the concept, Fusion Teams, defined by Gartner, are multidisciplinary teams that blend business and IT roles, where individuals work together to design and deliver digital initiatives - essentially “fusing” IT and the LOB.

Fusion teams are often comprised of data engineers, analytics engineers, and BI developers. The data engineer’s job is to ensure that all of the data relevant to their domain are ingested, transformed, and modeled correctly for the LOB use cases. The BI developer’s job is to take the data that the data engineer has exposed and turn them into the data products that are needed by the LOB. Lastly, but maybe most importantly (in this delivery model), the analytics engineer’s job is to rapidly prototype data models and business logic leveraging the exposed data models to meet business requirements.  Once the BI developer has developed and validated these rapid prototypes, the data engineer will validate and implement the requirements. Without the role of the analytics engineer, these Fusion Teams are still operating in the IT & LOB limbo of the 1990s. 

Fusion Teams generally work in domain-centric teams that are focused on delivering key data products for specific LOBs.  Because of their focus on specific domains and the same resources working on those domains, these Fusion Teams have become more efficient over time, leading to a lower cost of digital initiatives and faster delivery time. This model has shown to be very efficient for businesses and can help build trust in Central IT, and ultimately make the CDW data assets more valuable to the organization. 

How Sigma enables analytics engineers

As we’ve discussed, an effective analytics engineer needs to understand both business use cases and technical data processes. Because of these unique requirements, analytics engineers need to leverage tools that allow them to access underlying data from the CDW, visualize data for validation and testing, and drill down to specific data points to investigate discrepancies or validate KPIs. Many tools can do one or some of these, but no tool is built for the niche needs of the Analytics Engineer more than Sigma. 

Sigma’s high-performing direct queries to the CDW allow analytics engineers to work with real-time data without worries that the data in their model does not reflect data that has been meticulously validated. The easy-to-learn syntax allows Analytics Engineers to use Sigma to create calculations for business logic and validation purposes quickly. Additionally, Sigma’s drill-through capabilities will enable users to investigate individual data points throughout a data product with ease and quickness, which is absolutely necessary for the high-paced day of an Analytics Engineer.

‍