The Data Apps Conference
WATCH ON-DEMAND
A globally recognized private equity firm with a four-decade track record of acquiring and scaling businesses across diverse industries has elevated its data strategy through the seamless integration of LLMs through Sigma and Databricks.
Faced with the growing challenge of handling large amounts of unstructured data—including financial documents, term sheets, credit agreements, and resumes—the firm sought a solution to streamline data extraction and structuring. Sigma has played a crucial role in addressing these challenges, enabling teams to seamlessly interact with larger datasets, bridge the gap between technical and non-technical users, and leverage AI-powered models to extract structured insights and enable error correction. To achieve this, the firm adopted Sigma as a core component of its data strategy, integrating AI-powered models into its workflow alongside Databricks.
Prior to implementing Sigma, the firm relied heavily on Excel for data analysis. While widely used across the organization, Excel struggled to scale with the increasing volume and complexity of data generated by portfolio companies. Financial documents, often spanning hundreds of pages, required significant manual effort to extract meaningful insights, slowing down decision making and increasing the risk of errors.
The head of the data team also needed a solution that could integrate with its existing data infrastructure, particularly Databricks. Existing business intelligence tools failed to provide the right balance between robust analytics capabilities and accessibility for non-technical users. This created inefficiencies, limiting the ability of investment teams to quickly access, analyze, and act on critical financial data.
The firm sought a solution that would allow investment teams to interact with large datasets while integrating seamlessly with Databricks. Sigma stood out due to its ability to function as an intuitive front-end interface while leveraging the full power of a modern data warehouse - as well as pull through LLMs.
Key factors that made Sigma stand out included:
A core aspect of the firm’s implementation of Sigma has been its ability to extract and structure data from unstructured sources efficiently. Using Sigma on Databricks, the firm can process complex financial agreements and extract critical details through custom LLMs. For example, PDFs that contain financial metrics, entities, and date-time data are able to be extracted. This allows the firm to process complex financial agreements and extract critical details such as:
By automating this process, the firm has significantly reduced the time spent manually reviewing and extracting information from lengthy documents. Databricks serves as the data warehouse and processing engine, where raw financial documents and datasets are stored. Sigma provides an interface for visualization and human-in-the-loop verification. AI models process documents within Databricks, while Sigma allows users to validate, refine, and write back structured outputs directly into the warehouse.
The firm also utilizes AI-powered models for resume parsing, automatically extracting key qualifications while removing personally identifiable information to maintain privacy. Through Unity Catalog in Databricks, these AI models remain fully auditable, with permissions programmatically controlled to prevent unauthorized access to financial data.
Sigma’s ability to integrate LLMs directly into workflows ensures that unstructured data can be structured efficiently, reducing manual processing and enabling real-time decision-making.
The firm has witnessed a fundamental shift in how data is processed and utilized. Investment teams now have direct access to AI-enhanced insights, empowering them to make data-driven decisions faster and with greater confidence. The ability to error-correct AI-generated outputs directly within Sigma ensures that human oversight remains integral to the process, maintaining accuracy while reducing manual labor.
Furthermore, the firm's data security has been strengthened through programmatic control of permissions, ensuring that only authorized personnel can access sensitive financial information. By leveraging Sigma’s write-back capabilities, users can also continuously refine AI models.
By integrating Sigma with Databricks, the firm has positioned itself at the forefront of AI-driven financial analysis. The synergy between advanced analytics, machine learning, and human expertise is allowing teams to focus on what truly matters—strategic decision-making and value creation across their investments.
Read more about LLMs in Sigma here.
