Objective
Our US Government Department of Defense Analytics Provider client’s data-driven analytics application faced a significant challenge with the cost and performance of its analytics queries. These queries, run hundreds of times per day, often returned no results—yet still incurred substantial compute costs. We were brought in to research and implement a solution that could proactively determine whether a query was likely to return any data before it ran—saving both time and money.
Conclusion
This feature addition not only solved a pressing cost issue for the client’s application but also enhanced its scalability, performance, and user experience. The success of this initiative highlights the value of smart infrastructure design and cross-functional collaboration in delivering business impact.
Challenge
- High-cost queries: Each query processed large volumes of data, and even empty-result queries came at a high cost—potentially thousands of dollars daily.
- Scalability: The solution had to work at scale with hundreds of terabytes of data.
- Integration: The new solution needed to plug into the existing application seamlessly and provide real-time feedback on query viability.
Solution
We designed and implemented a solution using Apache Druid, a high-performance analytics database, as a pre-query filter system. The goal was to summarize and index the data in a way that could quickly tell whether a given query might return results—before passing it to the main analytics engine, Google BigQuery.
Key Components
- Druid Cluster on Kubernetes:
Set up a highly available and scalable Apache Druid cluster within new Kubernetes (GKE) infrastructure, ensuring scalable handling of massive data loads. - Data Ingestion Pipeline:
Built a robust pipeline using Apache Airflow to ingest hundreds of terabytes of existing and ongoing data into Druid, optimizing for real-time ingestion and summarization. - Query Viability Layer:
Developed a system that used Druid to pre-evaluate queries—quickly checking if a query would likely return data. If not, it would short-circuit the process, saving unnecessary compute costs. - Application Integration:
Collaborated closely with the application team to integrate the new query viability layer directly into the product, ensuring a seamless user experience and a transparent performance gain.
Impact
- Cost Savings:
The system now prevents unnecessary queries, saving hundreds of dollars daily in compute and processing costs. - Performance Boost:
Application performance improved significantly, with fewer long-running queries clogging the system. - Enhanced UX:
Users receive immediate feedback when a query would return no data, improving transparency and usability.