FlexQuery: A Major Update to our Vision for Analytics and BI

"We’re going to stop fixing data in the BI layer starting today!"

I have heard some version of this refrain countless times in my career, and the last decade brought a ton of innovations that at times made it seem possible. Innovations including the wide adoption of cloud computing, cloud warehouse and lakehouse architectures, and the shift from ETL to ELT. At every step of the process, well-meaning data teams swore off the practice of solving data problems in BI due to the obvious issues it creates: Metrics locked in BI, inconsistent and low-quality data, and a high volume of rework.

Did anyone succeed though?

Almost universally, no. Business reality changes too fast for even the most adept data team. Data models start losing relevance the moment they are deployed. End users run faster than your governance program can keep up with. BI-layer data transformation, modeling, and calculation is an inevitable and inescapable part of data, regardless of our attempts to prevent it.

The First Step

We took on this challenge at GoodData by introducing one of the industry’s first and most powerful open business intelligence semantic layers. No longer are these important ‘last mile’ transformations or calculations locked into a proprietary front end. Thanks to our SDKs, APIs, and SQL access point, any system can benefit from the excellent work of your data analysts and BI engineers. It was an important first step, but our customers still struggled with complex data structure, speed, and cost issues that semantics alone simply could not solve.

The Next Stage of our Vision

It’s for that reason, we’re excited to announce FlexQuery, the next stage of our vision to provide powerful, open tools to data developers solving ‘last mile’ data problems that simply can’t be addressed in data warehouses. FlexQuery is a composable data service layer built on popular open-source technologies like Apache Arrow, Apache Iceberg, and DuckDB. We’ve pre-built the most useful business intelligence services into it, such as caching, data federation, transformation, and computation. However, in keeping with our commitment to developer-first approaches, FlexQuery is open for you to build modules that meet your specific needs.

So what can you actually do with FlexQuery?

Save money with advanced caching

FlexQuery automatically caches full and partial result sets for re-use, saving you significant money on your underlying cloud data architecture. Based on Apache Arrow, the ‘FlexCache’ supports super-fast in-memory computation on large volumes of data. Moving forward, we plan to provide more and more granular cache controls to help you balance data recency vs. cost savings.

Transform and federate data with agility

FlexQuery’s built-in transformation and federation capabilities make it easy to combine data from all of your applications via SQL or APIs. You can materialize data in memory or on disk query it live from your databases, or even mix the two modes together. It’s easy to do data engineering directly within FlexQuery to create the complex pipelines necessary to prep data for consumption.

Build with the best open-source tools

FlexQuery is built on the best open-source analytics tools, including Apache Arrow, Iceberg, and DuckDB, pre-configured for core BI use cases. However, at GoodData we always give you the keys to the kingdom: It’s easy to develop your own FlexQuery modules to address data engineering or data science needs at low cost directly in your BI platform.

Welcome to the Analytics Lake

FlexQuery is now at the computational heart of what we call the ‘Analytics Lake.’ Unlike a data lake, the Analytics Lake contains data, metadata, data models, metrics, and reports/visualizations in a single platform, making it easy to create and govern any kind of analytics application without complex software integration challenges. Moreover, thanks to our semantic layer, each user accesses the Analytics Lake through their favorite interface — Python for data scientists, React for application developers, SQL for data engineers and, no code/low code and chat interfaces for business users.

FlexQuery and the Analytics Lake concept represent the next step forward for business intelligence platforms, moving beyond simple point-and-click desktop interfaces to embrace the full complexity of data development in the modern era. With a combination of intuitive user interfaces, powerful analytics-as-code tools, and now the FlexQuery analytics engine, GoodData offers organizations large and small the world’s most powerful, flexible, and efficient business intelligence platform.

Ready To Learn More?

To learn more about FlexQuery and the Analytics Lake head here, or to explore it hands-on along with the rest of the GoodData analytics platform, request a demo.