This event triggers the creation of two ECS tasks in an ECS Cluster (Step 2). In Step 1, an EventBridge cron rule is triggered on a daily basis.The different stages are shown in the high-level architecture diagram below: They then use Glue to ingest the data into Glue DB in a format that Athena can query, and finally Athena is used to query the data and visualise it using QuickSight. To accomplish this, they built a serverless data pipeline that uses CloudQuery to collect data from all of their cloud accounts, and then stores it in S3. Lastly, they wanted to be able to explore the history of their cloud configuration over time. They also wanted to be able to query this data using SQL, and to be able to visualise it using their existing tools (like AWS QuickSight).

The Hexagon Cloud Center of Excellence (CCoE) team (co-authors of this post) wanted to build a fully serverless data lake that would be able to collect data from all of Hexagon’s cloud accounts and store it in a single location: their data lake. Today we are excited to share the story of how Hexagon, the global leader in digital reality solutions, combining sensor, software and autonomous technologies, was able to use CloudQuery to build their own serverless infrastructure data lake using AWS S3, Glue and Athena. Here at CloudQuery we’ve been working on an open source framework to enable companies to build their own infrastructure data platforms – a single source of truth for all of your infrastructure data, stored in a database or data lake.

Hexagon data