Project Type: Direct Engagement
Project Timeline: April 2024 – July 2024
Client Region: USA
Industry: IT Security

Context & Objectives

The client is a cost-conscious organization seeking a multi-tenant analytics platform similar to StackGenius. They required a solution that was lightweight, serverless-friendly, and capable of supporting governed, tenant-aware analytics with minimal operational overhead.
Their existing analytics setup could not efficiently manage schema evolution, provide tenant-level access controls, or scale cost-effectively for diverse data sources. Dashboards were slow to refresh, and governance was inconsistent, creating potential compliance risks.

The client wanted a Lakehouse solution that could handle multiple tenants securely, while keeping operations simple and cost-effective. The key goals were to:

Provide governed, tenant-aware analytics: Ensure that each user or tenant could only access their own data, while maintaining central control over permissions and policies.
Use serverless AWS services: Minimize the need for managing servers or infrastructure, reducing operational overhead and keeping costs predictable.
Support schema evolution and growth: Allow the data model to change over time such as adding new columns or tables without breaking existing dashboards or pipelines.
Deliver actionable dashboards in Metabase: Make insights easily accessible through dashboards, with filters and views that respected tenant-level access, enabling users to quickly act on data.

Business Challenge:

The client’s existing analytics platform faced several critical limitations:

• Dashboards often had high latency, and there was no strict enforcement of tenant-level isolation, meaning users could potentially access data they shouldn’t. Changes to data sources or schema updates frequently risked breaking downstream queries, creating delays and operational headaches.

• At the same time, maintaining cost discipline was challenging because the pay-per-query model could quickly become expensive without pre-aggregated views or partitioned datasets.

Overall, the operational complexity made it difficult for the team to quickly onboard new tenants or integrate additional data sources, slowing the delivery of actionable insights.

Project Goal:

The goal of this project was to design and implement a serverless Lakehouse on Apache Iceberg using AWS services that could meet the client’s needs for speed, governance, and scalability.

• The platform needed to enable tenant-aware analytics, ensuring that each tenant could only access their own data, while supporting near real-time dashboards with P95 query response times under 5 seconds.

• In addition, the solution had to allow cost-efficient, auditable ingestion and storage of data, leveraging serverless and pay-per-query services to keep operational costs predictable.

Finally, the architecture was designed to provide a clear foundation for future growth, including potential streaming pipelines and machine learning workloads, without causing downtime or complex rework.

Challenges

The project faced several technical and operational challenges that needed careful design.

• First, security and tenancy were critical: the platform had to enforce tenant-aware access control using AWS Lake Formation grants, with row-level security (RLS) and attribute-based access control (ABAC) in queries and views.

• Second, freshness and latency were important for user experience dashboards needed to be updated daily, with P95 query response times under 3–5 seconds for default filters.

• Third, reliability and scalability were key: the Lakehouse had to support multiple heterogeneous data sources, handle schema evolution gracefully, provide durable ingestion, and allow easy backfills without downtime.

Finally, cost management was a priority, as Athena’s pay-per-query model required predictable spending; the design minimized always-on compute to maintain financial control.

Solution & Implementation

We implemented a serverless Lakehouse on AWS using Apache Iceberg tables to create a multi-layered architecture (Bronze → Silver → Gold) with tenant-aware governance.

Data Ingestion & Orchestration

Data was ingested from a variety of sources, including SaaS APIs, CSV or JSON file drops, operational database extracts, and streaming logs or events. Batch ingestion and processing were handled using AWS Lambda functions and AWS Glue jobs, with schedules managed through EventBridge to automate regular data loads.
To ensure reliability and prevent data loss, dead-letter queues (DLQs) captured failed events, and retry mechanisms were implemented to guarantee idempotent processing, so each record was processed exactly once even in the event of failures..

Storage & Lakehouse Layer

Raw data was stored in Amazon S3, while Apache Iceberg tables organized the data into Bronze, Silver, and Gold layers, providing durable snapshots and efficient partition management. The AWS Glue Catalog maintained metadata for all tables, enabling schema evolution, versioning, and easy tracking of changes over time.
Policy tags were applied to enforce governance rules and control access, ensuring that data visibility and operations complied with tenant isolation and security requirements.

Transformation & Modeling

We applied SQL-based transformations on the Iceberg tables to create pre-aggregated datasets for commonly used queries, improving performance and reducing query costs. We built tenant-aware views and implemented row-level security (RLS) to ensure that each user could only access the data they were authorized for, maintaining strict tenant isolation.
Our team designed the data pipeline to flow through Bronze → Silver → Gold layers, enforcing data quality, consistency, and readiness for analytical consumption in dashboards and ad-hoc queries.

Serving & Analytics

We leveraged Amazon Athena to enable ad-hoc SQL queries directly on the Iceberg tables, allowing analysts and data engineers to interact with the data without moving it. For visual analytics, we built Metabase dashboards with embedded tenant filters, ensuring secure and isolated access for each tenant.
By combining pre-aggregated views and partition pruning, we were able to maintain low-latency dashboard performance, achieving sub-5-second P95 response times even under typical query loads..

Governance, Observability & Cost Management

Governance, security, and cost management were tightly integrated into the Lakehouse architecture. Lake Formation grants, KMS encryption, and least-privilege roles provided secure, compliant access, ensuring that each user could only interact with the data they were authorized to see.
We continuously monitored ingestion pipelines, query performance, and partition health using CloudWatch, giving full visibility into operational metrics.
At the same time, we leveraged cost dashboards in AWS Cost Explorer to monitor Athena query spend and storage usage, helping maintain predictable costs while scaling the platform efficiently.

Implementation Highlights

We implemented a Lakehouse solution that combined security, flexibility, and efficiency.

• Tenant isolation was enforced using RLS and ABAC patterns at both the table and query layers, ensuring users could only access authorized data.

• Iceberg tables supported schema evolution, allowing columns to be added or removed without impacting downstream views. By leveraging serverless services like Athena for queries and Lambda for ingestion, we reduced operational overhead and fixed costs.

• EventBridge and Glue orchestration automated batch pipelines with retry and DLQ support, simplifying operations.

• Governance was maintained through policy tags and Glue Catalog metadata, providing auditable access and controlled growth. Pre-aggregated views, partition pruning, and Athena optimizations ensured fast query performance and responsive dashboards.

Deliverables

Data Lake & Catalog Blueprint: Detailed design of the data lake, including S3 storage, and Iceberg layers, showing how data is organized and managed.

Ingestion Jobs & Scheduling: Automated pipelines using Lambda, Glue, and EventBridge, including retry logic and dead-letter queues to ensure reliable and fault-tolerant data ingestion.

Tenant Access Examples: Sample configurations for row-level security (RLS) and attribute-based access control (ABAC) in Metabase dashboards, ensuring each tenant sees only their own data.

Pre-Aggregated Views & Governance: Catalog of pre-computed views and policy tags that enforce access controls, improve query performance, and maintain data governance.

Operational Runbook: Step-by-step procedures for managing backfills, performing audits, and tracking costs to ensure smooth day-to-day operations.

Outcomes & Business Impact

The Lakehouse solution delivered measurable business impact across multiple dimensions.

• The MVP was implemented in just 6–8 weeks, providing tenant-aware dashboards that enabled executives and operational teams to access timely insights.

• By leveraging a serverless architecture, pay-per-query costs remained predictable, and minimal always-on compute reduced operational overhead. Centralized catalog management, policy tags, and tenant-aware views strengthened governance, lowered risk, and simplified compliance.

• Additionally, the architecture was designed with extensibility in mind, offering a clear path to support more advanced workloads, including streaming and machine learning.

Performance-wise, dashboards consistently achieved sub-5-second P95 response times, supporting daily to sub-daily analytics cycles and empowering users to make faster, data-driven decisions.

Tech Stack

Data Sources:

SaaS APIs, CSV/JSON drops, operational DB extracts

Ingestion & Orchestration:

AWS Lambda, EventBridge scheduling

Storage / Lakehouse:

S3 Data Lake, Apache Iceberg tables, AWS Glue Catalog

Transformation & Modeling:

SQL transformations on Iceberg Bronze/Silver/Gold layers

Query & Serving:

Amazon Athena, Metabase dashboards

Governance & Security:

AWS Lake Formation grants, KMS encryption,

Observability & Quality:

CloudWatch , reconciliation tables

DevOps & CI/CD:

versioned SQL artifacts

Cost & FinOps:

Athena pay-per-query

Target Architecture (High-Level)

Conclusion: Enabling Cost-Efficient, Governed Multi-Tenant Analytics

Through this project, we successfully built a serverless, Iceberg-backed Lakehouse on AWS that strikes a strong balance between performance, cost-efficiency, and governance. Tenant-aware dashboards and pre-aggregated views delivered lightning-fast, reliable analytics, all while enforcing strict access controls to protect sensitive data.

By harnessing the power of Athena, Lambda, Glue, and S3, we significantly reduced operational overhead and created a platform that scales effortlessly as new tenants or data sources are added.

What makes this architecture truly powerful is its future-proof design: it lays the foundation for streaming ingestion, real-time analytics, and advanced machine-learning workflows. This ensures the client can continuously expand their analytical capabilities, respond instantly to business needs, and innovate rapidly, all without disrupting existing operations or inflating costs.