Data lake modernisation on AWS — 96% less data scanned, 74% faster queries for claims and member analytics.
96% reduction in data scanned per query, 74% faster query execution, and 55% lower monthly storage costs — with POPIA-aligned access controls on sensitive claims and member data.
Challenge
Afrocentric's claims and member-data reporting ran against raw tables, with every query scanning far more data than the question actually required. As claims volumes grew, this drove up both query latency and pay-per-scan query costs, while the underlying storage layer carried more redundant data than necessary.
Afrocentric needed a modern data lake architecture that could cut cost and query time at scale, without compromising the fine-grained access control and data protection that sensitive medical-scheme claims and member data demand under POPIA.
Solution
CloudZA designed and implemented a data lake on Apache Iceberg. AWS Database Migration Service moves source data into Amazon S3, where an AWS Glue PySpark job deduplicates incremental records before Iceberg MERGE statements combine them into partitioned target tables — replacing raw, unpartitioned storage with a compacted, query-optimised layer.
AWS Glue Data Catalog, with an integrated crawler, provides a centralised schema catalogue, and AWS Lake Formation layers on fine-grained table- and column-level access control. Amazon Athena queries the Iceberg tables directly with standard SQL, and Amazon QuickSight sits on top for interactive, drill-down dashboards — including ML-backed auto-narrative summaries — covering claims volume trends, processing-rate bottlenecks, and geographic claims analysis.
Result
Benchmarked against the same query on the raw table, the Iceberg-partitioned table cut data scanned from 571.06 MB to 22.65 MB — a 96.03% reduction — and query run time from 18.024 seconds to 4.743 seconds, 73.68% faster.
Storage requirements fell roughly 40% by moving base tables to Parquet with Snappy compression, driving a 55% reduction in monthly storage costs. Because Athena is billed per data scanned, the same drop in scanned volume lowers query costs directly — and partition-level, POPIA-aligned access control in Lake Formation lets Afrocentric extend self-service QuickSight analytics to more teams without loosening data governance.