Taming S3 Shuffle at Scale
How we fixed the GET request explosion, prefix throttling, and threading edge cases that emerge when S3 shuffle meets production scale.
spark s3-shuffle performance data-engineering scale
$ echo "building the future of ad tech"
Deep dives into architecture, data engineering, and the tools we build.
Building a Composable ETL Framework for Spark
How we replaced bespoke PySpark scripts with a config-driven, hook-based framework inspired by Rust's composition model.
Recent Posts
View all → 
Debugging a Spark Job with 🔥DataFlint
How an open-source Spark UI replacement helped us find data skew, partition bloat, and shuffle spill.
spark dataflint performance data-engineering
Spark on Spot with S3 Shuffle
How S3 shuffle lets us run Spark executors 100% on spot instances with Karpenter, cutting compute costs 70-85%.
spark kubernetes spot-instances karpenter s3-shuffle cost-optimization
Running Spark on Kubernetes: An Ops-First Approach
How we run production Spark jobs on Kubernetes with one SparkApp YAML, pre-baked images, and sub-10-second warm starts.
spark kubernetes data-engineering airflow developer-experience
From 60 Minutes to 4: Optimizing Spark MERGE INTO on a 2 Billion Row Iceberg Table
How we cut our daily upsert pipeline from an hour to under 4 minutes using storage partition joins and shuffle hash hints.
spark iceberg performance data-engineering
Building Our Lakehouse with Apache Iceberg
How we designed a modern lakehouse architecture using Iceberg, Lakekeeper, and Spark on AWS.
data-engineering iceberg lakehouse architecture