Dhaval

Senior Data Engineer

Feb 2026 - Present • 8 mos

• Leading reliability and scalability for a distributed messaging analytics platform processing billions of daily events across WhatsApp, SMS, and RCS channels on a Delta Lake lakehouse architecture. • Architected idempotent ingestion controls for Spark archival ELT pipelines on Delta Lake — eliminating duplicate records and enabling safe reprocessing across high-volume workflows. • Engineered a job-level cost attribution system for Flink streaming and Spark batch workloads, improving infrastructure cost visibility and driving resource optimisation decisions. • Developed Python-based pipeline monitoring and automation utilities — covering schema validation, data quality alerting, and reconciliation checks across ELT workflows. • Designed dynamic Kubernetes configuration management for distributed services, enabling runtime config updates with zero service downtime.

Data Engineer

Dec 2022 - Dec 2025 • 3 yrs

• Designed and operated high-throughput stream processing pipelines using Java, Apache Flink, and Kafka — handling billions of daily events with sub-second latency and end-to-end exactly-once delivery guarantees. • Architected Delta Lake lakehouse data models following Medallion Architecture (Bronze–Silver–Gold), enabling reliable multi-hop ELT transformation from raw ingestion through curated analytics layers. • Built data ingestion and transformation workflows integrating Kafka event streams with AWS S3 and Parquet storage, enforcing schema consistency and data quality via Apache Avro and custom validation logic. • Led data migration and reconciliation from legacy analytics systems — validating multi-terabyte historical datasets and ensuring end-to-end data integrity across the new lakehouse platform. • Delivered ~20% improvement in pipeline throughput through Flink job optimisation — reducing shuffle overhead, tuning RocksDB state backends, and right-sizing parallelism while lowering infrastructure consumption. • Built Spring Boot microservices with REST APIs enabling self-service analytics query execution, eliminating ad-hoc data requests and improving analyst productivity. • Wrote Python scripts for pipeline health monitoring — automated row count checks, schema drift detection, and SLA breach alerting across batch and streaming workloads. • Production load-tested Flink and Spark pipelines at peak scale — identified bottlenecks pre-launch and hardened fault-tolerance and checkpoint configurations.

Senior Business Analyst

Nov 2021 - Dec 2022 • 1 yr 1 mo

• Analysed large-scale messaging datasets using PostgreSQL and Amazon Redshift, delivering delivery and engagement insights for enterprise clients. • Built archival pipelines and developed Tableau and Metabase dashboards enabling strategic and operational decision-making.