Asindu H

Lead Data Engineer

Dec 2023 - Aug 2025 • 1 yr 8 mos

MILLENNIUM IT ESP – Azure Data Lake, Hemas Holdings Designed and implemented a centralized Azure-based lakehouse supporting seven Strategic Business Units, integrating data across Sales, Supply Chain, Finance, and Marketing to enable unified analytics and reporting. Partnered with BI stakeholders to gather requirements, define KPIs, and align dashboards with business objectives. Built and maintained a scalable Delta Lakehouse architecture using Azure Synapse serverless Spark pools, Synapse Notebooks, and Azure Data Factory pipelines. Extracted data from SAP, SQL Server, and external sources, transforming it with PySpark and Spark SQL to ensure efficient, high-performance ingestion. Developed expertise in SAP Finance and MM tables (BSIS, BSEG, ACDOCA, MKPF, MSEG, BSIK, BKPF) and created downstream datasets powering key dashboards such as GR/IR reporting, cash advances, stock coverage, service levels, production, inventory, and sales analytics. Implemented SCD Type-2 for product dimensions to track pricing changes and built a unified product model supporting both primary and secondary sales reporting. Improved deployment efficiency by parameterizing notebooks, pipelines, and linked services, enabling CI/CD through Azure DevOps with reusable, environment-agnostic configurations. Ensured secure credential management using Azure Key Vault across environments. Optimized Power BI performance by introducing daily snapshot tables for GR/IR reporting, improving load times by over 30% and enabling efficient trend analysis. Built logic to track open and closed financial items and maintain accurate reporting cutoffs. Provided operational support through workflow monitoring, troubleshooting, and journal reconciliations, ensuring data accuracy and compliance. Collaborated continuously with stakeholders to validate transformations and adapt solutions to evolving business needs.

Data Engineer

Jan 2025 - May 2025 • 4 mos

Azure Databricks ETL & Lakehouse Project Designed and implemented an end-to-end ETL pipeline using Azure Databricks (serverless) and PySpark, processing data from SAP systems and file-based sources into a scalable lakehouse architecture. Built event-driven data ingestion pipelines by orchestrating workflows with Azure Data Factory and leveraging Databricks Auto Loader for incremental file detection and near real-time processing. Developed a robust Medallion Architecture (Bronze, Silver, Gold) using Delta Lake, ensuring structured data transformation, enrichment, and high-quality datasets for analytics. Stored and managed data in Azure Data Lake Storage Gen2 (ADLS Gen2) with optimized Parquet/Delta formats for performance and cost efficiency. Implemented advanced data quality frameworks, including validation checks, deduplication logic, reconciliation reporting, and monitoring for invalid or missing data. Enhanced pipeline reliability by configuring retry mechanisms, timeout controls, and failure handling to manage data delays and transient issues. Parameterized Databricks notebooks and pipelines to enable reusable, scalable deployments and efficient historical backfills across multiple environments. Applied Delta Lake optimization techniques such as OPTIMIZE, Z-Ordering, and VACUUM to improve query performance and storage management. Established Unity Catalog for centralized data governance, fine-grained access control, and end-to-end data lineage across the lakehouse. Designed CI/CD workflows for automated deployment and version control of Databricks assets across development and production environments. Integrated alerting and notification systems to inform stakeholders of pipeline status, data quality issues, and successful BI refreshes. Tools & Technologies: Azure Databricks, PySpark, Delta Lake, Auto Loader, Unity Catalog, Azure Data Factory, ADLS Gen2, SAP, Key Vault, CI/CD

Data Engineer

Apr 2023 - Dec 2023 • 8 mos

Lanka Tiles Data Warehouse Project Developed and maintained scalable ETL pipelines using SSIS and XtractIS to extract data from SAP systems and load it into a centralized SQL Server data warehouse. Applied robust data validation, cleansing, and filtering techniques to ensure high data accuracy, consistency, and reliability across all ingestion processes. Designed complex ETL workflows using SSIS components such as Derived Columns, Lookups, Merge Join, Sort, Script Tasks, and Slowly Changing Dimensions (SCD Type 2) to support historical tracking. Deployed and managed ETL packages in SQL Server, automating execution through SQL Server Agent for reliable and scheduled data processing. Optimized a high-volume Product Dimension pipeline by implementing efficient joins, early-stage filtering, and duplicate elimination, while removing unnecessary SAP tables—reducing execution time to ~30 minutes without compromising SCD Type 2 logic. Ensured maintainability and performance by following ETL best practices, modular design, and performance tuning techniques. Tools & Technologies: SSIS, SQL Server, SAP, XtractIS