v
vivekpandey600

Vivek Pandey

@vivekpandey600

Senior Data Engineer

Inde
Hindi, Anglais
Certaines informations sont présentées en anglais.
À propos de moi
I am a Senior Data Engineer with over 6 years of experience designing and optimizing end-to-end pipelines on Azure Databricks and PySpark. I specialize in Medallion Architecture, real-time streaming with Kafka, and data governance using Unity Catalog for global brands in retail and energy.... Plus d’infos

Compétences

v
vivekpandey600
Vivek Pandey
hors ligne • 

Voir mes services

ETL de données
I will be your databricks and pyspark data engineer for etl and delta lake pipelines
De75 $US / projetPlus de détails

Expérience professionnelle

Infosys

Senior Associate Consultant

Infosys • Temps plein

Feb 2025 - Present • 1 yr 8 mos

Building Databricks Lakehouse pipelines for global retail finance and consumer-service data. Designing end-to-end batch ETL pipelines using Medallion Architecture with Delta Lake and Unity Catalog. Ingesting data from 1,200+ global stores into AWS S3. Implemented CDC for incremental loads and optimized PySpark transformations, reducing batch runtime by 35%. Managing Amazon Connect call-transcript pipelines processing 200K JSON files daily.

Tata_Consultancy Services

Azure Data Engineer

Tata Consultancy Services • Temps plein

Feb 2020 - Jan 2025 • 4 yrs 11 mos

Data Engineer for TotalEnergies' digital drilling operations. Built real-time streaming pipelines using Kafka and PySpark to ingest sensor and rig performance data. Processed 1 TB/day of sensor data into Azure Data Lake and Cosmos DB. Reduced pipeline execution time by 50% through performance optimization. Developed batch pipelines with Azure Function Apps and automated job monitoring, saving 2 hours of manual effort daily.