
Sami Ullah
Senior Data Engineer Airflow, Spark, AWS
Compétences

Voir mes services

Expérience professionnelle
Senior Data Engineer
TRADERWARE • Freelance
Dec 2024 - Present • 1 yr 11 mos
Own end-to-end data feeding and ingestion for a social trading network and an AI-powered trading analytics platform Build and maintain financial market data pipelines across equities, forex, crypto and commodities using Airflow, Kafka, PyFlink, Spark and Python Converted a batch pipeline to real time with Kafka and PyFlink, with Redis topic routing and TimescaleDB sinks Migrated PostgreSQL infrastructure to AWS RDS and pricing data pipelines to QuestDB Enterprise Migrated about 25 PostgreSQL tables between market data providers and migrated Airflow (CeleryExecutor) to AWS EC2 with Docker Compose Built a real-time pipeline monitoring dashboard with Prometheus and Grafana, and maintained 99% uptime across critical pipelines Stack: Python, Airflow, Kafka, PyFlink, Spark, PostgreSQL, MongoDB, Redis, TimescaleDB, AWS (S3, RDS, EC2, Lambda, Secrets Manager), Docker
Data Engineer
Ehsaan Technologies • Temps plein
Aug 2022 - Dec 2024 • 2 yrs 4 mos
- Built a real-time stock data pipeline using Kafka, PyFlink and QuestDB - Developed Airflow pipelines ingesting market data from the FMP API, including a PostgreSQL-to-MongoDB migration - Built Python and Playwright web scrapers for financial data websites - Worked with PostgreSQL, MongoDB and Docker in production environments