kartikeyhadiya

GIS Data Engineer II

HERE Solutions India • Temps plein

Nov 2023 - Present • 2 yrs 6 mos

• Data Pipeline Creation & Maintenance: Designed and maintained data pipelines to transform, validate, and load data from multiple sources using AWS services, including Lambda, SQS, SNS, and Aurora. Also implemented CICD for automated testing and deployment to AWS using Gitlab CI. The automated data pipelines saved 3M Euros for the company by reducing the manual work. • Query Optimization: Collaborated with various teams to optimize SQL queries, reducing execution times by 50% and resolving memory issues to improve performance across applications. • Data Product Management: Managed data product containing billions of data points, ensuring data quality, scalability, and accessibility for analytical processes. The data product helped us reduce load on our production database by 50% as it served as a MV for multiple dashboards. • Large-Scale GIS Data Analysis: Analyzed extensive GIS datasets using distributed computing tools such as Ray, and Apache Spark to derive actionable insights from spatial data. • AI Agent: Created and custom AI Agent for Text to SQL generation. The AI Agent features schema awareness, custom functions and stored procedures awareness. The Agent also explained how the SQL query works. It helped PM’s get self-service business statistics and save countess hours for developers and analysts. • AWS Cost Monitoring: Managed Cloudability Dashboards and reports for monitoring AWS costs.

Senior Geospatial Engineer

ESRI India Technologies

Jan 2021 - Nov 2023 • 2 yrs 10 mos

• Performance Optimization: Enhanced existing codebases using multiprocessing and concurrency, achieving speed improvements of 10-20 times. • Custom Geoprocessing Tools: Created geoprocessing tools and scripts for ArcGIS Desktop and Pro, including tools for calculating the shortest distance to multiple data stations and generating map layouts, custom mosaic algorithm using Dask and Xarray. • ArcGIS Server Administration: Managed services on ArcGIS Server using ArcGIS API for Python and ArcGIS REST API to streamline geospatial operations. • Data ETL Processes: Created ETL pipelines (APIs to PostgreSQL databases), ensuring efficient data integration. • Web Scraping: Utilized web scraping techniques for data collection and analysis, enhancing data sources for geospatial projects. • Automated Property Card Generation: Developed a comprehensive product for automated property card generation using Python libraries (Jinja2, Selenium) and shell scripts for installation and online licensing processes. • Data Synchronization: Implemented data cloning processes from ArcGIS Online to ArcGIS Enterprise using the ArcGIS Python API and ArcGIS REST API.