Working on a data warehousing project to integrate various source streams using both batch and streaming processes on Databricks with Apache Spark, storing data in ADLS as Parquet for faster retrieval. This project aims to eliminate multiple database and ETL hops, reducing costs by replacing Azure Streaming Analytics with Spark and Databricks for improved cost and time efficiency.

Designed and optimized ETL pipelines using ADF, ADLS Gen2, Databricks, and Azure SQL/Cosmos DB, while creating Databricks pipelines and Spark notebooks for batch and streaming data from IoT Hub.

Developed real-time data streams with Azure Streaming Analytics, implemented CI/CD pipelines for ADF using GitHub, and established robust alert and monitoring systems in ADF and Databricks to ensure reliable data workflows and streamlined deployments.

Led the development of an in-house data ingestion product with Python, Flask, DuckDB, Postgres, Grafana for dynamic visualization, and Prefect for ETL workflow management. Moreover, successfully containerized the entire application using Docker for enhanced scalability and manageability in the data engineering workflow.

Orchestrated end-to-end ETL pipelines, incorporating audit logging and data integrity checks along with failure/data discrepancy alerts.

Pioneered bug tracking, resolution, and new feature development in the data model.

Developed ETL projects with SQL, PySpark, Scala, ADF, ADLS, and Azure Logic Apps, seamlessly extracting data from ERP systems and loading into the data models.

Implemented streaming data pipelines utilizing Azure Streaming Analytics to process real-time data and maintain data integrity. Implemented a robust monitoring framework using PySpark, Postgres, and Grafana to ensure data correctness. Executed ETL projects with SQL, PySpark

Crafted impactful data visualization reports, providing insights into key business metrics.

Built and managed a DWH architecture, and wrote automated ETL scripts using HiveQL, HDFS, HBase, Python, and Shell on a cloud platform for data ingestions.

Developed BI dashboards on Power BI, vShow, and FBI to gauge important metrics related to domains like customer funnel, marketing, and logistics.

Developed and maintained an enterprise data warehouse and monitored data ingestion pipelines on a daily basis using SQL, Python, Flink, ODPS, and ETL flows.

Designed and developed end-to-end data ingestion pipelines to ensure data flow daily.

Implemented and managed data flow jobs for data modeling solutions relevant to the health and life science industry, using tools like SQL Server Integration Services (SSIS) and Microsoft SQL Server.

Developed SQL queries, stored procedures, and dynamic SQL and optimized existing complex SQL queries to speed up day-to-day processes.

Designed and developed a database architecture and data model for a business flow using Talend Open Studio, SSIS, and MySQL Workbench.

Performed large-scale data conversions, migrations, and optimization to reduce resource and time costs while maintaining data integrity.

Wrote SQL stored procedures and Python scripts for data quality checks and ad-hoc analyses.