Redesigned an end-to-end pipeline for orchestrating channel schedules and their formats, replacing a legacy .NET-based system with Apache Airflow, AWS S3, AWS SQS, and AWS Glue, improving efficiency by 50%.

Worked on migrating on-premises data from SQL Server to AWS using Talend, ensuring seamless integration and better data accessibility in the cloud.

Partnered with the team to architect and document a video popularity scoring pipeline on AWS EMR, Apache Airflow, and AWS S3.

Led development on approximately 20% of project modules, driving enhancements and feature development within an Agile framework.

Providing technical guidance and mentorship to junior developers, conducting hands-on training for new team members to foster a collaborative learning environment.

Securely ingested data from on-premises SQL Server into Azure Data Lake Storage (ADLS) using Azure Data Factory.

Designed and documented the architecture for key features like Purchase Order Consolidation and an Automated Trigger Workflow, which automatically updates requisition statuses from draft to approved or directly creates orders based on predefined parameters using Azure Databricks and ADLS Gen2.

Optimized 40% of existing Databricks pipelines using Spark optimization techniques, reducing costs and improving efficiency.

Implemented a CI/CD pipeline for Databricks using GitHub Actions, enabling automated deployments and streamlined workflow management.

Contributed to UAT and production support by addressing critical and blocker bugs, providing timely fixes for issues raised by clients.

Developed modules/enhancements of smart products as per business requirements.