Responsible for the architecture of the data platform and development of different services to process data from Kafka using Scala and Java.

I’m responsible for designing and implementing Spark structured streaming jobs in Scala to ingest data from Kafka into S3 using Delta lake as storage layer. I’m also responsible for development of Spark batch jobs in Scala to generate other Delta tables using different file types as a source. Currently, I use Databricks to run Spark workloads, but in the past I also used EMR.

I’m responsible for designing and implementing Flink streaming jobs in Scala to process and correlate data from different Kafka topics. Runtime: EMR

I’m also responsible for designing and implementing micro-services using Java to consume data from Kafka and store them on MongoDB. Most of services are developed using SpringBoot and run on top of EKS.

Responsible for the architecture of the data platform following security standards like PCI-DSS. I’ve been working to define the stack and infrastructure required to run the data platform from scratch.

I'm also responsible for designing and implementing core components and tools used by other data engineering teams to ingest, process and deliver data.

I’m currently working on a project to build a data lake and a couple of data pipelines for one of the biggest e-commerce companies in India. The first delivery reduced the update time for one of their largest catalogs from over 15 hours to approximately 2 hours.

I also have been creating, reviewing and solving assessments for clients like Glider.ai and Codility UK on subjects like Spark, Databricks, Java, Scala, Chef, Elasticsearch, Azure and AWS.

I also work for other clients in data engineering and software engineering needs, most of them using Scala, Java, Spark and Flink on top of different cloud providers.

In the past I also worked in another long-term freelancer contract for Viewdeck Consulting where I was responsible for infrastructure automation using Chef on top of Azure, AWS and UKCloud.

Responsible for designing and implementing solutions to process data using Spark with Scala on top of Hadoop. and previously using MapReduce with pure Java and also Apache Crunch, Apache Pig and RHadoop.

I was also responsible for designing and implementing APIs and libraries to work with Hadoop (HDFS) and HBase, as well as creating pipelines to index data from HBase into Apache Solr in NRT.

I also worked with our infrastructure team to automate the entire Hadoop infrastructure using Chef, as well as to build a logging solution for the entire company using the ELK stack.

From 2009 to 2012 I worked developing and tuning web/EE and standalone applications using Java platform with different technologies and frameworks like EJB and Spring. During this time I also worked on solutions to provide digital certification and cryptography capabilities using physical HSM as underlying storage and OpenPGP standards through JCA/JCE.