Wassim Mraihi

Context :

• Definition of needs with the Data Analysts.

• Development of a Framework of Collection and Ingestion of the Datalake on Azure with Azure Data Factory and Spark Scala on Databricks

• Unity Catalog Roles Management

• Building of ADF (Azure Data Factory) pipelines

• Optimization of Pipeline execution time

• Data modeling in the Datalake in delta and parquet format

• Development of Databricks notebooks (with Scala, Python, and SQL) in order to perform transformations and calculations of indicators

• Development of unit tests.

• Branches managing, Pull Requests, and build pipelines on Azure Devops

• Documentation on Confluence: writing of Release Delivery Notes, DATe, DEX, and Monitoring

• Using Jira for ticket management.

• Working in Agile mode: Sprint planning, Demo, Daily

Technical Environment :

• Databricks, Azure Data Factory, Unity Catalog, Spark Scala, Spark Python, SQL, EventHub, Stream Analytics, Azure Devops, Jira, Confluence. Realizations :

• Creation of ADF pipelines

• Development in Spark Scala, Python, SQL, Unity Catalog

• Job Streaming Development: Stream Analytics, EventHub

• Creation and deployment of releases and pipelines

• Documentation on Jira, Confluence DATA ENGINEER - [email protected] - +33781000238

Context : Collect of credit data for the calculation of credit risk on behalf of the Banque de France and the Central Bank of Europe.

Realizations :

• Collect, Loading, Cleaning, Transformation and Storage of Data on Datalake

• Data Export and Data Import with Sqoop.

• Development of Spark Job with Java (Maven)

• Storing of data in Hive and PostgreSQL

• Development of Shell scripts to create and monitor Oozie Workflows

• Development of REST API with Spring Boot

• Continuous Integration with Jenkins Technical Environments : Hortonworks, Hadoop, Spark, Java, Maven, Service Web Rest, Maven, Hive, Sqoop, Oozie, Shell, Eclipse, Postgres DATA ENGINEER - [email protected] - +33781000238

Context :

• Preparation of the infrastructure on Azure, and developement with Spark Scala on Databricks. Technical Environment :

• Azure, Azure Devops, Databricks, Azure Portal, Spark Scala (Sbt), Citrix, IntellijIdea, LogAnalytics, Cloudera, Hadoop Power BI, Github Realizations : Infrastructure :

• Creation of Data Teams and Data Sources with AzureDevops and Citrix (Power Shell)

• Creation of Lab and UseCases BI on AzureDevops and Citrix

• Creation of Landing Zone and TakeOffZone

• Role management Development :

• Processing of AXA employees data with Spark Scala on Cloudera and migration of the project to Azure

• Development of Notesbooks on Databricks, and deployment of Spark jars with Azure Devops

• Creation of releases, pipelines

• Creation of cubes on Power BI

• Log monitoring on LogAnalytics on Azure Portal

• Monitoring of projects in Production Environment