Alejandro Hernan Colmenarejo

ETL using AWS services with Github repository. The purpose of the project is a pipeline which uses Glue Job in Pyspark and Python Scripts stored in Github repository. The pipeline is scalable , just needs a new configuration file with the new implementation and this is stored in AppConfig. Every data is getting from S3 and it is stored in S3 too using layers as bronze, silver and gold. In each layer the data has its own properties and transformations. Services used: AppConfig, Step Function, AWS Glue, Athena.

Use of GCP to retrieve files from AWS S3, store them in Google CloudStorage, normalize and transform the data using VertexAI, and upload the processed data to BigQuerytables. Use of AWS Glue and AWS Lambda to get data from differents sources(Redshift, Sharepoint, CSV, etc) in different formats and stored them in S3. Use of SNS and SQS to pass between buckets using Lambda Transformations. Orchestrate scripts from Glue and Lambda with StepFunctions and EventBridge events. Use of AWS Athena to data revision. Create graphics in AWS Quicksight using the data processed to createreports. Extraction data from Google Analytics using AWS AppFlow and storing them into S3 for Athena queries. Using AWS Services to fix some ETL processes in Java and Python.Transform JavaSparkwith Hive to Pyspark script. Use of AWS Glue to get tables from Saleforce and FASTstored in S3. Create graphics in Quicksight using the relation between both tables data.

Data extraction from HDFS, transform and clean the data extrated with querys using PySpark. Modeling data extracted into tables to create an automatic process for reporting