Data Engineer

1d1 day ago

GFT Technologies

Medellín, CO · Full-time

About this role

The Data Engineer role centers on designing and developing ETL/ELT pipelines for large volumes of financial information using Apache Spark and AWS Glue. Work takes place in AWS cloud environments where accounting data transformations and integrity validations are implemented.

Daily tasks include managing data catalogs in AWS Glue Data Catalog and optimizing queries over S3, Redshift, and RDS. Python scripts are created to automate data cleaning, validation, and processing across all pipeline stages.

Emphasis is placed on guaranteeing data quality, traceability, and lineage while collaborating with functional teams. Technical information is organized and documented to support effective data management practices.

Opportunities arise to handle Big Data processing and apply financial or accounting data expertise. Data Science practices can be incorporated into transformation and catalog workflows.

Requirements

Apache Spark for Big Data and distributed processing.
AWS Glue for ETL/ELT workflows.
AWS services including S3, Redshift and RDS.
Python for data processing and automation.
Design and implementation of data pipelines.
Data quality, traceability and lineage management.
AWS Glue Data Catalog.

Responsibilities

Design, develop and maintain ETL/ELT pipelines using Apache Spark and AWS Glue.
Process, transform and load large volumes of financial information in AWS cloud environments.
Implement transformation logic for accounting data including account mapping, conversion rules and integrity validations.
Manage data catalogs in AWS Glue Data Catalog.
Optimize queries and processing on S3, Redshift and RDS.
Ensure data quality, traceability and lineage across all pipeline stages.
Develop Python scripts for data automation, cleaning and validation.