Data Engineer

Agama Solutions • Charlotte, NC

5d • Hybrid

About The Position

The Data Engineer will join the Information Security organization and be responsible for data ingestion, transformation, provisioning, and consumption. This role involves building and maintaining scalable and efficient data pipelines to support data processing and analytics, developing and managing ETL processes to integrate data from various sources into a central repository, and using Apache Airflow to create and manage complex workflows and ensure timely execution of data jobs. The engineer will optimize data processing workflows and database queries for performance and scalability, monitor data pipelines and systems to ensure reliability, data quality, and performance, and create and maintain comprehensive documentation for data processes, workflows, and systems. Strong communication and collaboration skills are essential for working closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.

Requirements

• 5+ years of Data Engineering experience
• Hands-on experience with Apache Airflow for workflow orchestration
• Proficiency in Hadoop ecosystem (HDFS, Hive, MapReduce, etc.)
• Advanced skills in Python for data processing and ETL tasks
• Strong experience with PySpark for large-scale data processing
• Deep understanding of relational databases (e.g., MySQL, PostgreSQL) and SQL
• Experience with Git and GitHub for version control and collaboration

Nice To Haves

• Experience working in a GCP environment

Responsibilities

• Build and maintain scalable and efficient data pipelines to support data processing and analytics.
• Develop and manage ETL processes to integrate data from various sources into a central repository.
• Use Apache Airflow to create and manage complex workflows and ensure timely execution of data jobs.
• Optimize data processing workflows and database queries for performance and scalability.
• Monitor data pipelines and systems to ensure reliability, data quality, and performance.
• Create and maintain comprehensive documentation for data processes, workflows, and systems.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.

What This Job Offers

• Career Level: Mid Level
• Education Level: No Education Listed