Data Engineer -Iceberg

American IT Systems•Atlanta, GA

3d•Onsite

About The Position

We are seeking an experienced Data Engineer with strong hands-on experience in SQL, Python, SSIS, Airflow, Apache Spark, Apache Iceberg, and Databricks. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data processing solutions in a cloud-based environment.

Requirements

Strong hands-on experience with SQL
Strong programming experience with Python
Hands-on experience with Apache Spark / PySpark
Experience with Databricks
Strong experience with Apache Airflow
Hands-on experience with SSIS
Experience with Apache Iceberg
Strong understanding of ETL/ELT and data pipeline development
Experience working with large-scale datasets and distributed data processing
Strong troubleshooting and performance-tuning skills
Good understanding of modern data lake/lakehouse architectures

Responsibilities

Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, SSIS, Airflow, and Spark.
Develop complex SQL queries, stored procedures, data transformations, and performance optimization solutions.
Build and maintain data pipelines and workflows using Apache Airflow, including scheduling, dependencies, monitoring, retries, and failure handling.
Develop and optimize Apache Spark/PySpark applications for large-scale data processing.
Work with Databricks to develop, deploy, and optimize data engineering workloads.
Work with Apache Iceberg tables and modern data lake/lakehouse architectures.
Migrate and modernize legacy SSIS ETL workflows into cloud-based and Spark/Databricks-based data pipelines.
Implement data ingestion, transformation, validation, and integration processes from multiple sources.
Optimize Spark jobs, SQL queries, data partitioning, and storage strategies for performance and scalability.
Implement data quality checks, error handling, logging, monitoring, and pipeline recovery mechanisms.
Collaborate with data architects, developers, analysts, and business teams to understand requirements and deliver reliable data solutions.
Follow best practices for data security, governance, performance, and operational reliability.