Sr. Data Engineer

Cargill • Atlanta, GA

2d

About The Position

The Senior Data Engineer designs, builds, and maintains complex data systems that enable data analysis and reporting. With minimal supervision, the role ensures large datasets are processed efficiently and made accessible for decision-making. The position develops scalable, sustainable, and robust data products and solutions using advanced engineering and cloud-based technologies, and builds systems that support data processing and advanced analytics.

Requirements

• At least 4 years of relevant work experience; the role typically reflects 5 or more years of relevant experience.

Nice To Haves

• Experience developing data systems on major cloud platforms such as AWS, GCP, or Azure.
• Hands-on experience building modern data architectures, including data lakes, data lakehouses, and data hubs, and related ingestion, governance, modeling, and observability capabilities.
• Proficiency with data collection and ingestion tools such as Kafka and AWS Glue, and storage formats such as Iceberg and Parquet.
• Experience developing streaming data pipelines and using streaming tools such as Kafka and Flink.
• Expertise in data transformation and modeling using SQL-based frameworks and orchestration tools such as dbt, AWS Glue, and Airflow; deep experience with SCD and schema evolution.
• Strong background using Spark for data transformation, including streaming, performance tuning, and debugging with Spark UI.
• Advanced programming skills in Python, Java, Scala, or similar languages, and expert-level SQL skills for data manipulation and optimization.
• Experience with DevOps practices, including code management, CI/CD, and deployment strategies.
• Strong background in data governance principles, including data quality, privacy, and security for data product development and consumption.
• Previous Cargill experience or experience with Animal Health Nutrition.

Responsibilities

• Prepare data infrastructure to support efficient storage and retrieval of data.
• Examine and resolve appropriate data formats to improve data usability and accessibility across the organization.
• Develop complex data products and solutions using advanced engineering and cloud-based technologies, ensuring they are scalable, sustainable, and robust.
• Develop and maintain streaming and batch data pipelines to ingest data from various sources, transform it into information, and move it to data stores such as data lakes and data warehouses.
• Review existing data systems and architectures to identify opportunities for improvement and optimization.
• Collaborate with multifunctional data and advanced analytics teams to gather requirements and ensure solutions meet partners’ functional and non-functional needs.
• Build complex prototypes to test new concepts and implement data engineering frameworks and architectures that improve data processing and support advanced analytics initiatives.
• Develop automated deployment pipelines that improve code deployment efficiency with fit-for-purpose governance.
• Perform complex data modeling in accordance with datastore technology to support sustainable performance and accessibility.

Benefits

• The position is eligible for a discretionary incentive award dependent on company and personal performance.
• Minnesota Sick and Safe Leave accruals of one hour for every 30 hours worked, up to 48 hours per calendar year unless otherwise provided by law.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed