Senior Data Engineer

Intercontinental Exchange Holdings, Inc. • Atlanta, KS

3d • Onsite

About The Position

The Senior Data Engineer will join the Enterprise Architecture team in a cross-cutting role, helping define and implement the next-generation data platform. The role leads the design and implementation of scalable, self-service data pipelines, with a strong focus on data quality and governance. It involves working with key stakeholders to build mission-critical ML and AI data workflows and shaping the organization’s data engineering practice. The position emphasizes maintainable, scalable systems that enable self-service data access while preserving high standards for data quality and governance.

The ideal candidate is a problem-solver who enjoys complex data systems and cares about data quality. They can collaborate or work independently, communicate technical concepts to technical and non-technical stakeholders, and build systems that help others work with data efficiently and confidently.

Requirements

• At least 5 years of professional experience in data engineering, including at least 2 years working on enterprise-scale data platforms.
• Bachelor's degree in computer science or equivalent.
• Deep expertise in workflow orchestration, performance optimization, and operational management.
• Strong understanding of data transformation techniques, including testing frameworks and deployment strategies.
• Experience with stream-processing frameworks and technologies.
• Proficiency with SQL and Python for data transformation and pipeline development.
• Familiarity with containerized application deployment.
• Experience implementing data quality frameworks and automated testing for data pipelines.
• Ability to work cross-functionally with data scientists, ML engineers, and business stakeholders.

Nice To Haves

• Experience with self-hosted data orchestration platforms rather than managed services.
• Background implementing data contracts or schema governance.
• Knowledge of ML/AI data pipeline requirements and feature engineering.
• Experience with real-time data processing and streaming architectures.
• Familiarity with data modeling and warehouse design principles.
• Prior experience in a technical leadership role.

Responsibilities

• Design, build, and maintain an on-premises data orchestration platform using open-source tools.
• Create self-service capabilities that allow teams across the organization to build and deploy data pipelines without extensive engineering support.
• Implement data quality testing frameworks to ensure data integrity throughout the data lifecycle.
• Establish data engineering best practices, including version control, CI/CD for data pipelines, and automated testing.
• Collaborate with ML/AI teams to build scalable feature engineering pipelines for batch and real-time data processing.
• Develop reusable patterns for common data integration scenarios across the organization.
• Work with infrastructure teams to optimize the Kubernetes-based data platform for performance and reliability.
• Mentor junior engineers and advocate for engineering excellence in data practices.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior