Manager, Data Engineering

The J.M. Smucker Co.•Orrville, OH

10d•Hybrid

About The Position

The Manager, Data Engineering is responsible for leading a team of Data Engineers focused on building and operating high-quality data pipelines within a modern cloud data platform. This role combines people leadership, hands-on technical oversight, and operational excellence to deliver trusted, scalable, and efficient data solutions. As part of the Enterprise Data Platforms & Enablement team, this role partners closely with Cloud Engineering, Governance, Architecture, and business stakeholders to support a large-scale transformation from on-premise systems to cloud-native platforms. The Manager ensures that data engineering practices align with modern best practices, including Databricks-based development, medallion architecture (bronze/silver/gold layers), and automated data ingestion frameworks (e.g., Fivetran).

Requirements

Bachelor’s Degree or equivalent experience
7+ years of experience in data engineering or data platform roles
3+ years of experience leading and developing technical teams
Experience operating in modern cloud data environments (Databricks, data lakes, or lakehouse platforms)
Proven experience delivering enterprise-scale data pipelines and platforms
Experience supporting cloud migration or modernization initiatives
Strong expertise in Databricks (notebooks, jobs, cluster management)
Proficiency in Python and PySpark for distributed data processing
Advanced SQL skills and experience working with large-scale datasets
Experience designing and operating cloud-based data platforms (AWS preferred)
Experience with data ingestion tools (e.g., Fivetran or similar)
Deep understanding of data modeling and medallion architecture patterns

Nice To Haves

AWS services such as S3, IAM, and data storage architectures
Data governance tools such as Atlan or similar
Reporting and visualization tools such as Tableau
CI/CD tools and automation frameworks
Monitoring and observability tools for data platforms

Responsibilities

Lead, coach, and develop a team of Data Engineers, fostering strong technical skills and ownership
Set clear priorities, manage workload, and ensure timely delivery of data engineering initiatives
Establish engineering standards, code quality expectations, and best practices across the team
Partner with stakeholders to translate business needs into scalable data solutions
Oversee the design, development, and operation of data pipelines built in Databricks
Ensure pipelines are scalable, reliable, and aligned to medallion architecture standards (bronze, silver, gold)
Guide implementation of ingestion frameworks using tools like Fivetran and custom ingestion patterns
Drive consistent development patterns for batch and near real-time data pipelines
Provide technical leadership for the Databricks platform, including workspace, jobs, clusters, and performance optimization
Collaborate with Cloud Engineering to ensure seamless integration with AWS services (e.g., S3)
Ensure alignment with enterprise architecture, including data modeling, partitioning strategies, and storage optimization
Optimize compute usage for performance and cost efficiency
Establish and enforce data quality standards, including testing, validation, and monitoring frameworks
Ensure robust observability across pipelines (monitoring, alerting, lineage visibility)
Partner with Governance teams to support metadata management and lineage through tools like Atlan
Enforce security, compliance, and data access standards across all data engineering assets
Own production support processes, including incident management, root cause analysis, and prevention
Establish proactive monitoring and health checks for pipelines and platform performance
Drive continuous improvement in automation, CI/CD, and release management practices
Support and lead data migration efforts from legacy on-prem systems to cloud platforms