Sr Data Engineer

McGraw Hill LLC.•UNAVAILABLE, UNAVAILABLE

2d•Remote

About The Position

The Senior Data Engineer will advance McGraw Hill Education’s product eventing, data platforms, and analytics capabilities by designing scalable, reliable, high-quality data solutions that turn learning interactions and product events into actionable learning insights. Embedded within product platform teams, the engineer will collaborate with product managers, software engineers, and data teams to define meaningful events, build end-to-end event pipelines, and maintain data quality from event creation through insight delivery. The role centers on AWS, Databricks, and modern event-driven data architectures, including real-time or near-real-time pipelines, Delta Lake, and medallion (Bronze/Silver/Gold) patterns. The engineer will develop distributed ETL/ELT pipelines, data models, orchestration, governance, and quality practices, and contribute throughout the software development lifecycle. The position is remote and open to applicants authorized to work for any employer within the United States.

Requirements

At least 5 years of Data Engineering experience.
Deep expertise in event-driven data and lakehouse architecture, including product event design, event schemas, real-time or near-real-time processing, Delta Lake, medallion patterns, Unity Catalog governance, and Databricks cloud-native data solutions.
Hands-on experience with Databricks, including Delta Lake, Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, Databricks SQL, and MLflow.
Experience designing and implementing event-driven architectures, schemas, ingestion, streaming pipelines, and real-time or near-real-time processing using Apache Kafka, cloud-native streaming services, or equivalent platforms.
Experience with AWS services including S3, Redshift, Glue, Lambda, EMR, Athena (with Iceberg), Step Functions, and IAM, integrated with Databricks where applicable.
Proficiency in Python (PySpark), Scala (Spark), or SQL for pipeline development, event processing, and data transformation.
At least 3 years working with cloud platforms, primarily AWS, and architecting and operating Databricks environments, including workspace configuration, cluster policies, instance profiles, security, and cost optimization.
At least 1 year of experience with workflow automation and pipeline orchestration using Databricks Workflows, Apache Airflow with the Databricks provider, or equivalent tools, including dependency management, retries, monitoring, and alerting.
Experience collaborating with product platform teams to define event requirements and schemas, implement event instrumentation, and build reliable pipelines connecting learning interactions to curated data products and insights.
Strong understanding of end-to-end event data quality, including schema validation, event completeness, data consistency, observability, and monitoring across event generation, ingestion, processing, and consumption.
Experience optimizing high-volume event processing and distributed data pipelines using Spark, Delta Lake, and cloud-native technologies.

Nice To Haves

Experience in the EdTech domain.

Responsibilities

Collaborate with product platform teams to define learning events, establish event standards, and build scalable pipelines connecting learning interactions to actionable insights.
Design, develop, and optimize distributed ETL/ELT pipelines using Apache Spark (PySpark/Scala) and Databricks.
Build and maintain real-time and near-real-time event-driven pipelines for product and learning interactions.
Establish end-to-end data quality across schemas, ingestion, transformations, and curated data products; implement validation, monitoring, and alerting.
Develop and maintain Databricks Workflows, including dependencies, retries, scheduling, monitoring, and alerting.
Optimize pipeline performance using partitioning, caching, broadcast joins, and Delta Lake capabilities.
Design scalable Delta table data models, including fact and dimension tables, star and snowflake schemas, and aggregations.
Apply data architecture and governance principles using Unity Catalog, Delta Sharing, and AWS-native lakehouse patterns.
Use Git-based version control and project management tools such as Jira within Agile/Kanban delivery frameworks.
Translate product and data requirements into technical designs and production-grade solutions across requirements gathering, architecture, implementation, testing, deployment, and ongoing support.
Create solution design documentation, including event architecture, data flow diagrams, pipeline specifications, data mappings, and Unity Catalog data asset definitions.
Communicate with product, engineering, analytics, and business stakeholders.