Lead Data Engineer (Databricks, Big Data)
About The Position
Norfolk Southern Corporation is seeking a Lead Data Engineer who enjoys collaborating across the organization to deliver reliable, scalable data solutions. Join a skilled team focused on modern data platforms, where you will lead the definition of architecture, help shape and operate our Databricks-based lakehouse and streaming analytics capabilities, and guide the team through technical delivery. This role blends strong business and analytical judgment with hands-on engineering and technical leadership: you will partner with stakeholders and the engineering team to understand problems, translate them into clear requirements, define and drive pragmatic architectures, and deliver production-grade ingestion and transformation pipelines - with ownership through deployment and steady-state operations. You will also work closely with the team to proactively identify and remove technical blockers that stand in the way of delivery. In this role, you will work with teams across the company to clarify Data and Analytics needs, perform structured requirements discovery alongside Data Modelers, align with BI development on scope and estimates, and lead and guide delivery through completion while promoting engineering quality, consistency, and best practices across the team.
Requirements
- Databricks Lakehouse Expertise (Required): 10+ years in data engineering with deep, hands-on experience across the Databricks ecosystem including Spark/PySpark pipelines, Delta Lake (merges, schema evolution), Delta Live Tables, Unity Catalog governance, Genie for self-service analytics, and LakeFlow Designer for visual ETL orchestration.
- Streaming & Cloud Infrastructure: 4+ years building reliable batch/streaming Spark workloads, 3+ years with Kafka (or Confluent) for high-volume event processing, and 3+ years working with AWS analytics services (S3, IAM, Glue/Lambda/MSK).
- Data Modeling & SQL: Proven ability to design conceptual, logical, and physical data models with strong governance practices, paired with advanced SQL skills for building reliable, business-ready datasets.
- Technical Leadership: Track record of leading technical design, mentoring engineers, driving architectural consensus, and unblocking teams on complex data engineering challenges.
- Delivery & Collaboration: Experience delivering ETL/ELT pipelines end-to-end in a lakehouse environment, working within Agile frameworks (Scrum/Kanban/SAFe) to manage iterative delivery and cross-team dependencies.
Nice To Haves
- Prior experience as a technical lead, staff engineer, or similar role with formal responsibility for architecture decisions and team unblocking.
- Python (and/or Scala) and PySpark/Scala-Spark.
- Database solutions like Snowflake, Delta Lake or BigQuery.
- NoSQL databases, including HBASE and/or Cassandra.
- Azure, AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue.
- Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.
- Strong understanding of Relational & Dimensional modeling.
- Experience with GIT code versioning software.
- Experience with REST API and Web Services.
Responsibilities
- Lead the definition of data architecture in partnership with the team and stakeholders, setting technical direction for the lakehouse, ingestion, and transformation patterns.
- Identify, escalate, and remove technical blockers for the team - unblocking delivery through hands-on problem solving, design decisions, or coordination with other teams.
- Lead the design and development of enterprise data models (conceptual, logical, and physical) across domains, ensuring they align with lakehouse and pipeline architecture.
- Define and enforce data modeling standards, best practices, and governance processes, leveraging tools such as Erwin.
- Define data requirements, gather, and wrangle large scale of structured and unstructured data, and validate data by running various data tools in the Data Environment.
- Support the standardization, customization and ad-hoc data analysis, and will develop the mechanisms to ingest, analyze, validate, normalize and clean data.
- Create data policy and develop interfaces and retention models which require synthesizing or anonymizing data.
- Implement statistical data quality procedures on new data sources, and by applying rigorous iterative data analytics, supports Data Scientists and analytics and insights creation in data sourcing and preparation to visualize data and synthesize insights of commercial value.
- Develops and maintains data engineering best practices and contributes to Insights on data analytics and visualization concepts, methods and techniques.
- Lead Data Engineering team to use state-of-the-art Big Data tools and technologies to build scalable data architecture and performant data pipelines to acquire high-velocity real-time and batch data of different sizes and scales
- Work closely with the data science and business intelligence teams to develop data models and pipelines for research, reporting, and machine learning.
- Build data pipelines that clean, transform, and aggregate data from disparate sources.
- Employ a variety of languages and tools (e.g. scripting languages) to marry systems together.
- Apply knowledge of Data Architecture components, leads project teams from requirements to implementation.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
