Manager Data Engineering (Big Data & Databricks)

Norfolk Southern•Atlanta, GA

34d•Hybrid

About The Position

The Manager Data Engineering will lead a team responsible for building various cutting-edge Big Data solutions and Big Data pipelines in collaboration with the Data Science, Business Intelligence (BI), software development, engineering, and business teams. They will be focused on driving part of the strategic horizontal Big Data initiatives that advance the skills and capabilities of the organization, while improving the level of enablement for partners and stakeholders. The candidate will participate in all phases of the Data Engineering life cycle and will independently and collaboratively write project requirements, architect solutions, perform data ingestion development, support duties, coach and mentor Data Engineers, and set the data vision for the organization.

Requirements

Experience as a People Leader, directly managing technical teams, including hiring, developing, motivating, and directing people as they work.
5 years of experience in Big Data, BI, and Advance Analytics.
Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python.
Advanced knowledge of Big Data, Data Engineering, and BI with hands-on and technical experience in tools/technologies like Apache Kafka, Spark, Hadoop, Spotfire, Tableau, PowerBI, etc.
5 years of experience in building data ingestion pipelines using: python/PySpark and/or Spark-Streaming, etc.
Experience with relational databases e.g. Teradata, Oracle, SQL Server, as well as NoSQL databases including HBase and Cassandra.
Bachelor's Degree, preferably in Information Systems, Computer Science, Computer Information Systems or related field.

Nice To Haves

Experience with data engineering productivity toolchains and concepts (CI/CD, DevOps, GitHub, Jira, Confluence, Agile Scrum / Kanban, etc.)
Hands-on coding experience in at least one modern programming language: Python, Scala, etc. Strong SQL and Python skills, including knowledge of Python libraries/frameworks.
Knowledge of Unix/Linux platform and shell scripting.
Strong project management skills, including work prioritization, planning, and task delegation.
Ability to effectively communicate (both written and verbally) technical information at all levels of the organization.
Experience in successfully building and deploying a new data platform on Azure/ AWS.
Experience in Azure / AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue.
Strong knowledge of Messaging Platforms like Confluent Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.
Lake with Spark SQL, Delta Live Tables, Unity Catalog.
Experience with Data governance tools like Collibra, etc.

Responsibilities

Develop the vision and manage the execution of the Data Engineering roadmap by collaborating with cross-functional teams, products, analytics, engineering, and machine learning experts.
Oversee various parts of the data lifecycle from acquirement to insights, in particular: data ingestion, data warehouse, Data Lakehouse, and data engineering.
Hire, coach, and mentor talented Data Engineers.
Lead Data Engineering team to use state-of-the-art Big Data tools and technologies to build scalable data architecture and performant data pipelines to acquire high-velocity real-time and batch data of different sizes and scales.
Design, implement, and support scalable data infrastructure solutions to integrate with multi-heterogeneous data sources, aggregate and retrieve Big Data in a fast and safe mode, curate data that can be used in BI reporting, analysis, machine learning models and ad-hoc data requests.
Ensure appropriate levels of governance and testing procedures for developed solutions.
Stay up to date with technology, trends, and tools in Big Data.
Take part in development of data pipelines and solutions.
Optimize and tune data ingestion and data retrieval pipelines.
Implement the solutions in production environment and implement efficient and timely support models.
Manage multiple projects concurrently.