Senior Data Engineer (Databricks or equivalent)

Metropolitan Council•390 Robert St. N St. Paul, MN

3d•Hybrid

About The Position

This position is eligible for a hybrid telework arrangement (both remote and onsite). Candidate's permanent residence must be in Minnesota or Wisconsin. We are the Metropolitan Council, the regional government for the seven-county Twin Cities metropolitan area. We plan 30 years ahead for the metropolitan area and provide regional transportation, wastewater, and housing services. Information Services is the central IT department supporting all divisions of the Metropolitan Council. Our 140 team members provide technology, practices, and innovative solutions that enable the Council's core services. The Senior Data Engineer designs, develops, and maintains data solutions. This includes databases, data platforms, data lakes, data warehouses, and data marts, as well as data movement (data pipelines/ETL/ELT – extracting, transforming, and loading data). They collaborate with technology and business units to create solutions that make data available to systems and/or data consumers, such as developers, data scientists, business analysts, or line-of-business users. They may also integrate data, including analytics and data science outputs, into existing business processes or systems. The Senior Data Engineer ensures the security, performance, and availability of these data solutions. They administer data tools and identify and resolve data quality issues. The ideal candidate for the position has a high degree of expertise in data platforms and database technologies, data modeling, structured and unstructured data, and strong programming skills. They have strong communication skills and experience working on small to large, highly complex initiatives. The ideal candidate has a proven track record of developing successful data strategies and is a strong teammate who can listen, lead, and mentor others effectively.

Requirements

Master's degree and three (3) years of experience.
Bachelor's degree and five (5) years of experience.
Associate's degree and seven (7) years of experience.
High school diploma/GED and nine (9) years of experience.
Experience developing data solutions for the cloud
Modern cloud data platform experience in tools such as Databricks, Snowflake, or an equivalent.
Proficiency in Python, Scala, or other equivalent object-oriented programming languages
Gathering, understanding, and analyzing stakeholder requirements, needs, and objectives.
Data development methodologies, best practices, standards, and procedures.
Experience writing and tuning complex SQL queries and defining SQL best practices.
Installation and configuration of database software and drivers on servers and clients.
Data modeling, database normalization, and denormalization techniques.
Designing, developing, and maintaining data solutions for operations, reporting, and analytics (databases, data platforms, data lakes, data warehouses, and data marts).
Designing, developing, and maintaining data integrations (data pipelines/ETL/ELT – extracting, transforming, and loading data) using various types of data sources and targets.
ITIL, Incident, Service, and Change Management, including SLAs and metrics.

Nice To Haves

Experience with APIs, Web Services, and Integration Platform as a Service.
Data streaming and messaging experience.
Reporting and data visualization experience.
Experience leading a technical team.
Experience working and collaborating with individuals from diverse backgrounds.

Responsibilities

Collaborate with technology and business stakeholders to determine data needs.
Develop data solutions using various software development methodologies, including waterfall and agile.
Write and understand complex SQL queries and define best practices.
Develop and support ELT/ETL solutions utilizing Python, Scala, or other object-oriented programming languages
Design, develop, and maintain data integrations between source and target systems.
Understand and implement data security and data classifications.
Innovate and leverage emerging technologies and participate in strategic planning.
Set up monitoring and alerting for data solutions.
Script, automate, configure, and monitor data processes.
Troubleshoot database, data integration, or application issues related to data.
Build and maintain modern cloud data platforms for reporting and analytics.
Partner with business intelligence and analytics developers on metric development, reporting, and visualizations.
Work with a variety of internal and external data sources, including APIs, Web Services, and integration platform-as-a-service.
Work with data streaming and messaging services to move data between systems.
Provide technical leadership on projects and initiatives
Mentor peers and partners across the organization