About the position
As a Senior Data Engineer on the MMA National IT Data and Analytics Team, we will count on you to develop and maintain scalable data pipelines, enhance data models, and provide subject matter expertise in data acquisition and consumption pipelines on Azure, Databricks and Confluent. It involves developing best practices, reusable code, libraries, and frameworks for cloud-based data warehousing and ETL, and uses multi-cloud, programming languages like Java, Scala, Python, RDBMS, NoSQL databases, and design enterprise data warehouse platforms. By fostering collaboration and aligning with business objectives, you will have the opportunity to elevate data models, drive data-driven decision-making, and enhance data accessibility throughout the organization.
Responsibilities
- Develop and maintain scalable data pipelines.
- Enhance data models.
- Provide subject matter expertise in data acquisition and consumption pipelines on Azure, Databricks, and Confluent.
- Develop best practices, reusable code, libraries, and frameworks for cloud-based data warehousing and ETL.
- Design enterprise data warehouse platforms.
- Foster collaboration and align with business objectives to drive data-driven decision-making.
Requirements
- 7+ years of hands-on experience designing and implementing data engineering solutions using Azure cloud offerings including Data Factory, Azure Synapse, ADLS, and Azure Functions and Logic Apps.
- In-depth understanding and experience in data engineering and data warehousing best practices, data modelling, data security and governance principles throughout data lifecycle.
- Hands-on development experience with Relational DBs (MS SQL Server, PostgreSQL, Oracle), as well as complex stored procedures and functions using SQL and optimization of SSIS packages and Redshift database queries.
- Experience designing and building data pipelines using APIs, BigQuery and Streaming ingestion methods.
- Proficiency in Spark, Python and ability to work with telemetry using Azure monitor, App insights.
- Solid understanding of DevSecOps, Git integration, Code deployment using CI/CD pipelines, managing work in an agile environment using Azure Boards, Confluence.
- Ability to troubleshoot and deploy using Redgate tools and Visual Studio.
Nice-to-haves
- Familiarity with Modern Data Platforms technologies like Snowflake, Matillion, Bryte Flow, and Tableau.
- Knowledge in Azure IoT, Azure HDInsight + Spark, Kafka and Azure Stream Analytics.
- Experience with Informatica - IICS task flows development and maintenance.
- Data bricks, Microsoft Azure Data Engineer, and/or other cloud certifications.
Benefits
- Generous time off, including personal and volunteering.
- Tuition reimbursement and professional development opportunities.
- Remote work.
- Charitable contribution match programs.
- Stock purchase opportunities.
