Senior Azure Databricks Data Engineer

MasterApp Labs•Indianapolis, IN

11h•Hybrid

About The Position

The Senior Azure Databricks Data Engineer assists with data technology strategy and data integration issues, and provides direction and oversight for the design and implementation of data solutions across secure systems. The role helps define and apply data management policies and engineering standards, advises and supports project teams, and participates in requirements gathering, solution reviews, and communication of technical concepts to non-technical personnel. It works closely with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation. The engineer extracts, transforms, and loads data from varied sources; designs, builds, and maintains batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake; and develops and tunes dimensional data models and SQL workloads in Azure Synapse Analytics. The role orchestrates and monitors workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery. It applies governance and security controls to protect PHI and PII in accordance with HIPAA and agency policy, documents solutions, and supports operations through systems analysis and recommendations for policy and procedural changes. Other duties and responsibilities may be assigned or changed at any time. The position is a contract role based in Indianapolis, Indiana, with a hybrid arrangement requiring onsite work one day per week. Flexible work-from-home options are available.

Requirements

Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or a related field; alternatively, a bachelor's degree in a related field plus 2 or more years of relevant experience, or 4 or more years of hands-on experience in a similar data engineering role. Relevant certifications and coursework may substitute for formal education or experience requirements.
Experience developing Databricks notebooks with PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions.
Knowledge of Delta Lake ACID transactions, MERGE upserts, schema enforcement and evolution, time travel, and table maintenance such as OPTIMIZE, Z-order, and VACUUM.
Experience with bronze/silver/gold medallion architecture and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT.
Experience using Unity Catalog for workspace catalogs, schemas, permissions, lineage, external locations, and storage credentials.
Ability to right-size clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues using Spark UI.
Experience scheduling and managing dependencies with Azure Data Factory pipelines and Databricks Workflows (Jobs), and using Git folders (Repos) for source control.
Experience designing MPP warehouse tables, including hash, round-robin, and replicated distribution, columnstore indexes, and partitioning.
Experience querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views.
Experience building star-schema facts and dimensions, stored procedures, and views, including SCD Type 1 and Type 2.
Experience with COPY INTO, PolyBase, CETAS, and parameterized Synapse/Azure Data Factory pipelines, including copy activities, data flows, and triggers.
Knowledge of workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling or pausing.
Experience with ADLS Gen2 folder and file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault.
Experience with Git and CI/CD in Azure DevOps or GitHub, including branching, pull requests, and deploying Synapse artifacts and Databricks Asset Bundles.
Experience monitoring pipelines and workloads with Azure Monitor/Log Analytics, configuring alerts, and troubleshooting failed jobs.
Strong written and verbal communication skills.
Knowledge of Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks.

Nice To Haves

Databricks Certified Data Engineer Associate certification; Databricks Certified Data Engineer Professional certification is a plus.
Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.
Experience with Power BI connectivity to Synapse (Import versus DirectQuery) and understanding of how model design affects dashboard performance.

Responsibilities

Assist with data technology strategy and resolve strategic data integration issues, including data quality, stewardship, real-time and event-based integration, and development of a universal data integration vision.
Provide direction and oversight for designing and implementing data integration and synthesis solutions across multiple secure systems.
Assist with defining and implementing data management policies and design standards, and support ongoing adherence to them.
Provide advice, training, and technical support for projects.
Participate in requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel.
Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation.
Extract, transform, and load data from a wide variety of sources using data engineering tools and methods.
Design, build, and maintain batch and streaming Azure Databricks pipelines using PySpark, Spark SQL, and Delta Lake with a bronze/silver/gold medallion lakehouse architecture.
Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics dedicated and serverless SQL pools for reporting and dashboards.
Orchestrate and monitor workflows with Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery.
Apply data governance and security controls, including Unity Catalog permissions, role-based access, row- and column-level security, and data masking, to protect PHI/PII in accordance with HIPAA and agency policy.
Document and demonstrate solutions using documentation, flowcharts, layouts, diagrams, charts, code comments, and clear code.
Support operations by conducting systems analysis and recommending changes to policies and procedures.
Perform other duties, responsibilities, and activities as assigned.