Senior Data Engineer – Azure & Microsoft Fabric

INSPYR Solutions • Deerfield Beach, FL

3d • Hybrid

About The Position

The Senior Data Engineer designs, builds, tests, and supports data pipelines and data products for analytics, reporting, and machine learning on Azure and Microsoft Fabric. The role ingests, transforms, and models structured and unstructured data from Azure Synapse, Azure SQL, SQL Server, APIs, and file-based sources. Data is delivered to Fabric Lakehouse and Warehouse on OneLake using Azure Data Factory, Fabric Data Factory pipelines, Dataflow Gen2, and PySpark notebooks.

The engineer partners with the Data Science team to build machine-learning pipelines, automates builds and releases through Azure DevOps, and provisions supporting Azure resources with Bicep. The role also applies secure data practices and incorporates data quality, lineage, and governance into pipelines. It collaborates with Data Scientists, Solution Architects, Software Engineers, DevOps Engineers, QA Engineers, Project Managers, Information Security Engineers, and Delivery Managers.

This is a six-month-plus role based in Deerfield Beach, Florida, with a hybrid 3/2 schedule. Compensation is $75–$80 per hour. Work authorization is required: U.S. citizens, green card holders, or people otherwise authorized to work in the United States.

Requirements

• 15+ years of software development experience with relational databases, including 5+ years of Azure data engineering.
• Must have developed and deployed at least 15 Azure Data Factory or Fabric Data Factory pipelines to production following SDLC processes.
• Expert SQL/T-SQL skills, strong Python and PySpark skills, and demonstrable debugging skills in MS SQL.
• Hands-on experience with Microsoft Fabric Lakehouse, Warehouse, Data Factory pipelines, Dataflow Gen2, Notebooks, OneLake, and semantic models.
• Demonstrable experience with Azure Data Factory, Azure Synapse, Azure SQL, and ADLS Gen2.
• Experience building ML pipelines and supporting data scientists, including feature engineering, training and scoring pipelines, and MLflow, Azure Machine Learning, or Fabric Data Science.
• Experience with Azure DevOps Repos, YAML Pipelines, and Boards, and CI/CD for data solutions.
• Experience writing infrastructure as code with Bicep.
• Strong knowledge of data modeling, medallion architecture, Delta Lake, and relational and non-relational databases.
• Knowledge of data security and governance, including Entra ID, Managed Identity, Key Vault, Microsoft Purview, and private networking.
• Bachelor's degree in computer science, engineering, or a related technical field, or equivalent experience.
• Must be a team player with a get-it-done attitude, strong initiative, willingness to share knowledge and learn, and an interest in improving solutions, systems, and processes.
• Work authorization as a U.S. citizen, green card holder, or otherwise authorized to work in the United States.

Nice To Haves

• ARM experience; Terraform is a plus.
• Azure Synapse Analytics and Azure Databricks.
• Microsoft Purview.
• Power BI semantic models and Direct Lake.
• Fabric Real-Time Intelligence, including Eventstream and KQL.
• GitHub Enterprise and fabric-cicd.
• Experience migrating SAS workloads.
• One or more certifications: Microsoft Certified Fabric Data Engineer Associate (DP-700), Microsoft Certified Fabric Analytics Engineer Associate (DP-600), Azure Data Scientist Associate (DP-100), Azure DevOps Engineer Expert (AZ-400), or Azure Data Engineer Associate (DP-203, retired but still recognized).

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using Azure Data Factory and Microsoft Fabric Data Factory, including pipelines, Copy activity, and Dataflow Gen2, sourcing data from Azure Synapse, Azure SQL, SQL Server, APIs, and files.
• Build bronze, silver, and gold medallion architecture in Fabric Lakehouse and Warehouse on OneLake using Delta Parquet tables, shortcuts, and Mirroring where appropriate.
• Develop transformations in PySpark, Spark SQL, and T-SQL in Fabric Notebooks, and create reusable, parameterized, metadata-driven pipeline frameworks.
• Replace legacy SAS and file-based ETL processes with governed ADF and Fabric pipelines that support alerting, retries, and reruns.
• Develop ML pipelines with the Data Science team, including data preparation, feature engineering, model training, and batch scoring; publish model outputs to Lakehouse tables for Power BI Direct Lake reporting.
• Manage the model lifecycle using MLflow experiment tracking and the model registry, and help migrate Azure Machine Learning workloads to Fabric Data Science.
• Build and maintain Azure DevOps YAML CI/CD pipelines for ADF, Fabric items, notebooks, SQL objects, and ML assets, including Git integration, branching, pull-request validation, and promotion between Dev and Prod workspaces.
• Write and maintain Bicep templates to provision supporting Azure resources consistently across environments, including Data Factory, ADLS Gen2, Key Vault, managed identities, private endpoints, and networking.
• Build dimensional star-schema data models and Fabric semantic models for analytics and reporting.
• Apply data security and compliance controls, including Managed Identity and Entra ID authentication, Key Vault secrets, OneLake and workspace roles, row- and column-level security, data masking, and sensitivity labels.
• Implement data quality checks, lineage, and cataloging with Microsoft Purview in data-movement processes.
• Monitor, troubleshoot, and tune pipelines, Spark jobs, and queries for performance and capacity cost using Fabric Monitoring Hub and Azure Monitor.
• Translate stakeholder business requirements into technical specifications and maintain design specifications and component documents.
• Track the Azure and Microsoft Fabric roadmap and recommend generally available features and best practices.
• Follow coding standards and best practices, including code reviews, logging and instrumentation, code analysis, code coverage, and unit and integration testing.
• Collaborate with Product Managers, Solution Architects, DevOps, and Information Security teams to understand data requirements in an Agile environment.
• Share knowledge with teammates, learn from them, and identify improvements to solutions, systems, and processes.

What This Job Offers

• Career Level: Senior