Azure Data Integration Engineer

i4DM•Millersville, MD

About The Position

The Azure Data Integration Engineer will support ATLAS, the VA’s open-architected, multi-cloud enterprise data management and analytics platform. This hands-on role designs, develops, and maintains scalable Azure pipelines and ETL/ELT solutions that ingest, condition, curate, and transform data from multiple sources into standardized enterprise data products. The engineer uses Azure Data Factory, Synapse, Databricks, Azure Data Lake, and related services to improve interoperability and provide consistent, secure access to trusted data. Working with analysts, data scientists, product teams, enterprise users, BI developers, cloud engineers, software engineers, architects, and DevOps teams, the engineer supports analytics and AI/ML use cases and helps deliver insights for evidence-based decisions and improved outcomes for Veterans and their families. The work spans data integration, cloud data architecture, governance, performance optimization, analytics enablement, deployment automation, and documentation across the ATLAS ecosystem. The engineer participates in an Agile team and contributes to technical discussions, design reviews, code reviews, sprint planning, and Agile/Scrum ceremonies.

Requirements

Bachelor’s degree in computer science or a related technical field.
At least five years of practical experience in cloud application development with Microsoft Azure.
Expertise in Azure Data Factory, Synapse Analytics, Data Lake, Databricks, DAX, Power Query (M), and data modeling.
Strong SQL skills, including T-SQL, query optimization, and database performance tuning.
Experience with big data processing using Apache Spark, Python, or Scala.
Experience integrating Power BI with Azure SQL, Data Lake, and Synapse Analytics.
Understanding of ETL processes, data warehousing, and BI best practices.
Strong knowledge of Row-Level Security (RLS) and Power BI service administration.
Hands-on experience with real-time data streaming using Kafka, Event Hub, or Stream Analytics.
Familiarity with DevOps for data pipelines, CI/CD, and Infrastructure as Code, including Terraform or Bicep.
Knowledge of Azure AD authentication, security best practices, and cloud monitoring.
Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements.
Knowledge of Agile, Scrum, and SAFe methodologies.
Careful, reliable, and detail-oriented when performing high-impact administration tasks.
Experience with Synapse Pipelines, Starburst, Azure Data Lake Service, Collibra for data cataloging, Azure ML/AI services, and Pyramid Analytics for visualization.

Nice To Haves

Microsoft Certified: Azure Data Engineer Associate (DP-203).
Microsoft Certified: Power BI Data Analyst Associate (PL-300).
Experience with CosmosDB, Delta Lake, and NoSQL solutions.
Knowledge of machine learning and AI integration with data pipelines.
Familiarity with Power BI, Looker, or Tableau for data visualization.
Knowledge of Azure architecture and services, Azure portal, Azure Government Cloud, SQL administration, SQL reporting, APIs, application and API authorization methods, and migrating SQL schemas and data between environments.
Experience in embedded analytics and Power BI API integration.

Responsibilities

Design and implement scalable, secure, and efficient data pipelines using Azure Data Factory.
Develop and optimize ETL processes to ingest, transform, and store data in Azure Data Lake, SQL, and Synapse Analytics.
Automate ingestion of structured and unstructured data from sources such as APIs, databases, and IoT streams.
Build and maintain data lakes, warehouses, and lakehouses using Azure Data Lake, Synapse, and Databricks.
Optimize data storage and retrieval using Parquet, Delta Lake, and columnar storage formats.
Tune large-scale data solutions for availability, scalability, and performance.
Implement batch and real-time streaming solutions using Azure Stream Analytics, Event Hub, and Kafka.
Use big data processing frameworks such as Apache Spark with Databricks and HDInsight.
Integrate data from sources such as SQL Server, CosmosDB, SAP, and third-party APIs.
Implement security practices including RBAC, Azure AD, Managed Identities, and Key Vault.
Use Purview, Data Catalog, and metadata management tools to support data governance.
Monitor and troubleshoot pipeline performance using Azure Monitor, Log Analytics, and Application Insights.
Optimize query performance in Synapse Analytics, SQL, and CosmosDB.
Automate data quality checks, anomaly detection, and alerting.
Work with BI Developers and Cloud Engineers to support analytics and AI/ML use cases.
Participate in Agile/Scrum ceremonies, design reviews, and technical discussions.
Document data flows, best practices, architecture decisions, and technical solutions.
Design, develop, and optimize Power BI dashboards and reports for business intelligence and analytics.
Create interactive visualizations using DAX, Power Query (M), and custom visuals.
Develop paginated reports using Power BI Report Builder.
Ensure compliance with cloud security best practices and standards including ISO 27001, GDPR, and HIPAA.
Automate deployments using Azure DevOps, GitHub Actions, and CI/CD pipelines.
Work with Cloud Engineers and DevOps teams to optimize deployment strategies.
Collaborate with software engineers, architects, and DevOps teams to build robust applications.
Participate in code reviews and sprint planning.