Data Engineer (Microsoft Fabric and Databricks)

TekWissen•Reston, VA

5d•Onsite

About The Position

The Senior Data Engineer designs, develops, administers, governs, and supports enterprise data platforms built on Microsoft Fabric. The role combines data engineering expertise and platform administration to deliver scalable, secure, reliable, and cost-effective data solutions for business operations, analytics, AI initiatives, and project delivery. It requires 10–15 years of relevant experience in data engineering, data platforms, cloud technologies, and enterprise data management. The role designs, builds, and supports data ingestion, ETL/ELT pipelines, lakehouse architectures, and analytical datasets using Microsoft Fabric, Databricks, Python, PySpark, SQL, Spark, Delta Lake, OneLake, and Data Pipelines. It also governs Databricks and Fabric environments, including security, access management, monitoring, performance tuning, capacity management, and platform operations. The Senior Data Engineer collaborates with business stakeholders, architects, and analytics teams, and drives continuous improvement in data quality, governance, automation, performance, and cost optimization.

Requirements

B.Tech in Computer Science, Information Technology, Engineering, or an equivalent degree.
10–15 years of overall relevant experience in data engineering, data platforms, analytics, and cloud technologies.
At least 5 years of hands-on Databricks experience and 2 years of experience with Microsoft Fabric environments.
Proven experience designing and supporting production-grade ETL/ELT and data integration solutions.
Experience administering enterprise-scale Databricks and Microsoft Fabric platforms.
Strong experience with Azure data services and cloud-native architectures.
Demonstrated experience in stakeholder engagement, solution ownership, and technical leadership.
Strong expertise in Databricks, Spark, Delta Lake, Microsoft Fabric, OneLake, lakehouse, and data warehouse architectures.
Advanced proficiency in Python, PySpark, SQL, and notebook-based development.
Expertise in ETL/ELT frameworks, data ingestion patterns, incremental processing, CDC, and data transformation techniques.
Strong understanding of dimensional modeling, data warehousing, and semantic data structures.
Experience with batch processing, streaming architectures, and real-time data integration.
Strong knowledge of Databricks Workspace Administration and Microsoft Fabric Administration.
Experience managing capacities, workspaces, governance controls, security policies, and platform operations.
Expertise in Microsoft Entra ID (Azure AD), RBAC, access controls, service principals, and security integration.
Experience with secrets management, key vaults, credential management, and secure data access.
Knowledge of platform monitoring, observability, troubleshooting, incident management, and operational excellence.
Strong understanding of data governance, metadata management, lineage, data cataloging, and compliance controls.
Experience with Git, Azure DevOps, CI/CD pipelines, release management, and automated testing.
Knowledge of Infrastructure as Code, automation frameworks, and deployment standards.
Experience implementing enterprise data quality and reliability frameworks.
Experience tuning Spark applications, SQL workloads, storage structures, and distributed compute environments.
Knowledge of Fabric capacity optimization, Databricks cluster optimization, workload management, and FinOps practices.
Ability to balance performance, scalability, reliability, security, and cost considerations.
Ability to work effectively in a fast-paced enterprise environment supporting multiple business priorities.

Nice To Haves

Experience mentoring engineers and supporting enterprise platform operations.
Microsoft Certified: Fabric Analytics Engineer Associate.
Microsoft Certified: Azure Data Engineer Associate.
Databricks Certified Data Engineer Professional.
Databricks Certified Data Engineer Associate.

Responsibilities

Own the end-to-end technical design, development, and delivery of enterprise data engineering solutions.
Design and implement scalable ETL/ELT, integration, and transformation pipelines for structured and unstructured data.
Develop and maintain batch, near-real-time, and streaming ingestion pipelines from databases, APIs, files, cloud storage, and enterprise applications.
Build reusable notebooks, transformations, and processing frameworks using Databricks, Microsoft Fabric, Python, PySpark, and SQL.
Design and implement lakehouse architectures using Delta Lake, OneLake, and Bronze, Silver, and Gold medallion layers.
Create trusted datasets, curated data products, and semantic-ready structures for analytics, reporting, AI, and machine learning.
Implement data quality frameworks, validation rules, reconciliation controls, and exception management processes.
Design scalable data models, dimensional schemas, and data marts for reporting and analytical workloads.
Optimize Spark workloads, SQL queries, storage strategies, partitioning, and data-processing performance.
Administer Databricks workspaces, Fabric capacities and workspaces, Lakehouses, Data Warehouses, Dataflows, and Pipelines.
Configure and maintain user access, security groups, RBAC permissions, service principals, and identity integrations.
Manage secrets, credentials, key vault integrations, and secure access to enterprise data sources.
Monitor platform health, resource utilization, workload performance, and capacity consumption.
Perform platform upgrades, feature enablement, release validation, and environment maintenance.
Implement capacity planning, workload management, and resource allocation strategies.
Enforce workspace standards, naming conventions, tagging, and compliance controls in support of governance initiatives.
Develop operational runbooks, support procedures, and disaster recovery documentation.
Monitor data pipelines and platform services to ensure availability and reliability.
Establish and maintain metadata management, data lineage, cataloging, and governance processes.
Implement observability, logging, monitoring, and alerting across the data platform ecosystem.
Drive cost optimization across compute, storage, Fabric capacities, and Databricks workloads.
Establish Git-based source control and branching strategies for notebooks, pipelines, and infrastructure components.
Implement CI/CD pipelines for automated deployment, testing, environment promotion, and release management.
Develop reusable deployment templates and engineering accelerators for platform consistency and scalability.
Communicate technical solutions, trade-offs, risks, dependencies, and delivery plans to technical and non-technical stakeholders.