Principal Data Engineer

Microsoft•Redmond, WA

About The Position

Microsoft is seeking a Principal Data Engineer to lead the architecture, development, and operation of scalable and reliable data platforms supporting experimentation and scorecards. The role calls for deep expertise in ETL, data pipelines, data modeling, and distributed systems, as well as hands-on proficiency in Apache Spark, Python, and SQL. The engineer will shape architecture across team boundaries, anticipate customer needs, and balance immediate delivery with long-term scalability, reliability, security, and cost. They will own complex, ambiguous problems from design through production and guide others toward high engineering standards. The role also emphasizes AI-native engineering: applying AI-assisted practices across design, coding, testing, and troubleshooting while validating the quality, correctness, and responsible use of AI-generated work. The position includes building and evaluating data agents for pipeline diagnostics and performance investigation, with careful controls on access and production remediation.

Requirements

Bachelor's degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

Master's degree in Computer Science or a related technical field and 8+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
Bachelor's degree in Computer Science or a related technical field and 12+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
Experience building data platforms for experimentation, metrics, and scorecards, including understanding success metrics, guardrail metrics, and data quality.
Experience with Databricks-like platforms.

Responsibilities

Lead data-platform architecture by designing and evolving ETL workflows, data pipelines, and data models using Spark, Python, and SQL.
Build end-to-end data solutions using Spark or similar distributed systems, open table formats such as Delta Lake, and Parquet files.
Develop scalable, secure APIs and data access controls, including role-based access control (RBAC).
Provision resources through Infrastructure as Code and deliver production-grade deployments through CI/CD, with observability and operational runbooks.
Own complex system-design decisions and evaluate alternatives with attention to cross-team dependencies, scalability, resiliency, disaster recovery, and cost.
Gather evidence, validate assumptions, test hypotheses, and use telemetry and investigation to resolve complex problems.
Apply AI across design, coding, testing, and troubleshooting; validate AI-generated outputs and help the team adopt effective, responsible practices.
Establish monitoring and operational readiness, drive incident resolution and root-cause analysis, and implement preventive improvements.
Automate safe deployments, targeting zero-touch deployment where possible.
Build data agents that investigate pipeline failures and performance using permitted logs, metrics, traces, lineage, and execution plans.
Detect anomalies and regressions, gather evidence, and propose root causes, actionable fixes, and tuning.
Evaluate and validate agents in production by measuring diagnosis and recovery time, reliability, and performance.
Automate bounded diagnostic and runbook workflows using least-privilege access, auditable actions, controlled remediation, and human approval for risky production changes.