Principal Data Engineer
About The Position
The Principal Data Engineer leads the architecture, development, and operation of scalable, reliable data platforms, with deep expertise in ETL, data pipelines, data modeling, and distributed systems and hands-on proficiency in Apache Spark, Python, and SQL. The role shapes architecture across team boundaries, anticipates customer needs, and balances immediate delivery with long-term scalability, reliability, security, and cost. It owns complex, ambiguous problems from design through production and guides others toward high engineering standards. The engineer takes an AI-native approach, using and improving AI-assisted development practices to accelerate delivery while remaining accountable for the quality, correctness, and responsible use of AI-generated work. The platform supports experimentation and scorecards. The position includes production operations, incident resolution, agent-assisted diagnostics, and careful validation and control of automated remediation.
Requirements
- Bachelor's degree in Computer Science or a related technical field and at least 6 years of technical engineering experience coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
- Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.
Nice To Haves
- Master's degree in Computer Science or a related technical field and at least 8 years of technical engineering experience coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python; or a bachelor's degree in a related field and at least 12 years of such experience; or equivalent experience.
- Experience building data platforms for experimentation, metrics, and scorecards, including understanding success metrics, guardrail metrics, and data quality.
- Experience with Databricks-like platforms.
Responsibilities
- Design and evolve ETL workflows, data pipelines, and data models using Spark, Python, and SQL for a platform supporting experimentation and scorecards.
- Build end-to-end data solutions using Spark or similar distributed systems, open table formats such as Delta Lake, and Parquet files.
- Develop scalable, secure APIs and data access controls, including role-based access control (RBAC).
- Provision resources through Infrastructure as Code and deliver production-grade deployments through CI/CD, with observability and operational runbooks.
- Own complex system-design decisions, evaluate alternatives, and account for cross-team dependencies, scalability, resiliency, disaster recovery, and cost.
- Gather evidence, validate assumptions, test hypotheses, and use telemetry and investigation to resolve complex problems.
- Apply AI across design, coding, testing, and troubleshooting; validate AI-generated outputs and help the team adopt effective, responsible practices.
- Establish monitoring and operational readiness, drive incident resolution and root-cause analysis, and implement preventive improvements.
- Automate safe deployments, targeting zero-touch deployment where possible.
- Build data agents that investigate pipeline failures and performance using permitted logs, metrics, traces, lineage, and execution plans; detect anomalies and regressions; and propose evidence-based root causes, fixes, and tuning.
- Evaluate and validate agents in production by measuring diagnosis and recovery time, reliability, and performance.
- Automate bounded diagnostic and runbook workflows with least-privilege access, auditable actions, controlled remediation, and human approval for risky production changes.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
