Data Engineer Jobs

About The Position

Requirements

• Ability to obtain and maintain a Public Trust clearance.
• Eight or more years of experience in data engineering, ETL/ELT pipeline development, or distributed data processing within enterprise cloud environments, including seven years within the last ten years.
• Bachelor's or master's degree in computer science, information systems, engineering, or a related field; alternatively, twelve total years of equivalent experience may be substituted for a degree.
• Databricks Data Engineer Professional Certification.
• At least one cloud-focused certification, such as AWS Developer Associate or Solutions Architect Associate/Professional, or equivalent.
• Expertise designing and implementing Databricks Lakehouse solutions, including Delta Lake table design, schema evolution, expectations, and bronze/silver/gold medallion architecture.
• Experience with Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models.
• Experience with Databricks performance and cost optimization, including Photon, file sizing, Z-ORDER, clustering/partitioning, caching, AQE, and shuffle/skew mitigation.
• Hands-on experience building distributed data pipelines using Python, PySpark, and cloud-native services across AWS and/or Microsoft Azure.
• Experience administering and optimizing relational and NoSQL data systems, such as PostgreSQL, Aurora, and DynamoDB, and designing efficient, secure schemas.
• Experience developing and maintaining CI/CD pipelines using Git-based workflows, such as GitHub Actions or Azure DevOps, Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks.
• Experience supporting federal environments and compliance-aligned development.
• Experience applying agile development methodologies, including backlog management, sprint planning, and iterative delivery.
• Experience in AI/ML DevSecOps or agentic-AI engineering, including Databricks Model Serving, Genie, Agent Bricks, or integration with AWS Kiro or other government-approved platforms; stated as mandatory.

Nice To Haves

• Databricks Lakehouse Architect certification.
• Generative AI Associate/Professional or related ML/AI certifications.

Responsibilities

• Serve as the principal technical authority for Databricks data engineering, pipeline optimization, streaming, and Lakehouse standards across programs.
• Lead design reviews, technical roadmaps, and architectural decisions with federal SMEs and platform/security teams.
• Build and optimize batch and streaming pipelines using PySpark, Lakeflow Spark Declarative Pipelines, Delta Lake, Auto Loader, and Workflows, with error handling, retries, alerting, and monitoring.
• Lead and enforce medallion-architecture ingestion and refinement patterns, data lifecycle management, and governed data storage tiering.
• Implement performance tuning and cost-efficiency strategies, including cluster sizing, serverless compute adoption, workload health monitoring, and resource governance.
• Perform root cause analysis, disruption and outage reporting, and production support for Databricks jobs, pipelines, applications, and platform features.
• Coordinate with Privacy & Governance teams to enforce Unity Catalog RBAC/ABAC, row- and column-level security, governed tags, lineage, and compliance controls.
• Mentor mid-level and junior engineers, perform code reviews, and uphold engineering quality through testing strategies, quality gates, reproducible builds, and DevSecOps practices.