Sr Data Engineer

SiloSmashers • Arlington, VA

11d • Onsite

About The Position

SiloSmashers is hiring a Senior Data Engineer to support a government customer. The role architects, develops, optimizes, and maintains high-performance data ingestion, transformation, streaming, and analytical pipelines in Databricks-enabled Lakehouse environments. The senior technical expert will guide Databricks Lakehouse engineering, Delta Lake optimization, Unity Catalog governance, medallion architecture, structured streaming, and cloud data engineering across enterprise platforms. The position collaborates with the Project Manager / Contractor Lead, federal technical teams, and platform, SRE, governance, privacy, and development personnel. Regular physical attendance at customer sites is required.

Requirements

• Ability to obtain and maintain a Public Trust clearance.
• Eight or more years of experience in data engineering, ETL/ELT pipeline development, or distributed data processing within enterprise cloud environments, including seven years within the last ten years.
• Bachelor's or master's degree in computer science, information systems, engineering, or a related field; alternatively, twelve total years of equivalent experience may be substituted for a degree.
• Databricks Data Engineer Professional Certification.
• At least one cloud-focused certification, such as AWS Developer Associate or Solutions Architect Associate/Professional, or equivalent.
• Expertise designing and implementing Databricks Lakehouse solutions, including Delta Lake table design, schema evolution, expectations, and bronze/silver/gold medallion architecture.
• Experience with Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models.
• Experience with Databricks performance and cost optimization, including Photon, file sizing, Z-ORDER, clustering/partitioning, caching, AQE, and shuffle/skew mitigation.
• Hands-on experience building distributed data pipelines using Python, PySpark, and cloud-native services across AWS and/or Microsoft Azure.
• Experience administering and optimizing relational and NoSQL data systems, such as PostgreSQL, Aurora, and DynamoDB, and designing efficient, secure schemas.
• Experience developing and maintaining CI/CD pipelines using Git-based workflows, such as GitHub Actions or Azure DevOps, Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks.
• Experience supporting federal environments and compliance-aligned development.
• Experience applying agile development methodologies, including backlog management, sprint planning, and iterative delivery.
• Experience in AI/ML DevSecOps or agentic-AI engineering, including Databricks Model Serving, Genie, Agent Bricks, or integration with AWS Kiro or other government-approved platforms; stated as mandatory.

Nice To Haves

• Databricks Lakehouse Architect certification.
• Generative AI Associate/Professional or related ML/AI certifications.

Responsibilities

• Serve as the principal technical authority for Databricks data engineering, pipeline optimization, streaming, and Lakehouse standards across programs.
• Lead design reviews, technical roadmaps, and architectural decisions with federal SMEs and platform/security teams.
• Build and optimize batch and streaming pipelines using PySpark, Lakeflow Spark Declarative Pipelines, Delta Lake, Auto Loader, and Workflows, with error handling, retries, alerting, and monitoring.
• Lead and enforce medallion-architecture ingestion and refinement patterns, data lifecycle management, and governed data storage tiering.
• Implement performance tuning and cost-efficiency strategies, including cluster sizing, serverless compute adoption, workload health monitoring, and resource governance.
• Perform root cause analysis, disruption and outage reporting, and production support for Databricks jobs, pipelines, applications, and platform features.
• Coordinate with Privacy & Governance teams to enforce Unity Catalog RBAC/ABAC, row- and column-level security, governed tags, lineage, and compliance controls.
• Mentor mid-level and junior engineers, perform code reviews, and uphold engineering quality through testing strategies, quality gates, reproducible builds, and DevSecOps practices.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior