Senior Staff Data Engineer, Databricks (R6508)

Shield AI

4d•$145,000 - $220,000

About The Position

Shield AI is hiring a Senior Staff Data Engineer to help build and operate its enterprise lakehouse on Databricks. The role will create a governed data foundation supporting multiple business domains and downstream analytics, with hands-on ownership of scalable ingestion, reliable data processing, and technical controls across the Bronze and Silver layers of the medallion architecture. The engineer is expected to understand the meaning, sensitivity, classification, and intended use of the data handled by their pipelines. They will apply that understanding to pipeline controls, access patterns, data quality checks, and segregation boundaries appropriate for a highly regulated environment. The role also establishes reusable engineering patterns and standards, supports secure platform delivery and automated deployment, and partners with analytics and domain teams to provide reliable, well-governed data for downstream modeling. Shield AI is a venture-backed defense-technology company whose mission is to protect service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation and synthetic-reality technologies. The company has offices and facilities across the U.S., Europe, the Middle East, and Asia-Pacific, and its technology supports operations worldwide.

Requirements

12 or more years of data engineering experience, including hands-on ownership of production data pipelines.
Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.
Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access-control concepts.
Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with them.
Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.
Strong SQL skills and a solid grounding in data-modeling fundamentals.
Ability to understand the business and regulatory context of data being processed, beyond pipeline mechanics.
Experience applying data classification, segregation, security, retention, or compliance requirements in data-engineering workflows in a regulated or security-sensitive environment.
Ability to design pipelines with awareness of data-domain sensitivity, ownership, permitted use, and downstream impact.
Comfort working in a fast-moving platform build where patterns are still being established and engineers are expected to shape standards.

Nice To Haves

Databricks certification such as Data Engineer Associate or Professional.
Experience establishing or maturing Unity Catalog structures and access models in an enterprise setting.
Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms.
Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment.
Experience enabling multiple business domains on a shared data platform while maintaining strong governance and ownership boundaries.

Responsibilities

Design and build batch and streaming ingestion pipelines from enterprise source systems into the Databricks lakehouse using Delta Lake.
Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.
Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement without embedding unauthorized or undocumented KPI logic.
Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards for other domains to adopt.
Implement and maintain Databricks platform constructs for secure delivery, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.
Build and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and promotion across development, test, and production environments.
Apply data classification, segregation, and handling requirements in pipeline design so sensitive and regulated data is processed according to enterprise controls and access policies.
Build data quality controls that address business meaning, record integrity, completeness, and expected domain behavior in addition to technical failures.
Quarantine, flag, and route problematic records or datasets according to defined quality and compliance rules rather than silently dropping or obscuring issues.
Maintain documentation for source objects, ingestion logic, transformations, data quality rules, and known limitations.
Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.
Collaborate with domain engineering teams on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains join the platform.