Senior Databricks Data Engineer

N2IA Technologies•Washington, DC
•$160 - $200•Onsite

About The Position

N2IA Technologies is seeking a Senior Databricks Data Engineer to build, refine, and maintain core data pipelines on the Databricks Lakehouse Platform. These pipelines enable enterprise AI, analytics, and mission systems by making data accessible, reliable, and usable across the agency. The engineer will work with cloud engineers, data architects, and AI developers to ensure data is clean, connected, secure, and optimized for advanced analytics and AI/ML workflows. The role supports a Quantico, Virginia customer. Although described as remote, full-time onsite work at the client site is required.

Requirements

  • At least 3 years of hands-on data engineering experience building production pipelines on Databricks.
  • At least 10 years of technical professional experience.
  • Strong proficiency with Apache Spark, including PySpark and Spark SQL, Delta Lake, and the Databricks Lakehouse Platform.
  • Hands-on experience with Databricks on Azure and/or AWS, such as Azure Databricks, Azure Data Factory, AWS Glue, or similar orchestration tooling.
  • Experience with Unity Catalog for data governance, access control, and lineage.
  • Experience integrating structured and unstructured data sources.
  • Knowledge of SQL, schema design, data modeling, and cloud-based storage patterns such as Delta tables and cloud object storage.
  • Secret clearance.

Nice To Haves

  • Databricks certification, such as Databricks Certified Data Engineer Associate or Professional.
  • Experience supporting AI/ML pipelines, vectorization, or embedding generation, ideally using MLflow, Databricks Mosaic AI, Model Serving, or Vector Search.
  • Experience with Delta Live Tables and streaming pipelines using Structured Streaming or Auto Loader.
  • Familiarity with Azure, AWS, or multi-cloud data ecosystems.
  • Experience with data governance, access controls, or sensitive data protection, including Unity Catalog and row- or column-level security.
  • Exposure to DevOps and CI/CD patterns for data workflows, such as Databricks Asset Bundles, Repos, or Terraform.
  • MuleSoft implementation experience.

Responsibilities

  • Design, build, and maintain scalable Databricks data pipelines and ETL/ELT workflows using Delta Lake, Delta Live Tables, and the medallion architecture across cloud and on-premises systems.
  • Develop and orchestrate workflows using Databricks Workflows and Jobs.
  • Implement data quality checks, validation, lineage tracking, and metadata management using Unity Catalog, Lakehouse Monitoring, and Delta Live Tables expectations.
  • Collaborate with other teams to ensure datasets meet requirements and are discoverable through Unity Catalog.
  • Automate ingestion and transformation using Databricks-native patterns such as Auto Loader, Structured Streaming, and COPY INTO.
  • Support API strategy, data modeling, and domain-driven data structures within the Lakehouse.
  • Troubleshoot pipeline issues and optimize Spark jobs and clusters for performance and cost, including Photon, cluster sizing, Z-ordering, liquid clustering, and workload tuning.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service