Senior Databricks Engineer - 100% Remote

Valueaddedsolutions•Richland, WA

2d•Remote

About The Position

The Pacific Northwest National Laboratory (PNNL) requires an experienced Senior Databricks Engineer contractor to augment the enterprise data engineering team during the implementation of a new ERP platform. The contractor will directly develop code, build and optimize data pipelines, and work agilely across a variety of day-to-day engineering tasks to support technical execution within the team. The contractor will work alongside an existing technical contractor team and internal staff to develop, refine, and operationalize five inaugural domain lakehouse pipelines. As a collaborative and adaptable technical contributor, the contractor will lead development across the Silver through Platinum medallion layers, implement dimensional data models, establish automated testing and quality assurance frameworks, maintain technical documentation (including Architecture Decision Records (ADRs) in GitHub), and mentor incoming staff on daily operations. The contractor will work under the technical and operational direction of the Senior Manager, Data & Analytics and the Data Architecture Workstream Lead to maintain alignment with critical ERP project milestones and enterprise architecture standards.

Requirements

7+ years of data engineering/platform engineering experience, with 3-5+ years focused on production cloud data architectures.
5+ years of production experience with Azure Databricks, Delta Lake, PySpark, Spark SQL, Workflows, and Unity Catalog.
Proven ability to write clean, modular, and maintainable production code in Python/PySpark and SQL while agilely adapting to diverse technical tasks.
Demonstrated expertise in dimensional data modeling (Kimball methodology), star schemas, fact/dimension structures, and Platinum layer curation.
Proven background ingesting, transforming, and modeling large-scale, complex transactional or ERP data structures.
Proficient in CI/CD pipelines (e.g., Databricks Asset Bundles, GitHub Actions), automated testing frameworks, and managing ADR documentation.
Exceptional interpersonal skills, an amenable and collaborative team-first mindset, and a proven ability to mentor and upskill fellow engineers.
U.S. Citizenship: The contractor must be strictly a United States Citizen.
Information Security: The contractor must execute agreements to safeguard sensitive data in accordance with laboratory requirements and federal stipulations.
Mandatory Onboarding Training: Prior to being granted access to PNNL networks and computing environments, the contractor must successfully complete all prerequisite onboarding courses, including Cyber Security, Human Resources, and System Access Training, administered online via the PNNL Web Portal.

Nice To Haves

Demonstrated familiarity or hands-on experience with legacy data warehouses, SQL Server databases, data marts, and migrating legacy SQL/ETL workloads to Databricks.
Prior experience delivering data pipelines in highly regulated or security-conscious environments.
Experience leveraging GenAI/LLM developer tools (e.g., GitHub Copilot, Databricks Assistant) to accelerate development and test creation.
Databricks Certified Data Engineer Associate/Professional or Microsoft Certified: Azure Data Engineer Associate.

Responsibilities

Hands-on development, tune, and operationalize high-throughput batch and streaming ETL/ELT pipelines in PySpark and SQL across the Medallion Architecture (Bronze → Silver → Gold/Platinum).
Ingest complex, enterprise transactional data from the ERP platform and a legacy data warehouse into Azure Data Lake Storage (ADLS Gen2) and Delta Lake.
Implement governed data distribution patterns optimized for downstream applications, analytics, reporting layers, and Power BI semantic models.
Lead the implementation of enterprise dimensional models, star schemas, slowly changing dimensions (SCDs), and curated analytical aggregates.
Collaborate with data architects and business analysts to translate functional data mappings into performant, query-optimized physical data structures.
Design and implement automated data quality validations, schema enforcement, and end-to-end integration tests across development, test, and production environments.
Establish automated monitoring, alerting, error-handling routines, and pipeline telemetry to guarantee data accuracy, operational resilience, and SLA compliance.
Operationalize and optimize Databricks Workflows and Jobs for cost efficiency, scalability, and enterprise fault tolerance.
Standardize deployment workflows using Git/GitHub CI/CD patterns such as Databricks Asset Bundles (DAB).
Apply fine-grained access controls, object governance, and lineage tracking leveraging Databricks Unity Catalog.
Document pipeline designs, runbooks, and Architecture Decision Records (ADRs) directly within GitHub.
Mentor and train incoming Databricks Engineers to transfer platform domain knowledge and ensure smooth handover of day-to-day lakehouse operations.