Lead DevOps/Platform Engineer IV - Richland, WA

Pacific Northwest National Laboratory•Richland, WA

19h•$161,300 - $255,000•Onsite

About The Position

At PNNL, our core capabilities are divided among major departments that we refer to as Directorates within the Lab, focused on a specific area of scientific research or other function, with its own leadership team and dedicated budget. Our Science & Technology directorates include National Security, Earth and Biological Sciences, Physical and Computational Sciences, and Energy and Environment. In addition, we have an Environmental Molecular Sciences Laboratory, a Department of Energy, Office of Science user facility housed on the PNNL campus. The National Security Directorate (NSD) drives science-based, mission-focused solutions to take on complex, real-world threats to our nation and the world. The AI and Data Analytics Division, part of NSD, combines profound domain expertise and creative integration of advanced hardware and software to deliver computational solutions that address complex data and analytic challenges. Working in multidisciplinary teams, we connect foundational research to engineering to operations, providing the tools to innovate quickly and field results faster. Our strengths are integrated across the data analytics lifecycle, from data acquisition and management to analysis and decision support. We are seeking a Lead DevOps/Platform Engineer to join PNNL's advanced AI engineering initiatives, contributing to next-generation systems spanning agentic AI platforms, large-scale data orchestration, and real-time intelligence processing. In this role, you'll apply your expertise in scalable system design and AI/ML engineering to build mission-critical capabilities while developing your technical leadership and establishing yourself as a key contributor to our engineering community.

Requirements

Expert-level proficiency in Python and at least one additional language (Go, C#/.NET, C++) with proven ability to establish infrastructure automation standards, architect scalable tooling platforms, and guide teams in developing sophisticated automation frameworks
Mastery of Infrastructure as Code principles with deep expertise in Terraform, CloudFormation, Pulumi, or ARM templates and demonstrated ability to design enterprise-wide IaC strategies, module libraries, and governance frameworks that enable consistent and secure infrastructure deployment
Proven track record of architecting and leading implementation of enterprise-grade CI/CD platforms with ability to define build/release strategies, establish deployment patterns, and drive continuous delivery adoption while designing internal developer platforms that abstract complexity and accelerate team velocity
Expert proficiency with GitOps methodologies (ArgoCD, Flux), infrastructure testing frameworks (Terratest, InSpec), and policy-as-code (OPA, Sentinel) with strategic application of AI assist tools to drive team productivity, accelerate automation development, and optimize operational efficiency
Demonstrated expertise architecting and leading multi-cloud infrastructure strategies across AWS, Azure, and GCP with deep expertise in containerization and Kubernetes ecosystem including production-grade container platforms, custom operators, CRDs, and multi-cluster strategies at organizational scale
Expert ability to architect sophisticated event-driven systems using cloud-native services (EventBridge, Event Grid, Pub/Sub, SNS/SQS) with advanced knowledge of service mesh architectures (Istio, Linkerd, Consul) and API gateway patterns for zero-trust networking and complex microservice environments
Mastery of cloud and container networking including CNI design, custom ingress implementations, advanced load balancing, service discovery patterns, and network security policies with ability to troubleshoot complex distributed system networking issues
Experience architecting edge computing solutions, hybrid cloud strategies, and secure enclave deployments with understanding of data sovereignty, latency optimization, and security requirements for geographically distributed infrastructure
Proven ability to architect comprehensive observability platforms integrating metrics (Prometheus, Thanos, Cortex), distributed tracing (Jaeger, Tempo), and logging systems (ELK, Loki, Splunk) with deep expertise in SRE principles including SLO/SLI frameworks, error budgets, and incident management
Expert implementation of security-first infrastructure including secrets management (Vault, AWS Secrets Manager, Azure Key Vault), automated vulnerability scanning, DevSecOps toolchains, and security policy enforcement across all infrastructure layers
Strategic capability to design enterprise disaster recovery and business continuity strategies including multi-region architectures, automated backup systems, RPO/RTO optimization, and regular DR testing with advanced chaos engineering practices to systematically improve system resilience
Deep understanding of compliance frameworks (SOC 2, HIPAA, FedRAMP, PCI-DSS, GDPR) with proven ability to implement automated compliance controls, audit logging, and infrastructure hardening standards that meet regulatory requirements
Expertise in architecting end-to-end MLOps platforms with proven ability to design and implement model lifecycle management infrastructure including experiment tracking (MLflow, Weights & Biases), model versioning, model registries, feature stores (Feast, Tecton), and automated ML pipeline orchestration supporting continuous training and deployment
Deep expertise in building infrastructure for ML model serving and deployment including real-time inference APIs, batch prediction systems, A/B testing frameworks, model monitoring for drift detection, and automated model retraining pipelines with canary deployments and rollback capabilities
Advanced knowledge of distributed ML training infrastructure including multi-GPU and multi-node training orchestration, resource scheduling, and optimization for frameworks like PyTorch, TensorFlow, and JAX on Kubernetes-based platforms (Kubeflow, Ray, Spark ML) with deep understanding of compute resource management and cost optimization
Proven ability to architect cloud-native data platforms with expertise in ETL/ELT orchestration frameworks (Airflow, Prefect, Dagster, AWS Step Functions), production data storage systems (S3, Redshift, Databricks Delta Lake, PostgreSQL, MongoDB, Snowflake), and distributed data processing frameworks (Spark/Databricks, Kafka, Flink, Ray) supporting petabyte-scale data systems and real-time ML feature pipelines
Exceptional problem-solving and troubleshooting abilities with proven track record of resolving complex infrastructure incidents spanning ML pipelines, data platforms, and distributed systems while leading incident response and root cause analysis, combined with outstanding communication skills to translate technical complexity into business impact for executive leadership and stakeholders
Demonstrated ability to establish infrastructure and MLOps documentation standards, create comprehensive runbooks for ML system operations and DR procedures, develop technical training programs, and build knowledge sharing practices while mentoring and developing platform engineering teams through technical guidance and architecture reviews
Proven capacity to lead multiple concurrent infrastructure and MLOps initiatives while maintaining production reliability for both traditional applications and ML systems, managing competing priorities, balancing technical debt, and establishing on-call practices, incident response frameworks, and blameless post-mortem processes that drive systemic improvements
Strategic ability to balance immediate operational needs with long-term infrastructure and MLOps vision, evaluate emerging ML infrastructure technologies, drive platform modernization initiatives including ML democratization, and establish technical roadmaps that enable organizational scaling, AI/ML innovation, and operational excellence
PhD and 3 years of Software Engineering experience -OR- MS/MA and 5 years of Software Engineering experience -OR- BS/BA and 7 years of Software Engineering experience -OR- AA and 16 years of Software Engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development -OR- HS/GED and 18 years of Software Engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development
U.S. Citizenship

Nice To Haves

Degree in computer science, software engineering, or related field.
Track record of architecting and operating large-scale infrastructure supporting significant user bases, high-volume transaction systems, petabyte-scale data platforms, or production ML systems serving millions of predictions.
Experience building and leading high-performing platform engineering, DevOps, or MLOps teams through hiring, mentoring, technical guidance, and career development.
Experience establishing infrastructure practices, platform strategies, MLOps frameworks, and DevOps transformation initiatives at organizational scale.
Background in mission-critical, regulated, or high-security environments (government, defense, financial services, healthcare) with understanding of compliance requirements for both traditional systems and ML/AI applications.
Demonstrated success leading complex, multi-team infrastructure and MLOps initiatives from architecture through production deployment, operational handoff, and continuous improvement.

Responsibilities

Design and deploy scalable agentic AI systems with dynamic reasoning and decision-making capabilities
Architect LLM orchestration frameworks using LangChain, LlamaIndex, and emerging agent platforms
Build MLOps platforms spanning experiment tracking, model versioning, deployment, and governance
Develop developer-focused tooling, adapters, and interfaces for AI-native frameworks
Integrate multi-modal data sources (text, vision, structured/sensor data) into cohesive reasoning pipelines
Design microservices architectures coordinating across multiple domains and security enclaves
Lead distributed system design processing data from hundreds of sources simultaneously
Architect real-time streaming platforms handling terabytes per hour with event-driven architectures
Build robust data pipelines for petabyte-scale ETL, data lake/mesh architectures, and real-time analytics
Design container orchestration (Kubernetes) and CI/CD pipelines for classified and edge environments
Deploy AI systems in highly secure environments with resilient agent-to-agent communications
Create monitoring and observability systems (logging, metrics, tracing) across secure enclaves
Ensure compliance with ethical AI standards and security-first DevOps practices
Build geospatial processing, time-series, and intelligence data fusion capabilities
Lead a team of engineers to deliver on high risk / high impact ambiguous technical scope
Drive technical strategy and architectural decisions across cross-functional teams
Translate ambiguous requirements and cutting-edge research into actionable technical roadmaps
Lead design discussions shaping team-wide engineering standards
Mentor engineering teams and guide junior scientists/engineers