Staff Engineer, Test Automation - San Mateo (R5792)

Shield AI•San Mateo, CA

19d•$188,000 - $282,000

About The Position

Shield AI is seeking a Staff Automated Test Engineer for the Hivemind Software Engineering Integration and Test team. This hands-on technical leadership role defines and evolves automated test architecture, validation infrastructure, MLOps quality systems, and CI/CD pipelines for the Hivemind autonomy platform. The work supports resilient autonomy capabilities for unmanned aircraft and robotic platforms operating in complex, contested, and GPS-denied environments. The role spans production flight code, machine learning models, simulation and synthetic environments, mission planning and orchestration systems, Ground Control Station applications, telemetry and data pipelines, cloud-native developer infrastructure, and hardware-in-the-loop systems. The engineer will lead cross-functional initiatives, establish automation and verification standards, identify systemic quality risks, and build scalable testing infrastructure. The position calls for deep Python automation and distributed-systems testing expertise, technical leadership, and experience validating machine learning systems across their lifecycle. The listed annual salary range is $188,000–$282,000. A full-time regular employee offer package includes pay within the range, bonus, benefits, and equity. Offers are contingent on a cleared background check and may include a reference check. Compensation depends on factors including skill set, experience, credentials, and work location.

Requirements

Typically 8 or more years of relevant experience in software engineering, test infrastructure, developer tooling, MLOps, systems integration, or systems verification, or an equivalent combination of experience and demonstrated impact.
At least 5 years of experience building scalable automation frameworks or developer tooling in Python.
Demonstrated success designing test, CI/CD, or MLOps infrastructure used across multiple engineering teams.
Experience validating machine learning systems across the data, training, evaluation, packaging, deployment, and monitoring lifecycle.
Understanding of ML quality risks, including data leakage, training-serving skew, nondeterminism, distribution shift, drift, model regression, and statistical acceptance criteria.
Experience defining model-performance baselines, automated evaluation suites, release thresholds, and candidate-to-production comparison workflows.
Experience testing GPU-accelerated infrastructure and workloads, including GPU scheduling, allocation, utilization, and resource contention in Kubernetes environments.
Experience with performance benchmarking, profiling, and observability for GPU workloads, including identifying compute, memory, storage, networking, and data-loading bottlenecks.
Experience validating multi-tenant Kubernetes environments, including RBAC, resource quotas, workload isolation, and scheduling behavior.
Experience qualifying integrated hardware and software systems, including automated validation of compute, GPU, storage, networking, drivers, firmware, and deployed software configurations.
Strong system-design skills and experience testing distributed systems, backend services, APIs, and integrated hardware and software environments.
Experience developing integration and regression strategies for internally developed, third-party, open-source, and partner software, including dependency management, compatibility testing, and upgrades.
Strong understanding of asynchronous and concurrent Python programming for scalable automation and parallel test execution.
Experience with package and dependency management, reproducible environments, and build systems such as Conan, pip, setuptools, Poetry, Nix, or similar.
Experience with automated observability, log collection, analytics, reporting, and root-cause analysis in complex software, data, and infrastructure systems.
Experience working in Linux-based development environments.

Nice To Haves

Experience with model registries, experiment tracking, dataset or feature versioning, model serving, and automated artifact promotion using MLflow, Kubeflow, Weights & Biases, SageMaker, Vertex AI, or similar platforms.
Experience with GPU scheduling and orchestration platforms such as Run:ai, NVIDIA GPU Operator, KAI Scheduler, Kueue, Volcano, or similar technologies.
Experience with NVIDIA GPU infrastructure, including CUDA, drivers, container runtimes, Multi-Instance GPU, GPU fractionalization, and hardware/software compatibility testing.
Experience with GPU profiling and performance-analysis tools such as NVIDIA Nsight, PyTorch Profiler, or similar technologies.
Experience validating perception, planning, decision-making, reinforcement learning, or other autonomy models in simulation and on deployed systems.
Experience testing models on embedded or edge-compute platforms, including latency, memory, power, accelerator compatibility, quantization, and hardware-specific behavior.
Experience qualifying production servers or appliances, including hardware validation, burn-in, provisioning, firmware, networking, storage, and software-stack validation before deployment.
Experience with reliability, fault-injection, and recovery testing across distributed compute, storage, networking, and GPU infrastructure.
Experience validating reproducible installation, operation, upgrades, and rollback in cloud, on-premises, disconnected, or air-gapped environments.
Experience with containers, Kubernetes, cloud infrastructure, infrastructure as code, and reproducible test environments.
Proficiency with Go or TypeScript for automation tooling or UI test development.
Experience integrating Python with native C or C++ applications through bindings, wrappers, subprocess interfaces, or similar interoperability tooling.
Aerospace, robotics, autonomy, embedded systems, or safety-critical software experience.
Familiarity with software-in-the-loop, hardware-in-the-loop, requirements-based verification, configuration management, artifact traceability, or standards such as DO-178C and MIL-STD-882.

Responsibilities

Own the technical strategy, architecture, and roadmap for automated testing, verification, and MLOps quality across the Hivemind ecosystem.
Design and maintain scalable test frameworks for autonomy software, backend services, APIs, operator-facing applications, and distributed hardware environments.
Lead functional, integration, regression, system, performance, reliability, and end-to-end testing across simulation, edge-compute, software-in-the-loop, and hardware-in-the-loop environments.
Build automated ML validation pipelines for data quality, training reproducibility, model accuracy, robustness, regression, latency, resource utilization, and system integration.
Establish CI/CD and continuous training workflows that version and track datasets, models, configurations, evaluation results, and deployment artifacts.
Develop scenario-based validation for autonomy models, including edge cases, degraded sensing or communications, distribution shifts, and representative mission conditions.
Create observability, analytics, and failure-triage capabilities for software behavior, model and data drift, inference health, test results, and production performance.
Build Python automation to improve test execution, parallelization, reporting, environment setup, experiment comparison, and developer productivity.
Create test harnesses, simulators, stubs, mocks, and synthetic-data capabilities to improve system testability and coverage.
Collaborate with software, autonomy, machine learning, data, simulation, and systems engineers to define verification strategies and improve designs before implementation.
Develop and govern AI-assisted engineering workflows using coding agents and LLM-based tools for test generation, log analysis, debugging, and failure triage while maintaining security, reproducibility, and traceability.