Staff Software Engineer ML Platform (6244)

Shield AI•San Mateo, CA

About The Position

Shield AI is hiring a Staff Software Engineer to build backend services for machine-learning experiment tracking, model registries, and related workflows. The role connects training runs to the data, code, and configuration used to produce them; manages checkpoints and model artifacts; and helps engineers reproduce results and move trained models toward use. The work includes integrating tools such as MLflow where appropriate and developing reliable services for internal engineering teams and customer environments. The position is part of the AI Factory effort, which provides a reference architecture for Shield AI’s engineering teams and customers building AI capabilities. The work supports researcher iteration, efficient use of compute resources, and the progression of AI systems from experimentation to operational deployment, drawing on distributed systems, Kubernetes, and open-source AI technologies.

Requirements

Experience building production backend services for machine-learning platforms or data-intensive workflows.
Strong software engineering skills in Python or a comparable backend language.
Experience designing APIs, metadata models, and persistent state for services used by other engineers.
Experience handling storage, partial failures, retries, and consistency in distributed systems.
Understanding of training runs, checkpoints, and model versioning.
Ability to work directly with machine-learning users and deliver maintainable software that solves their workflow problems.

Nice To Haves

Experience with MLflow, Weights & Biases, model registries, or comparable experiment-management tools.
Experience integrating Kubernetes jobs, object storage, or machine-learning workflow systems.
Experience with reproducibility, data provenance, or evaluation records.
Experience supporting open-source software or on-premises and air-gapped deployments.

Responsibilities

Build experiment-tracking and model-registry services, integrating tools such as MLflow where appropriate.
Connect exact dataset, code, and configuration versions to training runs, checkpoints, and model versions.
Develop APIs and adapters connecting training jobs, data services, and artifact storage.
Make interrupted or failed workflows recoverable through reliable state handling, retries, and useful diagnostics.
Make checkpoints, model artifacts, and evaluation results easy to find and retrieve for further training, evaluation, or deployment.
Work directly with machine-learning engineers to simplify workflows and reduce manual record keeping.
Deliver tested services with observability and compatible upgrades across supported cloud and customer-managed environments.