Staff Software Engineer ML Platform (6244)

f7d8749a-72d6-407c-bf98-292ea630c73a•San Diego, CA

About The Position

Shield AI is hiring a Staff Software Engineer to build backend services for reliable machine-learning experiment and model-management workflows. The role connects training runs to their data, code, and configuration; manages model artifacts; and helps machine-learning engineers reproduce results and move trained models into use. The position contributes to the AI Factory reference architecture used internally and delivered to customers building their own AI capabilities. The engineer will build experiment-tracking and model-registry services, develop APIs and integrations across training jobs, data services, and artifact storage, and make workflows recoverable when jobs fail or are interrupted. The work includes ensuring checkpoints, artifacts, and evaluation results can be found and retrieved, and delivering tested, observable services with compatible upgrades across supported cloud and customer-managed environments. The role works directly with machine-learning engineers to simplify workflows and reduce manual record keeping.

Requirements

Experience building production backend services for machine-learning platforms or data-intensive workflows.
Strong software engineering skills in Python or a comparable backend language.
Experience designing APIs, metadata models, and persistent state for services used by other engineers.
Experience handling storage, partial failures, retries, and consistency in distributed systems.
Understanding of training runs, checkpoints, and model versioning.
Ability to work directly with machine-learning users and deliver maintainable software that solves their workflow problems.

Nice To Haves

Experience with MLflow, Weights & Biases, model registries, or comparable experiment-management tools.
Experience integrating Kubernetes jobs, object storage, or machine-learning workflow systems.
Experience with reproducibility, data provenance, or evaluation records.
Experience supporting open-source software or on-premises and air-gapped deployments.

Responsibilities

Build experiment-tracking and model-registry services, integrating tools such as MLflow where appropriate.
Connect exact dataset, code, and configuration versions to training runs, checkpoints, and model versions.
Develop APIs and adapters connecting training jobs, data services, and artifact storage.
Make interrupted or failed workflows recoverable through reliable state handling, retries, and useful diagnostics.
Make checkpoints, model artifacts, and evaluation results easy to find and retrieve for further training, evaluation, or deployment.
Work directly with machine-learning engineers to simplify workflows and reduce manual record keeping.
Deliver tested services with observability and compatible upgrades across supported cloud and customer-managed environments.