Staff Software Engineer ML Platform (6244)

Shield AI•San Diego, CA

5d•$200,000 - $300,000

About The Position

Shield AI is hiring a Staff Software Engineer to build backend services that make machine-learning model development workflows reliable. The role focuses on connecting training runs to their data, code, and configuration; managing model artifacts; and helping machine-learning engineers reproduce results, compare approaches, and move trained models into use. The engineer will build the AI Factory reference architecture used internally by Shield AI engineering teams and delivered to customers building their own AI capabilities. The work spans experiment tracking, model registries, distributed systems, Kubernetes, and open-source AI tools, supporting workflows from experimentation through operational deployment across cloud and customer-managed environments.

Requirements

Experience building production backend services for machine-learning platforms or data-intensive workflows.
Strong software engineering skills in Python or a comparable backend language.
Experience designing APIs, metadata models, and persistent state for services used by other engineers.
Experience handling storage, partial failures, retries, and consistency in distributed systems.
Understanding of training runs, checkpoints, and model versioning.
Ability to work directly with machine-learning users and deliver maintainable software that solves their workflow problems.

Nice To Haves

Experience with MLflow, Weights & Biases, model registries, or comparable experiment-management tools.
Experience integrating Kubernetes jobs, object storage, or machine-learning workflow systems.
Experience with reproducibility, data provenance, or evaluation records.
Experience supporting open-source software or on-premises and air-gapped deployments.

Responsibilities

Build experiment-tracking and model-registry services, integrating tools such as MLflow where appropriate.
Connect exact dataset, code, and configuration versions to training runs, checkpoints, and model versions.
Develop APIs and adapters connecting training jobs, data services, and artifact storage.
Make interrupted or failed workflows recoverable through reliable state handling, retries, and useful diagnostics.
Ensure checkpoints, model artifacts, and evaluation results can be found and retrieved for further training, evaluation, or deployment.
Work directly with machine-learning engineers to simplify workflows and reduce manual record keeping.
Deliver tested services with observability and compatible upgrades across supported cloud and customer-managed environments.