ML Cloud Infrastructure Engineer

HavocAI

4d • $150,000 - $175,000

About The Position

As a Machine Learning Cloud Infrastructure Engineer, you will build and operate the infrastructure HavocAI teams use to train, evaluate, deploy, and monitor machine learning models safely and reliably. You will develop the pipelines, services, platforms, and integrations connecting data lakes, telemetry stores, simulation environments, training workloads, cloud compute, and deployed models. Your work will enable autonomy, data, and software engineers to move efficiently from raw field and simulation data to reproducible datasets, scalable training, rigorous evaluation, and production deployment. This role is ideal for a strong software, infrastructure, or data engineer with hands-on ML experience who enjoys building production systems at the intersection of data, models, compute, and autonomy. You should be comfortable operating in a fast-paced environment, solving ambiguous infrastructure problems, and building systems that must remain scalable, observable, secure, and reliable.

Requirements

• 3+ years of experience in software engineering, infrastructure engineering, data engineering, ML infrastructure, or a related field.
• Strong programming experience in Python, with experience in Go, C++, or another systems-oriented language preferred.
• Experience building and operating production services, APIs, data pipelines, developer platforms, or infrastructure.
• Hands-on experience with ML workflows such as dataset preparation, model training, evaluation, or deployment.
• Experience with cloud infrastructure, preferably AWS, and Infrastructure as Code.
• Hands-on experience with Kubernetes and containerized environments.
• Strong understanding of production engineering fundamentals, including reliability, observability, testing, automation, and maintainability.
• Ability to work effectively across engineering disciplines and solve ambiguous technical problems with a high degree of ownership.
• U.S. Citizenship and ability to obtain and maintain a U.S. Government security clearance.

Nice To Haves

• Experience with MLOps and workflow platforms such as MLflow, Weights & Biases, Kubeflow, Ray, Airflow, or Dagster.
• Experience with GPU/accelerator scheduling, distributed training, or large-scale ML workloads.
• Experience working with multi-modal datasets including imagery, video, telemetry, sensor, or simulation data.
• Experience supporting autonomy, robotics, simulation, or real-time systems.
• Experience deploying ML models to edge or embedded environments.
• Experience with AWS GovCloud, GCP Assured Workloads, or compliance-driven environments such as FedRAMP or IL4/IL5.

Responsibilities

• Build pipelines that transform raw multi-modal data—including telemetry, imagery, video, sensor, and simulation data—into curated, versioned training datasets.
• Develop reproducible training and evaluation workflows that scale across cloud compute and GPU resources.
• Build and maintain model deployment infrastructure for packaging, serving, inference, versioning, and rollback.
• Implement experiment tracking, dataset lineage, model versioning, and other capabilities required for reproducible ML development.
• Own data schema versioning and migration across pipelines, data lakes, and services as datasets and models evolve.
• Design, build, and operate scalable AWS infrastructure using Infrastructure as Code.
• Build and maintain Kubernetes/EKS workloads and containerized environments for training, batch processing, evaluation, and model serving.
• Develop self-service tooling and paved paths for compute scheduling, storage, data access, training, and deployment.
• Improve utilization, scalability, and cost efficiency across cloud and accelerator infrastructure.
• Build infrastructure that enables engineering teams to launch workloads safely without unnecessary operational overhead.
• Build evaluation frameworks and regression testing for model quality, dataset integrity, and pipeline correctness.
• Establish quality and reliability signals that help determine whether models are ready for production use.
• Develop monitoring, logging, tracing, and observability across training jobs, data pipelines, and deployed models.
• Diagnose and resolve performance, scaling, reliability, and infrastructure bottlenecks.
• Maintain high standards for automation, testing, documentation, and operational readiness.
• Partner with Autonomy, Software, Data, Simulation, and Security teams to build ML infrastructure spanning edge data capture through cloud training and model deployment.
• Contribute to CI/CD and release processes for models, datasets, and ML pipelines.
• Translate engineering requirements into scalable platform capabilities that can support multiple teams and use cases.
• Incorporate feedback from engineers and users to continuously improve ML development workflows.
• Implement secure infrastructure practices, including IAM least privilege, secrets management, access controls, and secure handling of sensitive and defense-related data.
• Build data and ML workflows with reproducibility, traceability, and appropriate controls from the start.
• Partner with security and infrastructure teams to ensure ML systems meet applicable operational and compliance requirements.

Benefits

• 100% Employer paid Health, Dental and Vision Insurance for you and your families
• Life Insurance (Employer Paid)
• Ability to participate in the companies 401k program (Matching)
• Unlimited PTO policy with an enforced 2 week minimum
• Equity Package
• Work / Home Office Stipend
• Global Entry
• 16 Week Paid Parental Leave
• Monthly Health and Wellness Stipend

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed