AI Infrastructure Engineer – Agents & ML Systems

HavocAI

21h • $175,000 - $200,000

About The Position

As an AI Infrastructure Engineer – Agents & ML Systems, you will help build the internal AI infrastructure that allows HavocAI teams to use modern AI systems safely, reliably, and effectively. You will develop tools, services, pipelines, and integrations that connect large language models, agentic workflows, internal data sources, engineering systems, and ML workflows.

This role is ideal for a strong software or infrastructure engineer who is excited about the practical application of AI. You need not have worked on every part of the AI stack, but you should be curious, hands-on, and comfortable building production systems that connect models, tools, data, and users.

You will work on systems that help internal teams search and reason over company data, automate engineering workflows, support simulation and autonomy development, curate data for future model training, and evaluate AI systems before they are trusted in critical workflows. This is a high-impact role at the intersection of software engineering, AI infrastructure, developer tooling, data systems, and applied ML.

Requirements

• Bachelor's degree in Computer Science, Engineering, Machine Learning, Data Science, Applied Mathematics, or a related technical field.
• 3+ years of experience in software engineering, infrastructure engineering, ML infrastructure, backend systems, data engineering, developer tools, or related technical roles.
• Strong programming experience in Python, TypeScript, Go, C++, or similar languages.
• Experience building production software systems, APIs, services, data pipelines, or internal platforms.
• Experience with, or strong interest in, LLM applications, AI agents, tool-using systems, RAG pipelines, or AI developer tools.
• Familiarity with modern AI infrastructure concepts such as embeddings, vector search, prompt management, evaluation, model serving, fine-tuning, or MLOps.
• Ability to integrate systems across APIs, databases, object stores, documents, logs, internal tools, and structured or unstructured data sources.
• Strong understanding of production engineering fundamentals, including reliability, observability, testing, and maintainability.
• Strong grounding in securing AI and agentic systems, including least-privilege tool access, prompt-injection and misuse mitigation, secrets management, and safe handling of sensitive data.
• Ability to work across Software, Data, ML, Infrastructure, and Product teams.
• Strong debugging skills and comfort working with complex distributed systems.
• U.S. citizenship and the ability to obtain and maintain a security clearance.

Nice To Haves

• Experience with MCP, including MCP servers, clients, tools, resources, prompts, or connector patterns.
• Experience with LangGraph, LangChain, LlamaIndex, Semantic Kernel, OpenAI APIs, Anthropic APIs, local/open-weight models, or similar AI tooling.
• Experience with vector databases, embeddings, retrieval systems, knowledge graphs, document processing, search, or RAG systems.
• Experience with LLM fine-tuning, supervised fine-tuning, preference tuning, synthetic data generation, evaluation datasets, or model benchmarking.
• Experience with Kubernetes, Docker, Ray, Airflow, Dagster, MLflow, Weights & Biases, Kafka, Postgres, S3-compatible storage, or similar infrastructure.
• Experience building internal platforms, developer tools, workflow automation systems, or data/ML infrastructure.
• Experience with human-in-the-loop workflows, approval systems, audit logs, policy enforcement, or safe tool invocation patterns.
• Experience integrating AI tools with engineering workflows such as GitHub, CI/CD, issue trackers, documentation systems, simulation platforms, or data lakes.
• Familiarity with autonomy, robotics, simulation, telemetry, perception, or defense technology workflows.
• Experience building secure AI systems for sensitive, regulated, government, defense, or enterprise environments.
• Active or prior security clearance.

Responsibilities

• Build internal AI infrastructure that connects LLMs and AI agents with internal tools, APIs, data sources, data lakes, telemetry stores, simulation tools, code repositories, documentation systems, logs, and engineering workflows.
• Develop and maintain agentic AI systems for task automation, data analysis, engineering support, simulation workflows, and internal productivity.
• Build tool integration and connector infrastructure for AI agents, including MCP and other emerging tool-use standards, spanning servers, tools, resources, prompts, connectors, and secure tool-use patterns.
• Create pipelines for retrieval, RAG, context management, document processing, embeddings, and internal knowledge search.
• Support ML infrastructure workflows such as data preparation, dataset curation, experiment tracking, model evaluation, fine-tuning support, and model deployment.
• Build evaluation frameworks for agent performance, tool-use reliability, task success, model quality, regression testing, and failure analysis.
• Develop observability, logging, tracing, auditability, monitoring, and debugging tools for AI agents, model calls, MCP tools, and ML pipelines.
• Partner with Autonomy, Software, Data, Simulation, Product, and Operations teams to identify high-value AI use cases and turn them into reliable internal tools.
• Secure agentic AI systems end-to-end with least-privilege tool access, sandboxed tool execution, prompt-injection and misuse mitigation, secrets management, human-in-the-loop approvals, and safe handling of sensitive and defense data.
• Maintain documentation, reusable examples, templates, and best practices that help internal teams adopt AI tools safely and effectively.

Benefits

• 100% Employer paid Health, Dental and Vision Insurance for you and your families
• Life Insurance (Employer Paid)
• Ability to participate in the companies 401k program (Matching)
• Unlimited PTO policy with an enforced 2 week minimum
• Equity Package
• Work / Home Office Stipend
• Global Entry
• 16 Week Paid Parental Leave
• Monthly Health and Wellness Stipend

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level