Senior Infrastructure Engineer, Developer Platform (Copy)

Vantaca + HOAi•Palo Alto, CA

2h•$200,000 - $260,000

About The Position

HOAi, acquired by Vantaca in fall 2024, is hiring an engineering-focused AI infrastructure and optimization professional to improve the performance and economics of millions of AI tasks running in production. The role owns the systems, experiments, and feedback loops that determine how tasks use AI models, with the objective of maximizing successful task completion per dollar while meeting quality, reliability, and latency requirements. TypeScript is the primary implementation language. The position builds production infrastructure for model execution, routing, retries, fallbacks, structured outputs, observability, evaluations, experimentation, and safe deployment. It benchmarks models and configurations, analyzes production traces, develops task-specific quality measures and evaluation datasets, and turns experimental findings into production improvements. It also designs continuous optimization approaches for prompts, model selection, routing, agent strategies, reasoning budgets, context, tool use, and execution plans. The role works across model providers, prompts, agent architectures, routing strategies, production traces, and optimization systems. It establishes cost, quality, latency, and reliability performance for major workloads; supports controlled production experiments and rollouts; detects regressions; and ensures optimization systems remain safe and aligned with user and business outcomes. The focus is on how models are used and AI workloads are operated, rather than the internals of model-serving infrastructure.

Requirements

Strong software engineering skills in TypeScript and Node.js.
Experience building production systems that use LLMs or other machine-learning models.
Experience operating high-volume or high-reliability services.
Experience designing and running rigorous model evaluations.
Strong understanding of modern LLM APIs and model behavior.
Experience working with production traces, datasets, and experimentation infrastructure.
Ability to reason quantitatively about quality, latency, reliability, and cost.
Strong SQL and data-analysis skills.
Experience designing controlled experiments and interpreting noisy results.
Experience building optimization, search, experimentation, or feedback-loop systems.
Ability to translate experimental findings into production engineering changes.
Sound judgment about when benchmark differences matter and when they do not.
Ability to keep systems reliable while prompts, models, or execution policies change continuously.

Nice To Haves

Python experience for analysis and evaluation tooling.
Experience with model routing, cascading, or agent evaluation.
Experience with LLM-as-judge systems and human evaluation.
Experience with online experimentation and A/B testing.
Experience with bandits, Bayesian optimization, evolutionary search, or other optimization methods.
Experience with automated prompt optimization, reward modeling, or feedback-loop design.
Experience guarding against reward hacking and grader exploitation.
Experience with fine-tuning, distillation, or open-weight models.
Experience with AI observability, tracing, prompt optimization, or context optimization.
Experience with semantic caching or prefix caching.
Experience with high-volume distributed systems, queues, scheduling, rate limits, or provider capacity management.
Experience with internal benchmarking, experimentation, AI infrastructure, or inference platforms.
Experience with CUDA, GPU kernels, vLLM, TensorRT-LLM, or low-level inference optimization.

Responsibilities

Build infrastructure to measure, operate, and improve millions of AI tasks reliably.
Build TypeScript services and libraries for model execution, routing, retries, fallbacks, and structured outputs.
Instrument AI workflows for quality, latency, reliability, token usage, and cost.
Create consistent interfaces across model providers and configurations.
Provide end-to-end traceability across prompts, models, tools, agent steps, retries, and outcomes.
Define durable schemas for tasks, evaluations, experiments, model versions, and production results.
Design systems for safe rollout, rollback, and comparison of model and workflow changes.
Improve reliability and operational efficiency of high-volume AI workloads, and partner with platform and product engineers to support adoption of optimization capabilities.
Build and maintain evaluation suites that represent production workloads and predict real production performance.
Define task-specific quality metrics and acceptance thresholds; build golden datasets from production workloads, human-reviewed examples, edge cases, and synthetic scenarios.
Develop deterministic checks, automated graders, and LLM-as-judge evaluations where appropriate, and calibrate automated evaluations against human judgment.
Build regression tests, sample and analyze production tasks for failure modes, update evaluations as behavior changes, and connect offline evaluation results to online task, user, and business outcomes.
Evaluate models and configurations across providers, model sizes, reasoning effort, prompts, context strategies, tools, agent architectures, retries, fallbacks, execution parallelism, task decomposition, and planning.
Run reproducible, statistically sound experiments connected to production outcomes and produce clear recommendations and deployable changes.
Build and operate systems that continuously search for improved AI configurations, selecting appropriate methods such as manual experimentation, grid or random search, Bayesian optimization, bandits, evolutionary approaches, and production feedback loops.
Build safe optimization guardrails against flawed graders, narrow-set overfitting, segment-level regressions, reward hacking, and hidden operational costs.
Develop candidate-generation and evaluation pipelines, experiment queues, champion/challenger comparisons, promotion and rollback criteria, exploration and exploitation policies, segment-level monitoring, human-review escalation, and versioning for prompts, models, tools, datasets, graders, and optimization policies.
Establish quality, cost, latency, and reliability trade-offs for major workloads; reduce unnecessary context, tokens, agent steps, tool calls, retries, and model calls.
Optimize model selection, routing, reasoning and token budgets, prompts, caching, reuse, cascades, fallbacks, parallelism, provider selection, workload placement, and operational overhead.
Assess fine-tuning or distillation against frontier-model APIs where appropriate, with the objective of minimizing cost per successful task while preserving required quality and reliability.
Turn experimental results into measurable production improvements and build dashboards for quality, success rate, latency, reliability, and AI economics by workflow, model, configuration, and customer segment.
Run controlled production experiments and A/B tests, roll out model migrations and configuration changes safely, detect regressions after model or provider updates, and establish automated gates for model, prompt, tool, and agent changes.
Monitor optimization systems for reward hacking and grader exploitation, identify divergence between production behavior and offline evaluation, and partner with product and engineering teams on the value of additional model capability.
Quantify optimization results in successful tasks, user outcomes, and spend.
Use production traces to identify failure modes, generate candidates, validate changes through evaluations and controlled rollouts, and use production feedback to inform subsequent iterations.

Benefits

Medical, dental, and vision insurance effective day one.
Unlimited PTO, with a requirement to take at least one continuous week per year.
401(k) plan with company match.
Remote-flexible work, with office attendance when needed.
Parental leave benefits.
Ongoing industry and professional development trainings.
Health Savings and Flexible Spending Accounts.
Life and short-term disability insurance at no cost to employees.
Telemedicine through Teladoc at no cost to employees.
Employee Assistance Program with 24/7 access to counseling, financial planning, legal services, and family support resources.