Senior AI Infrastructure Engineer

Hellyeah AI•San Francisco, CA

4d•Onsite

About The Position

Hellyeah AI is hiring two Senior AI Infrastructure Engineers, one in San Francisco and one in Shanghai. The role owns the platform that Hellyeah’s products and production agents run on, including the model gateway, agent runtime, clusters, data stores, pipelines, secrets, and release paths. One engineer will lean toward the AI platform and the other toward the core platform. They share ownership and on-call responsibilities and work closely with the CTO. The position is onsite five days a week; on-call includes nights and weekends, and workdays often exceed 10 hours. The AI platform work includes operating a multi-provider model gateway and production agents; managing routing, failover, rate limits, rollouts, tenant access, usage metering, and provider-bill reconciliation; maintaining shared coding-agent access and remote workspaces; and building safe autonomous infrastructure workflows. The company runs production AI across text, speech, and video, primarily using third-party model APIs with some self-hosted serving. It does not train models. The core platform work includes operating Kubernetes clusters and cloud infrastructure across AWS, Azure, and GCP; managing Cloudflare services; shipping services through GitOps; leading migrations; operating databases and data pipelines; and maintaining reliability, observability, and infrastructure security. The role also builds paved roads for product teams, reviews infrastructure changes, and shares runbooks and agent skills. Product feature development, IT support, marketing sites, and SEO are outside the role.

Requirements

At least 5 years of experience running production infrastructure, including real on-call experience.
Deep Kubernetes debugging experience, including stateful workloads, and hands-on experience running Kubernetes on bare metal, including storage, networking, upgrades, and node failures.
Hands-on production experience with LLM APIs, including routing, rate limits, streaming, failure modes, and cost.
Experience delegating real operational work to agents or automation, with guardrails, approvals, and recovery when a run fails.
Enough Go or Python experience to read an unfamiliar codebase and ship a fix.
Hands-on observability experience: maintaining Grafana, Prometheus, and OpenTelemetry collectors; writing metric and log queries; and building useful dashboards.
Production experience with GitOps and Terraform, ideally across more than one cloud.
Networking fundamentals, including DNS, TLS, load balancing, WAF, and routing.
Experience taking a third-party service to production end to end, including SSO, backups, and upgrades.
Experience leading a region, cloud, or database migration with a tested rollback.
Experience reducing a real cloud or AI bill and ability to show the results.
Security judgment covering least privilege, secret handling, credential rotation, and safe rollouts.
Daily use of coding agents and clear views on what they should not be allowed to touch without review.
Familiarity with the current agent landscape, including Hermes Agent, OpenClaw, open-source Codex, and Vercel's eve, and ability to explain agent harness best practices such as durable runs, sandboxing, scoped credentials, human approval, memory, tracing, and evaluations.
Confidence to reject risky changes and explain why, including to a founder.
Written English proficiency for daily work with teammates in the US, China, and India.

Nice To Haves

Experience with LiteLLM or a similar LLM gateway.
LLM serving experience, such as SGLang or vLLM.
Experience with multi-tenant API platforms, including metering and billing.
Data pipeline experience, such as Airbyte, Airflow, or CDC into BigQuery.
Experience with realtime audio or WebSocket services.
Agent sandboxing experience, such as gVisor, Landlock, or agent-sandbox.
Experience with Langfuse or other LLM tracing tools.
Experience with remote development environments, such as Coder.
FinOps experience across several clouds.
Mandarin.

Responsibilities

Operate the model gateway, including routing, fallbacks, provider failover, rate limits, and rollouts for more than 90 model deployments across OpenAI, Anthropic, Azure, Google, and AWS Bedrock.
Benchmark model providers on latency, quality, cost, and failure behavior before directing production traffic to them, and escalate vendor issues.
Make the gateway multi-tenant by managing team and customer keys, routes, limits, usage metering, and reconciliation of usage against provider invoices.
Operate the AI developer platform, including shared coding-agent seats and quotas, per-user limits, and remote engineering workspaces.
Run production agents safely on Kubernetes through sandboxing, credential scoping, and pinned releases.
Build autonomous infrastructure workflows and agents for routine requests and alerts, with human approval for risky actions.
Manage infrastructure and AI spending by reconciling bills, attributing spending to teams and products, detecting anomalies, right-sizing clusters, using vendor credits, and handling quota and billing escalations.
Debug and upgrade EKS, AKS, and bare-metal k3s using Argo CD and Kustomize; manage AWS, Azure, and GCP through reviewed Terraform.
Operate Cloudflare across approximately a dozen domains, including DNS, WAF, TLS, CDN, object storage, GeoDNS, load balancers, egress IPs, and email authentication.
Deploy services through GitOps, including manifests or Helm, pinned images, Postgres and Redis, SSO, DNS, TLS, secrets, and backups; provide high availability when needed and assign each service a named owner.
Build production infrastructure for new product launches and solve infrastructure blockers by reading source code, patching or working around issues, or building missing tools.
Lead migrations across regions, clouds, platforms, and databases with tested rollbacks; retire obsolete resources and bring unmanaged services under GitOps.
Upgrade and migrate production PostgreSQL, MySQL, ClickHouse, and Redis; manage safe schema changes and verify backups through restoration.
Operate Airbyte, Airflow, Dataform, and CDC pipelines into BigQuery, keeping product and finance data fresh and identifying stalled syncs.
Set SLOs and trustworthy alerts, lead infrastructure and product-service incidents, including realtime speech incidents, write incident reviews, and automate recurring triage.
Maintain observability across clusters and the gateway using Grafana, Prometheus, Loki, Datadog, and OpenTelemetry; manage telemetry, retention, and cost.
Secure infrastructure by rotating credentials, replacing static keys with workload identity, finding and responding to exposed secrets and vulnerable dependencies, and controlling access to customer data.
Make access self-service through SSO, workload identity, and scoped secrets; own OAuth clients and API apps on advertising and commerce platforms, including their reviews.
Build release pipelines, canaries, preview environments, and guardrails that enable product teams to ship and operate their own services.
Review infrastructure changes from engineers, founders, and their agents, and share runbooks and agent skills.
Join on-call, run an incident review, take a new model from benchmark to production, reconcile a month of gateway usage against provider invoices, reduce manual access requests by half, put a recurring request or alert under agent handling with human approval for risky actions, and ensure a second person can operate each critical system during the first 90 days.