Site Reliability Engineer - Fleet Command & Control

Etched • San Jose, CA

4d • Onsite

About The Position

Etched is hiring one of its first Site Reliability Engineers to build the command-and-control layer for its fleet of frontier inference systems. The role will establish alerting, automated recovery, fleet data, and event routing to keep clusters serving and direct problems to the right experts. The Fleet SRE function owns detection, first response, and handoff, while domain SREs embedded in teams own deeper product fixes. The goal is to detect user-impacting problems, triage them with agents, and provide engineers with relevant logs, dashboards, and recent changes.

The SRE will define system-health signals, alerting standards and SLOs; reduce alert noise; automate guarded recovery; improve tracing and telemetry; and lead incident management. The role also partners with Supercomputing and Infrastructure on fleet telemetry, canaries, rolling deployments, and graceful drains, and reviews service designs for failure modes and graceful degradation. On-call is shared with SREs in other time zones in a follow-the-sun rotation.

This is a hands-on operational role involving incident response and debugging across the serving stack, hardware, network, and facilities layers. The engineer will investigate production symptoms, triage hardware faults on live racks, coordinate on-site technicians, and document evidence and hypotheses for handoffs. Manual fixes should be captured as runbooks and progressively turned into agent-executable tasks.

Etched is a fully in-person team in San Jose (Santana Row). Its technical staff are expected to contribute across engineering and research disciplines as needed.

Requirements

• At least 2 years of experience in site reliability, production engineering, or infrastructure for large-scale production systems; 8 or more years for the San Jose lead seat.
• Experience defining alerting standards adopted by several teams using Prometheus, Grafana, or similar tools, and strong knowledge of SLOs, error budgets, and alert design.
• Real on-call experience for services with external users, including leading incidents and writing blameless postmortems.
• Demonstrated ability to debug production incidents in unfamiliar systems under time pressure using logs, metrics, traces, and core dumps.
• Experience with application and scripting languages such as Python, Go, or Rust.
• Hands-on experience with Linux, containers, Kubernetes, and infrastructure as code such as Terraform or Ansible.
• Fluency with Linux debugging tools such as perf, strace, gdb, dmesg, journalctl, lspci, and tcpdump.
• Comfort working with hardware-related failures, including PCIe device dropouts, ECC errors, power and thermal events, and issues spanning software and physical layers.
• Sound judgment about when automation should act independently and when it should page a person.
• Clear writing and the ability to get five or more teams to agree on shared standards.

Nice To Haves

• Experience operating GPU, accelerator, or HPC fleets.
• Experience with LLM inference serving, including prefill and decode, KV cache, batching, and latency metrics such as time to first token.
• Network debugging from switch to edge, including tracing issues across switches, load balancers, and Cloudflare, and using edge controls such as rate limits to shed or shift traffic.
• Experience with out-of-band management such as BMC, Redfish, or IPMI, and fleet-wide firmware rollouts.
• Experience with data center telemetry, including power distribution, liquid cooling, CDUs, or DCIM.
• Experience with PagerDuty or similar incident tooling, including event correlation and automated remediation.
• Experience working with colocation providers and remote-hands technicians.
• Experience founding or scaling an SRE function at a startup.

Responsibilities

• Define and continuously refine signals that indicate an unhealthy site, rack, node, or chip, and determine which recovery actions can run automatically versus which require a person.
• Establish shared alerting standards and SLOs with teams, focusing on user-facing symptoms and ensuring alert metrics accurately represent the conditions they claim to measure.
• Reduce alert noise by suppressing expected deploy and maintenance alerts, correlating related alerts, and retiring pages that do not prompt action.
• Automate hardware and service recovery, including restarts, node drains, traffic shifts, and spare swaps, with scoped permissions, rate limits, blast-radius checks, and audit trails.
• Classify events, route them to the appropriate domain SRE with context, and dispatch on-site technicians for physical work.
• Partner with Supercomputing and Infrastructure on telemetry signals, establish OpenTelemetry tracing standards, and report uptime and reliability trends.
• Own incident severity levels, channels, communications, postmortems, corrective-action tracking, on-call rotations, handoffs, and onboarding.
• Review service designs for failure modes and graceful degradation, and partner on canaries, rolling node-by-node deployments, and graceful drains.
• Participate in a follow-the-sun on-call rotation with SREs in other time zones.
• Acknowledge incidents, assess user impact, open incidents, and make timely decisions about restarting, draining, rolling back, or escalating.
• Trace latency or throughput regressions through the serving stack to identify bottlenecks.
• Triage live-rack hardware errors, use recovery tools, and determine when a node should be removed from service.
• Correlate distributed traces with deploys, firmware events, host and BMC logs, network data, and facilities telemetry to distinguish software, hardware, coolant, and power issues.
• Investigate discrepancies between alerts and dashboards, including capped metrics, stale exporters, and failed scrapes.
• Guide on-site technicians through reseats, PSU and tray swaps, and leak response.
• Provide clean incident handoffs with timelines, evidence, and current hypotheses.
• Convert manual fixes into runbooks and then into agent-executable tasks.

Benefits

• Medical, dental, and vision packages with generous premium coverage.
• $500 per month credit for waiving medical benefits.
• Housing subsidy of $2,500 per month for those living within walking distance of the office.
• Relocation support for those moving to San Jose (Santana Row).
• Wellness benefits covering fitness, mental health, and more.
• Daily lunch and dinner in the office.
• Unlimited compute budget, subject to ROI justification.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed