Principal Engineer Capacity/Compute

Anthropic•San Francisco, CA

5d•$320,000 - $485,000•Hybrid

About The Position

Anthropic is hiring a hands-on engineer for its Capacity Engineering team, which makes sure the company’s infrastructure resources are accounted for, well-utilized, and efficiently allocated across accelerator and CPU families and multiple cloud providers. The team builds the data, tooling, and operational systems used to plan, measure, and maximize utilization across first-party and third-party compute. The role involves building production systems for telemetry ingestion and normalization, fleet observability, and workload performance instrumentation. The engineer will work with Kubernetes-native infrastructure at meaningful scale and influence decisions involving a major area of company spending. Work spans data engineering, systems engineering, and observability, in a high-autonomy and high-ambiguity environment. Depending on background and business priorities, the primary focus may be data platform, planning, efficiency, or attribution and forecasting, with fluid boundaries between these areas. The position collaborates with research engineering, infrastructure, inference, and finance teams. It includes gathering requirements, defining data contracts, designing for internal consumers, and owning reliable, load-bearing systems with appropriate on-call coverage and service-level objectives.

Requirements

A strong track record building and operating production systems; this is a hands-on engineering role with a DevOps component.
Production-quality Python and SQL skills, including idiomatic, well-tested, maintainable code and BigQuery SQL involving table-valued functions and views.
Deep experience operating at least one major cloud provider: Amazon Web Services, Google Cloud, or Microsoft Azure.
Experience with observability tooling, including Prometheus, PromQL, and Grafana; ability to write recording rules and build monitoring relied upon by engineering teams.
Ability to gather requirements independently and work across organizational boundaries in an ambiguous environment with limited direction.
A bachelor’s degree or an equivalent combination of education, training, and experience.
A field of study relevant to the role, demonstrated through coursework, training, or professional experience.

Nice To Haves

Experience with capacity planning, resource management, or cost attribution systems at a hyperscaler or in a large-scale machine-learning environment.
Product engineering or developer experience, including experience making internal systems useful to their consumers.
Scheduling and packing efficiency experience, or profiling-driven optimization of large distributed workloads.
Multi-cloud data ingestion experience, particularly normalizing billing exports, reservation APIs, on-demand capacity reservations, commitments, and vendor telemetry across providers with different billing arrangements.
Total cost of ownership and forecasting experience, including analyzing whether infrastructure growth is causal or correlated with business drivers.
Familiarity with accelerator infrastructure, such as GPU metrics (DCGM), TPU utilization, Trainium power and utilization metrics, or machine-learning training and inference systems at the hardware level.
Experience building internal data products with self-service access, schema contracts, API serving, documentation, and discoverability.
Experience with storage efficiency, retention, and lifecycle programs at exabyte scale.

Responsibilities

Build the planning and allocation stack used by leadership to allocate capacity, teams to plan against allocations, and schedulers to enforce allocation decisions; support cross-region and cross-provider placement, guardrails, queueing, and occupancy KPIs.
Drive efficiency programs, including stranding and rightsizing, unused-capacity recovery, and job-level utilization across training, inference, and evaluation workloads.
Establish per-configuration utilization baselines and work with system-owning teams to improve performance.
Own attribution and forecasting by reconciling billing across more than ten providers against telemetry and internal systems, attributing spending to workloads, and translating demand signals and research roadmaps into compute and supply plans.
Build and own data pipelines that ingest occupancy, utilization, and cost data from a diversifying fleet into BigQuery, with responsibility for completeness, latency SLOs, and gap detection.
Integrate data from new cloud providers and normalize their billing and usage information.
Operate Kubernetes-native systems at scale, including collection agents, workload labeling, and taint, reservation, and scheduling behavior that determines usable capacity.
Gather requirements, define schema contracts, and design data products for consumers ranging from research engineers to finance and company leadership.
Provide on-call support and maintain SLOs for critical data and operational systems.