Member of Technical Staff, AI Infrastructure Reliability

Fireworks • San Mateo, CA

1d

About The Position

Fireworks AI is a platform for specialized intelligence that enables companies to build, train, and serve AI models tailored to their data, workflows, and products. The company serves more than 40 trillion tokens a day across open-model workloads. Reliability Engineering helps ensure the platform remains dependable as it grows.

In this role, you will work across cloud infrastructure, AI systems, and product teams to make sure the platform’s components fit together, fail gracefully, and withstand load. You will define what reliability means at Fireworks—including SLOs, error budgets, production readiness, and on-call expectations—and drive adoption across engineering. You will own reliability processes and tooling end to end while enabling every team to take practical ownership of the reliability of its systems.

The role has broad scope and latitude to focus on the work that most improves outcomes. It includes making telemetry, alerting, and dashboards consistent and useful; addressing customer-impacting failures that may not be visible in per-service SLOs; finding and resolving failures at system boundaries; coordinating incidents and postmortems; and reducing operational toil through automation. You will partner with infrastructure, inference and training, performance, product, and control-plane teams.

Requirements

• At least 5 years of experience with Linux internals, system performance troubleshooting, and networking fundamentals, including TCP/IP, HTTP, and gRPC.
• At least 5 years of software engineering experience in Python, Go, C++, or Rust, including production-grade tools and systems code.
• Experience operating and debugging Kubernetes, Terraform, and Docker in high-throughput production environments.
• Experience with distributed systems such as high-throughput control planes, microservices, or multi-region setups.
• Knowledge of fault-tolerant design, SLO/SLA management, automated failover, and high-availability architecture.
• Ability to influence other teams to adopt standards through credibility and useful tooling rather than mandates.
• Willingness to investigate unfamiliar parts of the stack when problems cross boundaries.
• Bachelor's or Master's degree in Computer Science or Computer Engineering, or equivalent practical experience.

Nice To Haves

• Experience with observability tools such as Prometheus, Grafana, and OpenTelemetry, and with actionable alerting.
• Exposure to GPUs, inference serving, or distributed training.
• Experience building agents or LLM-based tools for investigation, triage, or automation.
• Contributions to open-source infrastructure, systems, or ML-serving projects.
• Comfort working in a startup environment where pragmatism and teamwork take precedence over process.

Responsibilities

• Define reliability standards, including SLOs, error budgets, and production-readiness criteria, informed by system instrumentation and real telemetry.
• Own reliability tooling, including logging and telemetry pipelines, alerting standards, failure injection, load testing, self-healing automation, and AI-assisted investigation tools.
• Ensure customer-impacting reliability issues are assigned owners and resolved, including problems not captured by individual service SLOs.
• Identify failures at system boundaries, such as load-amplifying retries, incompatible timeouts, and unmapped dependencies, and drive fixes with the teams responsible.
• Coordinate live production incidents, lead blameless postmortems, and track follow-up actions through completion.
• Automate repetitive operational work to reduce toil and keep on-call demands sustainable as the company grows.
• Partner with cloud infrastructure teams on capacity and multi-region risk.
• Partner with inference and training teams on failure modes in serving and training systems.
• Partner with performance teams on zero-downtime rollouts.
• Partner with product and control-plane teams on customer-facing reliability.

Benefits

• Opportunity to work on challenging AI infrastructure problems, including low-latency inference and scalable model serving.
• Opportunity to work with cutting-edge technology that supports businesses and developers globally.
• Ownership and direct impact in a fast-growing team, with an emphasis on results over bureaucracy.
• Collaboration with experienced engineers and AI researchers.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior