Head of Platform Reliability

Cielo Projects•New York, NY

7d•$165,000 - $210,000•Onsite

About The Position

Quant is seeking a Head of Platform Reliability to join their team. This role will be responsible for the reliability of the production environment for The Clearing House's On-Chain Money Initiative, which will allow US financial institutions to clear and settle tokenized deposits on chain. The responsibilities include managing the full production environment (OP Stack, a private Besu L1, Paladin nodes, the Overledger Gateway, the Settlement Bridge and the Audit Store), defining service levels and error-budget policy, and having the authority to stop a release when the budget is spent. The role also involves building and leading a team to run this environment on a three-shift rotation designed and participated in by the Head of Platform Reliability. The position requires an understanding of how chain platforms fail differently, considering that state is expensive, restarts are not a strategy, and much of what appears to be infrastructure is actually protocol behavior.

Requirements

Experience leading site reliability or production engineering where failure had significant consequences.
Experience running error budgets in practice, including using one to stop a release and defending that decision.
Honest experience of 24x7, multi-shift coverage, understanding what causes burnout and what constitutes a sustainable rotation.
Proficiency in Kubernetes and Terraform.
Experience debugging an observability stack, not just configuring it.

Nice To Haves

Experience with blockchain nodes in production, particularly Besu or another Ethereum client.
Experience in banking, payments, or a regulated environment where an auditor required proof of operations.

Responsibilities

Own the reliability of the full production environment, including OP Stack, a private Besu L1, Paladin nodes, the Overledger Gateway, the Settlement Bridge, and the Audit Store.
Define and manage service levels and error-budget policy.
Exercise authority to stop a release when the error budget is spent.
Build and lead the team responsible for running the production environment.
Design and participate in a three-shift rotation for 24x7 coverage.
Address the unique challenges of chain platform reliability, including state costs, restart strategies, and protocol behavior.