Senior Incident Commander

Box•Redwood City, CA

1d•$187,000 - $233,500•Hybrid

About The Position

Box is looking for a Senior Incident Commander with a strong Site Reliability Engineering (SRE) skillset and proficient Python for tooling and automation. The GTOC (Global Technical Operations Center) team protects the customer experience, and responsible for detecting customer-impacting failures early, drive rapid mitigation when they occur, coordinate production change risk, and continuously improve how Box runs incidents. In this role, you will be leading critical and blocker incidents and driving them to swift resolution, while leveraging your software engineering expertise to design and build the tools, automation, and processes that define the next generation of cloud operations. You will shape how we improve manageability, observability, and resiliency of our most critical services, and drive down MTTx so that when incidents happen, we respond faster and smarter than ever before.

Requirements

5+ years in SRE, production operations, reliability engineering, or equivalent high-scale internet/SaaS operations with repeated experience leading or co-leading major production incidents.
Demonstrated Incident Commander / Technical Duty Officer (or equivalent) skill: calm under uncertainty, clear communication, ability to delegate, and judgment on when to escalate vs. when to dig deeper.
Strong SRE fundamentals: SLIs/SLOs and error budgets (practical use), observability (metrics, logs, traces), golden signals, blameless postmortems, toil reduction, and “you build it, you run it” partnership with product teams.
Proficient in Python for automation and tooling (not just one-off scripts): readable code, APIs, packaging or service-style tools others can run, and comfort reviewing others’ automation.
Solid Linux/Unix troubleshooting; comfort in multi-tier distributed systems (app, data, network, cloud).
Networking literacy sufficient for real incidents (DNS, TLS, load balancing, HTTP, basic routing/firewall concepts).
Experience with cloud environments (GCP preferred; AWS/Azure also valued) and container/orchestration concepts (Kubernetes or equivalent).
Excellent written and verbal communication: exec-ready impact statements, precise Slack/bridge facilitation, and documentation that others can trust under stress.
Proven ability to coach and uplift others — training, mentoring, runbook quality, drills.

Nice To Haves

Experience in a 24x7 NOC / GTOC / follow-the-sun operations center, or partnering closely with one.
Hands-on with Prometheus-compatible observability (e.g. Chronosphere, Grafana), distributed tracing(e.g. SignalFx), synthetic monitoring (e.g. Catchpoint), and PagerDuty (or similar).
Experience improving incident tooling (ChatOps, workflow automation, status/comms templates, drill frameworks).
Familiarity with change management under pressure (emergency changes, break-glass, CAB/async approval patterns).
Additional languages or tooling useful for ops automation (Go, shell, Terraform, CI/CD).
Prior work mapping dependency graphs, service catalogs, or Tier 1 journey documentation.

Responsibilities

Own and direct live-site Critical and Blocker (and other high severity) incidents from identification and escalation through mitigation and recovery.
Triage, refine, and verify the problem and customer-impact statements. Organize the incident bridge, establish clear swim lanes, coordinate SME’s resources, and lead cross-functional Incident Bridges to mitigate issues and restore service quickly.
Improve Incident Platform tooling - Optimize templates, automate repetitive response steps, and develop tooling to reduce operational overhead and time to mitigate.
Partner with SRE and engineering teams to deepen shared knowledge of Box's dependencies, Tier 1 journeys, failure modes. Work together to implement secure, automated workflows that close operability gaps.
Lead daily change reviews of planned changes in Jira, partnering collaboratively with engineering teams to evaluate and minimize change risk.
Provide day to day technical expertise and experience to the organization to address issues in globally diverse, 24x7 environments - influencing from policy and procedural decisions to key architectural and tooling insights to improve Box's Incident, Change, and Problem Management engineering capabilities.
Lead projects to improve tools and processes related that enhance overall site resiliency, service manageability and observability.
Represent GTOC/NOC in problem management, readiness reviews, and reliability in cross functional forums.
Turn incident learnings into actionable engineering asks (such as SLOs, signal quality, rollback readiness, dependency docs) to reduce customer impact.
Leverage observability APIs, PagerDuty/Jira, and internal services to make response workflows measurable and repeatable.
Continuously mentor and uplift the team by running tabletop exercises and improving runbooks.