Staff Engineer, Command Center Insights & Actions

Crusoe•San Francisco, CA

13d•$210,000 - $255,000

About The Position

Crusoe is seeking a Staff Engineer to serve as the detection authority for its Command Center platform. This role involves defining the heuristics, thresholds, and rules that power alerting and anomaly detection systems, transforming raw infrastructure telemetry into precise, actionable signals. It is a full software engineering position where the individual will ship customer-facing features, build systems from the ground up, and scale existing services. The ideal candidate will possess deep expertise in anomaly detection, heuristics, and machine/reinforcement learning applied to real infrastructure at a global scale. Beyond technical contributions, this role requires being a cultural bar-raiser and force multiplier, elevating the team's overall craft. The position reports to the Engineering Manager of the CCIA team and operates as a peer to the Distributed Systems Staff Engineer.

Requirements

Deep experience building anomaly detection systems, heuristics-based rule engines, or ML/RL systems for infrastructure or data-intensive domains.
Demonstrated ability to reason about precision/recall trade-offs and build feedback loops that keep detection systems accurate over time.
Strong foundations in the building blocks of reliable, scalable backend systems.
5+ years shipping production software.
Experience with modern compiled or systems languages (Go, Rust, C++, Java, or similar).
Comfortable with time-series data, telemetry pipelines, and observability primitives.
Ability to explain detection logic, trade-offs, and system behavior clearly to both engineers and non-technical partners.
A force multiplier mindset: making the team better through mentorship, clear technical vision, and investment in people.

Nice To Haves

Experience with GPU profiling tools (Nsight, NCCL Inspector) or hardware-level infrastructure diagnostics.
Background in observability platforms or products.
Experience with reinforcement learning applied to operational or infrastructure problems.
Familiarity with large-scale fleet management or cloud infrastructure.
Passion for building team culture and engineering quality of life.

Responsibilities

Own the full detection stack, including heuristics, threshold calibration, precision/recall tuning, and rule systems that define "something is wrong" for the platform.
Design and maintain anomaly detection systems such as straggler node detection, GPU health signals, and fleet-level behavioral baselines.
Drive detection fidelity by reducing false positives, increasing signal coverage, and building feedback loops for accurate threshold maintenance as the fleet grows.
Evaluate and integrate machine learning and reinforcement learning techniques where they outperform rule-based approaches, understanding when not to use a model.
Ship customer-facing features end-to-end across the CCIA stack, including the alert rule engine, control plane APIs, automated action systems, and insights delivery surfaces.
Build new systems from scratch and scale existing ones to support Crusoe's rapidly growing global fleet.
Collaborate with product counterparts to shape requirements early and partner with the data science team to develop and validate detection models.
Participate in design discussions across teams, contribute architectural perspective, and help evaluate technical trade-offs.
Mentor engineers at all levels through code review, design feedback, and direct coaching.
Contribute to hiring by helping define what great looks like.