Staff Software Engineer, OTA Services

Horizon3

1d•$229,500 - $260,000•Remote

About The Position

Horizon3 is hiring a Staff Software Engineer for its Over-the-Air (OTA) Services team. The role is the technical anchor for how the NodeZero platform updates across thousands of customer deployments without customer intervention or downtime and while avoiding new vulnerabilities. NodeZero runs as a container in customer infrastructure, and the OTA system must locate deployed instances, negotiate and deliver updates safely, validate them, and roll back when needed. The system must operate across diverse customer environments, including air-gapped and restricted networks. This is a hands-on staff-level position involving production coding, architecture design, technical direction for update delivery and version management, and mentorship of P4 engineers. The OTA Services team partners closely with the Image Build & Release team, which produces the artifacts delivered by OTA Services. The position reports to the Client Platform Engineering Manager. Horizon3 is a remote-first cybersecurity company. The role may require up to 5% travel, and employees need a consumer-grade broadband connection of at least 25 Mbps.

Requirements

8 or more years of professional software engineering experience, with deep specialization in distributed systems, remote update systems, or fleet management infrastructure.
Deep understanding of remote update system design, including delivery protocols, version negotiation, delta delivery, health verification, and rollback semantics.
Experience designing rollout strategies for software deployed across heterogeneous, customer-managed environments.
Strong proficiency in Go, Python, or Rust for systems-level infrastructure work.
Experience building software for constrained or air-gapped environments and designing delivery systems that do not assume reliable connectivity.
Deep understanding of Kubernetes, including how deployments are managed, updated, and monitored in running clusters.
Experience with container image distribution, including OCI artifacts, image manifests, layer caching, and registry protocols.
Strong understanding of mTLS, certificate lifecycle, and secure channel establishment for update delivery.
Track record of setting engineering standards adopted by other engineers, including protocol designs, reliability requirements, and distributed-systems testing standards.
Ability to produce clear architecture documents and drive technical decisions across team boundaries, particularly with the Image Build team.
Experience mentoring mid-level engineers through design reviews and code reviews.

Nice To Haves

Experience with Kubernetes operators or controller patterns.
Background in cybersecurity or experience building software for regulated or high-security environments.
Familiarity with Fleet or similar open-source fleet management tools, including understanding when custom infrastructure is preferable.
Experience with signed artifact delivery and end-to-end chain-of-custody verification.

Responsibilities

Design and own the over-the-air update system, including delivery protocols, version negotiation, rollout orchestration, health verification, and automatic rollback.
Own deployment fleet management systems that track the version state of every NodeZero deployment, including what is running, pending, or stuck.
Design safe rollout strategies, including phased rollouts, canary deployments, customer pinning, and emergency hold mechanisms.
Design delivery mechanisms that work across diverse customer environments, including air-gapped networks, strict egress policies, and minimal-footprint deployments.
Own OTA system reliability end-to-end and participate in on-call response for delivery failures, stuck updates, and version drift incidents.
Mentor P4 engineers through design reviews, code reviews, and technical pairing.
Write production code, produce architecture designs, and set technical direction for update delivery and version management.