Principal Software Engineer - Agentic SOC

Huntress

1d•$215,000 - $240,000•Remote

About The Position

Huntress is building an Agentic SOC, a system where an LLM-based agent investigates security signals end-to-end. This agent gathers telemetry, reconstructs events, and determines if activity is malicious. The goal is to augment human analysts by handling high-volume, repetitive tasks, allowing them to focus on complex cases. The system will also pre-process tool outputs to provide analysts with a clearer picture. This is a critical system that investigates numerous signals daily across thousands of organizations, including live intrusions where speed is essential. The agent must be accurate, provide auditable reasoning, and know when to escalate to a human analyst. The Principal Software Engineer will lead the architecture and direction of this system, deciding on investigation structures, the balance between deterministic code and model reasoning, agent capabilities, performance measurement, and autonomous decision-making. Collaboration with SOC analysts, detection engineers, product, and other engineering teams is key. The role involves deep coding and mentoring other engineers, directly shaping Huntress's scaling of protection. AI coding tools are encouraged to accelerate development, with no token usage caps, but responsible use is expected.

Requirements

15+ years of experience developing complex software products, including significant time as the technical lead on production systems.
Hands-on experience designing, shipping, and operating LLM-based agents in production, covering tool use, context management, structured output, guardrails, and evaluation, along with a clear-eyed understanding of how they fail.
A track record of building evaluation for non-deterministic systems, such as labeled datasets, offline evals, and regression gates, and of measuring quality over time rather than at launch.
A security mindset: experience building systems that handle untrusted input, enforce multi-tenant isolation, and produce an audit trail that holds up under scrutiny.
Experience building systems that make automated decisions with real consequences, and the judgment to know what to automate and what to leave to people.
Curiosity about how security analysts think and work, and a willingness to spend real time learning from them.
Deep backend expertise in distributed systems, queues, and durable workflows, and concurrency, with a track record of designing for throughput and correctness under load.
Strong skills in one or more backend languages and the ability to pick up new ones quickly. Our primary stack is Ruby on Rails, and we expect you to become fluent in it.
Experience with AWS, Azure, or other public cloud environments, and with data stores such as Postgres and Redis.
Experience with AI coding tools, such as Claude Code.
BS or MS in Computer Science or Engineering, or equivalent experience.

Nice To Haves

Experience working in or building for a SOC, incident response, threat hunting, or detection engineering.
Familiarity with endpoint and identity telemetry and with attacker tradecraft.
Experience attacking or adversarially testing ML or LLM systems.
Experience with Ruby on Rails.

Responsibilities

Own the architecture of agentic investigation, defining signal flow, state management, tool exposure to agents, and the components of a finished investigation (verdict, timeline, scope, evidence).
Build agents that emulate strong analyst behavior by learning triage and investigation processes from SOC and product researchers.
Develop LLM-powered pre-processing for analyst tools and data sources to summarize, correlate, and highlight key information, reducing manual assembly.
Incorporate a security mindset into the agent's design, considering untrusted input, multi-tenancy, and auditability due to the agent's access to attacker-influenced data and customer environments.
Treat evaluation as a core product feature, defining metrics for investigation correctness, reasoning, and documentation, and ensuring changes are evidence-based.
Use measured performance data to determine the system's autonomous actions, human review points, and how these evolve over time.
Design for production reliability, latency, and cost at scale, ensuring the system functions even when models are slow, incorrect, or unavailable, and that all actions are traceable.
Design the agent-analyst handoff to enhance analyst speed and information, considering how analyst roles evolve and how to build trust with partners and customers regarding agent decisions.
Prototype solutions for complex and uncertain problems, derisking approaches before handing them off.
Serve as the technical lead for the Agentic SOC, representing the team's technical perspective to leadership, other teams, and executives, driving alignment, and managing scope.
Balance technical rigor with pragmatism, pushing back against risky shortcuts and over-engineering, while favoring action and feedback.
Improve the team's overall performance through thoughtful code reviews, pairing, and creating leverage via better patterns, tooling, and shared understanding, partnering with the engineering manager on team health and feedback.