Principal Engineer - Agentic Security Operations (remote)

First American•Santa Ana, CA

1d•Remote

About The Position

Our Security Operations Center (SOC) protects the enterprise by detecting, investigating, and responding to cybersecurity threats. As a member of our Threat Detection Engineering team operating within our SOC, you will build governed agent capabilities to support data onboarding, detection engineering, threat hunting, investigation, response, automation, and platform operations. Our goal is to expand threat visibility, accelerate security outcomes, reduce repetitive work, and improve operational consistency and resilience without compromising sound security judgment, human accountability, or control. The Principal Engineer provides hands-on technical leadership to turn this strategy into secure, reliable, and supportable production capabilities across the SOC. We are open to remote candidates for this role.

Requirements

Extensive experience designing and operating enterprise security, software, data, or distributed systems at scale.
Demonstrated success as a principal-level technical leader while remaining actively involved in architecture, prototyping, coding, testing, and production delivery.
Hands-on experience delivering production AI-assisted or agentic systems that use tools, memory, retrieval, planning, and structured workflows.
Strong understanding of security operations, including telemetry management, detection engineering, threat hunting, triage, investigation, response, and automation.
Experience with high-volume streaming and batch data pipelines, object-based storage, elastic analytics, distributed query, schema management, and data-quality controls.
Experience designing modular agents, reusable capabilities, workload routing, local and managed execution, and human-in-the-loop workflows.
Deep understanding of grounding, retrieval, source attribution, context freshness, permission inheritance, and protection against untrusted or manipulated context.
Experience evaluating nondeterministic systems using versioned test sets, historical replay, synthetic scenarios, regression testing, and measurable outcome criteria.
Strong software-engineering practices, including source control, automated testing, continuous delivery, infrastructure automation, peer review, and rollback.
Experience designing secure APIs, event-driven integrations, workload identities, authorization boundaries, secrets management, and auditable action execution.
Knowledge of AI security risks such as prompt injection, data poisoning, excessive agency, sensitive-data exposure, insecure tool use, and model or workflow drift.
Strong production-engineering experience covering observability, service objectives, capacity, failure isolation, continuity, recovery, and operational support.
Ability to evaluate emerging capabilities critically, distinguish demonstrated functionality from product claims, and validate architectural assumptions through controlled testing.
Excellent written and verbal communication skills, including the ability to document architectures, operating models, engineering standards, and executive recommendations.
Ability to lead through influence and work effectively across organizational and technical boundaries.
Hands-on experience with cloud-scale data and analytics platforms supporting security operations, including distributed query, high-volume data processing, object storage, schema management, data pipelines, and the operational patterns required to run security analytics at scale (e.g., Databricks, Snowflake, etc.)
Experience operating and migrating security analytics and detection workloads across modern platforms, along with strong working knowledge of major public cloud environments (AWS, Azure, GCP) and the identity, networking, logging, data protection, and workload-security controls required to secure them.
A minimum of 10 years’ experience in cybersecurity or relevant fields.
Bachelor’s degree in computer science, Information Technology, Cybersecurity, a related field, or equivalent work experience.

Nice To Haves

Preferred certifications such as, GASAE, GAIPS, GDSA, GCDA, CEH, OSCP, CISSP, etc.
Experience modernizing a centralized security analytics environment into a decoupled architecture with customer-controlled data and replaceable compute or control layers.
Experience with knowledge graphs, entity resolution, semantic normalization, retrieval-augmented systems, or multi-agent coordination.
Experience migrating security content while validating semantic equivalence, historical fidelity, and operational parity.
Background in detection engineering, incident response, threat hunting, adversary emulation, or security automation.
Experience defining governance and autonomy frameworks for AI-enabled production systems.
Experience building internal platforms or shared engineering services used by multiple security functions.

Responsibilities

Lead the architecture and hands-on implementation of Agentic Security Operations capabilities across the SOC.
Define and evolve the reference architecture for governed data services, agent capabilities, workload routing, execution environments, and deterministic orchestration.
Build reusable capabilities that translate operational intent into tested, reviewable, and auditable security outcomes.
Establish engineering standards for creating, testing, registering, approving, releasing, monitoring, and retiring agents and their capabilities.
Advance detection-as-code practices, including versioning, simulation, deployment, tuning, migration, and rollback.
Build governed knowledge and context services that provide authoritative documentation, policies, operational history, citations, freshness information, and permission-aware retrieval.
Design workload identity, least-privilege access, short-lived authorization, spend controls, approval gates, audit trails, rollback, and emergency shutdown mechanisms.
Define autonomy levels that determine which activities may execute automatically, require human approval, or remain analyst led.
Establish agent evaluation and observability through historical replay, synthetic scenarios, expected-versus-actual comparisons, policy tests, traces, outcome verification, and regression testing.
Measure quality, security, reliability, latency, cost, analyst acceptance, and operational impact.
Engineer reliable integrations across search, case management, workflow, communications, and response systems, including retries, idempotency, partial failures, and reconciliation.
Ensure consequential response actions remain deterministic, attributable, reversible, and subject to appropriate human oversight.
Design production reliability, continuity, recovery, and support models for agent capabilities and security data services.
Lead proofs of concept and architecture evaluations using measurable criteria for security, fidelity, scale, reliability, portability, operating effort, and economic value.
Collaborate with enterprise architecture, infrastructure, data, risk, privacy, legal, and compliance teams.
Mentor engineers, establish technical standards, review designs, and influence Threat Detection Engineering strategy.
Present complex technical decisions, risks, tradeoffs, and investment recommendations clearly to engineering and executive stakeholders.
Partner across Threat Detection and Response groups, including engineering, threat hunting, security analysis, investigation, and incident response stakeholders on initiatives and to translate operational feedback into improvements within our agentic harness, evaluations, safeguards, workflows, and other platforms.