Software Engineer, AI for Incident Response (Technical Leadership)

Meta•Menlo Park, CA

3d•$219,000 - $301,000

About The Position

This is a deeply hands-on technical leadership role at the intersection of large-scale distributed systems, incident response, applied AI, and product engineering. The role sets technical direction and drives execution for an AI agent that gathers context, investigates incidents, identifies likely root causes, develops safe mitigation plans, and executes them while humans supervise and can intervene. The goal is to build an accurate, trusted agent that supports expert incident responders as a transparent collaborator and guides less frequent responders toward safe outcomes. The role leads two connected areas: agent quality—including skills, tools, context, memory, offline evaluation, backtesting, online experimentation, and learning from production outcomes—and responder experience, including a multi-turn interface that makes the agent's work visible and brings together operational data for decision-making. The engineer will personally design and ship critical system capabilities while leading senior engineers across team and organizational boundaries. The long-term opportunity is to help create self-healing infrastructure that safely resolves production incidents at Meta scale.

Requirements

Bachelor's degree in Computer Science, Computer Engineering, a relevant technical field, or equivalent practical experience.
10+ years of software engineering experience, including building and operating large-scale distributed, infrastructure, or production AI systems.
Experience setting technical direction and leading complex engineering efforts across multiple teams or organizational boundaries.
Experience applying AI to production problems at scale, including evaluating system quality and failure modes.
Recent hands-on experience designing, coding, debugging, and shipping complex production systems.
Experience building systems that support critical, always-on production workloads.
Experience diagnosing complex production systems using telemetry, code, configuration, deployments, and dependency information.
Experience translating customer needs into technical architecture and driving adoption of a platform or developer product.
Experience coding in C++, Java, Python, Rust, or equivalent languages.
Experience influencing senior engineers and leaders without direct organizational authority.

Nice To Haves

Experience improving agent quality through context engineering, skills, memory, evaluation, automated experimentation, hill climbing, fine-tuning, or reinforcement learning.
Demonstrated ability to integrate AI tools to optimize or redesign workflows and drive measurable impact, such as efficiency gains or quality improvements.
Experience building multi-turn human-agent experiences for complex or high-stakes workflows.
Experience building production AI agents that use tools and reason across telemetry, code, configuration, deployments, and operational knowledge.
Track record of taking an ambitious technical product from early adoption to trusted, broad production use.
Experience designing autonomous or semi-autonomous production actions with identity, authorization, independent validation, auditability, rollback, and escalation.
Experience adhering to and implementing responsible, ethical AI practices, such as risk assessment, bias mitigation, and quality and accuracy reviews.
Experience with incident management, SRE, observability, on-call operations, or automated remediation at significant scale.
Demonstrated ongoing AI skill development, such as prompt/context engineering or agent orchestration, and staying current with emerging AI technologies.
Experience building extensible platforms that allow domain teams to contribute specialized capabilities while preserving a coherent user experience.

Responsibilities

Define and drive the technical vision, architecture, and execution roadmap for AI-powered incident investigation and mitigation.
Personally design, code, evaluate, and ship major capabilities in a production agentic system.
Improve timely root-cause accuracy through better reasoning, skills, tool use, context, memory, planning, and domain knowledge.
Build offline evaluation, historical replay, and online experimentation systems to measure changes and continuously improve agent behavior.
Create a transparent, multi-turn experience for collaborative investigation with expert responders and agent-led guidance for less frequent responders.
Enable the agent to reason across telemetry, code, configuration, deployments, service dependencies, operational knowledge, and incident history.
Build closed-loop learning systems that turn production outcomes and user feedback into evaluations, experiments, and product improvements.
Advance the system toward progressive autonomy with explicit authorization, validation, auditability, rollback, and human oversight.
Build an extensible platform combining shared capabilities with domain-specific skills, tools, and agents contributed by infrastructure and product teams.
Partner with on-call engineers, reliability experts, product groups, data science, and product management to drive accuracy, adoption, and operational impact.
Ensure the system meets high standards for reliability, latency, scale, security, privacy, and cost.
Influence technical strategy across organizations and mentor senior engineers working across distributed systems and applied AI.