Staff Triage & RCA Engineer

Agility Robotics•Fremont, CA

3d•Hybrid

About The Position

Agility Robotics is at the forefront of humanoid robotics, redefining how robots interact with and navigate the human world. Our robots rely on a complex intersection of hardware and advanced software to perform tasks in dynamic environments. We are seeking a Staff Triage & RCA Engineer to provide technical leadership for software issue triage and root cause analysis across our robotic systems. In addition to leading complex investigations, this role will define the processes, tooling, and technical systems used to diagnose and resolve failures at scale. You will serve as a technical authority for ambiguous hardware-software failures, establish standards and methodologies for triage and RCA, and drive the development of diagnostic tools and infrastructure that improve the efficiency and effectiveness of the broader engineering organization. You will also mentor engineers and help set technical direction for the discipline.

Requirements

10+ years of experience in Systems Integration, Software-Hardware interface, R&D, or similar roles involving complex mechatronic or autonomous systems.
Proven track record of leading complex, ambiguous technical investigations and defining scalable engineering processes or tooling.
Experience with monitoring and observability platforms (e.g., Datadog, Splunk, or New Relic).
Experience working with cloud storage and databases (e.g., AWS S3, SQL, or NoSQL) to retrieve and analyze large-scale telemetry and video datasets.
Experience influencing cross-functional engineering teams and mentoring technical contributors.
Mastery of log parsing and proficiency with Python or similar scripting languages for data analysis and automation.
Strong understanding of robotics software stacks and communication protocols such as EtherCAT and CAN.
Experience correlating video and/or hardware symptoms with system telemetry to identify software and system failures.
Experience developing diagnostic, observability, or automation tooling for complex systems.
Ability to define technical direction and solve ambiguous, unprecedented problems with scalable solutions.
Ability to balance hands-on technical investigation with longer-term process, tooling, and systems development.
Exceptional ability to synthesize complex data into clear, actionable insights.
Strong technical leadership, communication, and mentoring skills.
Bachelor’s or Master’s degree in Computer Science, Robotics, Electrical Engineering, or a related field.
Applicants must have current authorization to work in the United States.

Nice To Haves

Experience with HW/SW integration and design on HiL.
Experience troubleshooting HW/SW interactions involving cameras, encoders, IMUs, or other sensors.

Responsibilities

Serve as a technical lead for complex software, firmware, and system-level failure triage.
Define and improve standardized triage processes, workflows, and disposition criteria.
Lead prioritization and escalation of critical software-related failures, ensuring engineering teams receive clear, actionable investigations.
Identify systemic failure trends across the fleet and drive appropriate corrective actions.
Lead end-to-end investigations of novel and ambiguous failures at the hardware-software interface.
Use logs, telemetry, video, and system behavior to identify definitive root causes and contributing factors.
Establish and promote rigorous RCA methodologies and standards across the organization.
Author high-quality RCA reports and troubleshooting guidance that enable broader engineering teams to independently investigate failures.
Define the technical direction and roadmap for triage, diagnostics, and RCA tooling.
Develop scalable diagnostic scripts, automation, and data analysis frameworks to accelerate investigations.
Drive improvements to system observability, telemetry, logging, and diagnosability.
Identify opportunities to automate recurring investigations and improve the overall engineering feedback loop.
Serve as a technical subject matter expert and mentor for engineers performing triage and RCA.
Establish technical standards and best practices for failure investigation and documentation.
Influence Software, Hardware, Controls, and Systems teams on approaches to fault isolation, diagnosability, and system reliability.
Provide technical direction for initiatives that improve the organization's ability to detect, diagnose, and resolve failures.

Benefits

401(k) Plan: Includes a 6% company match.
Equity: Company stock options.
Insurance Coverage: 100% company-paid medical, dental, vision, and short/long-term disability insurance for employees.
Benefit Start Date: Eligible for benefits on your first day of employment.
Well-Being Support: Employee Assistance Program (EAP).
Time Off: Exempt Employees: Flexible, unlimited PTO and 12 company holidays, including a winter shutdown. Non-Exempt Employees: 10 vacation days, paid sick leave, and 12 company holidays, including a winter shutdown, annually.
On-Site Perks: Catered lunches four times a week and a variety of healthy snacks and refreshments at our Salem and Pittsburgh locations.
Parental Leave: Generous paid parental leave programs.
Growth Opportunities: Professional development and tuition reimbursement programs.
Relocation Assistance: Provided for eligible roles.
Annual Discretionary Bonus: Provided for eligible roles.