Staff Software Engineer - Agent Observability (Galileo)
About The Position
As a Software Engineering Technical Leader on Cisco’s Agent Resilience Team, you will lead technical design and development of scalable, customer-facing products that help enterprises monitor, evaluate, and control AI agent behavior. The role combines hands-on engineering with technical leadership, including driving complex initiatives across multiple engineers, guiding architectural decisions, and ensuring reliable delivery. The team is advancing AI observability, distributed tracing, multimodal evaluation, and agent control capabilities, with opportunities to integrate with Splunk AI Observability and Cisco’s broader technology portfolio. You will collaborate with backend and frontend engineers, AI researchers, product managers, and designers to develop enterprise-scale products. You will also participate in a shared on-call rotation, typically one week approximately every two months, depending on team size.
Requirements
- Bachelor’s degree with 8+ years of related experience, Master’s degree with 6+ years of related experience, or equivalent practical experience.
- 8+ years of experience developing production backend software applications and services using Python and/or Go.
- Experience leading the technical design and delivery of complex software systems or projects involving multiple engineers, including coordinating technical work and providing engineering guidance.
- Experience designing and developing distributed systems, backend architectures, and APIs for scalable, high-throughput, data-intensive applications.
- Experience making technical architecture decisions involving system scalability, reliability, performance, and maintainability.
Nice To Haves
- Experience developing customer-facing products involving AI/ML, generative AI, large language models (LLMs), or agentic applications.
- Experience building AI observability, evaluation, or monitoring capabilities, including LLM tracing, distributed tracing, or OpenTelemetry.
- Experience designing or operating distributed data-processing systems using Kafka, ClickHouse, or similar streaming and analytical data technologies.
- Experience developing enterprise observability, analytics, or monitoring products that process and analyze telemetry from multiple services.
- Experience leading cross-team technical initiatives, mentoring engineers, and influencing architectural decisions across engineering teams.
Responsibilities
- Lead the architecture, design, and delivery of large-scale backend systems for AI agent observability, distributed tracing, multimodal evaluation, AI metrics, and agent control.
- Drive complex engineering initiatives from technical design through production deployment, coordinating work across engineers, defining technical approaches, and guiding implementation.
- Design and develop distributed systems, APIs, and data-processing capabilities that ingest telemetry from multiple services and generate insights from generative AI traces, metrics, and events.
- Contribute hands-on production code in Python and/or Go.
- Lead technical design and code reviews, mentor engineers, and drive architectural decisions that improve performance, reliability, and engineering quality.
- Collaborate with product managers, AI researchers, and engineering teams across Splunk and Cisco to deliver customer-facing capabilities.
- Participate in a shared on-call rotation, typically one week approximately every two months, depending on team size.
Benefits
- Medical insurance
- Dental insurance
- Vision insurance
- 401(k) plan with Cisco matching contribution
- Paid parental leave
- Short- and long-term disability coverage
- Basic life insurance
- Cisco restricted stock unit grants may be available, subject to eligibility and continued employment vesting conditions
- 10 paid holidays per full calendar year
- One floating holiday for non-exempt employees
- One paid day off for the employee’s birthday
- Paid year-end holiday shutdown
- Four paid personal wellness days
- Non-exempt employees receive 16 days of paid vacation per full calendar year, accrued at 4.92 hours per pay period for full-time employees
- Exempt employees participate in a flexible vacation time off program with no defined limit, subject to availability and business limitations
- 80 hours of sick time off provided on hire date and each January 1 thereafter, with up to 80 unused hours carried forward
- Additional paid time away may be requested for critical or emergency family issues
- Optional 10 paid volunteer days per full calendar year
- Employees in non-sales roles are eligible to earn annual bonuses, subject to Cisco policies
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
