The Principal Software Engineer will define the technical trajectory for agentic investigation in Splunk Observability Cloud, ensuring AI solutions achieve production-grade trust and efficiency. The role architects evaluation systems and retrieval layers that enable models to reason over large telemetry datasets, improving accuracy and time-to-answer for incident resolution. It establishes production standards for AI agent latency, cost, and security, including guardrails for safe autonomous remediation. The engineer mentors team members, partners with chief architects, raises applied-AI capabilities across the observability portfolio, and sets technical direction that bridges AI research and mission-critical product features. The role is part of the Observability Cloud AI team, which builds an AI Assistant and Troubleshooting Agent to investigate production problems across metrics, traces, logs, and infrastructure. The team works with a large observability data estate and uses a customer-zero methodology, deploying internally before public release. It prioritizes rigorous evaluation and customer reliability.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Job Type
Full-time
Career Level
Principal
Education Level
No Education Listed