About The Position

The Principal Software Engineer will define the technical trajectory for agentic investigation in Splunk Observability Cloud, ensuring AI solutions achieve production-grade trust and efficiency. The role architects evaluation systems and retrieval layers that enable models to reason over large telemetry datasets, improving accuracy and time-to-answer for incident resolution. It establishes production standards for AI agent latency, cost, and security, including guardrails for safe autonomous remediation. The engineer mentors team members, partners with chief architects, raises applied-AI capabilities across the observability portfolio, and sets technical direction that bridges AI research and mission-critical product features. The role is part of the Observability Cloud AI team, which builds an AI Assistant and Troubleshooting Agent to investigate production problems across metrics, traces, logs, and infrastructure. The team works with a large observability data estate and uses a customer-zero methodology, deploying internally before public release. It prioritizes rigorous evaluation and customer reliability.

Requirements

  • 12 or more years of professional software engineering experience.
  • At least 3 years of experience designing, building, and deploying production-grade LLM-based systems, including at least one agentic system with active tool use.
  • Experience developing AI/ML evaluation frameworks, including dataset creation, offline and online scoring, and release validation.
  • Experience engineering retrieval and context pipelines for structured and semi-structured data, such as telemetry, logs, or time-series databases.
  • Knowledge of production-grade engineering principles, including latency optimization, cost management, multi-tenancy, and AI security guardrails.

Nice To Haves

  • Experience in observability, SRE, or incident management domains, including distributed tracing and alerting workflows.
  • Experience with cloud-native infrastructure at scale, specifically Kubernetes.
  • Active open-source contributions in AI, agent frameworks, or observability ecosystems such as OpenTelemetry or MCP.
  • Experience leading customer-zero or field-deployed programs to gather actionable product feedback.
  • Strong communication skills and a track record of setting technical direction and mentoring engineers through design reviews and technical documentation.

Responsibilities

  • Define the technical trajectory for agentic investigation and long-term technical direction for AI systems.
  • Architect evaluation systems and retrieval layers for reasoning over large telemetry datasets.
  • Improve the accuracy and time-to-answer of incident resolution.
  • Establish production standards for AI agent latency, cost, and security, with guardrails for safe autonomous remediation.
  • Mentor team members and partner with chief architects.
  • Raise applied-AI capabilities across the observability portfolio and set the technical bar for the organization.
  • Bridge AI research and mission-critical product features.

Benefits

  • Medical, dental, and vision insurance.
  • 401(k) plan with Cisco matching contribution.
  • Paid parental leave.
  • Short- and long-term disability coverage.
  • Basic life insurance.
  • Cisco restricted stock unit grants may be available, subject to eligibility and vesting conditions.
  • Paid holidays and a floating holiday for non-exempt employees.
  • Birthday day off, year-end holiday shutdown, and personal wellness days.
  • Paid vacation or flexible vacation time off, depending on employee classification.
  • Paid sick time, with limited carryforward of unused sick time.
  • Additional paid time away may be requested for critical or emergency family issues.
  • Optional paid volunteer days.
  • Annual bonuses may be available for non-sales roles.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service