Principal AI Engineer

LogicMonitor•San Francisco, CA

23h•Hybrid

About The Position

LogicMonitor® is the AI-first hybrid observability platform powering the next generation of digital infrastructure. LogicMonitor delivers complete visibility and actionable intelligence across on-premises, cloud, and edge environments. By anticipating issues before they strike, optimizing resources in real time, and enabling faster, smarter decisions, LogicMonitor helps IT and business leaders protect margins, accelerate innovation, and deliver exceptional digital experiences without compromise. Our customers love LogicMonitor's ability to bring cloud and traditional IT together into one view, as seen in minimal churn rates, expansion business, and exciting new customer references. In fact, LogicMonitor has received the highest Net Promoter Score of any IT Infrastructure Management provider. LogicMonitor also boasts high employee satisfaction. We have been certified as a Great Place To Work®, and named one of BuiltIn's Best Places to Work for the seventh year in a row! Edwin AI is LogicMonitor’s enterprise AIOps platform, designed to help organizations detect, understand, and resolve incidents faster by reasoning across metrics, logs, topology, and change intelligence. As Edwin AI evolves toward AI-driven action and remediation, we are building enterprise-grade AI agents that integrate deeply with systems like ServiceNow to automate incident workflows with safety, reliability, and auditability. We are seeking a Principal AI Engineer to design and build AI agents that operate across observability data and ITSM systems. This role focuses on turning AI insights into safe, deterministic actions—building systems that can enrich incidents, orchestrate workflows, and reduce MTTR in large-scale, multi-tenant enterprise environments. This is a hands-on senior individual contributor role with high ownership and technical authority. Here's a closer look at this key role:

Requirements

8+ years of backend or platform engineering experience
Proven experience building distributed, microservices-based systems at scale
Hands-on experience integrating with ServiceNow or enterprise ITSM platforms
Production experience building AI agents or LLM-powered systems that invoke tools or workflows
Experience with AIOps, observability platforms, or cloud management systems
Experience building AI systems that trigger real operational actions
Familiarity with Model Context Protocol (MCP) or equivalent agent-tool architectures
Strong proficiency in Java and/or Python
Experience with Kubernetes, containers, and cloud-native systems
Deep understanding of async execution, retries, idempotency, and failure handling
Experience designing audit-safe, compliance-friendly enterprise automation

Responsibilities

Design and implement AI agents within Edwin AI that reason over metrics, logs, topology, and change data
Build and operate deep integrations with ServiceNow (Incidents, Problems, Changes, CMDB)
Translate observability insights and RCA outputs into ServiceNow-native actions and workflows
Implement event-driven, asynchronous pipelines for long-running and approval-based actions
Define and enforce guardrails, validation, auditability, and human-in-the-loop controls for AI actions
Ensure enterprise standards for security, reliability, and tenant isolation
Partner with Product, Platform, and Observability teams to shape Edwin AI agent capabilities
Provide technical leadership and mentorship on agentic AI system design