AI Observability Engineer

Moring Ai•Raleigh, NC
•$60,000 - $90,000•Onsite

About The Position

Moring.ai is hiring a hands-on individual contributor to own observability for an AI platform used by developers at a large financial services enterprise. The platform enables developers to build and deploy Python-based agents that call LLMs, use tools, and run on Kubernetes. The engineer will make agent and platform behavior visible to engineers, customer operations teams, and auditors, including model calls, inputs, tool use, execution time, and cost. The role focuses on observability architecture and operations: logs, traces, metrics, and AI-specific signals. It includes deploying LangSmith and related tools in the customer's environment, instrumenting services and agents with OpenTelemetry, building collector pipelines and developer tooling, integrating with the customer's existing observability stack, and supporting evaluation, cost accounting, audit reporting, and operation in restricted enterprise networks. The engineer will also define platform SLOs and on-call runbooks, improve them based on incidents, and teach the team and customer engineers to use the observability capabilities.

Requirements

  • Six or more years in software or platform engineering with a deep focus on observability.
  • Strong fundamentals in logs, traces, and metrics, including how they are produced, collected, stored, queried, and correlated, and the strengths and limitations of each.
  • Deep OpenTelemetry experience with SDKs, collectors, instrumentation libraries, semantic conventions, sampling, and context propagation; experience designing OTel pipelines rather than merely enabling an agent.
  • Experience building or operating observability platforms for cloud-native systems, including Kubernetes and microservices, using tools such as Prometheus, Grafana, Loki or Elastic, Jaeger or Tempo, Datadog, Splunk, or comparable tools.
  • Hands-on experience with LangSmith or a comparable LLM observability tool, and understanding of LLM application observability concerns such as non-determinism, token cost, prompt and response capture, tool calls, and evaluation.
  • Strong Python skills for building tooling, instrumentation, and collector components.
  • Good working knowledge of AWS, including EKS, CloudWatch, IAM, and networking, and of Kubernetes.
  • On-call experience for systems instrumented by the candidate, with evidence that dashboards and alerts improved as a result.
  • Must be authorized to work in the United States (GC/EAD/US Citizen).
  • Ability to explain to non-engineers what a trace shows and why it matters.

Nice To Haves

  • Experience in large enterprises, especially regulated industries such as financial services, insurance, or healthcare.
  • Understanding of data redaction and retention requirements.

Responsibilities

  • Own end-to-end observability architecture for the AI platform, including logs, traces, metrics, and AI-specific signals.
  • Deploy and operate LangSmith and related tools such as Langfuse or Arize Phoenix in the customer's environment, integrating them with the existing observability stack.
  • Instrument the platform and agents with OpenTelemetry, including agent reasoning, LLM calls, tool and MCP calls, and retrieval steps, using consistent semantic conventions.
  • Build OpenTelemetry collector pipelines for collection, processing, sampling, sensitive-data redaction, and export to appropriate backends under the customer's data-handling rules.
  • Build observability tooling, including SDK helpers and decorators, dashboards for cost, latency, token usage, and quality, alerting for error rates and spend, and audit-ready reports.
  • Connect observability to evaluation by capturing production traces for eval datasets and surfacing quality regressions.
  • Build cost and usage accounting for tokens and spend by model, team, feature, and agent.
  • Support observability in restricted enterprise networks, including private endpoints, no-egress environments, self-hosted backends, and data-retention rules.
  • Define platform SLOs and on-call runbooks and improve them based on real incidents.
  • Teach the team and the customer's engineers how to use the observability tools and systems.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service