AI Observability Engineer

Moring Ai•Newark, NJ
•$60,000 - $90,000•Onsite

About The Position

Moring.ai builds enterprise AI for regulated industries, deploying AI applications and agents in customers’ cloud environments. The AI Observability Engineer will own observability for an AI platform used by developers at a large financial services enterprise to build and deploy Python agents that call LLMs, use tools, and run on Kubernetes. This hands-on individual contributor role focuses on enabling engineers, customer operations teams, and auditors to understand agent behavior, including model and tool calls, timing, and cost. The engineer will own end-to-end observability architecture across logs, traces, metrics, and AI-specific signals. The role includes deploying and integrating LangSmith and related tools, instrumenting agents and platform services with OpenTelemetry, creating collector pipelines, and developing developer tooling, dashboards, alerts, and audit reports. It also covers connecting production traces to evaluations, accounting for token usage and spend, making observability work in restricted enterprise networks, defining platform SLOs and on-call runbooks, improving them based on incidents, and teaching engineers how to use the systems. The position is based on-site in Newark, New Jersey, with up to 10% travel to customer sites.

Requirements

  • At least 6 years in software or platform engineering with a deep focus on observability.
  • Very strong understanding of logs, traces, and metrics, including how they are produced, collected, stored, queried, and correlated, and the strengths and limitations of each.
  • Deep OpenTelemetry experience with SDKs, collectors, instrumentation libraries, semantic conventions, sampling, and context propagation; experience designing OTel pipelines.
  • Experience building or operating observability platforms for cloud-native systems, including Kubernetes and microservices, using tools such as Prometheus, Grafana, Loki or Elastic, Jaeger or Tempo, Datadog, Splunk, or comparable tools.
  • Hands-on experience with LangSmith or a comparable LLM observability tool, and understanding of LLM observability concerns such as non-determinism, token cost, prompt and response capture, tool calls, and evaluation.
  • Strong Python skills for building tooling, instrumentation, and collector components.
  • Good working knowledge of AWS, including EKS, CloudWatch, IAM, and networking, and of Kubernetes.
  • Experience being on call for systems the candidate instrumented, and improving dashboards and alerts based on that experience.
  • Ability to explain to a non-engineer what a trace shows and why it matters.
  • Must be authorized to work in the United States (GC/EAD/US Citizen).

Nice To Haves

  • Experience inside large enterprises, especially regulated industries such as financial services, insurance, or healthcare.
  • Understanding of data-redaction and data-retention requirements.

Responsibilities

  • Own the AI platform’s end-to-end observability architecture, including logs, traces, metrics, and AI-specific signals.
  • Deploy and operate LangSmith and suitable related tools, and integrate them with the customer’s existing observability stack.
  • Instrument the platform and agents with OpenTelemetry, including agent reasoning, LLM calls, tool and MCP calls, and retrieval steps, using consistent semantic conventions.
  • Build OpenTelemetry collector pipelines for collection, processing, sampling, sensitive-data redaction, and export in accordance with customer data-handling rules.
  • Build SDK helpers and decorators, dashboards for cost, latency, token usage, and quality, alerts for error rates and spend, and reports for audit teams.
  • Connect production observability traces to evaluation datasets and surface quality regressions.
  • Build cost and usage accounting for tokens and spend by model, team, feature, and agent.
  • Adapt observability to restricted enterprise networks, including private endpoints, no-egress environments, self-hosted backends, and data-retention rules.
  • Define platform SLOs and on-call runbooks, and improve them based on real incidents.
  • Teach the team and customer engineers how to use the observability tools and systems.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service