AI Observability Engineer

Moring Ai•Dallas, TX
•$60,000 - $90,000•Onsite

About The Position

Moring.ai is hiring a hands-on individual contributor to own observability for an AI platform used by developers at a large financial services enterprise. The platform runs Python-based AI agents that call LLMs and tools on Kubernetes. The engineer will enable developers, customer operations teams, and auditors to understand agent behavior, including model calls, inputs, tool use, execution time, and cost. The role applies deep observability expertise to AI systems. It includes designing and operating logs, traces, metrics, and AI-specific signals; deploying LangSmith and related tools in the customer's environment; instrumenting agents and platform services with OpenTelemetry; building collector pipelines and observability tooling; connecting production traces to evaluation; and accounting for model usage and cost. The work must support restricted enterprise networks, customer data-handling rules, and audit needs. The engineer will define platform SLOs and on-call runbooks, improve them through incident experience, and teach the team and customer engineers to use the observability systems. The position is on-site in Dallas, Texas, with up to 10% travel to customer sites. Applicants must be authorized to work in the United States.

Requirements

  • 6+ years in software or platform engineering with a deep focus on observability.
  • Strong fundamentals in logs, traces, and metrics, including how they are produced, collected, stored, queried, and correlated, and the strengths and limitations of each.
  • Deep OpenTelemetry experience with SDKs, the collector, instrumentation libraries, semantic conventions, sampling, and context propagation; experience designing OTel pipelines rather than merely enabling an agent.
  • Experience building or operating observability platforms for cloud-native systems, including Kubernetes, microservices, and tools such as Prometheus, Grafana, Loki or Elastic, Jaeger or Tempo, Datadog, Splunk, or comparable products.
  • Hands-on experience with LangSmith or a comparable LLM observability tool, and understanding of LLM application observability concerns such as non-determinism, token cost, prompt and response capture, tool calls, and evaluation.
  • Strong Python skills for building tooling, instrumentation, and collector components.
  • Good working knowledge of AWS, including EKS, CloudWatch, IAM, and networking, and of Kubernetes.
  • On-call experience for systems the candidate instrumented, with evidence that dashboards and alerts improved as a result.
  • Must be authorized to work in the United States (GC/EAD/US Citizen).

Nice To Haves

  • Experience inside large enterprises, especially regulated industries such as financial services, insurance, or healthcare.
  • Understanding of data redaction and retention requirements.
  • Ability to explain to non-engineers what a trace shows and why it matters.

Responsibilities

  • Own end-to-end observability architecture for the AI platform, including logs, traces, metrics, and AI-specific signals.
  • Deploy and operate LangSmith and related tools such as Langfuse or Arize Phoenix in the customer's environment, integrating them with the existing observability stack.
  • Instrument the platform and agents with OpenTelemetry, including traces for agent reasoning, LLM calls, tool and MCP calls, and retrieval steps, using consistent semantic conventions.
  • Build OpenTelemetry collector pipelines for collection, processing, sampling, sensitive-data redaction, and export to appropriate backends under customer data-handling rules.
  • Build observability tooling, including SDK helpers and decorators, dashboards for cost, latency, token usage, and quality, alerts for error rates and spend, and audit-readable reports.
  • Connect observability to evaluation by capturing production traces for evaluation datasets and surfacing quality regressions.
  • Build cost and usage accounting for tokens and spend by model, team, feature, and agent.
  • Support restricted enterprise networks, including private endpoints, no-egress environments, self-hosted backends, and data-retention rules.
  • Define platform SLOs and on-call runbooks, and improve them based on real incidents.
  • Teach the team and customer engineers how to use the observability systems.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service