AI Observability Engineer

Moring Ai•Alpharetta, GA

3d•Onsite

About The Position

Moring.ai is hiring a hands-on individual contributor to own observability for an AI platform being built for a large financial services enterprise. The platform enables developers to build and deploy Python agents that call LLMs, use tools, and run on Kubernetes. The engineer will make it possible to inspect agent behavior, including model calls, inputs, tools used, step durations, and costs. The role applies deep observability-platform experience to AI applications. It includes operating LangSmith and related tools in a customer environment, instrumenting agents and platform services with OpenTelemetry, creating collector pipelines and supporting tooling, and integrating with existing enterprise observability systems. The work must support customer data-handling rules, restricted networks, audits, and operational needs.

Requirements

6+ years in software or platform engineering with a deep focus on observability.
Strong understanding of logs, traces, and metrics, including how they are produced, collected, stored, queried, and correlated, and the strengths and limitations of each.
Deep OpenTelemetry experience with SDKs, the Collector, instrumentation libraries, semantic conventions, sampling, and context propagation.
Experience designing OpenTelemetry pipelines, not just enabling an agent.
Experience building or operating observability platforms for cloud-native systems, including Kubernetes, microservices, and related tooling.
Strong Python skills for building tooling, instrumentation, and collector components.
Good working knowledge of AWS, including EKS, CloudWatch, IAM, and networking, and of Kubernetes.
Experience being on call for systems instrumented by the candidate, with dashboards and alerts improved through operational experience.
Ability to explain to non-engineers what a trace shows and why it matters.
Authorization to work in the United States; the posting specifies GC, EAD, or U.S. citizen.

Nice To Haves

Experience with observability tools such as Prometheus, Grafana, Loki or Elastic, Jaeger or Tempo, Datadog, Splunk, or comparable platforms.
Hands-on experience with LangSmith or a comparable LLM observability tool, such as Langfuse, Arize Phoenix, Braintrust, or Weights and Biases Weave.
Understanding of LLM application observability, including non-determinism, token cost, prompt and response capture, tool calls, and evaluation.
Experience in large enterprises, especially regulated industries such as financial services, insurance, or healthcare.
Understanding of data redaction and retention requirements.

Responsibilities

Own the AI platform’s end-to-end observability architecture for logs, traces, metrics, and AI-specific signals.
Deploy and operate LangSmith and, where appropriate, related tools such as Langfuse or Arize Phoenix within the customer environment, integrating them with the existing observability stack.
Instrument the platform and agents with OpenTelemetry, tracing agent reasoning, LLM calls, tool and MCP calls, and retrieval steps using consistent semantic conventions.
Build OpenTelemetry Collector pipelines for collection, processing, sampling, sensitive-data redaction, and export to appropriate backends under customer data-handling rules.
Build SDK helpers, decorators, dashboards, alerts, and reports that support engineering teams, customer operations, and auditors.
Create dashboards for cost, latency, token usage, and quality; configure alerts for error rates and spend; and produce audit-readable reports.
Connect observability to evaluation by capturing production traces for evaluation datasets and surfacing quality regressions.
Build cost and usage accounting for tokens and spend by model, team, feature, and agent.
Support observability in restricted enterprise networks, including private endpoints, no-egress environments, self-hosted backends, and data-retention rules.
Define platform SLOs and on-call runbooks, and improve them based on incidents.
Teach the team and customer engineers how to use the observability systems and tooling.