AI Observability Engineer

Moring Ai•Alpharetta, GA

1d•$60,000 - $90,000•Onsite

About The Position

Moring.ai is hiring a hands-on individual contributor to own observability for an AI platform being built for a large financial services enterprise. The platform enables developers to build and deploy Python-based agents that call LLMs, use tools, and run on Kubernetes. The role focuses on making agent and platform behavior visible, including model and tool calls, timing, token usage, cost, and quality. The engineer will own the platform’s end-to-end observability architecture and apply established observability practices to AI applications. This includes operating LLM observability tools, instrumenting agents and services with OpenTelemetry, building collector pipelines and developer tooling, and integrating with the customer’s existing stack and data-handling requirements. The role also supports evaluation, cost accounting, enterprise auditability, SLOs, on-call readiness, and training for the team and customer engineers. The position is based on-site in Alpharetta, Georgia, with travel to customer sites of up to 10%. It is a full-time individual contributor role. Candidates must be authorized to work in the United States (GC/EAD/US Citizen).

Requirements

6+ years of software or platform engineering experience with a deep focus on observability.
Strong fundamentals in logs, traces, and metrics, including how they are produced, collected, stored, queried, and correlated, and the strengths and limitations of each.
Deep OpenTelemetry experience with SDKs, collectors, instrumentation libraries, semantic conventions, sampling, and context propagation; experience designing OTel pipelines.
Experience building or operating observability platforms for cloud-native systems, including Kubernetes and microservices, using tools such as Prometheus, Grafana, Loki or Elastic, Jaeger or Tempo, Datadog, Splunk, or comparable systems.
Hands-on experience with LangSmith or a comparable LLM observability tool, and understanding of LLM application observability, including non-determinism, token cost, prompt and response capture, tool calls, and evaluation.
Strong Python skills for building tooling, instrumentation, and collector components.
Good working knowledge of AWS, including EKS, CloudWatch, IAM, and networking, and of Kubernetes.
On-call experience for systems the candidate instrumented, with evidence that dashboards and alerts improved through operational experience.
Must be authorized to work in the United States (GC/EAD/US Citizen).

Nice To Haves

Experience in large enterprises, especially regulated industries such as financial services, insurance, or healthcare.
Understanding of data redaction and retention requirements.

Responsibilities

Own the AI platform’s end-to-end observability architecture for logs, traces, metrics, and AI-specific signals.
Deploy and operate LangSmith and related tools such as Langfuse or Arize Phoenix in the customer environment, integrating them with the customer’s existing observability stack.
Instrument the platform and agents with OpenTelemetry, tracing agent reasoning, LLM calls, tool and MCP calls, and retrieval steps using consistent semantic conventions.
Build OpenTelemetry collector pipelines for collection, processing, sampling, sensitive-data redaction, and export to appropriate backends under customer data-handling rules.
Build observability tooling, including SDK helpers and decorators, dashboards for cost, latency, token usage, and quality, alerts for error rates and spend, and audit-readable reports.
Connect observability with evaluation by capturing production traces for evaluation datasets and surfacing quality regressions.
Build cost and usage accounting for tokens and spend by model, team, feature, and agent.
Adapt observability to restricted enterprise networks, including private endpoints, no-egress environments, self-hosted backends, and data-retention rules.
Define platform SLOs and on-call runbooks, and improve them based on real incidents.
Teach the team and customer engineers how to use the observability systems and tools.