Software Engineer 5, AI Observability

Netflix•Remote, OR
•$500,000 - $860,000

About The Position

The AI Observability team builds end-to-end observability for ML and GenAI workloads, capturing model inputs, features, predictions, outcomes, and behavior across online and batch systems. Its platform helps teams monitor model performance, data quality, drift, latency, and failures, and provides libraries, dashboards, and alerts to make systems easier to debug and operate. This hands-on senior engineering role builds reusable infrastructure for Netflix’s AI Observability platform, model performance, evaluation, and vendor integrations. The platform enables ML and AI practitioners to monitor production model quality, evaluate LLM and agentic systems, and adopt vendor tooling through a consistent self-service interface. The AI Platform team owns generic infrastructure, while domain teams own their domain-specific evaluations and remediation. The engineer partners with engineering, product, machine learning, and data teams to turn their needs into reusable platform capabilities.

Requirements

  • Experience in software, AI/ML, or platform engineering, including hands-on production observability, monitoring, or ML/LLM evaluation.
  • A proven track record of designing standards, libraries, SDKs, or frameworks adopted by other teams.
  • Strong Python coding skills and proficiency in at least one of Java, Go, or Scala.
  • Experience building production services.
  • Practical experience operating ML models in production, online and/or batch, including drift, model performance, and data quality.
  • Hands-on experience with modern observability stacks such as Prometheus/Grafana, Datadog, OpenTelemetry, ELK/OpenSearch, Jaeger/Tempo, or similar.
  • Solid understanding of distributed systems, microservices, and at least one major cloud platform: AWS, GCP, or Azure.
  • Ability to work cross-functionally with ML, data, infrastructure, and product teams, and communicate clearly about system behavior, quality, and risk.
  • Experience with vendor integration and VPC deployment.
  • An AI-native engineering mindset, using AI tools as a core part of the workflow to accelerate design, development, testing, and code review.

Nice To Haves

  • Hands-on experience with ML/LLM observability and evaluation tools such as Arize, Braintrust, LangFuse, Weights & Biases, Galileo, Vertex AI Model Monitoring, or SageMaker Model Monitor.
  • Experience building or shipping LLM/GenAI applications and evaluating them, including prompt/result logging, evaluation metrics, LLM-as-a-judge, and human-in-the-loop review.
  • Experience evaluating agentic systems, including tool use, multi-step reasoning, and trajectory/task-success measurement.

Responsibilities

  • Build observability frameworks and platform capabilities that provide metrics, logs, and distributed traces across online inference, batch scoring, feature pipelines, and agent orchestration.
  • Build primitives that enable teams to monitor model performance, data quality, drift, and degradation in their own systems.
  • Build and extend evaluation frameworks for LLM and agentic systems, including reusable building blocks for response quality, grounding, hallucination, task success, tool use, trajectory correctness, LLM-as-a-judge, and human-in-the-loop scoring.
  • Lead build-versus-buy evaluations for observability and evaluation tooling.
  • Own SDKs, connectors, and APIs that integrate vendor platforms into a consistent, well-supported interface.
  • Build reusable libraries, SDKs, and templates that make observability and evaluation the default for new systems.
  • Provide dashboarding, alerting, and SLO/SLI building blocks and out-of-the-box templates for tracking model performance, latency, cost, and reliability.

Benefits

  • Health Plans
  • Mental Health support
  • 401(k) Retirement Plan with employer match
  • Stock Option Program
  • Disability Programs
  • Health Savings and Flexible Spending Accounts
  • Family-forming benefits
  • Life and Serious Injury Benefits
  • Paid leave of absence programs
  • Flexible time off for full-time salaried employees
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service