Software Engineer II, AI Performance

The Walt Disney Company • New York, NY

4d • $123,000 - $165,000 • Onsite

About The Position

Disney Entertainment and ESPN Product & Technology is a global organization focused on building and advancing the technological backbone for Disney’s media business. The team combines technology with creativity to create world-class products, enhance storytelling, and drive innovation and scalability. This role is within the Product Engineering group, specifically the Observability & Insights team, which ensures the reliability, performance, and transparency of Disney Streaming’s distributed systems. They build telemetry, dashboards, alerting, insights pipelines, and developer experience tooling.

As a Software Engineer II on the Reliability Insights team, you will be responsible for designing and building data and reporting pipelines. These pipelines will evaluate and demonstrate the real-world performance, precision, and business impact of Disney’s operational AI and observability tools. You will work at the intersection of backend engineering and data-driven insights, transforming raw telemetry, AI agent interactions, and operational logs into clear metrics related to AI reliability, model drift, and incident mitigation. Your work will involve building automated ground-truth pipelines and model evaluation frameworks to quantify the effectiveness of AI agents and anomaly models in production. You will also capture end-user feedback, correlate AI signals with incident resolution timelines (e.g., Mean Time to Mitigation), and develop single-pane dashboards for engineering and leadership. By collaborating with SRE, product, and platform engineering teams, you will address visibility gaps, reduce alert noise, and establish clear standards for AI model performance across Disney+, Hulu, and ESPN.

Requirements

• 3+ years of applicable experience in backend and data pipeline engineering, including building scalable APIs (e.g., FastAPI, Flask) and metrics/insight generation platforms.
• Strong experience with data platform engineering and distributed processing tools (e.g., PySpark, SQL, Pandas, Databricks, Snowflake) to process and analyze telemetry data.
• Practical understanding of model evaluation concepts, such as precision, recall, model drift, and ground-truth labeling systems.
• Hands-on experience with modern development practices, including version control (GitHub), containerization (Docker), and cloud-native deployments (AWS/EKS).
• Proficiency with AI-assisted development tools (e.g., Cursor, Claude Code) to accelerate engineering velocity.
• Strong analytical skills to translate complex system telemetry and operational logs into meaningful reliability metrics and business insights.
• Excellent cross-functional collaboration and communication skills, with a track record of partnering with operational teams (SRE, DevOps) to turn raw data into actionable insights.

Nice To Haves

• Experience with observability, telemetry, and monitoring platforms (e.g., Datadog, Grafana, Conviva).
• Familiarity with agentic workflows and foundation models (e.g., GPT-4, Claude) and how to evaluate their output accuracy.
• Knowledge of incident management workflows and SRE metrics (e.g., MTTR, MTTD, alert suppression).
• Prior experience working on data attribution, metrics aggregation, or financial/performance visibility tooling in high scale environments.

Responsibilities

• Design, build, and maintain automated feedback pipelines that capture operational outcomes (confirmed incidents, false positives, dismissed alerts) and correlate them directly back to AI model predictions and agent actions.
• Develop programmatic evaluation frameworks to track model precision, recall, and drift over time, establishing automated threshold alerts to proactively trigger model tuning and prevent alert fatigue.
• Build end-to-end data pipelines linking AI outputs to incident lifecycles, quantifying business impact through metrics such as Service to Incident correlation, notification success rates, and Mean Time to Mitigation (MTTM).
• Design and deploy scalable APIs, metrics layers, and single-pane model health dashboards that present clear AI ROI, operational health, and performance narratives to leadership and technical stakeholders.
• Build mechanisms to capture structured feedback from end users to continuously feed real-world usage signals back into model improvement cycles.

Benefits

• A bonus and/or long-term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits, dependent on the level and position offered.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: Associate degree