Software Engineer II - AI Performance
About The Position
The Software Engineer II will join the Reliability Insights team within Disney Entertainment and ESPN Product & Technology’s Observability & Insights group. The role focuses on designing and building data and reporting pipelines to evaluate the real-world performance, precision, and business impact of Disney’s operational AI and observability tools. It combines backend engineering with data-driven insights by transforming telemetry, AI agent interactions, and operational logs into metrics for AI reliability, model drift, and incident mitigation. The engineer will build automated ground-truth pipelines and model evaluation frameworks to measure the performance of AI agents and anomaly models in production. The work includes capturing end-user feedback, correlating AI signals with incident resolution timelines, and creating dashboards for engineering teams and leadership. The role partners with SRE, product, and platform engineering teams to improve visibility, reduce alert noise, and establish standards for AI model performance across Disney+, Hulu, and ESPN.
Requirements
- At least 3 years of applicable experience in backend and data pipeline engineering, including building scalable APIs such as FastAPI or Flask and metrics or insight generation platforms.
- Strong experience with data platform engineering and distributed processing tools such as PySpark, SQL, Pandas, Databricks, or Snowflake for processing and analyzing telemetry data.
- Practical understanding of model evaluation concepts, including precision, recall, model drift, and ground-truth labeling systems.
- Hands-on experience with modern development practices, including GitHub, Docker, and cloud-native deployments such as AWS/EKS.
- Proficiency with AI-assisted development tools such as Cursor or Claude Code.
- Strong analytical skills to translate system telemetry and operational logs into reliability metrics and business insights.
- Excellent cross-functional collaboration and communication skills, including partnering with operational teams such as SRE and DevOps to turn raw data into actionable insights.
- Bachelor’s degree in Computer Science, Engineering, or equivalent experience.
Nice To Haves
- Experience with observability, telemetry, and monitoring platforms such as Datadog, Grafana, or Conviva.
- Familiarity with agentic workflows and foundation models such as GPT-4 or Claude, including evaluating output accuracy.
- Knowledge of incident management workflows and SRE metrics such as MTTR, MTTD, and alert suppression.
- Prior experience with data attribution, metrics aggregation, or financial/performance visibility tooling in high-scale environments.
Responsibilities
- Design, build, and maintain automated feedback pipelines that capture operational outcomes, including confirmed incidents, false positives, and dismissed alerts, and correlate them with AI model predictions and agent actions.
- Develop programmatic evaluation frameworks to track model precision, recall, and drift over time, with automated threshold alerts to trigger model tuning and help prevent alert fatigue.
- Build end-to-end data pipelines linking AI outputs to incident lifecycles and quantify business impact using metrics such as Service to Incident correlation, notification success rates, and Mean Time to Mitigation (MTTM).
- Design and deploy scalable APIs, metrics layers, and single-pane model health dashboards that communicate AI ROI, operational health, and performance to leadership and technical stakeholders.
- Build mechanisms to capture structured end-user feedback and feed real-world usage signals into model improvement cycles.
Benefits
- A bonus and/or long-term incentive units may be provided as part of the compensation package.
- Medical, financial, and/or other benefits may be provided, dependent on the level and position offered.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
