Sr Data Analyst-Agentic AI & GenAI Delivery

GM Financial•Irving, TX

1d•Hybrid

About The Position

Why GM Financial Technology Innovation isn’t just a talking point at GM Financial, it’s how we operate. From generative AI and cloud-native technologies to peer-led learning and hackathons, our tech teams are building real solutions that make a difference. We’re committed to AI-powered transformation, using advanced machine learning and automation to help us reimagine customer interactions and modernize operations, positioning GM Financial as a leader in digital innovation within a dynamic industry. Join us and discover a workplace where your ideas matter, your development is prioritized, and you can truly make a global impact. About the role: The Senior Data Analyst – Agentic AI & GenAI Delivery plays a critical role in operationalizing and scaling Agentic AI solutions across the enterprise. This role focuses on driving delivery, deployment validation, and continuous optimization of AI systems through data-driven insights, validation frameworks, and reporting mechanisms.Unlike traditional data analyst roles, this position operates at the intersection of AI systems, production delivery, and performance analytics, ensuring that Agentic AI solutions are functioning as intended, meeting business objectives, and operating reliably in production environments. This role partners closely with architects, AI engineers, product teams, and business stakeholders to: Validate that AI use cases align with real-world outcomes. Monitor agent behavior, performance, and reliability. Establish data-driven feedback loops for continuous improvement. The ideal candidate brings strong expertise in data analysis, AI system validation, observability, and reporting, along with a solid understanding of Agentic AI / GenAI workflows and production deployment challenges. In this role you will: Drive the delivery and operational validation of Agentic AI solutions through structured data analysis and reporting. Define and implement data-driven validation frameworks to evaluate AI system performance, accuracy, reliability, and business impact. Analyze production data from AI systems (agents, workflows, prompts, responses) to identify trends, issues, and optimization opportunities. Develop dashboards, reports, and metrics to track the health and effectiveness of Agentic AI deployments. Partner with architecture and engineering teams to validate feasibility outcomes and ensure solutions align with real-world system behavior. Monitor AI systems in production, identifying anomalies, failure patterns, hallucinations, and performance degradation. Support deployment efforts by validating readiness criteria, including performance thresholds, guardrails, and compliance requirements. Enable continuous improvement loops by feeding insights back into model tuning, prompt design, and system architecture. Support A/B testing and experimentation for AI workflows and use cases. Collaborate with business stakeholders to measure and report on AI-driven business outcomes and ROI. Ensure transparency and traceability of AI decisions through structured logging, trace analysis, and reporting. Contribute to the development of AI observability frameworks, including metrics, KPIs, and alerting strategies.

Requirements

Validate readiness of Agentic AI use cases for production deployment.
Track deployment success metrics and post-production performance.
Identify gaps between expected vs. actual outcomes.
Define metrics for: Accuracy and response quality, Task completion success rates, Hallucination and failure cases, Latency and throughput.
Build evaluation datasets and validation pipelines.
Analyze: Agent workflows and decisions, Prompt-response chains, Tool usage and orchestration behavior.
Develop observability dashboards using telemetry and logs.
Detect and escalate production issues and anomalies.
Perform root cause analysis on failures and performance issues.
Deliver executive-level reporting on AI system effectiveness.
Provide actionable insights to improve system design and outcomes.
Work closely with: Lead Architects for feasibility alignment AI/ML engineers for model/system improvements Product teams for use case refinement.
Translate technical findings into clear business insights.
Advanced SQL, Python (Pandas, NumPy), or similar tools.
Data visualization platforms (Power BI, Tableau).
Strong experience in data validation, anomaly detection, and statistical analysis.
Familiarity with: LLM workflows and prompt engineering, RAG pipelines and evaluation strategies, Agent orchestration and tool integration.
Understanding of AI failure modes (hallucinations, drift, inconsistency).
Experience with: Logging, tracing, and telemetry systems AI evaluation tools and frameworks Monitoring production systems (Azure Monitor, Application Insights).
Strong working knowledge of Azure ecosystem, including: Azure OpenAI / AI services Azure Databricks Data platforms (Azure SQL, Cosmos DB) Monitoring tools (Log Analytics, App Insights).
Strong analytical and problem-solving skills in complex AI-driven systems.
Ability to connect system behavior with business outcomes.
Expertise in translating data into actionable insights.
High attention to detail in validation, quality, and accuracy.
Strong communication skills across technical and non-technical stakeholders.
Ability to thrive in fast-evolving AI environments.
Ability to wrangle large datasets, structured and non-structured data, including data mining and manipulation.
6-8 years experience in data analytics, data science, or AI Systems analysis or similar role required.
Bachelor’s Degree in Data Science, Computer Science, Engineering or related quantitative field preferred.

Nice To Haves

Experience supporting AI/ ML or GenAI systems in production environments preferred.
Auto finance experience preferred, cross functional Agile team experience preferred.
Master’s Degree in related quantitative field preferred.

Responsibilities

Drive the delivery and operational validation of Agentic AI solutions through structured data analysis and reporting.
Define and implement data-driven validation frameworks to evaluate AI system performance, accuracy, reliability, and business impact.
Analyze production data from AI systems (agents, workflows, prompts, responses) to identify trends, issues, and optimization opportunities.
Develop dashboards, reports, and metrics to track the health and effectiveness of Agentic AI deployments.
Partner with architecture and engineering teams to validate feasibility outcomes and ensure solutions align with real-world system behavior.
Monitor AI systems in production, identifying anomalies, failure patterns, hallucinations, and performance degradation.
Support deployment efforts by validating readiness criteria, including performance thresholds, guardrails, and compliance requirements.
Enable continuous improvement loops by feeding insights back into model tuning, prompt design, and system architecture.
Support A/B testing and experimentation for AI workflows and use cases.
Collaborate with business stakeholders to measure and report on AI-driven business outcomes and ROI.
Ensure transparency and traceability of AI decisions through structured logging, trace analysis, and reporting.
Contribute to the development of AI observability frameworks, including metrics, KPIs, and alerting strategies.