AI Business Analyst

Allied Benefit Systems•Chicago, IL

1d•$105,000 - $125,000•Remote

About The Position

The AI Business Analyst will be responsible for evaluating whether Allied’s internal AI solutions are functioning as intended, identifying where they succeed and where they fall short, and translating those findings into clear, actionable feedback for product and engineering teams. This role plays a key role in Allied’s AI delivery team, helping ensure that AI capabilities are not only technically sound but also aligned with business needs and user expectations.

Requirements

8–12 years of experience in business analysis, quality assurance, or systems evaluation roles, ideally within the insurance or administrative healthcare industry
Deep subject matter expertise in claims, health call centers, provider operations, or health plan administration
Strong analytical mindset with the ability to identify subtle patterns, inconsistencies, or edge cases in complex systems
Comfortable working with structured and unstructured data, including reviewing logs, outputs, and raw files to trace issues or validate behavior
Experience writing clear, structured documentation such as user stories, test cases, and issue reports
Excellent communication skills and the ability to collaborate across technical and non-technical teams
Reliable internet service is essential for staying connected and productive. The internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds of at least 100Mbps download/25Mbps upload.

Nice To Haves

Familiarity with AI/ML systems and their evaluation is a plus, but not required
Familiarity with tools for data inspection and analysis (e.g., Excel, SQL, Power BI)
Experience with log analysis and debugging tools (e.g., Azure Monitor, Splunk)
Exposure to API testing tools (e.g., Postman) and basic scripting or query languages is helpful
Agile collaboration tools (e.g., Freshservice, Azure DevOps, or similar) for tracking issues and documenting requirements

Responsibilities

Evaluate AI capabilities integrated into core systems (e.g., Dynamics CRM, claims adjudication platform, document repositories) to determine whether they are functioning as intended and delivering expected business outcomes
Investigate edge cases, anomalies, and failure patterns by reviewing raw data, logs, and system outputs
Collaborate with technical product owners and SMEs to document what is and isn’t working, and help translate findings into user stories and product requirements
Participate in model and capability reviews, providing business-level validation of AI behavior and identifying gaps in functionality or alignment with intended use
Contribute to the development of test scenarios and evaluation frameworks that reflect real-world usage and business expectations
Recommend improvements or refinements to AI capabilities based on observed behavior, user feedback, and business context
Support the AI Product Owner and QA team in defining acceptance criteria and success metrics for new AI features
Help ensure that AI solutions are understandable, trustworthy, and aligned with Allied’s operational goals
Other duties as assigned