AI Change Analyst

Gallagher•Rolling Meadows, IL

About The Position

The AI Change Analyst owns change management across AJ Gallagher’s AI portfolio, covering changes to AI projects and production systems such as RAG pipelines, Azure AI Foundry and Copilot Studio agents, and M365 Copilot integrations. Reporting to the Director of AI Technology, the role manages change requests from intake and impact assessment through approval, scheduling, deployment, and post-implementation review. It ensures AI-related changes are appropriately assessed, approved, documented, and traceable, working with AI developers, engineering leadership, the AI Support Manager, and business stakeholders. The role also embeds change control into delivery tooling, including Azure DevOps pipelines, environment approvals, and work item traceability. The position governs changes such as model upgrades or retirements, prompt and agent instruction updates, Azure AI Search index and knowledge-source changes, and additions of agent tools or connectors. It coordinates changes across projects and with enterprise IT change management, maintains audit-ready records, and reports on portfolio change health to the Director of AI Technology.

Requirements

3 to 5 years of experience in IT change management, release management, or project change control, including hands-on change-request management in a software delivery environment.
Working knowledge of change enablement practices, including risk classification, standard/normal/emergency changes, approval workflows, change calendars, rollback planning, and post-implementation reviews.
Hands-on experience with Azure DevOps Boards and Pipelines, or an equivalent such as GitHub Actions or Jira with CI/CD, including work item queries and dashboards, linking work to code and deployments, and enough YAML fluency to read pipelines and configure approvals and checks with engineers.
Ability to read pull requests, pipeline logs, or release notes to understand changes and identify incomplete requests or understated impacts.
Proven ability to track a high volume of concurrent changes across multiple projects and stakeholders without losing detail or allowing approvals to stall.
Ability to write change records and approval summaries that non-technical approvers and auditors can act on without follow-up questions.
Bachelor’s degree in Information Systems, Computer Science, Business, or a related field, or equivalent practical experience.
Detail-driven approach to identifying missing rollback plans, unlinked work items, and impacts on shared systems before changes reach production.
Ability to enable safe changes to ship faster by automating and pre-approving routine work while directing review effort toward higher-risk changes.
Ability to hold incomplete or inadequately evidenced changes for further work, respectfully and with a clear path to approval.
Systems awareness to assess second-order impacts across connected AI portfolio components.
Clear communication tailored to executives, auditors, and engineers.
Comfort working in a regulated environment and treating approvals and audit trails as part of trustworthy AI delivery.

Nice To Haves

Familiarity with how generative AI systems are built and changed, including LLMs, RAG, agents, prompts, and evaluations, and the ability to recognize when a change needs evaluation evidence.
Experience in insurance, financial services, or healthcare, with exposure to SOX-type change controls, internal audit, model risk management, or NAIC AI compliance.
Experience with ServiceNow Change Management, Jira Service Management, or a similar platform, including CAB workflows.
Exposure to Copilot Studio solutions, Power Platform environment strategy, and pipelines for promoting agents between environments.
Familiarity with Terraform plan/apply workflows for reviewing infrastructure changes.
Experience measuring delivery health with DORA-style metrics, including change failure rate, lead time for changes, and deployment frequency.
ITIL 4 Foundation or AZ-900 certification, or progress toward AZ-400.
Familiarity with the listed technology stack: Azure DevOps, ServiceNow or Jira Service Management, Azure AI Foundry, Azure OpenAI Service, Azure AI Search, Microsoft Copilot Studio, Power Platform, Microsoft Entra ID, Azure RBAC, Power BI, Microsoft Teams, and SharePoint.

Responsibilities

Manage intake, logging, and tracking for change requests across AI projects and production systems, ensuring each request documents scope, rationale, affected systems, test evidence, and rollback plans, and track requests through closure.
Assess change impacts and risks across users, integrations, data access, cost, and compliance; classify changes as standard, normal, or emergency and route them to appropriate approvers.
For scope changes, quantify impacts on schedule, cost, and dependent projects, and work with the AI Business Analyst to maintain requirements traceability.
Govern AI-specific changes, including model versions and retirements, prompts, agent instructions, Azure AI Search indexes and knowledge sources, and new agent tools or connectors; require evaluation evidence for groundedness and accuracy before approval.
Maintain the AI team’s change policy, request templates, and catalog of pre-approved standard changes; facilitate review meetings, prepare approval evidence, record decisions, and maintain the change calendar, release windows, and freeze periods.
Identify conflicts and dependencies among changes across initiatives and sequence releases to avoid disruption to shared resources, indexes, connectors, and APIs.
Partner with the Senior AI Engineering Lead and AI developers to configure and maintain Azure DevOps environment approvals and checks, branch policies, and pipeline templates, enabling approved production deployments and end-to-end change traceability.
Coordinate expedited approvals with the AI Support Manager for emergency changes and conduct post-implementation reviews of failed, rolled-back, and emergency changes.
Represent the AI team in Gallagher’s enterprise IT change process and ensure changes affecting shared infrastructure, identity, networking, or line-of-business systems are submitted and approved through that process.
Produce recurring reporting for the Director of AI Technology on change volume, lead time, success and rollback rates, and emergency change frequency, and flag trends indicating delivery or quality risk.
Maintain change records, approvals, and evidence for AI compliance reviews, model risk assessments, and internal audit, and keep the AI system inventory accurate as changes are implemented.