Principal AI Operations Engineer (Remote)

Inspira Financial•Oak Brook, IL

3d•Remote

About The Position

The Principal AI Operations Engineer reports to the Sr. Director, AI Center of Excellence, in the Technology Department. This role defines how Inspira Financial’s AI systems are instrumented, measured, monitored, and evidenced, and drives enterprise adoption of those standards across vendor-provided and internally built AI capabilities. The role establishes enterprise telemetry and observability standards; designs AI evaluation, drift detection, and agent guardrail frameworks; translates AI risk and compliance frameworks into operational controls and audit evidence; maintains the operational AI system inventory; and enables engineering teams to adopt these standards independently. The position also builds operational reporting on AI usage, quality, cost, and risk, leads token and platform cost optimization, and partners with Risk, Compliance, Legal, Privacy, and Security.

Requirements

Bachelor of Science in Engineering or Computer Science.
At least 8 years of telemetry-driven engineering experience, including 2 years operating LLM, ML, or agentic systems in production, with hands-on ownership of instrumentation and monitoring.
Production-grade proficiency in Python and SQL, including building tooling, pipelines, and services used by other teams.
Demonstrated experience in AI evaluation and observability, including evaluation harnesses, tracing, scoring, drift detection, and guardrails.
Working command of at least one AI risk or governance framework, such as NIST AI RMF, ISO/IEC 42001, EU AI Act, or model risk management practice, and ability to convert it into controls and code.
Experience operating in a regulated environment where logging, retention, evidence, and access control are enforced.
Ability to influence without formal authority across engineering teams, legal counsel, and executive stakeholders.
Ability to occasionally work outside normal business hours to meet project deadlines as needed.
Prolonged periods of sitting at a desk and working on a computer.

Nice To Haves

Relevant certifications, such as Datadog Certified Professional, Google Cloud Professional Data Engineer for BigQuery ML, or PagerDuty’s AI Ops.
Financial services, healthcare, benefits administration, or other regulated industry experience, particularly in environments handling PHI, PII, or fiduciary data.
Depth of knowledge with telemetry platforms and related metric and measurement practices.
Hands-on experience integrating and operating enterprise platforms for AI and application observability, centralized logging, governance, risk, continuous compliance monitoring, AI system inventory, evaluation and testing, incident management, and cloud cost management.
FinOps or cloud cost management experience applied to token and inference spend.
AI security fluency, including prompt injection defense, OWASP LLM and agentic risk categories, red teaming, tool sandboxing, and least-privilege design for agent identities.
Track record of establishing a capability that other teams adopted voluntarily.

Responsibilities

Define enterprise telemetry standards for AI systems, including traces, prompts, tool calls, retrieval, tokens, latency, and cost, using a vendor-neutral schema such as OpenTelemetry GenAI semantic conventions.
Build a logging and observability layer that provides the AI Center of Excellence, business units, and AI Governance Committee with a unified view of AI activity across custom applications, agent platforms, and vendor-embedded AI.
Integrate AI telemetry with enterprise log aggregation, application performance monitoring, security monitoring, alerting, and incident response platforms.
Establish retention, redaction, masking, and access patterns for AI conversation and trace data that meet privacy, records retention, and legal hold obligations.
Publish reference implementations and starter kits for teams adopting the telemetry standard.
Design AI evaluation frameworks, including golden datasets, offline evaluation harnesses, automated and LLM-as-judge scoring, and regression gates in CI.
Define drift criteria and detection for output quality degradation, retrieval decay, input distribution shift, and silent provider or vendor model updates.
Build runtime guardrails and boundary enforcement for agents, including tool and action authorization, scope and spend limits, escalation paths, human-in-the-loop triggers, and hard-stop conditions.
Establish the AI incident model, including severity taxonomy, detection, alerting, rollback, postmortems, and converting incidents into test cases.
Translate NIST AI RMF, ISO/IEC 42001, model risk management practices, and applicable privacy and recordkeeping obligations into operational controls engineers can implement.
Build an evidence layer that automates capture of artifacts needed for internal audit, external examination, and client due diligence.
Integrate AI operational data with governance, risk, and compliance platforms to support continuous control monitoring, automated evidence collection, risk reporting, and audit readiness.
Maintain the operational AI inventory across custom builds, agents, integrations, and vendor AI features, including an owner, risk classification, and control coverage for each entry.
Define assurance patterns for vendor AI, including required telemetry, attestations, detection of vendor model changes, and compensating monitoring.
Build metrics connecting AI usage, quality, cost, and risk posture to business outcomes, and report them to CoE leadership, business unit leaders, and the AI Governance Committee.
Own token and platform cost visibility and attribution by team and use case, and lead optimization work.
Run working sessions, document patterns, review designs, and coach teams through their first fully instrumented AI deployment.
Partner with Risk, Compliance, Legal, Privacy, and Security to incorporate feedback from the risk triad and ensure adequate detection, response, and mitigation measures.
Perform other duties as assigned.