Applied AI Engineer

FICO

1d•$105,000 - $165,000

About The Position

The myFICO Scores B2C Software Engineering team is hiring an Applied AI Engineer to own the quality, safety, and performance of AI-powered consumer features. The role combines Python and Java backend engineering with hands-on LLM experience, with responsibility for prompting, evaluating, guarding, and improving AI assistants and related features. The engineer will build and maintain tools that analysts and QA engineers use to measure AI quality and improve products using production evidence. The role includes owning prompts and evaluation methods for Personalized Credit Insights and the AI chat assistant; building regression and evaluation frameworks; assessing evaluator reliability against human-labeled data; designing safety and compliance guardrails; leading model upgrades and A/B tests; and improving cost and latency. The engineer will also build analyst and QA workflows, partner on fairness and bias evaluation, and use AI coding assistants and agentic workflows while reviewing and refining their output. FICO describes an inclusive, people-first work environment with opportunities for professional development, work/life balance, employee resource groups, and social events. The targeted base pay range is $105,000 to $165,000, reflecting differences in candidate knowledge, skills, and experience.

Requirements

Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Demonstrated professional software engineering experience.
Strong Python skills and ability to read and contribute to Java (Spring Boot) microservices.
Proven experience building and operating production LLM-powered applications, including prompt design, tool calling, structured outputs, and streaming.
Experience building evaluations for LLM systems, including test sets, regression suites, and LLM-as-judge approaches, and measuring evaluator quality against human labels.
Working knowledge of Amazon Bedrock or a comparable managed LLM platform, and AWS services such as S3, Lambda, and DynamoDB.
Proficient SQL skills and comfort analyzing production data in a data warehouse such as Amazon Redshift.
Production ownership experience including code review, CI/CD, release management, and on-call support within Agile/Scrum teams.
Strong written and verbal communication skills, including the ability to explain evaluation results and tradeoffs to product, legal, and executive stakeholders.

Nice To Haves

Experience with guardrail tooling.
Experience with LLM security testing.
Experience with RAG and retrieval evaluation.
Experience with fairness and bias evaluation.
Experience with model migrations.
Experience with observability tools such as Datadog.
Experience with BI tools such as Tableau.
Experience in regulated domains, including FCRA, ECOA, UDAAP, or CFPB guidance.

Responsibilities

Own and improve system prompts, tool descriptions, and evaluation prompts for Personalized Credit Insights and the AI chat assistant using production evidence.
Build and expand an evaluation framework, convert production failures into test cases, run regression suites before prompt or model changes, and report results.
Measure and improve the reliability of LLM-based evaluators against human-labeled data, tracking precision, recall, and agreement over time.
Design, tune, and monitor guardrails, including input and output classifiers, PII redaction, prompt-injection defenses, and financial-advice compliance rules defined with Legal and Compliance.
Lead model upgrades by rerunning evaluations, comparing quality, cost, and latency, and releasing changes safely through A/B tests.
Reduce cost and latency using prompt caching, model routing, and context management.
Build tools for Conversation Analysts and QA engineers, including transcript viewers, labeling workflows, and dashboards.
Partner with Analytics, Legal, and Compliance on fairness and bias evaluation of AI outputs.
Use AI coding assistants and agentic workflows, including Claude Code, Cursor, and GitHub Copilot, to accelerate delivery while reviewing and refining their output.
Participate in production ownership, including code review, CI/CD, release management, and on-call support within Agile/Scrum teams.