AI Quality Assurance Analyst Associate(Fellowship Program)
About The Position
The Information Technology (IT) Division lays the foundation for TRS to deliver excellent service experiences across the organization and with our members. We serve with purpose through mentorship and collaboration across a broad variety of teams unified by innovation to create technology and information solutions that have a positive impact on our members’ lives. We invite you to join one of Austin’s Top Workplaces. TRS offers a best-in-class combination of technology and continuous learning opportunities to equip you to solve problems, expand your knowledge, and create impact for 1 in 20 Texans. The AI QA Analyst Associate will support the Testing and Quality Assurance team within the AI Technology Platform (AITP) program. The incumbent will focus on validating AI-enabled solutions, including LLM agents, and intelligent automation solutions. Responsibilities include evaluating model behavior, prompt effectiveness, response accuracy, system reliability, as well as testing AI workflows, data pipelines, and system integrations, assist in developing automated testing strategies, supporting CI/CD pipelines, and improving regression test coverage. This Fellowship Program position is a full-time, temporary, limited term assignment not to exceed three years from the date of hire. It is intended for applicants that have graduated with a bachelor's degree or a master's degree from an accredited college or university in Computer Science, Information Technology, Software Engineering, Data Science, or a closely related field awarded within the last 12 months. This position requires onsite work at our Austin TX headquarters.
Requirements
- Bachelor’s degree or Master's degree in Computer Science, Information Technology, Software Engineering, Data Science, or a closely related field awarded within the last 12 months.
- None.
Nice To Haves
- Experience with prompt engineering
- Proficiency in Python or a similar programming language
- Experience with database queries (e.g., SQL) for validating data integrity
- Familiarity with testing AI workflows or data-driven applications
- Exposure to Microsoft Copilot Studio, including testing conversational agents and integrations
- Familiarity with Azure AI services (e.g., Azure AI Foundry) and understanding of AI data flows
- Exposure to CI/CD tools and automated testing frameworks
- Fundamentals of Machine Learning and Large Language Models
- AI system evaluation techniques
- Prompt engineering concepts and conversational AI design
- Responsible AI principles
- Basic QA methodologies, software testing concepts, and automation approaches
- Software development lifecycle and modern DevOps practices
- Evaluating AI generated outputs using quantitative and qualitative methods
- Writing effective prompts for conversational AI systems
- Using evaluating datasets and defining expected AI behavior
- Writing and executing test cases (manual and automated)
- Analyzing data and identifying defects or inconsistencies
- Communicating technical findings to technical and non-technical stakeholders
- Managing multiple tasks in a structured, team-based environment
- Analyze AI system behavior and identify non deterministic issues
- Evaluate ambiguous outputs and apply judgement based validation
- Collaborate with data scientists and AI engineers on model improvements
- Work with business users on requirement clarification, user acceptance tests, and troubleshooting issues.
- Plan, organize, and coordinate work assignments to effectively meet frequent and/or multiple deadlines; handling multiple tasks simultaneously; and managing conflicting priorities and demands.
- Work in a team test environment.
- Analyze problems and devising effective solutions.
- Effective verbal and written communication of complex technical information
- Establish and maintain harmonious working relationships with co-workers, agency staff, and external contacts.
- Work effectively in a professional team environment.
Responsibilities
- Assists in designing, developing, and executing automated test strategies for AI-enabled applications.
- Supports testing of AI workflows, data pipelines, and system integrations.
- Assists with prompt testing, hallucination detection, and evaluation of AI model performance using defined metrics.
- Supports basic bias and safety validation activities and participates in human-in-the-loop review processes to ensure accurate and appropriate AI outputs.
- Identifies and documents defects, inconsistencies, and risks during testing cycles.
- Assists with validating AI model inputs, outputs, and data flows to ensure accuracy and reliability.
- Supports testing of AI-driven features, including conversational agents and workflow automation.
- Supports data analysis related to AI system performance, including reviewing evaluation outputs, identifying trends,
- Assisting with reporting model quality metrics.
- Participates in trainings and projects related to the TRS Fellowship Program including webinars, quarterly check-ins, classes or other learning activities, book group, projects, and the mentoring program.
- Meets TRS Fellowship Program deadlines and goals for completion of trainings in deliverables in a timely manner.
- Performs related work as assigned.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Entry Level
