Digital Transformation, Chatbot Evaluator and Live Agent - Student Assistant

California State University, San Bernardino•San Bernardino, CA

23d•Onsite

About The Position

The purpose of the Ask Cody Chatbot Evaluator and Live Agent is to support the continuous improvement of AI-powered student services at CSUSB by evaluating chatbot performance, ensuring information accuracy, and enhancing user experience. This role contributes to the reliability and effectiveness of Ask Cody by identifying content gaps, validating data sources, and providing direct user support when needed.

Requirements

Ability to learn and perform assigned work.
Ability to follow oral instructions.
Ability to work cooperatively with faculty, staff, and other students.
Ability to communicate effectively with all levels.
Ability to read and write English at a level appropriate to the position.
Admission or registration as a CSU student.
Maintain a 2.0 G.P.A. or greater per semester and cumulative.
Students must be enrolled in at least halftime units at CSUSB (6 units undergraduate; 3 units graduate).
International students must have full-time enrollment at CSUSB (12 units undergraduate; 6 units graduate).
Eligible to work in the United States.
Possession of a Social Security Card.
Must reside in the State of California.
Satisfactory completion of a background check.
Verification of identity and eligibility to work in the United States.
Completion of the required employment eligibility verification form upon hire.

Nice To Haves

Basic Experience with Natural Language Processing (NLP) platforms
Familiarity with cloud platforms or cloud-based tools
Understanding of generative AI concepts, including Retrieval Augmented Generation (RAG)
Basic Experience with reporting or dashboard tools (e.g., Looker Studio, Google Sheets)
Knowledge of computing security best practices
Analytical and critical thinking abilities
High attention to detail and accuracy in reviewing information
Ability to evaluate written content for clarity, tone, and effectiveness
Ability to work independently and take initiative
Comfort using spreadsheets or tracking tools for documentation

Responsibilities

Evaluating chatbot interactions to ensure responses are accurate, clear, and effective.
Identifying errors, inconsistencies, and low-confidence responses.
Validating that links and referenced sources are functional and relevant.
Documenting recurring issues.
Tracking patterns in chatbot performance and following up on previously identified issues to confirm that improvements have been successfully implemented.
Proactively testing the chatbot using realistic, student-centered scenarios to uncover weaknesses and gaps in knowledge.
Analyzing chatbot behavior, including confidence indicators and grounding.
Assessing the reliability of underlying data sources.
Recommending practical, data-driven improvements.
Anticipating recurring student needs based on seasonal or cyclical trends to help guide future enhancements.
Ensuring the accuracy and consistency of information by cross-checking chatbot responses against official university sources.
Collaborating with campus departments to verify content, clarify discrepancies, and recommend updates to websites or other information repositories.
Supporting alignment across systems to maintain a high standard of reliable and up-to-date information.
Serving as a live agent when needed, responding to user inquiries in real time and helping direct users to appropriate resources or departments.
Documenting issues and observations that reveal gaps in knowledge or processes during live agent interactions.
Regularly communicating with supervisors to share updates, highlight priority concerns, and contribute to ongoing improvements of the chatbot system.