Digital Transformation, Chatbot Evaluator and Live Agent - Student Assistant

CSU Careers•San Bernardino, CA

13d•Onsite

About The Position

The purpose of the Ask Cody Chatbot Evaluator and Live Agent is to support the continuous improvement of AI-powered student services at CSUSB by evaluating chatbot performance, ensuring information accuracy, and enhancing user experience. This role contributes to the reliability and effectiveness of Ask Cody by identifying content gaps, validating data sources, and providing direct user support when needed.

Requirements

Ability to learn and perform assigned work.
Ability to follow oral instructions.
Ability to work cooperatively with faculty, staff, and other students.
Ability to communicate effectively with all levels.
Ability to read and write English at a level appropriate to the position.
Admission or registration as a CSU student.
Maintain a 2.0 G.P.A. or greater per semester and cumulative.
Students must be enrolled in at least halftime units at CSUSB (6 units undergraduate; 3 units graduate).
International students must have full-time enrollment at CSUSB (12 units undergraduate; 6 units graduate).
Eligible to work in the United States.
Possession of a Social Security Card.
Reside in the State of California.
Ability to work in the United States.

Nice To Haves

Basic Experience with Natural Language Processing (NLP) platforms
Familiarity with cloud platforms or cloud-based tools
Understanding of generative AI concepts, including Retrieval Augmented Generation (RAG)
Basic Experience with reporting or dashboard tools (e.g., Looker Studio, Google Sheets)
Knowledge of computing security best practices
Analytical and critical thinking abilities
High attention to detail and accuracy in reviewing information
Ability to evaluate written content for clarity, tone, and effectiveness
Ability to work independently and take initiative
Comfort using spreadsheets or tracking tools for documentation

Responsibilities

Evaluating chatbot interactions to ensure responses are accurate, clear, and effective.
Identifying errors, inconsistencies, and low-confidence responses.
Validating that links and referenced sources are functional and relevant.
Documenting recurring issues in chatbot performance.
Tracking patterns in chatbot performance and following up on previously identified issues to confirm improvements.
Proactively testing the chatbot using realistic, student-centered scenarios to uncover weaknesses and gaps in knowledge.
Analyzing chatbot behavior, including confidence indicators and grounding.
Assessing the reliability of underlying data sources.
Recommending practical, data-driven improvements for the chatbot.
Anticipating recurring student needs based on seasonal or cyclical trends to help guide future enhancements.
Cross-checking chatbot responses against official university sources to ensure accuracy and consistency.
Collaborating with campus departments to verify content, clarify discrepancies, and recommend updates to websites or other information repositories.
Supporting alignment across systems to maintain a high standard of reliable and up-to-date information.
Serving as a live agent when needed, responding to user inquiries in real time.
Helping direct users to appropriate resources or departments.
Documenting issues and observations during live agent interactions that reveal gaps in knowledge or processes.
Communicating regularly with supervisors to share updates, highlight priority concerns, and contribute to ongoing improvements of the chatbot system.