Conversational AI Specialist, Vice President

Morgan Stanley•Purchase, NY

22d•Onsite

About The Position

Morgan Stanley is seeking a Conversational AI Specialist to join its Wealth Management Generative AI Team, a firm-wide Center of AI enablement. The team deploys generative AI systems to optimize business processes, oversees governance and controls for ethical and responsible AI use, and provides AI education across the company. In this role, you will design enterprise-level conversational AI solutions, develop automation tools, and refine generative AI models to produce reliable, high-quality outputs. You will work with business teams across the Firm, technology partners, vendors, and centralized AI teams to accelerate the development and adoption of AI-based solutions. The role focuses on employee and client virtual assistant experiences, prompt development, model testing and evaluation, troubleshooting, risk management, stakeholder communication, collaboration, and performance reporting.

Requirements

Experience with conversational AI solutioning, product development, experimentation, or innovation.
Experience with prompt engineering for commercial or open-source LLMs such as GPT, Claude, Gemini, or LLaMA.
Experience with conversational AI evaluation techniques such as F1, RAGAS, and human feedback analysis.
Strong understanding of artificial intelligence concepts and technologies.
Excellent communication, interpersonal, and leadership skills.
Proven experience in program or project management.
Ability to manage multiple projects simultaneously in a fast-paced environment.
Solid analytical and problem-solving abilities, with keen attention to detail.

Responsibilities

Develop and maintain conversational experiences for employee and client virtual assistants, enabling access to knowledge, data, and workflows.
Develop and refine context-aware prompts for different use cases, domains, and user intents to improve the accuracy, relevance, and efficiency of chatbot responses.
Ensure prompt solutions are scalable, sustainable, integrated with technical infrastructure, and meet business needs.
Develop and maintain data-driven methodologies to test and evaluate generative AI responses for coherence, bias, factual accuracy, and user satisfaction.
Create and manage manual and automated human-evaluation strategies, including LLM-as-a-judge, to ensure model outputs meet high standards.
Troubleshoot issues and anomalies in LLM outputs and recommend improvements and optimization.
Identify potential project risks and delivery issues, develop mitigation strategies, monitor performance, and implement corrective actions.
Maintain effective communication with internal stakeholders, including technology partners and business product owners; manage expectations and address concerns.
Collaborate with cross-functional teams, internal stakeholders, and vendor partners.
Track project performance metrics and KPIs to measure progress and identify areas for improvement.