Gen AI Search Engineer

Morgan Stanley•New York, NY

18d

About The Position

We are seeking a talented and passionate Gen AI Search Engineer to revolutionize our search capabilities. In this pivotal role, you will be at the forefront of integrating cutting-edge Generative AI models, particularly from OpenAI, with our enterprise search platform powered by Lucidworks Fusion (and Apache Solr). You will design, develop, and implement intelligent search solutions that provide users with intuitive, contextually rich, and highly relevant information. This is an exciting opportunity to shape the future of how our organization accesses and utilizes knowledge. In the Technology division, we leverage innovation to build the connections and capabilities that power our Firm, enabling our clients and colleagues to redefine markets and shape the future of our communities. This is a Lead Software Engineering position at the Director level, which is part of the job family responsible for developing and maintaining software solutions that support business needs. Morgan Stanley is an industry leader in financial services, known for mobilizing capital to help governments, corporations, institutions, and individuals around the world achieve their financial goals. Interested in joining a team that’s eager to create, innovate and make an impact on the world? Read on. Enterprise Technology & Services (ETS) delivers shared technology services for Morgan Stanley supporting all business applications and end users. ETS provides capabilities for all stages of Morgan Stanley's software development lifecycle, enabling productive coding, functional and integration testing, application releases, and ongoing monitoring and support for over 3,000 production applications. ETS also delivers all workplace technologies (desktop, mobile, voice, video, productivity, intranet/internet) in integrated configurations that boost the personal productivity of employees. Application and end user functions are delivered on a scalable, secure, and reliable infrastructure composed of seamlessly integrated datacenter, network, compute, cloud, storage, and database functions.

Requirements

Bachelor's or higher degree in Computer Science, Engineering, or a related field.
Proven experience (typically 5+ years) in search engineering or a related software development
5+ years of Programming experience preferably using Python.
Strong understanding of search engine principles, information retrieval, and relevance ranking.
Hands-on experience working with Search Engine like Lucidworks Fusion, including pipeline development, query processing, and analytics.
Significant experience working with OpenAI APIs and models (e.g., completions, embeddings, fine-tuning).
Solid programming skills in languages such as Python and Java or experience with UI development such as AngularJS, ReactJS will be a plus
Experience with NLP techniques and libraries (e.g., NLTK, spaCy, Transformers).
Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and containerization technologies (e.g., Docker, Kubernetes).
Excellent problem-solving, analytical, and communication skills.
Ability to work independently and collaboratively in a fast-paced environment.

Nice To Haves

Experience with other enterprise search platforms (e.g., Elasticsearch, Solr, Microsoft Search).
Knowledge of vector databases (Redis) and similarity search techniques.
Experience with MLOps practices for deploying and monitoring AI models.
Contributions to open-source projects in the search or NLP domains.
Familiarity with agentic solutions like Copilot Studio.
Familiarity with agile development methodologies.

Responsibilities

Design, develop, and support existing Gen AI-powered search solution leveraging OpenAI models (e.g., GPT-3, GPT-4, embeddings).
Integrate and fine-tune large language models (LLMs) with GenAI search pipelines to enhance query understanding, semantic search, summarization, question answering, and content generation.
Develop and implement strategies for indexing, retrieving, and ranking information optimized for Gen AI applications.
Build robust and scalable APIs and microservices to connect Gen AI models with the Fusion platform.
Work on potential transition from Fusion and Redis Vector DB to Elastic Search.
Explore other GenAI search and agentic capabilities with Microsoft Search and Copilot Studio.
Develop robust backend search infrastructure for an Embeddings Lake to enhance search efficiency and accuracy.
Design and build APIs that allow seamless access to multiple content repositories
Experiment with different Gen AI techniques and evaluate their effectiveness in improving search relevance and user experience.
Monitor and analyse the performance of Gen AI-enhanced search features, identifying areas for optimization and improvement.
Stay up to date with the latest advancements in Generative AI, natural language processing (NLP), and search technologies.
Contribute to the development of best practices and documentation for Gen AI integration within the search architecture.
Troubleshoot and resolve technical issues related to Gen AI and Fusion integration