Machine Learning Engineer - AI

Relias•Morrisville, NC

15d•Hybrid

About The Position

Are you looking for a high energy, strategic, and fast-paced position as an AI Engineer? Join Relias, the company changing lives throughout the world by helping healthcare organizations improve their clinical and financial outcomes! For 11,000+ health care and human service organizations, Relias helps clients deliver better clinical and financial outcomes by elevating the performance of teams. We help organizations across the continuum of care get better at maintaining compliance, developing staff and promoting consistent, high-quality care. Our platform employs assessments to reveal specific gaps in skills and addresses them with personalized and engaging learning, choosing from 7,000+ online courses that meet accrediting board, state and federal requirements. We are passionate about our products and our clients; what we deliver and the impact we have on the world is truly something you can be proud to represent. Join us and make a difference. WHAT CAN RELIAS OFFER YOU? Fantastic health and wellness benefits package, including an outstanding 401k match, a flexible PTO program, and a generous and inclusive parental leave policy. Additionally, Relias pays for the employee portion of the monthly healthcare premium! Flexible work environment with onsite and work from home options – you choose when you want to come into the office! Active Employee Resource Groups open to all employees! Comprehensive onboarding program – a great introduction to our company, customers and culture! Growth and career advancement opportunities! 20%+ annual employee promotion and transfer rate Multiple development program options – leadership development, professional development curriculums, and Nanodegree options in both technology and data science Professional development gained from conference attendance and participation in organizations like NC Tech Onsite 321 Coffee Shop providing free coffee and pastries to employees We are seeking a highly skilled and forward-thinking AI Engineer to design and deploy scalable LLM-based systems within our Data & Generative AI team. This role is instrumental in building robust, repeatable, and LLM-agnostic infrastructure that supports the full lifecycle of generative AI applications. The AI Engineer will leverage protocols such as the Model Context Protocol (MCP) and Agent-to-Agent (A2A) communication to enable intelligent, autonomous system behavior across our product ecosystem. This is a hands-on engineering role that demands fluency in model orchestration, rigorous validation frameworks, and a deep understanding of production-grade AI infrastructure.

Requirements

5+ years of experience in AI/ML engineering, with 2+ years specifically in building and deploying LLM-based systems in production.
Bachelor’s degree in Data Science, Computer Science, or a related field.
Protocol-Oriented LLM Design: Proficiency in leveraging Model Context Protocol (MCP) and Agent-to-Agent (A2A) architecture to enable intelligent coordination between AI systems.
LLM Infrastructure Engineering : Strong experience building prompt management, context orchestration, and evaluation frameworks that are model-agnostic and production-ready.
AI Evaluation & Monitoring : Skilled in designing and operationalizing LLM testing pipelines using synthetic evaluation, unit prompting, and human-in-the-loop feedback loops.
Role Model and Mentorship: Strong coding skills in Python and AI toolchains (e.g., LangChain, LlamaIndex, MLflow, FastAPI, etc.), with experience in cloud platforms (Azure preferred) and containerization (Docker, Kubernetes).

Nice To Haves

Master’s degree or Ph.D. in Data Science, Computer Science, or a related field.
Experience with evaluating multiple LLMs and integrating open-source model

Responsibilities

LLM-based System Implementation: Architecting, building, and deploying generative AI systems using large language models. Applying MCP and A2A protocols to ensure dynamic, context-aware, and multi-agent capabilities across AI workflows.
LLM-Agnostic Infrastructure Development: Developing modular, reusable infrastructure components that support integration with various foundation models (e.g., OpenAI, Azure OpenAI, open-source LLMs). Enabling prompt chaining, context caching, and retrieval-augmented generation (RAG) in a scalable manner.
Testing and Validation Frameworks: Designing and implementing frameworks for automated evaluation, regression testing, and performance monitoring of LLMs. Ensuring high standards for factual consistency, bias mitigation, hallucination control, and user alignment.
Cross-functional Integration : Collaborating with Data Scientists, Engineers, and Product Managers to translate AI capabilities into real-world applications. Supporting the integration of LLM functionality into user-facing tools and backend systems.
Documentation and Maintainability: Writing high-quality, maintainable code with clear documentation to support auditability, version control, and team collaboration. Contributing to internal knowledge sharing and onboarding materials.

Benefits

Fantastic health and wellness benefits package, including an outstanding 401k match, a flexible PTO program, and a generous and inclusive parental leave policy.
Additionally, Relias pays for the employee portion of the monthly healthcare premium!
Flexible work environment with onsite and work from home options – you choose when you want to come into the office!
Active Employee Resource Groups open to all employees!
Comprehensive onboarding program – a great introduction to our company, customers and culture!
Growth and career advancement opportunities!
20%+ annual employee promotion and transfer rate
Multiple development program options – leadership development, professional development curriculums, and Nanodegree options in both technology and data science
Professional development gained from conference attendance and participation in organizations like NC Tech
Onsite 321 Coffee Shop providing free coffee and pastries to employees