Staff Applied AI Product Engineer

Carrum Health

3h•Remote

About The Position

At Carrum, we are transforming how we pay for, deliver and experience healthcare. If you are passionate about changing healthcare and want to finally get rid of surprise bills, poor quality, and high prices, while thriving in an entrepreneurial, cutting-edge environment, we would love to connect with you. In 2014 Carrum reinvented the Centers of Excellence (COE) category in digital health. Today, 95% of the US population lives within 50 miles of a Carrum COE and our providers rank in the top 10% nationally. Our team’s execution has been recognized by the venture community and we’ve raised more than $96M in aggregate from investors like OMERS, Tiger Global Management and Wildcat Ventures. Our impact has been externally proven in a 2021 RAND Corporation study and featured as a Harvard Business School (HBS) case study. As a Staff Applied AI Product Engineer, you will bridge the gap between cutting-edge AI capabilities and real-world patient needs. You will take a hands-on leadership role in architecting and integrating LLM-driven features into our core platform. While our foundation is built on Ruby on Rails (SOA) and React, you will lead the evolution of our stack to support AI-native workflows and build agents to support key business initiatives such as improving care navigation quality and efficiency. You will work closely with Data and Product teams (including a dedicated Applied AI PM) to determine use cases that require LLMs rather than ML, build robust RAG (Retrieval-Augmented Generation) pipelines, select appropriate foundation models, iterate on prompts, and design the "glue" that turns raw model intelligence into reliable product features. You will partner with our DevOps team to leverage AWS/Azure for simple and cost-effective cloud infrastructure deployment, ensuring that systems are both performant and resilient. A key responsibility will be implementing comprehensive monitoring and observability practices to guarantee the high availability, security, and scalability of our new AI services. Additionally, you will collaborate directly with our internal clinical experts to validate model outputs, ensuring your AI agents are safe, accurate, and truly helpful in a healthcare setting. As a senior technical leader, you will own the AI engineering strategy, moving us beyond "demos" into scalable, production-ready systems. You will establish patterns for building “smart products,” model evaluation, latency optimization, and cost management, while ensuring we maintain strict data privacy and HIPAA compliance. You will champion Responsible AI by operationalizing the internal guidelines set by our AI Council, ensuring that fairness checks, bias detection, and safety guardrails are strictly implemented to serve all patient demographics equitably. You will mentor the wider engineering team on AI-native development practices (such as prompt engineering and context management) and collaborate with product managers to identify high-leverage opportunities where AI can radically simplify the healthcare experience. You will partner with the VP of Engineering and Chief Product Officer to staff and align delivery teams on your initiatives, while working with product teams to define project timelines and milestones from inception to go-live. This is a full time position, the salary range for this role is $190,000 - $260,000 depending on level of experience and geographic location.

Requirements

You have strong engineering roots: You have 10+ years of software development experience and have been the most senior engineer on a team. You know how to build robust, scalable, high availability systems, not just Jupyter notebooks.
You are an expert AI practitioner: You have 2+ years of dedicated professional experience architecting production-grade applications with LLMs (OpenAI, Anthropic, Llama). You haven't just built demos; you have gone deep into the complexities of RAG pipelines and vector databases (e.g., pgvector, Pinecone), solving for latency, hallucination, and retrieval accuracy at scale in customer facing products.
You go beyond AI/API basics: You know when to leverage prebuilt solutions versus custom LLM integrations and can apply fine-tuning and model optimization techniques when needed
You know the stack (or can learn it fast): You have experience working in a web application environment (Ruby on Rails, Python), have designed systems and know how to expose AI logic via performant APIs to frontend clients (React).
You include infrastructure in your designs: You are comfortable working with DevOps to design and configure Terraform for AI infrastructure, ensuring your services are deployable, reproducible, and scalable.
You act as a Tech Lead: You don't just write code; you drive execution and elevate the team. You enjoy technical leadership through code reviews, architectural design docs (RFCs), and teaching others, while also project managing cross-functional initiatives—breaking down ambiguous requirements, managing dependencies, and coordinating across teams to deliver on time.
You are AI-native: You use AI developer productivity tools (e.g., Cursor, GitHub Copilot) daily to accelerate your workflow and mentor the team on effective AI-assisted development.
You care about "The Last Mile": You understand that a model is only as good as the product wrapping it. You care about latency, error handling, monitoring/alerting and the user experience when the model hallucinates.
You are data-pragmatic: You are comfortable with SQL and data modeling, understanding that good AI requires good data infrastructure.
You are entrepreneurial: As a member of a startup, you excel at going from 0 to 1, finding creative ways to use AI to launch new products quickly and make a big impact with limited resources.
You manage complexity well: You have experience handling deadlines, keeping many balls in the air, and prioritizing work in a fast-paced, dynamic environment.
You value distinct human skills: You have interpersonal skills and don't have a hard time being empathetic, courteous, and friendly.

Responsibilities

Define the Gen AI technical roadmap: You will build the foundation of Applied AI function at Carrum and have a direct impact on how Carrum leverages Generative AI to automate complex healthcare coordination and improve patient outcomes.
Build "magic" features: Move fast to prototype, iterate and ship AI-powered experiences that feel magical to users, such as instant answer bots or automated appointment logistics.
Architect for the future: Be the sole expert on integrating vector databases, orchestration frameworks (like LangChain), and LLM APIs into a mature Service-Oriented Architecture.
Bridge the gap: Act as the translator between the "stochastic" world of AI models and the "deterministic" world of software engineering, ensuring reliability and trust.
Lead without ego: Mentor talented full-stack engineers on how to incorporate AI tools into their workflows, lifting the technical ceiling of the entire team.
Own the outcome: Take ownership of the full lifecycle of AI features—from prompt iteration and evaluation to production monitoring and user feedback loops.
Solve meaningful problems: Relish working with a diverse cross-functional group to solve actual pain points for patients, rather than just building tech for tech’s sake.
Prioritize quality: Hold yourself and others accountable to spending that extra 10% on a project to deliver great documentation in addition to the functionality itself.