Senior AI Engineer

Florence Healthcare - US•Atlanta, GA

11d

About The Position

Florence software advances cures by helping the world’s most important research sites do their best work. Our solutions are now used by over 30,000 research teams in 70 countries around the world—we’re the most widely deployed site workflow tool in the industry. By the end of the decade, we’ll double the pace at which new medicines get to market by doubling the output of trial site teams. At Florence, we are committed to make the world a better place by accelerating research while providing an environment for our employees where they can be happy in their lives, enjoy their jobs, and grow.

Requirements

4+ years of related work experience
Solid backend/system design fundamentals
Experience building and deploying production-grade systems
Ability to debug complex issues, including probabilistic outputs
Comfort working with APIs, pipelines, and data flows
Ability to translate user needs into effective AI solutions
Strong intuition for balancing quality, latency, and cost
Focus on delivering measurable product impact
Communicates clearly across engineering and product teams
Contributes to team knowledge and shared practices
Strong understanding of LLM capabilities and limitations
Experience with prompt engineering and structured output design
Hands-on experience with embeddings and vector search
Familiarity with RAG architectures and when to apply them
Experience designing agent-based architectures (AgentCore concepts)
Understanding of tool use, planning strategies, and memory mechanisms in LLM systems
Evaluate agent performance across multi-step tasks (task success rate, error propagation)
Debug and optimize agent decision-making and tool selection behavior

Responsibilities

Design and implement AI-powered features (LLM workflows, copilots, and agent-based systems with tool use and multi-step reasoning)
Own the full lifecycle: prototyping → evaluation → production deployment → iteration
Ensure solutions are reliable, performant, and aligned with product needs
Build and optimize: Prompt pipelines for specific use cases, Retrieval systems (embeddings, chunking, ranking), RAG-based workflows where needed
Iterate on outputs to improve quality, accuracy, and consistency
Design scalable and cost-efficient AI architectures for production workloads
Select and evaluate models (hosted vs open-source) based on use case constraints
Design and build agentic workflows capable of multi-step reasoning and decision-making
Integrate agents with tools, APIs, and internal systems to perform real-world actions
Implement planning, execution, and reflection loops for complex tasks
Manage context, memory, and state across multi-step interactions
Balance deterministic workflows vs. agent autonomy for reliability and control
Run structured experiments to compare approaches (prompting, retrieval, models)
Define and track key metrics for AI performance (quality, latency, cost)
Debug and improve non-deterministic system behavior
Build and maintain evaluation datasets and benchmarks
Implement automated evaluation pipelines for continuous improvement
Drive technical direction and influence AI adoption across teams
Partner with product managers and designers to scope AI features
Contribute to shared patterns and reusable components
Participate in code reviews and design discussions
Support and mentor mid-level engineers where needed
Design guardrails to ensure safe and reliable AI behavior
Mitigate hallucinations, prompt injection, and model misuse
Ensure compliance with data privacy and enterprise requirements
Implement monitoring and observability for AI systems in production
Implement guardrails for agent actions (tool access control, execution boundaries)
Prevent failure cascades in multi-step agent workflows