Senior Applied AI Engineer 2
About The Position
Drata is seeking a Senior Applied AI Engineer to improve the quality and effectiveness of its AI systems through rigorous experimentation, evaluation, and applied research. This research-focused role emphasizes experimentation and rigor over production engineering. The engineer will own how Drata's AI products retrieve, reason, and respond, working closely with AI and Software Engineers to translate validated approaches into production-ready systems. Drata's compliance platform is document-heavy, and products such as VRM Agent, AIQA, Trust Agent, and policy-to-control mappers rely on high-quality information retrieval and reasoning. The role involves designing and evaluating retrieval and reasoning strategies, prototyping GenAI workflows, exploring machine-learning approaches, maintaining evaluation frameworks, and validating improvements before production rollout.
Requirements
- 6+ years of experience in applied research, data science, or machine learning, focused on NLP, information retrieval, or knowledge systems.
- 2+ years of hands-on experience building or contributing to production AI/ML systems.
- Strong foundation in information retrieval, including dense and sparse retrieval, embedding models, and search relevance.
- Experience with RAG systems, including chunking strategies, vector databases, and retrieval optimization.
- Proficiency in evaluation methodology, including metrics design, golden dataset creation, A/B testing, and statistical significance.
- Strong Python skills and comfort with notebook-driven research workflows.
- Experience communicating research findings to engineering teams and translating insights into actionable improvements.
Nice To Haves
- Experience with compliance, legal, or document-heavy domains.
- Publications or contributions in information retrieval, NLP, or RAG evaluation.
Responsibilities
- Design and evaluate information access and reasoning strategies across RAG, agents, and classic machine learning, including chunking, embedding models, hybrid search, metadata filtering, and semantic routing.
- Prototype GenAI workflows, including agentic systems, that map and reason over compliance objects such as controls, risks, requirements, and evidence.
- Explore machine-learning and probabilistic approaches where GenAI is not the best fit, including classifiers, ranking models, graph/link prediction, calibration, and structured prediction.
- Build and maintain evaluation frameworks, including golden datasets, automated quality metrics, and regression detection.
- Implement and tune ranking and reranking systems, including cross-encoders, LLM-based rerankers, learning-to-rank, and custom scoring functions.
- Run experiments to validate hypotheses and quantify improvements before production rollout.
- Debug failure modes and build error taxonomies across retrieval, reasoning, and generation.
- Collaborate with AI and Software Engineers to hand off validated approaches for productionization.
- Stay current on applied research in RAG, agents, LLM evaluation, and relevance modeling, and bring innovations into the product.
Benefits
- Stock equity.
- Up to 100% employer-paid medical, dental, and vision premiums for employees and their dependents.
- Wellness benefits and healthcare concierge services.
- 401(k) plan.
- Company-paid life and disability insurance.
- Tax-advantaged spending accounts and discounted voluntary offerings.
- Paid parental leave after six months of employment.
- Kindbody fertility and family-building benefits.
- Dedicated leave specialists.
- Annual stipends for professional and personal development.
- Internal learning opportunities.
- Flexible vacation policy.
- Paid holidays.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
