AI Engineer - Quality and Post-Training

Glean • San Francisco, CA

5d • Hybrid

About The Position

Glean is hiring an AI Engineer to improve the quality of its enterprise AI models, assistants, and autonomous agents. The role sits at the intersection of applied research, quality, and product engineering, focusing on measuring system performance and closing gaps through improvements to models and agent harnesses. Work includes using evaluations and production traces to guide changes to the agent harness, model routing, prompts, and retrieval, while balancing quality, latency, and cost.

The engineer will post-train models for specialized use cases, build simulated enterprise workflow environments, and develop evaluation sets, LLM-based judges, and agentic verifiers. They will bring emerging research into the product and partner with product, platform, and research teams to turn evaluation results and customer and production learnings into quality improvements shipped to production.

This is a hybrid role requiring work four days per week in Glean’s Mountain View or San Francisco office. The standard annual base salary range is $200,000–$280,000. Compensation may also include variable compensation, equity, and benefits. Glean offers medical, vision, and dental coverage, a generous time-off policy, a 401(k) plan, a home office improvement stipend, annual education and wellness stipends, regular company events, and daily healthy lunches.

Requirements

• At least 5 years of industry experience in AI or Machine Learning Engineering.
• Hands-on experience post-training LLMs, including adapting open-weight models to specialized tasks.
• Experience building evaluations for AI assistant and agent systems, including generating evaluation sets and LLM-as-a-judge verifiers.
• Experience calibrating offline and online signals, with good judgment about whether a metric reflects quality improvements.
• Strong coding skills in Python and/or Go; TypeScript is a plus.
• Proficiency in an ML framework.
• A track record of shipping production ML systems.
• Product-minded approach and ability to turn vague problems into concrete experiments.
• Ability to thrive in a close-knit, cross-functional team.

Responsibilities

• Use evaluation results and production traces to improve the agent harness, model routing, prompts, and retrieval while considering quality, latency, and cost.
• Post-train models using supervised fine-tuning and preference- or reward-based optimization methods such as DPO, PPO, and GRPO, including generating training data.
• Build reinforcement-learning environments to simulate enterprise workflows, including evaluation sets, harness support, and LLM-based verifiers for iterative improvement and benchmarking.
• Improve representative evaluation sets, LLM judges, and agentic verifiers, and calibrate them against human graders for trusted release decisions and model training.
• Work with product, platform, and research teams to ship quality improvements to production.
• Work with product teams and customers to translate evaluation and production learnings into improved outcomes.

Benefits

• Medical coverage
• Vision coverage
• Dental coverage
• Generous time-off policy
• 401(k) plan
• Home office improvement stipend
• Annual education stipend
• Annual wellness stipend
• Regular company events
• Daily healthy lunches
• Potential variable compensation and equity, depending on role eligibility

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed