Senior AI Engineer
About The Position
Procore is hiring a Senior AI Engineer for its Evals & Goldsets team, which measures and improves the quality of AI agents developed for the construction industry. The role builds and operates systems that assess agent performance, including production tracing in Langfuse and Databricks, an online LLM judge, and goldsets and benchmarks used for offline evaluations. The engineer will partner with data and ML engineers to make evaluations faster, more cost-effective, trustworthy, and difficult to saturate, and will help turn production traces into feedback loops that improve agent behavior. The team combines hands-on development with architecture and design review, and values rigorous measurement and comfort with ambiguity. This is a hybrid position based in Procore’s Austin, Texas office, reporting to the Engineering Manager for Evals & Goldsets.
Requirements
- At least 5 years of experience in software, AI, or ML engineering.
- Strong proficiency in Python.
- Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.
- Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
- Experience building data pipelines, ETL processes, or streaming data systems.
- Experience with analytics platforms and data warehouses such as Databricks or Snowflake.
- Ability to communicate evaluation results clearly and influence the teams that build on them.
- A degree in computer science or a related technical field, or equivalent practical experience.
Nice To Haves
- Experience with TypeScript.
Responsibilities
- Improve online evaluations, including LLM judge accuracy, calibration, and the infrastructure that runs the judge in production.
- Build and maintain production trace infrastructure using Langfuse and Databricks.
- Build and maintain data pipelines for evaluation goldsets, including annotation and quality checks.
- Build agentic loops that mine production traces for failures and turn them into new evaluation cases and agent improvements.
- Improve the runtime, cost, and reliability of online and offline evaluation pipelines.
- Contribute code and architecture, review designs, and mentor other engineers on the team.
Benefits
- The role may be eligible for equity compensation and/or bonus incentive compensation.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
