AI Evals Engineer — Evaluation Datasets & Ground Truth

Prophetic Technologies Inc

39d • Hybrid

About The Position

Prophetic Software is seeking a full-time AI Evals Engineer to own ground truth for their AI-native platform. This role is crucial for producing, maintaining, and defending the evaluation datasets that measure the performance of various pipeline components, including prompts, models, classifier thresholds, and retrieval logic. The engineer will be responsible for defining what 'correct' means for each module, prioritizing validation efforts, and sourcing data through production sampling, human labeling, synthetic generation, or programmatic construction. The role involves versioning datasets, guarding against contamination, and reporting metrics accurately. This is a key position within a fast-scaling company that is disrupting the real estate development industry with AI.

Requirements

• Experience building or evaluating ML or LLM-powered systems in production, in a role where output quality was your problem.
• Have built evaluation or validation datasets before and can talk about one in detail: how you defined correctness, how you sourced labels, what went wrong, how you knew the labels were good.
• Working fluency in ML validation fundamentals: train/validation/test discipline, stratified sampling, precision/recall trade-offs, class imbalance, calibration, inter-rater agreement, basic significance testing and power.
• Understand feature engineering well enough to reason about why a classifier fails and what data would expose it.
• Strong Python and SQL; comfortable pulling and reshaping data yourself.
• Hands-on with LLM-based systems: prompting, structured outputs, agent/tool-use harnesses, and the specific ways they fail (non-determinism, prompt sensitivity, evaluator bias).
• Judgment about when LLM-as-judge is reliable and when it is not, and how to prove either.
• Can read a system design, understand the business logic it encodes, and translate that into a label schema without waiting to be told.

Nice To Haves

• Have run a human-labeling program end to end, including vendor selection, guideline authoring, QA sampling, and cost/quality trade-offs.
• Experience with eval tooling.
• Experience with data labeling platforms.
• Have used a frontier model as a distillation/oracle source and can articulate where that assumption breaks.

Responsibilities

• Decide what needs to be measured, and how.
• Read pipelines and system architecture, sit with product and engineering, and decompose each system into evaluable modules with explicit input → expected-output contracts.
• For each module, define what “correct” means in writing: rubrics, label schemas, edge-case policies, and the tolerances that matter to the business.
• Prioritize where a validation set unblocks the most iteration.
• Source the data by whatever means is cheapest and most trustworthy for that module (production sampling, human labeling, synthetic/oracle-generated ground truth, programmatic and adversarial construction).
• Version every dataset; track lineage, splits, and which model/prompt versions have seen which examples.
• Guard against contamination and leakage.
• Slice by customer segment, input type, and difficulty.
• Refresh sets as the product and traffic change; retire stale examples.
• Calibrate automated graders against human gold sets.
• Report metrics correctly: precision/recall/F1, confusion matrices, calibration, confidence intervals, sample sizes needed to detect a given effect.
• Partner with engineers who own the eval harness and CI, and with ML engineers on classifier features.

Benefits

• 100% medical, dental & vision insurance coverage for you; 30% coverage for dependents
• Competitive salary and meaningful early-stage equity
• Unlimited PTO
• Hybrid/Remote stipend
• In-office perks: snacks, drinks, coffee, ping-pong table, and more — plus cuddles from Olive, our in-office Doberman!
• Budget for intra-office travel
• 2–3 annual team meetups in person

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed