Manager, Software Engineering - AI/ML Evaluation Platform
About The Position
We’re seeking Engineering Managers with AI Evals Framework experience to drive and scale the validation of our AI systems while staying deeply hands-on with both infrastructure and product delivery. You’ll sit at the intersection of ML, data, and developer experience, building the tooling and platforms that help every team at Supio ship reliable AI faster. A bit about you Track record of shipping complex technical products, internal platforms, or developer tools at startup speed. Deep hands-on experience building infrastructure for LLM evaluation, including datasets, runners, and metrics. Experience designing APIs and SDKs that other engineers genuinely enjoy using. Strong data engineering fundamentals; you care about dataset versioning, schema validation, and reproducibility. Bachelor’s in CS or related field; 2+ years managing engineering teams or leading technical initiatives.
Requirements
- Track record of shipping complex technical products, internal platforms, or developer tools at startup speed.
- Deep hands-on experience building infrastructure for LLM evaluation, including datasets, runners, and metrics.
- Experience designing APIs and SDKs that other engineers genuinely enjoy using.
- Strong data engineering fundamentals; you care about dataset versioning, schema validation, and reproducibility.
- Bachelor’s in CS or related field; 2+ years managing engineering teams or leading technical initiatives.
Responsibilities
- Lead and grow a team of 6–10 engineers while architecting a self-service evaluation platform used across product teams.
- Create the tooling that allows product teams to define, run, and visualize their own tests without being blocked by your team.
- Integrate our HITL workflows and subject-matter experts into the model evaluation pipeline through intuitive UI/UX and clear developer surfaces.
- Define and implement company-wide metrics for latency, cost, and accuracy across diverse tasks and products.
- Own the full lifecycle of our evaluation platform—from roadmap and architecture to execution, rollout, and continuous improvement.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Manager
