Senior Data Scientist, AI Evaluation
Syllo
•$160,000 - $200,000•Remote
About The Position
Syllo is hiring a Senior Data Scientist to evaluate and improve the quality and performance of its AI systems. The role develops rigorous ways to measure system behavior, runs experiments, investigates issues, and translates complex data into actionable decisions. It is a hands-on position at the intersection of data science, engineering, and applied AI, working directly with production data and systems and partnering closely with Engineering, Product, and Analytics.
Requirements
- Master's degree in computer science, data science, machine learning, or a related field with substantial hands-on experience, or 2–4 years of relevant professional experience.
- Strong Python and SQL skills and experience working directly with production data.
- Comfort navigating production code and data systems that the candidate did not build.
- Strong foundation in applied statistics, experimentation, and quantitative analysis.
- Practical experience working with and evaluating modern AI/ML systems.
- Experience diagnosing complex issues involving data, models, or production systems.
- Ability to turn ambiguous questions into structured investigations and clearly communicate the results.
Nice To Haves
- Experience building AI/ML evaluation frameworks, benchmarks, or monitoring systems.
- Experience with information retrieval, search, or large-scale information systems.
- Familiarity with modern cloud data platforms and production data infrastructure.
- Experience building internal analytics or observability tooling.
Responsibilities
- Develop quantitative methods and tooling to evaluate AI system quality and performance.
- Design and run experiments to compare approaches, understand tradeoffs, and measure the impact of changes.
- Build repeatable processes to identify regressions, anomalies, and changes in system performance.
- Extract, transform, and analyze production data to support technical and product investigations.
- Diagnose complex issues involving data, models, and production systems, identifying root causes and potential solutions.
- Partner with Engineering to improve measurement of system quality, reliability, and performance.
- Build tools and dashboards that make system performance easier to understand and monitor.
- Communicate findings, tradeoffs, and recommendations to technical and non-technical stakeholders.
Benefits
- Health insurance
- Equity
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Senior
