Member of Technical Staff, Evals Platform

Fireworks • San Mateo, CA

18h

About The Position

Fireworks is seeking a Member of Technical Staff to help define how developers evaluate and improve models on its AI platform. This role combines scalable systems engineering, data science, and model quality, building infrastructure and workflows that connect evaluation with post-training. The position will advance an evaluation setup that has outgrown its original scope, improving its scale and programmatic access across backend systems, sandbox infrastructure, and user-facing products.

The role includes evolving internal evaluation infrastructure, rigorously investigating and reproducing benchmark results, creating new benchmarks for emerging or domain-specific use cases, and improving fine-tuning workflows across supervised fine-tuning (SFT), reinforcement fine-tuning (RFT), and related capabilities. The successful candidate will work with customers and internal stakeholders to understand needs, troubleshoot issues, and turn bespoke processes into reusable product solutions.

Fireworks describes itself as a platform for specialized intelligence, founded by the team behind PyTorch and backed by several investors. It is a Series D company valued at $17.5 billion. The company offers the opportunity to work on AI infrastructure and collaborate with engineers and AI researchers in a fast-growing team.

Requirements

• 1–7 years of software engineering or data science experience; hiring is at multiple levels.
• Strong system-design skills, including architecting scalable, programmatic systems and transitioning legacy setups into robust infrastructure.
• Hands-on experience building or working with sandbox environments and infrastructure for secure code execution and testing.
• Deep understanding of LLM evaluations, how to design them, and how to use results to guide model improvement.
• Meticulous attention to data and metrics.
• Understanding of the end-to-end generative AI lifecycle, from prompting a base model through dataset curation and fine-tuning to productionizing agents.

Nice To Haves

• 3+ years of software engineering or applied data science experience.
• Experience with the Harbor framework or similar container registry and orchestration tools.
• Interest in identifying where different models excel or fall short and publishing findings, such as technical blogs or whitepapers.
• Interest in AI hardware, including GPU constraints and inference optimization techniques, and how these relate to model performance.
• Experience in fast-paced environments with end-to-end feature ownership.

Responsibilities

• Take ownership of the internal evaluation setup and evolve it into scalable infrastructure.
• Design systems that eliminate single-host coordination bottlenecks, reduce log-synchronization latency, and provide seamless programmatic access.
• Track state-of-the-art benchmarks, review research papers, investigate data discrepancies, and rigorously reproduce published results internally.
• Design and build new benchmarks for complex, emerging, or domain-specific use cases.
• Build and improve user-facing post-training workflows, including fine-tuning experiences across SFT, RFT, and related model-improvement capabilities.
• Partner with customers and internal stakeholders to understand evaluation and fine-tuning needs, triage issues, and turn bespoke workflows into reusable product solutions.

Benefits

• Work on challenging AI infrastructure problems, including low-latency inference and scalable model serving.
• Work with cutting-edge technology that affects how businesses and developers use AI.
• Ownership and direct impact in a fast-growing team.
• Collaboration with engineers and AI researchers.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level
• Education Level: No Education Listed