Machine Learning Engineer, Speech - Joint Audio-Video Modeling
About The Position
Cantina Labs is a social AI company developing advanced real-time models for expression, personality, and realism to bring characters to life and transform storytelling, connection, and creation. Cantina, our flagship social AI platform, is the beginning of our mission to shape human creativity and social interactions through AI. We are seeking a Research / ML Engineer to join our Speech Team to build state-of-the-art speech and audio generation systems end-to-end, with a focus on joint audio-video modeling. This role involves owning the audio side of multimodal generation, including representations, generative backbones, and conditioning/alignment for characters to speak, sing, and emote in sync with visuals. Responsibilities include voice cloning, multi-speaker conditioning, cinematic dialogue with music and sound design, and adjacent speech tasks. The role drives the model-data-evaluation flywheel, collaborating with research, video, data, and infra teams to ship fast, reliable, and cost-aware models at the intersection of research and engineering, contributing to safe, steerable, and trustworthy AI systems.
Requirements
- Exceptional research/development experience with large-scale audio models (>8B parameters, >500k hours of data).
- Deep hands-on experience with diffusion and/or flow-matching transformers, including practical knowledge of samplers, schedules, conditioning mechanisms, and distillation.
- Deep hands-on experience training audio VAEs, neural audio codecs, and vocoders (latent/tokenizer design, reconstruction and perceptual objectives, adversarial training).
- Strong experience with multi-node, multi-GPU distributed training (FSDP/DeepSpeed or equivalent).
- Strong software engineering skills with a proven track record of building complex systems.
- Strong with PyTorch and performance work (profiling, CUDA/Triton/C++ as needed) and writing reliable production-quality code.
- Shipped large-scale speech/audio or multimodal generative models to production.
- Background in working with large-scale ML data, and the ability to iterate on data and triangulate quality using both subjective and objective signals.
- Experience with voice cloning, speech control/steerability, or expressive speech generation.
- Notable publications and/or open-source contributions in speech/audio/ML.
Nice To Haves
- Experience with multimodal audio-video modeling: joint AV generation of multi-shot, multi-speaker scenes with dialogue, music, and sound design generated jointly with video, and the cross-modal alignment that keeps them in sync.
- Experience with video generation: video diffusion/flow-matching transformers, video VAEs, conditioned and multi-shot generation, building data pipelines for video models.
- Streaming or real-time generation, causal distillation (e.g., Self Forcing / Self Forcing++).
Responsibilities
- Design, train, and improve audio VAEs, neural codecs, and vocoders for generative models, focusing on latent design, reconstruction, perceptual objectives, and compression-vs-fidelity tradeoffs.
- Architect, implement, pre-train, fine-tune, and post-train/align diffusion and flow-matching transformers for large-scale audio and video generation.
- Design audio conditioning and cross-modal alignment within joint AV models, including audio latents, reference-audio and multi-speaker conditioning, and multi-shot generation audio/video modeling.
- Design, run, and analyze scientific experiments to advance understanding of the models.
- Define data requirements and collaborate on acquisition, curation, AV-sync and quality filtering, annotation quality, and synthetic data strategies for paired audio-video and speech corpora.
- Design automated objective/subjective evaluations, including audio fidelity and intelligibility metrics, AV-sync, listening and viewing tests, robustness & bias checks, and red-team studies.
- Drive inference efficiency through distillation, step-count reduction, quantization, and kernel/memory optimization to meet interactive latency and cost targets.
- Harden the training → evaluation → inference pipeline; profile latency, memory, and cost; and meet production SLAs with robust monitoring and rollback.
- Partner with infrastructure to run distributed training/inference on cloud fleets and productionize models with reliability and observability.
- Independently lead small research projects while collaborating on larger team initiatives, including cross-team work with video generation.
- Develop and improve dev tooling to enhance team productivity.
- Contribute to safety/consent guardrails, watermarking, and misuse/abuse mitigation for responsible voice and likeness technology.
Benefits
- Competitive salary and generous company equity
- Medical, dental, and vision insurance – 99.99% of premiums covered by Cantina
- 42 days of paid time off, including: 15 PTO days, 10 sick days, 15 company holidays, 2 floating holidays
- Generous parental leave & fertility support
- 401(k) retirement savings plan
- Lifestyle spending account – $500/month to use however you’d like
- Complimentary lunch and snacks for in-office employees
- One Medical membership
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
