Model Engineer, Multimodal Understanding

SpaceXSI•Palo Alto, CA

3d•$160,000 - $250,000

About The Position

The Model Engineer, Multimodal Understanding will join the multimodal team to advance multimodal intelligence across image, video, audio, and text. The work spans the full stack, including data curation and acquisition, tokenizer training, large-scale pre-training, post-training and alignment, infrastructure and scaling, evaluation, tooling and demonstrations, and end-to-end product experiences. The role collaborates with pre-training, post-training, reasoning, data, applied, and product teams to deliver capabilities in multimodal reasoning, world modeling, tool use, agentic behaviors, and interactive human-AI collaboration. It contributes to building models that can perceive, reason about, and interact with the world in real time.

Requirements

Experience writing production-quality code in Python, PyTorch, JAX, Rust, or C++.

Nice To Haves

Expert-level proficiency in Python and strong experience with at least one of JAX, PyTorch, or XLA.
Track record building or optimizing large-scale distributed ML systems, including training or inference optimization, GPU utilization, multi-GPU/TPU setups, or hardware co-design.
Deep experience designing and running data pipelines at scale, including curation, filtering, generation, and quality studies, especially for noisy or real-world multimodal data.
Strong fundamentals in evaluation design, benchmarks, reward modeling, or reinforcement learning techniques, particularly for interactive or agentic behaviors.
Proactive, self-starting approach; ability to thrive in high-intensity environments and interest in advancing multimodal AI.
Willingness to own end-to-end initiatives and do what is needed to deliver breakthrough user experiences.
Experience leading major improvements in model capabilities through better data, modeling, algorithms, or scaling.
Familiarity with state-of-the-art multimodal LLMs, scaling laws, tokenizers, compression techniques, reasoning, or agentic systems.
Proficiency in Rust and/or C++ for performance-critical components.
Hands-on experience with large-scale orchestration tools such as Spark, Ray, or Kubernetes.
Background building full-stack tooling, including performant interfaces, real-time research demos or apps, or end-to-end product ownership.
Interest in end-to-end user experience for interactive, real-time multimodal AI systems.

Responsibilities

Design, build, and optimize large-scale distributed systems for multimodal pre-training, post-training, inference, data processing, and tokenization at web or petabyte scale.
Develop high-throughput pipelines for data acquisition, preprocessing, filtering, generation, decoding, loading, crawling, visualization, and management of images, videos, audio, and text.
Advance multimodal capabilities, including spatial-temporal compression, cross-modal alignment, world modeling, reasoning, emergent abilities, audio/image/video understanding and generation, real-time video processing, and handling noisy data.
Drive data-quality efforts and studies, including human and synthetic curation, filtering techniques, analysis, and scalable pipelines to support trillion-parameter models.
Create evaluation frameworks, internal benchmarks, reward models, and metrics that capture real-world usage, failure modes, interactive dynamics, and human-AI synergy.
Innovate on algorithms, modeling approaches, hardware/software/algorithm co-design, and scaling paradigms to achieve state-of-the-art performance.
Build research tooling, user-friendly interfaces, prototypes, demonstrations, and full-stack applications, enabling rapid iteration based on feedback.
Work across pre-training, SFT, RL, and post-training to enable reasoning, tool calling, agentic behaviors, orchestration, and seamless real-time interactions.