Model Engineer, Multimodal Understanding

xAI•Palo Alto, CA

About The Position

Join SpaceXSI’s multimodal team to advance multimodal intelligence across image, video, audio, and text. The role spans the full stack, including data curation and acquisition, tokenizer training, large-scale pre-training, post-training and alignment, infrastructure and scaling, evaluation, tooling and demonstrations, and end-to-end product experiences. Work cross-functionally with pre-training, post-training, reasoning, data, applied, and product teams to deliver frontier capabilities in multimodal reasoning, world modeling, tool use, agentic behaviors, and interactive human-AI collaboration. Contribute to models that can see, hear, reason about, and interact with the world in real time.

Requirements

Experience writing production-quality code in Python, PyTorch, JAX, Rust, or C++.

Nice To Haves

Expert-level proficiency in Python, with strong experience in at least one of JAX, PyTorch, or XLA.
A proven track record building or optimizing large-scale distributed ML systems, including training or inference optimization, GPU utilization, multi-GPU/TPU setups, or hardware co-design.
Deep experience designing and running data pipelines at scale, including curation, filtering, generation, and quality studies, especially for noisy or real-world multimodal data.
Strong fundamentals in evaluation design, benchmarks, reward modeling, or RL techniques, particularly for interactive or agentic behaviors.
A proactive, self-starting approach and ability to thrive in high-intensity environments, with a passion for advancing multimodal AI.
Willingness to own end-to-end initiatives and do what it takes to deliver breakthrough user experiences.
Experience leading major improvements in model capabilities through better data, modeling, algorithms, or scaling.
Familiarity with the state of the art in multimodal LLMs, scaling laws, tokenizers, compression techniques, reasoning, or agentic systems.
Proficiency in Rust and/or C++ for performance-critical components.
Hands-on experience with large-scale orchestration tools such as Spark, Ray, or Kubernetes.
A background building full-stack tooling, including performant interfaces, real-time research demos or applications, or end-to-end product ownership.
Passion for end-to-end user experience in interactive, real-time multimodal AI systems.

Responsibilities

Design, build, and optimize large-scale distributed systems for multimodal pre-training, post-training, inference, data processing, and tokenization at web or petabyte scale.
Develop high-throughput pipelines for data acquisition, preprocessing, filtering, generation, decoding, loading, crawling, visualization, and management of image, video, audio, and text data.
Advance multimodal capabilities, including spatial-temporal compression, cross-modal alignment, world modeling, reasoning, emergent abilities, audio/image/video understanding and generation, real-time video processing, and noisy-data handling.
Drive data quality studies, including human and synthetic curation, filtering techniques, analysis, and scalable pipelines supporting trillion-parameter models.
Create evaluation frameworks, internal benchmarks, reward models, and metrics that capture real-world usage, failure modes, interactive dynamics, and human-AI synergy.
Innovate on algorithms, modeling approaches, hardware/software/algorithm co-design, and scaling paradigms for state-of-the-art performance.
Build research tooling, user-friendly interfaces, prototypes and demonstrations, and full-stack applications; enable rapid iteration based on feedback.
Work across the stack, from pre-training through SFT, RL, and post-training, to enable reasoning, tool calling, agentic behaviors, orchestration, and seamless real-time interactions.