Research Engineer, Foundation Model Training, SeekrGEO

Seekr • Austin, TX

1d • Hybrid

About The Position

This role contributes to the foundation model program behind SeekrGEO, Seekr’s geospatial AI product. The focus is on pretraining and post-training of large multi-modal models on geospatial data, along with the distributed training systems required for large-scale operations. The role supports the full model lifecycle through deployment, with a primary focus on training systems that enable ambitious model programs, including large-scale distributed training, parallelism strategies, data infrastructure, and the operational rigor for multi-week runs. You will collaborate with Research Scientists on modeling and recipe decisions, translating research ideas into working code and determining their suitability for full training runs.

Requirements

• Strong background in modern ML systems, with deep familiarity with transformer architectures, multi-modal models, and the practicalities of training them at scale.
• Fluency with PyTorch and the distributed training ecosystem (FSDP, tensor/pipeline/sequence parallelism, ZeRO, checkpointing strategies).
• Hands-on experience with at least one large-scale training framework such as Megatron-LM, torchtitan, or DeepSpeed.
• Ability to move comfortably between engineering and research: read a paper, reproduce its core idea, and pressure-test its scalability.
• Demonstrated experience contributing to a large model run through pretraining or continued pretraining (not just fine-tuning).
• Comfort designing experiments and evaluating ambiguous technical tradeoffs.
• Strong Python and software engineering fundamentals, including testing, code review, CI/CD, debugging, and performance analysis.
• Fluency with AI coding assistants and modern developer workflows.
• Clear communication and strong collaboration skills across technical and non-technical partners.
• Reside near Austin, TX or Reston, VA and able to work 3 days per week in office.

Nice To Haves

• Experience with remote sensing data pipelines and storage/access patterns for different modalities (SAR, hyperspectral, multispectral, high-cadence EO).
• Experience with infrastructure for agentic systems or tool-using models (rollouts, evaluation harnesses, RL loops at scale).
• Familiarity with government and defense data handling, classification regimes, or air-gapped deployment.
• Experience deploying or distilling large models for inference under real latency and cost constraints.
• Open-source contributions to training stacks or geospatial ML libraries.
• Experience with AMD ROCm is a strong plus; CUDA/NVIDIA experience is welcome.
• Experience with data infrastructure for large training corpora: sharded formats, deduplication, streaming pipelines, mixture tuning.
• Experience with experiment tracking, model and data versioning, evaluation pipelines, and diagnosing production issues in trained models.

Responsibilities

• Build and harden training infrastructure on accelerator clusters, including data loaders, parallelism strategies, checkpointing, fault tolerance, and evaluation harnesses.
• Own the parallelism strategy for training workloads, encompassing FSDP, tensor/pipeline/sequence parallelism, ZeRO variants, activation and gradient checkpointing, and mixed precision, considering memory and throughput tradeoffs.
• Diagnose distributed training failures and implement fixes as reusable platform improvements.
• Design and operate the data pipeline for large training corpora, including sharded formats, streaming loaders, deduplication, mixture tuning, and versioning for reproducibility.
• Maintain the health of multi-week training runs through checkpoint management, fault-tolerant and elastic training, and operational hygiene for shared infrastructure.
• Conduct performance optimization on accelerators, including kernel-level profiling, attention kernel selection and tuning, and memory layout optimization.
• Build evaluation infrastructure for trustworthy and reproducible model comparisons during training and after deployment.
• Support deployed models throughout their lifecycle, monitoring production systems, diagnosing regressions, and feeding insights back into the training cycle.
• Contribute improvements to the SeekrFlow training platform.
• Partner with Research Scientists to test ideas, reproduce research claims, verify scalability of recipes, and transition research prototypes to production.
• Collaborate with the SeekrGEO product team and customer-facing teams to align training infrastructure with model workflow requirements.
• Effectively utilize AI coding assistants within a modern engineering workflow while maintaining strong judgment over code and infrastructure.
• Operate distributed training at scale across accelerator clusters, managing collective communication and large-scale run failure modes.
• Perform performance work on accelerators, including kernel-level profiling, mixed precision, activation and gradient checkpointing, attention kernels, and memory layout optimization.
• Manage checkpointing, fault-tolerant and elastic training, and operational hygiene for multi-week runs.
• Utilize experiment tracking, model and data versioning, evaluation pipelines, and diagnose production issues in trained models.
• Own ambiguous, long-horizon technical problems.

Benefits

• Meaningful Mission & Impact
• Equity Ownership – RSUs
• Unlimited PTO plus 14 paid company holidays
• Flexible hybrid work environment
• Competitive Total Rewards (base salary, bonuses, or commission plans)
• 401(k) with Company Match
• Comprehensive Health & Wellness (Medical, dental, vision, and life insurance coverage starting day one)
• Paid parental leave

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed