Member of Technical Staff, Distributed Training Performance

Hark•San Jose, CA
•$180,000 - $450,000

About The Position

Hark is an artificial intelligence company developing personalized, multimodal intelligence and next-generation AI hardware. The Omni team builds AI experiences beyond text, enabling models to understand and generate content across modalities such as text and audio. As a member of the Omni team, you will improve Hark's Megatron-based pretraining systems across thousands of GPUs. The role focuses on model FLOPs utilization (MFU), training throughput, and the performance of kernels, communication, and parallelism strategies. You will work with researchers on new architectures and with infrastructure engineers to resolve bottlenecks, maintain numerical stability, and keep large training runs reliable.

Requirements

  • Experience developing distributed training systems and working with the internals of Megatron-LM, NeMo, or a comparable framework, including its parallelism strategies.
  • Strong understanding of data, tensor, pipeline, sequence, context, and expert parallelism, including communication patterns, memory tradeoffs, and failure modes.
  • Experience using GPU profilers to analyze kernel execution, memory bandwidth, communication overhead, and data-loading stalls.
  • Working knowledge of mixed-precision training, including BF16 and FP8, and methods for diagnosing numerical instability.
  • Strong Python and C++/CUDA engineering skills, with experience building reliable tools for researchers.
  • Experience developing CUDA or Triton kernels, or using CUTLASS, for attention, normalization, or mixture-of-experts models.
  • A record of measurably improving MFU or throughput on large-scale training jobs.

Nice To Haves

  • Experience with mixture-of-experts training at scale, including expert parallelism, load balancing, and all-to-all communication optimization.
  • Experience with GPU communication libraries such as NCCL or network topology-aware scheduling.
  • Experience with fault-tolerant training, asynchronous checkpointing, in-memory recovery, or elastic training.
  • Experience balancing workloads across different encoders in multimodal training.
  • Familiarity with cluster schedulers such as Slurm or Kubernetes and GPU health monitoring at fleet scale.

Responsibilities

  • Own MFU and step time for pretraining runs across dense and mixture-of-experts models, setting targets per model size and tracking them across runs.
  • Write or integrate fused and custom kernels using CUDA, Triton, or CUTLASS for attention, normalization, and mixture-of-experts routing.
  • Overlap communication with compute, reduce pipeline bubbles, and tune activation recomputation and memory use to raise throughput per GPU.
  • Implement model and optimizer sharding, and optimize data, tensor, pipeline, sequence, context, and expert parallelism for dense and mixture-of-experts architectures.
  • Profile GPU kernels, collective communication, data loading, and mixed-precision training to identify performance and numerical stability bottlenecks.
  • Develop reusable training components that support new architectures and multimodal encoders for speech, vision, and text.
  • Investigate performance regressions and training failures involving communication, numerical changes, hardware faults, or straggling nodes.
  • Work with architecture researchers and compute infrastructure engineers to align model designs, training software, and cluster capabilities.
  • Improve checkpointing, restart time, failure detection, and observability to reduce lost training work.
  • Participate in on-call support during large runs.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service