AI Infrastructure Engineer
About The Position
Bright Vision Technologies is seeking an AI Infrastructure Engineer to design, build, and operate the platform layer powering large-scale AI training and inference workloads. The role focuses on GPU clusters, distributed training frameworks, scheduling, storage performance, and developer experience for ML engineers and researchers, with an emphasis on reliability, efficiency, and cost control. The ideal candidate has built or operated production AI infrastructure at scale, understands interactions among hardware, kernels, schedulers, and ML frameworks, and applies strong software engineering practices to platform work. The position is 100% remote in the United States and is full-time, direct W2, with an annual salary range of $100,000–$160,000. The employer encourages applications from U.S. citizens, Green Card holders, EAD holders, and H-1B transfer candidates, but cannot sponsor new H-1B visa petitions. Bright Vision Technologies is an equal opportunity employer. Its EEO commitment applies to all aspects of employment and prohibits workplace harassment and discrimination.
Requirements
- Bachelor’s or Master’s degree in Computer Science or a related field.
- Ten or more years of experience in infrastructure, platform, or HPC engineering.
- Hands-on experience operating GPU clusters or large-scale ML training infrastructure.
- Strong proficiency in Python and at least one systems language, such as Go or C++.
- Deep understanding of distributed training, accelerator architectures, and collective communication.
- Experience with Kubernetes, Slurm, Ray, or similar scheduling systems for ML workloads.
- Strong understanding of Linux internals, networking, and high-performance storage.
- Experience with at least one major cloud provider’s ML infrastructure offerings.
- Strong software engineering practices, including testing, CI/CD, and code review.
- Excellent communication and cross-functional collaboration skills.
Nice To Haves
- Experience operating InfiniBand or RDMA networking at scale.
- Contributions to open-source ML infrastructure projects.
- Familiarity with custom orchestrators or research-grade training stacks.
- Exposure to frontier model training operations.
- Experience with FinOps for AI workloads.
Responsibilities
- Design and operate GPU and accelerator infrastructure for training and inference across on-premises clusters, cloud-managed services, and hybrid configurations.
- Build scheduling, queueing, and resource-sharing systems to maximize accelerator utilization across teams.
- Integrate PyTorch, JAX, DeepSpeed, FSDP, Megatron-LM, Ray Train, and other frameworks into a unified platform.
- Operate high-performance storage systems and data pipelines that supply training data to accelerators at near-line-rate.
- Design networking architectures supporting RDMA, InfiniBand, NCCL, and high-bandwidth collective communication.
- Build observability for AI workloads, including utilization, throughput, training stability, and failure-mode analytics.
- Implement checkpointing, restart, and fault-tolerance patterns for large-scale, long-running training jobs.
- Optimize compute, storage, and networking costs through scheduling, spot capacity, and right-sizing.
- Develop developer tooling and paved-road workflows for safe and efficient research experimentation.
- Partner with research and applied ML teams to plan capacity for upcoming training runs.
- Implement security controls, isolation, and access management for multi-tenant AI infrastructure.
- Automate cluster provisioning, lifecycle management, and configuration enforcement.
- Maintain runbooks, capacity dashboards, and operational documentation for the AI platform.
- Stay current with AI infrastructure research, accelerator hardware, and emerging open-source AI tooling.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
