Software Engineer (Ray Data)

Anyscale•San Francisco, CA

11d•$215,000 - $230,000

About The Position

Anyscale is commercializing Ray, a popular open-source project that's creating an ecosystem of libraries for scalable machine learning. Ray Data is a Python-native data processing engine that serves as a one-stop shop for all AI data processing needs, providing performant, first-class integration with cutting-edge AI frameworks using both multi-modal and structured data. The Ray Data team is looking for exceptional engineers to build, optimize, and scale Ray for modern and increasingly complex AI workloads. As part of this role, you will improve the performance of Ray Data and multi-modal batch inference use cases, ensure efficient scaling across different stages of the Data pipeline in a heterogeneous environment, build data loading solutions for production training workloads, focus on stability and fault tolerance at high scale, and work with customers and new age AI native companies in scaling their AI workloads.

Requirements

At least 3-4 years of relevant work experience
Solid background in building scalable and fault-tolerant distributed systems
Experience with data processing, database internals.
Passionate about large scale systems and performance for AI.

Responsibilities

Improve the performance of Ray Data and multi-modal batch inference use cases.
Ensure efficient scaling across different stages of the Data pipeline in a heterogeneous environment.
Building data loading solutions for production training workloads.
Focus on stability and fault tolerance at high scale.
Working with customers and new age AI native companies in scaling their AI workloads.