Principal Software Engineer, Data Processing OSS
About The Position
DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing. DataPelago Nucleus is a universal data processing engine that is designed to accelerate the processing of diverse data – structured through unstructured – with any parallel processing framework – e.g., Apache Spark – on any infrastructure – including vectorized CPU and GPU. DataPelago Accelerator for Spark (DPA-S), our product based on DataPelago Nucleus, is running large-scale production applications of many globally renowned customers.
Requirements
- 15+ years of relevant experience, Bachelor's degree in Computer Science, or a related field OR a Master's degree in Computer Science or a related field.
- 3+ years of deep technical experience in developing core components of Apache Spark, Apache Flink, Apache Doris, Apache Gluten, Velox, Apache DataFusion, Apache DataFusion Comet, or equivalent platforms designed for large-scale data processing.
- 3+ years of deep technical experience in instrumenting, analyzing, and optimizing the performance of data processing engine components on benchmark and customer workloads.
- Sound knowledge of the architecture and internal operation of one or more of Apache Spark, Apache Flink, Presto/Trino.
- Demonstrated experience in the design, development, and successful release of high-performance data processing engines for large production deployments.
- Exceptional programming skills in C, C++, and Java.
- Extensive development experience in Linux environments.
- Strong analytical and problem-solving skills with a passion for performance optimization.
Responsibilities
- Influence the architecture of how our data processing engine interfaces with open-source platforms and engines.
- Lead design of functional and performance enhancements to open-source platforms such as Apache Gluten and Velox and their integration with our data processing engine.
- Individually design, implement, test, optimize, and maintain components of the data processing engine.
- Analyze technology roadmap of Apache Gluten, Velox, and equivalent platforms and identify opportunities for our engine to enhance technology and product leadership.
- Partner effectively with engineering, product management, open-source community and customer success teams.
- Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain highest product quality, security, efficiency, & productivity.
Benefits
- Health Insurance
- Life Insurance
- Retirement or Pension Plans
- Paid Time Off
- various Leave options
- Performance-Based Incentives
- employee stock purchase plan
- restricted stocks (RSU’s)
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
