Staff Software Engineer - Apache Spark

Cloudera•Washington, DC

1d•Remote

About The Position

At Cloudera, we empower people to transform complex data into clear and actionable insights. With as much data under management as the hyperscalers, we're the preferred data partner for the top companies in almost every industry. Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world’s largest enterprises. The Data Platform Pillar is the bedrock of Cloudera’s technology, where we design and build the core components that let our customers store, manage, and process data with unmatched scalability, security, and performance. Cloudera is searching for a visionary Staff Software Engineer with deep expertise in distributed systems to join the Apache Spark Team. You will be at the forefront of innovation, building our next-generation, enterprise-grade system designed to conquer data challenges at a massive scale—running Spark on thousands of nodes and crunching petabytes of data for the world's largest companies. This is your chance to directly influence the open-source community as a key contributor to Apache Spark while collaborating with a high-impact, distributed team that includes multiple Spark committers. If you're passionate about pushing the boundaries of distributed data processing, come build the impossible with us.

Requirements

5-7+ years of experience in professional software development.
Proven experience leading technical initiatives and delivering complex product enhancements from concept to production.
Strong proficiency in Java, Scala, or other JVM-based language.
Solid experience in the design and development of distributed systems.
Passion for clean coding, attention to detail, and a focus on software quality and maintainability.
Strong oral and written communication skills for effective collaboration across a distributed team.
Demonstrated ability to research, problem-solve, and operate independently without constant supervision.
An open-minded approach with a desire to learn new technologies and an unwavering passion for building exceptional products.

Nice To Haves

Spark & Ecosystem Experience with using/developing Apache Spark, Apache Iceberg, or other related technologies.
Deep experience with large-scale, distributed systems design and development, including a strong understanding of scaling, performance optimization, and scheduling.
Experience with SQL Planners and Optimizers
Prior experience as a contributor to open-source projects.

Responsibilities

Architect, implement, and deliver next-generation features for Cloudera’s Data Engineering Experience, operating at a massive scale on thousands of production nodes.
Be a core contributor to Apache Spark, directly shaping the future of distributed data processing in the open-source community.
Develop high-performance features using Scala, Java, and Python on modern data platforms.
Gain and apply expert-level knowledge in core distributed data processing concepts, including: SQL Planners and Optimizers, Data layout and modern table formats like Apache Parquet and Iceberg, Fault tolerance and resilience in large-scale distributed systems.
Develop a deep technical understanding of components across the Cloudera Data Engineering Experience, with a focus on Iceberg and Spark, applying this knowledge to your daily tasks.
Work hands-on with massive distributed systems, scaling from hundreds to thousands of nodes in live production clusters.
Conduct thorough root cause analysis, debug complex system-level deployment issues, and resolve failures to maintain high system quality.
Improve internal infrastructure and tooling to streamline development, testing, and deployment processes.
Work closely with a high-impact, distributed team and stakeholders to drive product vision and delivery.