Staff Software Engineer - Compiler & Spark Runtime
About The Position
The Spark Runtime Engineering Team brings Snowflake’s performance, efficiency, and reliability to Spark workloads. The team builds compiler and runtime integrations that translate Spark workloads into efficient execution on Snowflake’s cloud-native query engine, addressing compiler infrastructure, query-plan lowering, Spark compatibility, execution integration, and end-to-end observability. This is a hands-on technical leadership role focused on designing and building critical compiler and runtime capabilities for Spark workloads on Snowflake. The engineer owns complex projects from design through implementation and production rollout, contributes directly to the codebase, reviews critical designs and code, and investigates correctness, performance, and reliability issues. Working with compiler, query engine, runtime, and product teams, the engineer turns customer needs into technical priorities and measurable outcomes. Leadership is exercised through technical judgment, hands-on execution, mentorship, and influence rather than formal reporting relationships.
Requirements
- At least 10 years of hands-on software engineering experience, including substantial work on compilers, query-processing systems, or runtime infrastructure.
- Demonstrated experience serving as a technical lead for complex engineering initiatives, guiding engineers and coordinating delivery without requiring formal management authority.
- Track record of designing, implementing, and delivering production systems, with ownership of significant architectural and implementation decisions.
- Deep expertise in at least one relevant area, such as compiler architecture and IR transformations, relational query processing, language semantics and type systems, or runtime integration.
- Proficiency in Scala, Java, C++, or a comparable implementation language, with ability to write and review production code.
- Strong systems fundamentals, including concurrency, memory behavior, and performance analysis, and experience debugging complex production issues.
- Ability to translate technical and customer requirements into practical engineering priorities and collaborate effectively across teams and locations.
- Strong technical communication and mentorship skills, including the ability to build alignment around architectural decisions and explain complex trade-offs.
Nice To Haves
- Experience with Apache Spark internals, Catalyst, Spark Connect, or DataFrame APIs.
- Experience with relational algebra, SQL generation, query optimization, or database execution systems.
- Experience with Scala/JVM performance, native runtime functions, metadata caching, or compilation-performance optimization.
- Experience building compatibility-test frameworks, differential testing, performance benchmarks, or production observability for data systems.
- Experience technically leading systems through customer adoption, production hardening, and staged releases.
- A degree in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.
Responsibilities
- Lead technical design and implementation of Spark query compilation and runtime integration, including intermediate representations, type resolution, semantic compatibility, query-plan transformations, and efficient SQL generation.
- Own complex engineering initiatives from problem definition and architecture through implementation, validation, and production rollout.
- Provide technical leadership to project teams by clarifying engineering trade-offs, aligning contributors, reviewing designs and code, and resolving technical blockers.
- Improve end-to-end workload performance and cost-efficiency through profiling, compilation-latency reduction, metadata-access improvements, and collaboration with query-engine teams.
- Establish correctness and compatibility practices, including differential testing against Apache Spark, regression coverage, and clear handling of unsupported behavior.
- Drive production quality for owned components, including observability, incident investigation, fault recovery, and safe rollout practices.
- Mentor engineers through design discussions, code reviews, debugging, and knowledge sharing.
- Partner with senior technical leaders and Product Management to prioritize investments, identify dependencies, and scope deliverable engineering milestones.
- Validate changes against representative customer workloads and track measurable improvements in compatibility, latency, reliability, and cost-efficiency.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
