Software Development Manager, Open Data Analytics Engines

Amazon • Redmond, WA

1d

About The Position

The Open Data Analytics Spark Performance team is looking for a skilled manager
to help in our mission to provide the world’s fastest and easiest-to-use cloud
Big Data processing platform. We develop the AWS Runtime for Spark, which is
100% API compatible with Apache Spark and is designed to run with superior
performance and scalability on customer-managed EC2 clusters, EKS clusters, and
in serverless environments. Our runtime is featured on AWS services including
Athenaand EMR, which enable AWS customers to run millions of clusters performing
large scale analytics against vast datasets. In the last 3 years Open Data
Analytics has improved our engines by a factor of 5x by making changes to the
plan optimizer, query runtime, and storage connectors. We have also made
significant changes to the compiler to enable enterprise features like
fine-grained access control. This is a “must-win” strategic area in a growing
and very technical space.

As a manager in this space, you will own speeding customer workloads by shipping
query performance optimizations in the AWS Runtime for Apache Spark. Your team
will apply and extend our automated statistical analysis tools to ensure that
these optimizations can be delivered to all AWS Spark customers and allow them to
benefit from them out-of-the-box, with no configuration required. This team
sped the AWS Runtime for Spark’s 3 TB TPC-DS performance by 1.7x in 2023 through
techniques including modifying Spark’s internal data structure implementations,
enhancing CodeGen, and enhancing statistics/building advanced heuristics in
Spark Catalyst rules to improve query plans both before and during runtime
through Adaptive Query Execution. You will drive the team to identify new
optimization opportunities, prioritize them, and deliver them to demonstrate
speedups vs. Athena/EMR Spark customer workloads and industry benchmarks.

You will take requirements from the savviest customers in the business, work
with our Product Management group to identify and pursue market-differentiating
opportunities to enhance our Spark customers’ experience, and design and deliver
these opportunities, solving deep problems with a strong group of Senior and
Principal Engineers.

To learn more about Athena, EMR, and the Spark Performance team’s work:

- AWS Athena: https://aws.amazon.com/athena
- AWS EMR: http://aws.amazon.com/emr
- AWS Big Data Blog: https://aws.amazon.com/blogs/big-data/
- EMR/EKS widens performance gap:
https://aws.amazon.com/blogs/big-data/amazon-emr-on-eks-widens-the-performance-gap-run-apache-spark-workloads-5-37-times-faster-and-at-4-3-times-lower-cost/

Utility Computing (UC)

AWS Utility Computing (UC) provides product innovations — from foundational
services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute
Cloud (EC2), to consistently released new product innovations that continue to
set AWS’s services and features apart in the industry. As a member of the UC
organization, you’ll support the development and management of Compute,
Database, Storage, Internet of Things (Iot), Platform, and Productivity Apps
services in AWS, including support for customers who require specialized
security solutions for customers who require specialized security solutions for
their cloud services.

About AWS
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted
cloud platform. We pioneered cloud computing and never stopped innovating —
that’s why customers from the most successful startups to Global 500 companies
trust our robust suite of products and services to power their businesses.

Diverse Experiences
AWS values diverse experiences. Even if you do not meet all of the
qualifications and skills listed in the job description, we encourage candidates
to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying.

Work/Life Balance
We value work-life harmony. Achieving success at work should never come at the
expense of sacrifices at home, which is why we strive for flexibility as part of
our working culture. When we feel supported in the workplace and at home,
there’s nothing we can’t achieve in the cloud.

Inclusive Team Culture
Here at AWS, it’s in our nature to learn and be curious. Our employee-led
affinity groups foster a culture of inclusion that empower us to be proud of our
differences. Ongoing events and learning experiences, including our
Conversations on Race and Ethnicity (CORE) and AmazeCon conferences, inspire us
to never stop embracing our uniqueness.

Mentorship & Career Growth
We’re continuously raising our performance bar as we strive to become Earth’s
Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and
other career-advancing resources here to help you develop into a better-rounded
professional.

Requirements

• 3+ years of engineering team management experience
• 7+ years of working directly within engineering teams experience
• 3+ years of designing or architecting (design patterns, reliability and scaling) of new and existing systems experience
• Knowledge of engineering practices and patterns for the full software/hardware/networks development life cycle, including coding standards, code reviews, source control management, build processes, testing, certification, and livesite operations
• Experience partnering with product or program management teams

Nice To Haves

• Experience in communicating with users, other technical teams, and senior leadership to collect requirements, describe software product features, technical designs, and product strategy
• Experience in recruiting, hiring, mentoring/coaching and managing teams of Software Engineers to improve their skills, and make them more effective, product software engineers
• Master's degree in Computer Science
• 9+ years of software development experience
• 5+ years of software development manager, managing teams of more than 5 software engineers experience
• Knowledge of big data businesses and technical concepts
• Experience in writing complex, highly-optimized SQL queries across large datasets
• Experience in Database like NoSQL
• Experience in a high volume, transactional and distributed system environment
• 5+ years of managing technical teams with significant personnel management experience and examples of developing engineers and building successful teams experience
• Experience performing complex business case analysis to justify technical decisions and presenting the justification to management in a high-level review
• Experience managing multiple projects and priorities across teams in a fast-paced, deadline-driven environment
• Experience using Unix/Linux

Responsibilities

• Prioritize the efforts of a strong engineering team researching and delivering new/enhanced query operator implementation and query plan optimizations
• Ensure AWS customers’ workloads benefit from the optimizations your team delivers.
• Drive collaboration between your team, other AWS teams, and the open source community to identify system/workload insight enhancements and collaborate on optimization techniques.
• Ensure data consistency and durability while obtaining breakthrough performance and scalability.
• Drive team design review processes, ensuring both rapid delivery and a high quality bar.
• Lead technical/product discussions with other Software and Product managers.
• Interface with a diverse customer base to understand requirements, priorities, and processes.
• Lead your team to support and learn from Athena and EMR customers’ Spark usage - with a focus on performance.
• Hire, coach, and mentor query optimization SDEs.
• Interact and partner with the open source community.
• Propose and implement new projects or recommend engine improvements.
• Own all operational metrics and support.
• Help develop long-term development and business technology strategies.

Benefits

• health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
• 401(k) matching
• paid time off
• parental leave

What This Job Offers

• Job Type: Full-time
• Career Level: Manager