Senior Software Engineer, Workflow Orchestration

Airbnb•San Francisco, CA

5d•Remote

About The Position

Airbnb was born in 2007 when two hosts welcomed three guests to their San Francisco home, and has since grown to over 5 million hosts who have welcomed over 2 billion guest arrivals in almost every country across the globe. Every day, hosts offer unique stays and experiences that make it possible for guests to connect with communities in a more authentic way. The Community You Will Join: Data represents the voice of Airbnb’s users at scale. The Workflow Orchestration team is responsible for the data workflow orchestration platform, which powers a big range of data use cases from offline data analytics, experiments, to AI/ML model development and training. We are the founding team of Apache Airflow and a team with rich and diverse experience in different orchestration platforms. A Typical Day: Architect, design, and implement scalable, reliable, and highly available data infrastructure solutions to support workflow orchestration across the company. Develop and maintain automation tools, streamlining deployment and management of the workflow orchestration platform. Collaborate with infrastructure engineers, data engineers, and other users of the data warehouse to ensure efficient integration and smooth operation of our platform with other systems and services. Conduct performance testing and help optimize our orchestration platform to meet the evolving needs of internal users. Manage and prioritize multiple projects and tasks, communicating progress and status with key stakeholders. Participate in technical discussions, code reviews, and pair programming sessions to foster knowledge-sharing and team growth. Provide input on best practices to improve scalability, reliability, and usability for the data platform.

Requirements

5+ years of experience in software engineering, with a focus on data infrastructure or supporting data engineers and/or data scientists.
Strong programming skills in one or more of the following languages: Java and Python.
Experience building data applications that create and implement higher-level abstractions on top of lower-level infrastructure to simplify complex data operations, improve scalability, and enhance overall system performance.
Experience with data modeling, database design and various SQL dialects.
Experience with containerization technologies such as Docker and Kubernetes.
Extensive background in workflow orchestration platforms such as Apache Airflow, Luigi, Azkaban, Oozie, Prefect, Maestro, or Kubeflow.
Utilize experience in Data Engineering to optimize processes and bring forth new ideas and technologies that enhance our data capabilities.
Excellent communication skills and the ability to collaborate with cross-functional teams to build empathetic tools and platforms.
Excellent analytical and problem solving skills.
BS/MS/PhD in Computer Science, a related field, or equivalent work experience preferred.

Responsibilities

Architect, design, and implement scalable, reliable, and highly available data infrastructure solutions to support workflow orchestration across the company.
Develop and maintain automation tools, streamlining deployment and management of the workflow orchestration platform.
Collaborate with infrastructure engineers, data engineers, and other users of the data warehouse to ensure efficient integration and smooth operation of our platform with other systems and services.
Conduct performance testing and help optimize our orchestration platform to meet the evolving needs of internal users.
Manage and prioritize multiple projects and tasks, communicating progress and status with key stakeholders.
Participate in technical discussions, code reviews, and pair programming sessions to foster knowledge-sharing and team growth.
Provide input on best practices to improve scalability, reliability, and usability for the data platform.