About The Position

Are you passionate about building scalable and reliable big data pipelines for modern search engines? Join the Apple Services Engineering AI/ML Search Platform team! We build the common infrastructure that powers search and recommendations across Apple Media products, including the App Store, Music, TV, Podcasts, Books, and Fitness+. As part of our team, you will enhance thousands of compute and big data pipelines to deliver greater scalability, reliability, and efficiency. By leveraging innovative approaches with machine learning and large language models, you will improve pipeline quality, optimize Spark and Kubernetes resource utilization, and create automation that accelerates developer agility.

Requirements

  • Bachelor’s degree in Computer Science, Computer Engineering, or a related field.
  • 3+ years of experience with large-scale data processing and pipelines.
  • Proficiency in Scala, Python, and scripting languages.
  • Experience in and solid understanding of distributed systems, performance tuning, and resource optimization.
  • Strong hands-on expertise with Apache Spark and the Hadoop ecosystem.

Nice To Haves

  • Experience developing or applying machine learning techniques or LLM-based agentic workflows for data pipeline optimization and data quality improvements.
  • Knowledge of cost optimization strategies for big data infrastructure.

Responsibilities

  • Develop automation and LLM-based agents to automatically increase testing coverage for data pipelines in a monorepo environment.
  • Develop automation and LLM-based agents to optimize Spark job resource utilization, including both CPU and memory efficiency.
  • Develop LLM-powered agents to automatically diagnose failures in large-scale data pipelines.
  • Build tools and automation to accelerate engineer productivity across development, testing, and production deployment of new pipelines.
  • Design and maintain dashboards to improve observability of pipeline execution and verification.
  • Deliver cost-efficient solutions for storage and compute platform migrations through automation and advanced machine learning techniques.
© 2024 Teal Labs, Inc
Privacy PolicyTerms of Service