Data Engineer (Senior) ID53687

AgileEngine • Downey, CA

26d • Hybrid

About The Position

As a Data Engineer specializing in Java and Apache Spark, you will help build and evolve large-scale data processing systems that power experimentation and user insights. Working within a cloud-based AWS EMR environment, you’ll contribute to improving data infrastructure reliability, scalability, and cost efficiency. This role offers the opportunity to shape critical datasets and analytics capabilities while collaborating with platform and data teams to support high-impact experimentation and decision-making.

Requirements

• 4+ years experience in software development
• Bachelor’s degree in Computer Science or equivalent practical experience
• Significant practical experience with Java (4+ years)
• Practical experience implementing Apache Spark jobs, including partitioning, grouping, joins, importing data into the cluster, and exporting data from the cluster
• Practical experience working with AWS, specifically AWS EMR (or ability to pick it up fast)
• Upper-intermediate English level

Nice To Haves

• Basic knowledge of Kubernetes
• Experience with Spark Operator
• Experience with Airflow
• Experience with Scala

Responsibilities

• Develop and maintain index builder products, including user session index builders, user session–derived index builders, and experimentation platform index builders
• Investigate and resolve reported issues related to index builders
• Assist with user inquiries regarding the platform and its datasets
• Improve index builder stability and reliability
• Support efforts to optimize compute costs across the platform
• Contribute to the Central Exposure Dataset effort, including building a consolidated dataset for experiment analysis
• Work toward meeting code freshness goals
• Persist Yarn logs and Spark history for terminated clusters
• Capture metrics from UserCohort
• Optimize resource allocation for platform infrastructure
• Help reduce the number of core instances for platform clusters
• Support the deprecation of legacy index builders used for experiment analysis

Benefits

• Professional growth: Mentorship, TechTalks, and personalized growth roadmaps.
• Competitive compensation: USD-based pay with education, fitness, and team activity budgets.
• Exciting projects: Modern solutions with Fortune 500 and top product companies.
• Flextime: Flexible schedule with remote and office options.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level