Lead Assistant Manager

EXL•San Francisco, CA
•Onsite

About The Position

The Senior Data Engineer, Data Platform designs, builds, and operates mission-critical ETL pipelines that ingest petabyte-scale data into a Hadoop ecosystem. The role partners with engineering, analytics, and business intelligence teams to improve data delivery speed, pipeline reliability, and processing efficiency using Java, Spark, and AWS. It requires strong expertise in distributed systems, production operations, and troubleshooting complex data issues at scale. The position also manages data engineering projects, provides strategic guidance, oversees a data engineering team, and supports continuous improvement of data processes. The role requires onshore presence in the San Francisco office.

Requirements

  • Graduate in Computer Science, Data Science, or a related field.
  • 3–4 years of experience in data engineering or a related field.
  • Expert-level Java proficiency building production-grade ETL frameworks, custom actions, and reusable libraries.
  • Experience with Maven multi-module projects, dependency management with shaded JARs, and Hadoop classpath conflicts.
  • Strong Groovy scripting skills for configuration DSLs, row processing, and partition logic.
  • Proficiency with Git version control, branching strategies, pull-request workflows, code reviews, and Jira-integrated commit practices.
  • Scala functional-programming experience and ability to write idiomatic Scala for large-scale data processing.
  • Ability to write complex SQL queries across multiple databases, such as MySQL, PostgreSQL, Oracle, and SQL Server.
  • Proven experience building production ETL frameworks with reusable components and understanding of ETL concepts, load patterns, CDC processing, and data lineage.
  • Deep production-scale Apache Spark expertise, including job optimization, memory and shuffle tuning, and DataFrame/RDD work in Scala or Java.
  • Real- or near-real-time data-processing experience with Kafka, GoldenGate, or Spark Streaming.
  • Experience migrating from Oozie to modern workflow orchestration such as Airflow.
  • Production-scale Hadoop ecosystem experience, including HDFS operations, Hive query optimization, and management of partitioned tables; understanding of data locality, replication factors, and cluster resource management.
  • Experience writing complex Hive/HQL analytics queries on multi-million-row tables.
  • Proficiency designing Oozie coordinators and workflows with dataset dependencies, SLA monitoring, and late-data handling.
  • Hands-on experience configuring Sqoop bulk imports, parallel mappers, split-by strategies, incremental modes, and large database exports.
  • Production AWS experience with S3 data lakes, lifecycle policies, versioning and encryption; EMR cluster operations; IAM role-based access; and Secrets Manager.
  • Understanding of AWS networking, VPCs, and security groups.
  • Hands-on experience implementing PII encryption/decryption, SafeNet, and data masking in regulated environments.
  • Strong problem-solving and communication skills, including explaining data engineering concepts to nontechnical stakeholders.
  • Experience in client-facing roles.
  • Ability to work onshore in the San Francisco office.

Nice To Haves

  • Banking, lending, or other regulated financial-services experience.
  • Data modeling experience.
  • Experience with observability and monitoring frameworks for data pipelines.
  • Databricks Certified Data Engineer Professional certification.
  • Presto or Trino experience.
  • NoSQL database experience.
  • Exposure to Databricks, Delta Lake, and notebook-driven development.

Responsibilities

  • Design and develop Java ETL workflows to ingest data from databases, APIs, and file feeds into Hadoop and Hive.
  • Build and maintain reusable ETL framework components, including actions, UDFs, and SerDes.
  • Configure and optimize ingestion using Sqoop, Spark, and Oozie, including incremental loads, partitioning, and CDC strategies.
  • Monitor and troubleshoot production pipelines, resolve data-quality issues, investigate failures, and optimize performance bottlenecks.
  • Implement data security controls, including PII encryption and decryption, AWS Secrets Manager integration, and compliance requirements.
  • Collaborate with source-system owners, data analysts, BI teams, and upstream and downstream teams to ensure data availability and quality.
  • Participate in an on-call rotation for critical production data feeds and SLA compliance.
  • Manage data engineering projects, provide strategic guidance, oversee the data engineering team, and ensure continuous improvement of data processes.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service