Lead Assistant Manager

EXL•San Francisco, CA
•Onsite

About The Position

The Senior Data Engineer, Data Platform will design, build, and operate mission-critical ETL pipelines that ingest petabyte-scale data into a Hadoop ecosystem. The role partners with engineering, analytics, and business intelligence teams to improve data delivery speed, pipeline reliability, and processing efficiency using Java, Spark, and AWS. It calls for deep expertise in distributed systems, production operations, and troubleshooting complex data issues at scale. The position is based in the San Francisco Bay Area and requires an onshore presence in the San Francisco office. The role also includes managing data engineering projects, providing strategic guidance, overseeing a data engineering team, and improving data processes continuously.

Requirements

  • Expert-level Java proficiency for building production-grade ETL frameworks, custom actions, and reusable libraries.
  • Experience with Maven multi-module projects, dependency management with shaded JARs, and Hadoop classpath conflicts.
  • Strong Groovy scripting skills for configuration DSLs, row processing, and partition logic.
  • Proficiency with Git branching strategies, pull-request workflows, code reviews, and Jira-integrated commit practices.
  • Scala functional programming experience for Spark development and large-scale data processing.
  • Ability to write complex SQL queries across databases such as MySQL, PostgreSQL, Oracle, and SQL Server.
  • Proven experience building production ETL frameworks with reusable components and a deep understanding of ETL concepts, load patterns, CDC processing, and data lineage.
  • Deep production-scale Apache Spark expertise, including job optimization, memory and shuffle tuning, and use of DataFrames or RDDs in Scala or Java.
  • Experience with Kafka, GoldenGate, or Spark Streaming for real-time or near-real-time data processing.
  • Experience migrating from Oozie to modern workflow orchestration tools such as Airflow.
  • Strong production-scale Hadoop experience, including HDFS operations, Hive query optimization, and management of partitioned tables; understanding of data locality, replication factors, and cluster resource management.
  • Experience writing complex Hive/HQL analytics queries on multi-million-row tables.
  • Proficiency designing Oozie coordinators and workflows, including dataset dependencies, SLA monitoring, and late-data handling.
  • Hands-on experience configuring Sqoop bulk imports, parallel mappers, split-by strategies, incremental modes, and large-scale database exports.
  • Production experience with AWS S3 data lakes, EMR cluster operations, IAM role-based access, and Secrets Manager; understanding of S3 lifecycle policies, versioning, encryption, AWS networking, VPCs, and security groups.
  • Hands-on experience implementing PII encryption and decryption, SafeNet, and data masking in regulated environments.
  • Strong problem-solving and communication skills, including explaining data engineering concepts to nontechnical stakeholders.
  • Experience in client-facing roles.
  • Graduate degree in Computer Science, Data Science, or a related field.
  • 3–4 years of experience in data engineering or a related field.

Nice To Haves

  • Banking, lending, or other regulated financial services experience.
  • Data modeling experience.
  • Experience with observability and monitoring frameworks for data pipelines.
  • Databricks Certified Data Engineer Professional certification.
  • Experience with Presto or Trino.
  • Experience with NoSQL databases.
  • Exposure to Databricks, Delta Lake, and notebook-driven development.

Responsibilities

  • Design and develop Java ETL workflows to ingest data from databases, APIs, and file feeds into Hadoop and Hive.
  • Build and maintain reusable ETL framework components, including actions, UDFs, and SerDes.
  • Configure and optimize ingestion with Sqoop, Spark, and Oozie, using incremental loads, partitioning, and CDC strategies.
  • Monitor and troubleshoot production pipelines, resolve data-quality issues, investigate failures, and address performance bottlenecks.
  • Implement data security controls, including PII encryption and decryption, AWS Secrets Manager integration, and compliance requirements.
  • Collaborate with source-system owners, data analysts, and BI teams to ensure data availability and quality.
  • Participate in an on-call rotation for critical production data feeds and SLA compliance.
  • Manage data engineering projects, provide strategic guidance, oversee the data engineering team, and continually improve data processes.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service