Principal Data Engineer

LendingClub•San Francisco, CA

1d•Hybrid

About The Position

Our mission at LendingClub is to empower those who strive to achieve better financial health. The Data and Analytics team plays a crucial role in achieving our mission. We are seeking a Principal Data Engineer to lead the design and evolution of data systems that power batch processing, real-time streaming, pipeline orchestration, data lake management, data cataloging, and machine learning workflows. This role has a strong emphasis on enabling and scaling machine learning and analytics use cases, including feature engineering, model training, and inference data pipelines. In this role, you will apply deep technical expertise, architectural thinking, and hands-on development to solve complex big data and ML platform challenges. You will partner closely with Data Science, Product, and Platform teams to build reliable, scalable, and cost-efficient data foundations that support both analytics and production machine learning systems.

Requirements

8+ years of data engineering experience with deep hands-on experience with distributed data systems including Hadoop, Spark, Hive, DBT, and Airflow/Dagster
Bachelor’s degree in computer science or a related field, or equivalent work experience
5+ years of production-quality Python experience, building and maintaining large-scale data pipelines
Strong experience building machine learning use cases through data engineering (e.g., feature engineering pipelines, training/inference data flows)
Experience working with public cloud platforms, preferably AWS
Experience with Databricks and/or Snowflake in production environments
Strong working knowledge of Git, JIRA, Jenkins, shell scripting
Familiarity with Agile methodologies, test-driven development, source control management, and test automation
Proven ability to work across cross-functional teams in a fast-paced, dynamic environment
Excellent collaborative problem-solving and communication skills, with the ability to influence without authority
A track record of designing and delivering scalable, reliable, and high-quality data platforms

Nice To Haves

experience building data pipelines for Digital Marketing use cases

Responsibilities

Design, build, and own large-scale data and ML data pipelines that integrate directly with LendingClub’s products and external vendors
Design and operate data platforms where autonomous coding agents help maintain pipelines, schemas, and tests, while you own architecture and guardrails.
Lead the architecture and implementation of MLOps including feature stores, training datasets, batch and real-time inference pipelines
Work with modern data technologies such as Hadoop, Spark, DBT, Dagster/Airflow, Atlan, and modern data platforms like Databricks and Snowflake, across the AWS cloud stack
Partner with Data Scientists to productionize ML workflows and ensure data reliability, reproducibility, and performance
Identify, design, and implement automation of manual processes, optimizing data delivery, improving system reliability, and reducing cloud costs
Implement processes and systems to monitor Data Quality, Observability, Governance, Lineage, and ML data consistency
Define policies, workflows, and quality gates for using AI agents in production data systems.
Provide technical leadership and mentorship, influencing data engineering standards, best practices, and long-term platform strategy
Coach teams on decomposing work, supervising agents, and validating AI-generated changes.
Support operations to manage the production environment and lead root cause analysis (RCA) for complex data and ML pipeline issues
Write unit and integration tests, advocate for test-driven development, contribute to engineering documentation and design reviews