Principal Software Engineer, Data Infrastructure

LendingClub•San Francisco, CA

14h•$197,000 - $230,000•Hybrid

About The Position

Happen Bank is building a modern enterprise data and AI platform on Databricks. This principal-level platform engineering role owns the technical architecture and development of frameworks and infrastructure used by Data Engineering, Data Science, and Analytics teams. The focus is on enabling downstream teams to onboard data sources, protect sensitive information, and promote work to production through configuration rather than custom code; it is not a pipeline-delivery role. The role covers configuration-driven ingestion from relational databases and object-store files; reusable, versioned data-protection libraries; infrastructure-as-code and deployment foundations; and platform reliability, monitoring, reconciliation, recovery, and validation. The Principal Engineer makes evidence-based technology decisions, partners across engineering, governance, security, database, and source-system teams, and raises engineering standards through mentorship and shared practices. The position is based in San Francisco under a hybrid model, with in-office attendance Tuesdays, Wednesdays, and Thursdays. Remote placement will not be considered. The role primarily works local Pacific Time hours, with flexibility across time zones as needed, and travel to Happen Bank offices or other locations as needed. Relocation is offered based on actual job level. The target base salary range is $197,000–$230,000; the role may also be eligible for equity and an annual bonus.

Requirements

10+ years of software or data platform engineering experience, including designing and delivering platform-level frameworks, libraries, or infrastructure used across multiple engineering teams.
Bachelor’s degree or higher, or an equivalent combination of education and experience.
Strong experience with infrastructure as code, including Terraform and CI/CD pipelines for data platforms.
Deep experience building ingestion capabilities across diverse sources, including relational databases, files, APIs, and custom systems; expertise in incremental processing, change data capture (CDC), schema evolution, and protecting production source systems from excessive load.
Advanced programming skills in Python, SQL, and a JVM language such as Scala or Java; experience developing Spark UDFs and reusable libraries using strong testing, packaging, and versioning practices.
Deep hands-on expertise with Databricks—including Delta Lake, Unity Catalog, Auto Loader, workflows, compute configuration, and governance—or equivalent data platform experience.
Strong knowledge of security and privacy engineering for sensitive data, including encryption, key management, tokenization or format-preserving techniques, role-based access controls, and audit logging.
Experience designing and operating large-scale data systems processing billions of rows, including performance testing, cost optimization, monitoring, and recovery strategies.
Excellent written and verbal communication skills, including developing technical designs, architecture decision records, interface contracts, and documentation, and influencing senior stakeholders without direct authority.
Ability to navigate ambiguous technical challenges, thoughtfully evaluate tradeoffs, and own durable solutions that improve engineering effectiveness across teams.
Deep experience applying AI tools to complex engineering work, including architecture exploration, development, testing, and problem-solving.

Nice To Haves

Experience in banking, lending, or another regulated environment with model risk management, SOX, or similar audit requirements.
Experience with data quality, lineage, catalog tooling, and data contract approaches.
Prior experience as a Principal Engineer, Staff Engineer, or technical lead for a platform engineering team.
Experience with Databricks Asset Bundles or equivalent deployment tooling.

Responsibilities

Own the technical vision and architecture for a declarative, configuration-driven ingestion framework that brings relational-database and object-store data into the raw data layer; define reusable patterns and contracts and evaluate ingestion tradeoffs.
Lead design and implementation of reusable, versioned libraries for policy-driven data protection, including configurable rules, governed access, consistent outputs, secure-by-default failure behavior, and audit trails; maintain backward compatibility across shared capabilities.
Define infrastructure-as-code practices for safe deployment, replacement, rollback, and lifecycle management, including data ownership for managed pipeline objects.
Establish compute, cost, and sizing guidance for large initial loads and steady-state incremental processing at billion-row scale.
Establish monitoring, alerting, reconciliation, recovery, and data-validation capabilities for reliable source-to-destination operations; identify and address observability gaps in platform tools.
Conduct structured proofs of concept with measurable success criteria, document architectural decisions and tradeoffs, and engage technology vendors to resolve product gaps and influence future capabilities.
Translate ambiguous technical challenges into clear requirements, interfaces, dependencies, and phased delivery plans.
Partner with Data Engineering, Data Governance, Information Security, DBAs, and source-system owners to deliver early solutions, build scalable long-term capabilities, and drive framework adoption.
Mentor senior and staff engineers; lead technical design and code reviews; establish best practices for testing, documentation, release management, and maintainability across shared platform capabilities.
Help shape how AI is incorporated into engineering workflows, applying sound judgment about security, accuracy, maintainability, and necessary human oversight.