Principal Staff Software Engineer - Data Lake Infrastructure
About The Position
The Data Lake Infrastructure team builds and operates the foundational storage and table-management technologies powering LinkedIn’s data platform. It owns OpenHouse, LinkedIn’s open-source control plane for Apache Iceberg tables, and the broader data lake stack, including storage engines, table formats, metadata/catalog services, and compaction and retention systems. These services support thousands of internal teams, petabyte-scale datasets, and mission-critical analytics, ML, and AI workloads. The team works with open-source communities and partners across Data Infrastructure, Analytics, and AI Platform to provide reliable, cost-efficient, governable data access. The Principal Technical Lead will drive the technical vision and execution for OpenHouse and LinkedIn’s data lake technologies. The role sets architectural direction, leads complex cross-team initiatives, and mentors engineers building table-management and lakehouse infrastructure. It combines hands-on engineering with technical leadership, designing for scale, reliability, and openness, while representing LinkedIn in the open-source data lake ecosystem. The position is hybrid, based in Mountain View or Bellevue, with office attendance on select days as determined by business needs. The role owns the roadmap and architecture for OpenHouse and core data lake services, leads the design and implementation of highly available distributed systems handling petabyte-scale Iceberg tables, and drives ecosystem contributions. It partners with internal platform and infrastructure teams, evaluates emerging technologies, and ensures reliability, performance, and cost efficiency. The technical lead also drives design reviews and consensus, guides incident response and root-cause analysis, and provides mentorship and review leadership.
Requirements
- Bachelor’s degree in Computer Science or equivalent.
- 10+ years of industry experience in software design, development, and algorithm-related solutions.
- 10+ years of experience in technical leadership, including architecture ownership, mentoring, and cross-team influence.
- 8+ years of experience with table formats such as Iceberg, Delta, or Hudi; data lake/lakehouse architectures; and query engines such as Spark, Trino, or Presto.
- 8+ years building large-scale distributed data systems in storage, databases, data lakes, or data infrastructure.
- 10+ years of experience in systems programming using Java, Scala, or Go, and understanding of distributed systems fundamentals such as consistency, replication, and metadata management.
- Experience with communication and the ability to influence cross-functional teams.
- Demonstrated ability to align technical decisions with business outcomes and drive customer-centric innovation.
- Deep understanding of concurrency, distributed algorithms, and data structures.
Nice To Haves
- Master’s or PhD in Computer Science or a related field.
- Experience architecting, building, and operating large-scale distributed systems.
- Background in industry thought leadership.
- Production experience operating Iceberg or a similar table-format infrastructure at multi-petabyte, multi-tenant scale.
- Open-source contribution experience, especially in Apache Iceberg, Trino, Spark, or related lakehouse projects.
- Experience with catalog/metadata services, compaction, retention, and data governance.
- Background in storage cost optimization, including tiering, compaction, and format evolution.
- Track record leading cross-organizational technical initiatives as a technical lead or architect.
Responsibilities
- Own the technical roadmap and architecture for OpenHouse and core data lake services, including catalog, table format, compaction, retention, replication, and governance.
- Lead the design and implementation of large-scale, highly available distributed systems handling petabyte-scale Iceberg tables.
- Drive open-source contributions and engagement with Apache Iceberg and related ecosystem projects.
- Partner with Analytics, AI Platform, and Data Infrastructure teams to align requirements, integration points, and long-term platform strategy.
- Provide technical mentorship and code/design review leadership across the team, raising the engineering bar.
- Evaluate and adopt emerging lakehouse technologies and drive build-versus-adopt decisions.
- Ensure system reliability, performance, and cost efficiency through sound SLOs, monitoring, and capacity planning.
- Lead technical design reviews, write RFCs and design documents, and drive consensus across stakeholders.
- Guide incident response and root-cause analysis for critical data lake infrastructure issues.
Benefits
- Generous health and wellness programs.
- Time away for employees of all levels.
- Annual performance bonus, stock, benefits, and/or other applicable incentive compensation plans may be included in the total compensation package.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
