Senior Data Engineer
About The Position
Join an ambitious deep-tech initiative shaping the future of clean energy. As a Senior Data Engineer, you will design and scale cloud-native data platforms for a pioneering fusion energy company working on the world’s first commercial laser fusion power plant. You will contribute to greenfield data engineering initiatives, near-real-time pipelines, and large-scale data infrastructure modernization using Databricks and AWS technologies. The role offers the opportunity to support breakthrough research while advancing engineering excellence, automation, and AI adoption within the data platform ecosystem. The customer is a German-American deep-tech company developing direct-drive laser fusion technology and commercializing the 2021 Lawrence Livermore breakthrough in which scientists achieved fusion ignition for the first time.
Requirements
- At least 5 years of experience in data engineering.
- At least 3 years of production experience with Databricks and Delta Lake.
- Strong expertise in medallion architecture and event-driven asynchronous pipeline design.
- Advanced proficiency in Python, PySpark, and SQL.
- Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.
- Strong AWS expertise, including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
- Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles.
- Practical experience with Git workflows, pull request reviews, and automated testing.
- Experience configuring secure CI pipelines using OIDC or service principals.
- Practical MLflow experience, including experiment tracking, model registration, and reproducibility practices.
- Strong understanding of data quality, lineage, provenance, schema contracts, and versioning.
- Upper-Intermediate English level or higher.
Nice To Haves
- Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.
- Experience with monitoring, observability, and cloud cost optimization.
- Experience packaging workloads in Docker containers and running batch jobs on AWS.
- Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy.
- Experience building feature sets and train/test splits that avoid data leakage.
- Experience with regulated or high-security environments, including data residency requirements.
- Experience disabling or restricting AI features in secured environments.
Responsibilities
- Design and build scalable cloud-native data platforms from greenfield to production.
- Design and implement near-real-time ingestion pipelines using event-driven patterns.
- Define and enforce data platform standards, including Data Lake/Lakehouse principles, medallion architecture, and data contracts.
- Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability.
- Introduce best practices for code quality, testing, and CI/CD across data pipelines.
- Drive adoption of AI tooling and agentic workflows within the data engineering team.
- Ensure data quality, observability, and reliability across pipelines and platforms.
- Design and deliver self-service tooling and microservices that abstract platform complexity.
- Collaborate directly with the customer's scientific and engineering teams.
- Monitor platform performance, reliability, and cloud infrastructure costs.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
