Data Engineer

Sigma Software

About The Position

Join a project focused on building and evolving a modern cloud-based data platform for scientific research, analytics, and operational workflows in the fusion energy domain. The platform supports scalable distributed data processing while maintaining data quality, lineage, governance, reproducibility, and operational reliability. The Data Engineer will work closely with scientists and engineers to support experimentation, analytics, and machine learning workflows in a research-intensive environment. The role involves designing, developing, and maintaining cloud-native data platforms and pipelines using technologies including Databricks, Apache Spark, Python, AWS, and Azure. The engineer will integrate heterogeneous data sources, model and optimize data structures, and improve pipeline performance, reliability, scalability, and cost-efficiency. The role also contributes to CI/CD, infrastructure automation, engineering best practices, and evaluation of new data engineering technologies.

Requirements

At least 3 years of commercial experience in Data Engineering.
At least 2 years of hands-on production experience with Databricks.
Strong expertise with Delta Lake, including medallion architecture and table optimization.
Advanced proficiency in Python, PySpark, and SQL.
Deep practical knowledge of Databricks components, including Unity Catalog, Lakeflow Jobs / Workflows, and job optimization.
Hands-on experience with AWS services including S3, IAM, and KMS, plus at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
Experience designing event-driven and asynchronous data pipelines.
Strong knowledge of Infrastructure as Code and CI/CD practices.
Practical experience with Terraform and Databricks Asset Bundles.
Experience with Git workflows, pull request reviews, and automated testing.
Understanding of secure authentication approaches such as OIDC and service principals.
Practical experience with MLflow, experiment tracking, model registry, and reproducibility practices.
Strong understanding of data quality, lineage, schema contracts, and provenance concepts.
Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQX.
Experience monitoring and optimizing data platform reliability and cloud costs.
Upper-Intermediate or higher English level.

Nice To Haves

Experience packaging workloads using Docker and running batch workloads in AWS.
Familiarity with ECR and containerized data processing solutions.
Experience with experiment optimization libraries such as scikit-optimize, BoTorch, Ax, or scipy.
Understanding of ML feature engineering and leakage prevention techniques.
Experience working in regulated or high-security environments.
Knowledge of data governance, compliance, and data residency requirements.
Experience disabling or restricting AI-assisted tooling or features in secure environments.

Responsibilities

Design, develop, and maintain scalable and reliable data platforms and pipelines for analytics and operational use cases.
Build distributed data processing workflows using Apache Spark, PySpark, and Databricks.
Develop and evolve cloud-native data solutions leveraging Azure and AWS services.
Model and optimize data structures for analytics, reporting, and downstream consumption.
Integrate data from multiple heterogeneous sources while ensuring data quality, consistency, and traceability.
Implement data quality controls, schema validation, lineage tracking, and monitoring processes.
Optimize the performance, scalability, reliability, and cost-efficiency of data pipelines and infrastructure.
Collaborate with scientists, analysts, and engineering teams to translate business and research requirements into technical solutions.
Contribute to CI/CD processes, infrastructure automation, and engineering best practices.
Research and introduce new technologies and approaches within the data engineering ecosystem.