Senior Data Engineer

Sigma Software

7d

About The Position

Are you passionate about building cutting-edge, AI-ready data platforms from the ground up? We are looking for a Senior Data Engineer to join our Data Engineering Team and lead high-impact, greenfield initiatives. You will work on building modern cloud-native data platforms, migrating on-premises legacy systems to the cloud, and laying the architectural foundation for AI-ready data infrastructure. In this role, you will collaborate closely with Machine Learning, Data Science, and Product teams, serving as a key technical contributor and thought leader. You will also drive R&D efforts around agentic AI architectures, event-driven systems, and LLM-ready data pipelines – turning architectural concepts into production-grade solutions.

Requirements

• 5+ years of professional experience in Data Engineering
• Strong Python and SQL development skills for pipeline development and optimisation
• Proficiency in Apache Spark / PySpark, including query optimisation and performance tuning
• Hands-on experience with Databricks (preferred) or Snowflake
• Experience with at least one major cloud provider: Azure (preferred), AWS, or GCP
• Experience with stream processing technologies (Kafka, Spark Structured Streaming)
• Solid understanding of ETL/ELT patterns, data modelling (dimensional, Data Vault), and data warehousing
• Experience with orchestration tools (Apache Airflow, Azure Data Factory, or equivalent)
• Knowledge of Infrastructure as Code (Terraform or equivalent)
• Understanding of production-grade system requirements: reliability, scalability, observability, and performance
• Upper-Intermediate English level

Nice To Haves

• Familiarity with RAG pipeline design and LLM integration patterns
• Knowledge of data governance frameworks and tools (Unity Catalog, Apache Atlas, or similar)
• Experience with dbt for data transformation and modelling
• Familiarity with MLflow, Feature Stores, or ML platform integration

Responsibilities

• Design and build scalable, cloud-native data platforms from greenfield to production
• Implement near-real-time ingestion pipelines using event-driven patterns
• Define and enforce platform standards, including Data Lake / Lakehouse principles, medallion architecture, and data contracts
• Refactor and optimise existing Spark and PySpark scripts for performance and maintainability
• Introduce best practices for code quality, testing, and CI/CD across data pipelines
• Drive adoption of AI tooling and agentic workflows within the data engineering team
• Ensure data quality, observability, and reliability across all pipelines and platforms
• Develop self-service tooling and microservices to simplify platform usage for other teams

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed