Senior Data Engineer
About The Position
We are looking for an experienced Data Engineer to join a dynamic data engineering team and contribute to the development of modern, scalable data solutions within a Microsoft Azure and Databricks environment. You will play a key role in designing, developing and maintaining data pipelines and data platforms, working with large and complex datasets to support business intelligence, analytics and data-driven decision-making. This is an excellent opportunity for a Data Engineer who enjoys working with modern cloud technologies and is interested in data architecture, modelling and building robust data solutions.
Requirements
- Experience creating data pipeline using python on Databricks
- Experience working at the curated and product layers of data engineering including transforming data into modelling technique including Data Vault 2.0, Kimball (dimensional) and 3NF
- A good understanding of techniques to manage schema evolution, slowly changing dimensions, data harmonisation.
- Practical experience of using DABs (Declarative Automation Bundles, nee Databricks Asset Bundles) for deployment
- Practical experience using git repositories and CI/CD pipelines (preferably with GitHub and ADO)
- Practical experience developing pipeline in a IDE environment (Preferably with VSCode)
Nice To Haves
- Experience with Microsoft Fabric and OneLake, particularly OneLake shortcuts.
- Knowledge of data modelling in Azure Data Warehouse.
- Experience working within the financial services or wealth management sector.
- Familiarity with data governance, data quality and modern cloud data architecture principles.
Responsibilities
- Design, develop and maintain scalable data pipelines and ETL/ELT processes using Azure Data Factory (ADF) and Azure Databricks.
- Develop efficient data processing and transformation solutions using Python and Databricks.
- Work with large and complex datasets, ensuring data quality, reliability and performance.
- Contribute to the design and implementation of modern data architectures within the Azure ecosystem.
- Work with Medallion Architecture principles to build robust and scalable data solutions.
- Support the development of Data Vault and Business Vault architectures using Databricks on Azure.
- Work with Microsoft Fabric and OneLake, including the use of shortcuts to integrate and access data across different platforms.
- Contribute to data modelling activities within Azure Data Warehouse and related analytical environments.
- Collaborate with Data Architects, Developers, Analysts and other stakeholders to understand requirements and translate them into effective technical solutions.
- Identify opportunities to improve data processing, architecture, performance and automation.
- Ensure solutions follow best practices around scalability, security, maintainability and data governance.
Benefits
- Remote within Spain Time zone.
- Possibility to be part of a multicultural team and work on international projects.
- CET working hours
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
