Azure Data Engineer
About The Position
As an Azure Databricks Data Engineer, you will work with the team to design and deliver scalable data pipelines, Lakehouse solutions, and governed data platforms on Azure. You will apply your expertise in PySpark, SQL, Delta Lake, Unity Catalog, and Power BI enablement to create secure, reliable, and high-performing data solutions. This role provides the opportunity to develop scalable and governed data solutions that directly support enterprise analytics and business decision-making. You will combine hands-on engineering with technical leadership, helping to establish best practices across data architecture, governance, security, performance, and platform reliability.
Requirements
- Knowledge of Finance and Insurance business processes and terminology.
- Work independently, Self-motivated and organized, Manages own time to ensure task are completed with the ability to work to tight deadlines and under high pressure situations.
- Can communicate openly, honestly and independently with stakeholders at all levels of the organisation to understand their business goals and processes.
- Strong experience crafting and implementing scalable ETL and ELT pipelines on Azure Databricks using PySpark, SQL, Notebooks, Jobs, and Workflows.
- Expertise in Lakehouse architecture, Delta Lake, and bronze, silver, and gold data-layer patterns, with the ability to develop reusable data transformation frameworks and libraries.
- Experience implementing Unity Catalog, role-based access control, data access controls, auditing, and consistent catalog, schema, and table standards.
- Ability to develop performance-optimized datasets for Power BI and collaborate on semantic models, data contracts, refresh strategies, Direct Lake, and Direct Query solutions where applicable.
- Experience optimizing Databricks clusters and job scheduling, implementing monitoring and alerting, and contributing to CI/CD practices using Git and Azure DevOps or GitHub.
Responsibilities
- Facilitate workshops to discuss, review and document business reporting requirements.
- Establish best practices across data architecture, governance, security, performance, and platform reliability.
- Developing and implementing scalable data integration pipelines on Azure Databricks using PySpark, SQL, Notebooks, Jobs, and Workflows.
- Building and maintaining Lakehouse architectures using Delta Lake and bronze, silver, and gold data layers to support analytics and reporting.
- Crafting robust, reusable systems and collections for data transformation.
- Implementing and managing Unity Catalog to provide centralized governance of data, tables, and permissions.
- Defining and enforcing role-based access control, data access controls, and auditing in line with security and compliance requirements.
- Standardizing catalog, schema, and table naming conventions across environments.
- Delivering well-modelled, performance-optimized datasets for Power BI and other business analytics tools.
- Collaborating with BI developers and analysts to define semantic models, data contracts, and refresh strategies.
- Supporting design and development of new reports.
- Troubleshooting and optimizing query performance between Power BI and Databricks, including Direct Lake and Direct Query where applicable.
- Configuring and improving Databricks clusters, cluster policies, and job schedules for cost, performance, and reliability.
- Implementing monitoring, logging, and alerting for data pipelines and jobs.
- Contributing to and improving CI/CD pipelines for Databricks code using Git and Azure DevOps or GitHub.
- Explore AI/ML technologies and opportunities to deploy machine learning pipelines that generate measurable business value.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
