Senior Data Engineer (Azure/Databricks)

Acuity Analytics

2d • Remote

About The Position

We’re looking for a Senior Data Engineer to join a team supporting global claims reporting for a leading reinsurance organisation.
You’ll work across Azure and Databricks, building reliable data pipelines and scalable solutions that power reporting and analytics across the business.

Requirements

• Strong experience with Azure Databricks (PySpark / Spark SQL)
• Solid Python and SQL skills
• Experience building data pipelines and data models
• Good understanding of Delta Lake / Lakehouse architecture
• Experience working with Azure DevOps and CI/CD
• Comfortable working with large, complex datasets
• Able to work with different teams and stakeholders

Nice To Haves

• Experience in insurance, reinsurance, or claims reporting
• Knowledge of NoSQL (e.g. Cosmos DB)
• Familiarity with event-driven or microservices architecture
• Experience with monitoring tools like Datadog or Application Insights
• Exposure to Microsoft Cognitive Services

Responsibilities

• Build and manage Databricks Workflows to orchestrate large-scale data pipelines
• Design and deliver ETL/ELT pipelines using best practices
• Work with Delta Lake and Lakehouse architecture to structure data
• Create and manage data assets using Unity Catalog
• Develop data transformations using Python (PySpark) and SQL
• Build and deploy PyFunc models into production pipelines
• Use Azure services like Key Vault, Logic Apps, Service Bus, and APIM to support secure and scalable solutions
• Set up and improve CI/CD pipelines with Azure DevOps
• Work with stakeholders to turn business needs into practical, scalable data solutions

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed
• Number of Employees: 1,001-5,000 employees