Sr. Azure Data Architect – Databricks

Exavalu Solutions India • Dallas, TX

3d

About The Position

The Sr. Data Architect will design and implement end-to-end data architecture and ecosystems using Azure-native services and Databricks. The role builds scalable, high-performance data platforms and data frameworks, translating stakeholder requirements into reliable solutions. It implements data governance with Unity Catalog and develops ingestion and integration patterns using Medallion Architecture and data engineering best practices.

The architect develops PySpark notebooks and procedures for advanced data transformation, and tunes performance in Databricks and Azure SQL databases. The solutions support yield improvement, process control, predictive maintenance, and operational efficiency, while maintaining governance, security, scalability, and compliance across the platform.

The role requires broad data architecture and engineering experience, extensive Azure and Databricks expertise, and knowledge of ingestion, ELT, streaming and batch processing, data governance, security, and operational excellence. The preferred qualifications include insurance property-and-casualty cloud data and analytics experience, strong communication and problem-solving skills, and the ability to document architecture decisions and recommendations.

Requirements

• 10+ years of overall experience and 5+ years of architecture experience in data architecture or data engineering roles.
• Strong hands-on experience with major enterprise data platforms.
• Extensive experience with Azure-native services, including Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse Analytics, and Azure SQL.
• Strong hands-on experience with Azure Databricks and its modern capabilities, including Unity Catalog, Delta Live Tables, and other current best practices.
• Experience designing Unity Catalog catalogs and schemas, permissions and security, managed and external tables, lineage, and governance.
• Experience with data ingestion tools such as Fivetran and HVR or equivalent CDC and replication tools.
• Experience with ELT tools such as DBT.
• Familiarity with real-time and streaming data, batch processing, and sensor, OT, and ET data flows.
• Strong expertise in data ingestion, transformation, and orchestration frameworks.
• Hands-on experience building insurance-specific data models.
• Proficiency in Python, PySpark, and Spark SQL.
• Strong understanding of streaming versus batch processing and Spark job performance tuning.
• Experience optimizing data storage formats, partitioning strategies, and schema design.
• Experience with object stores, data lakes, cloud compute scaling, and Databricks cluster management.
• Deep understanding of data governance, security and compliance, RBAC and attribute-based access control, encryption and audit logs, and data privacy requirements.
• Experience with operational excellence practices, including monitoring and observability, reliability and metrics, and backup, failover, and disaster recovery strategies.

Nice To Haves

• Prior experience working on insurance P&C cloud-based data and analytics projects.
• Strong analytical and problem-solving skills.
• Experience defining and enforcing best practices for Databricks workloads and Spark-based pipelines.
• Solid understanding of distributed data processing concepts.
• Strong communication and stakeholder management skills.
• Ability to document architecture decisions, trade-offs, and technical recommendations.

Responsibilities

• Design and implement end-to-end data architecture and data ecosystems using Azure-native services and Databricks.
• Build scalable, high-performance data platforms and robust data frameworks.
• Partner with stakeholders to translate business requirements into reliable data solutions.
• Implement data governance principles using Unity Catalog.
• Create data ingestion and integration patterns using Medallion Architecture.
• Apply data engineering best practices and develop PySpark notebooks.
• Create procedures for advanced data transformation.
• Tune performance in Databricks and Azure SQL databases.
• Architect end-to-end data solutions supporting yield improvement, process control, predictive maintenance, and operational efficiency.
• Ensure governance, security, scalability, and compliance across the platform.
• Develop reliable Delta Live Tables pipelines, including CDC implementation, data transformations, data quality controls, scaling, and performance tuning.
• Apply Databricks and Spark workload best practices.
• Optimize data storage formats, partitioning strategies, and schema design.
• Support monitoring and observability, reliability and metrics, backup, failover, and disaster recovery strategies.
• Document architecture decisions, trade-offs, and technical recommendations.

Benefits

• Flexibility, including adapting work arrangements to employee, customer, and business needs.
• Welcome back program to help individuals return to work after a prolonged absence due to health or family reasons.
• Commitment to a diverse and inclusive workforce and a culture that embraces individuals and promotes diverse perspectives.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed