Director of Data Engineering

Ryan Specialty • Chicago, IL

1d • $132,000 - $165,000

About The Position

The Director of Data Engineering is responsible for building and operating the data engineering function within Ryan Specialty’s corporate data organization. This role is accountable for the silver and gold layer of the medallion data architecture thus transforming raw ingested data into clean, conformed, and business-ready data assets that serve as the foundation for downstream consumption. Beyond technical delivery, this role builds and scales a high-performing engineering team, establishes disciplined software development practices, and manages the contractor and external parties that augment internal capacity. An ideal candidate should be collaborative and dynamic and be able to work across the organization on a fast-paced and quickly evolving team. The leader will have five primary responsibilities: 1. Own and deliver the silver and gold layer of the medallion data architecture 2. Establish and enforce a standardized software development lifecycle for all data engineering work 3. Build, mentor, and develop a high-performing data engineering team 4. Partner with IT on data platform infrastructure, tooling, and integration What will your job entail? Own the Databricks medallion architecture from silver forward: transformation patterns, data quality enforcement, business logic application, gold-layer curation, and pipeline orchestration Establish and enforce a standardized SDLC that all team members must follow: version control, code review, testing, CI/CD, and documentation Design and implement a data quality framework in partnership with the VP of Data Governance who defines it from a business perspective Lead Master Data Management (MDM) technical implementation: entity resolution pipeline design, match-and-merge workflow configuration, golden record integration into the gold layer, and performance optimization Serve as the primary interface between the data organization and the IT data platform team Translate engineering requirements into platform specification across Azure and Databricks for the IT data platform team to deliver Conduct structured code reviews that are explicitly developmental for data engineers Build a library of engineering standards, patterns, and examples Own the technical planning for contractor-staffed project delivery pods Design and maintain the employee and contractor onboarding process Provide ongoing technical oversight for all data engineers Own technical compliance enforcement in the Databricks environment: data lineage tracking via Unity Catalog, access audit logs, encryption verification, and contractor access governance Hire, mentor, develop, and assess a team of data engineers and contractors

Requirements

• Bachelor’s degree in computer science, information systems, engineering, or a related technical field
• 7+ years in data engineering with at least 3 years in leadership or management role
• Proven experience owning large-scale data engineering development in a cloud-native environment
• Deep expertise in Azure Databricks, including Unity Catalog, Delta Lake, Delta Live Tables, PySpark, Spark SQL, Databricks SQL, Jobs/Workflows, and cluster management
• Hands-on experience with data quality frameworks, including implementing quality gates, monitoring pipelines, and designing alerting systems
• Strong SQL skills with experience in large-scale dimensional modeling (star schema, slowly changing dimensions, multi-source fact table design)
• Experience with cloud data lake architectures on Azure (ADLS Gen2, Azure Data Factory, Azure DevOps, Azure Key Vault). AWS experience is acceptable if Databricks experience is strong
• Proficiency in Python for data engineering (PySpark, pandas, orchestration scripts, SDK integrations)
• Hands-on experience with CI/CD pipelines for data (Azure DevOps or equivalent); comfortable reviewing infrastructure-as-code, deployment scripts, and PR pipelines
• Demonstrated ability to push back on senior stakeholders and enforce process discipline
• Track record of building and scaling data engineering teams in a high-growth environment: establishing code review norms, architecture decision record, knowledge-sharing sessions, and technology onboarding programs
• Experience managing contractors and external development partners
• Ability to understand and communicate with technical and business experts
• Intellectual curiosity with a strong drive to learn
• Critical thinker with an open mind to innovative solutions
• High emotional intelligence
• Exceptional communication skills
• Strong collaboration and relationship building skills
• Ability to navigate ambiguity
• Highly structured in execution
• Skilled at simplifying complex environments
• Applicants must be authorized to work for any employer in the U.S.

Nice To Haves

• Experience in the insurance industry, particularly E&S lines preferred; or a strong willingness to learn

Responsibilities

• Own and deliver the silver and gold layer of the medallion data architecture
• Establish and enforce a standardized software development lifecycle for all data engineering work
• Build, mentor, and develop a high-performing data engineering team
• Partner with IT on data platform infrastructure, tooling, and integration
• Own the Databricks medallion architecture from silver forward: transformation patterns, data quality enforcement, business logic application, gold-layer curation, and pipeline orchestration
• Establish and enforce a standardized SDLC that all team members must follow: version control, code review, testing, CI/CD, and documentation
• Design and implement a data quality framework in partnership with the VP of Data Governance who defines it from a business perspective
• Lead Master Data Management (MDM) technical implementation: entity resolution pipeline design, match-and-merge workflow configuration, golden record integration into the gold layer, and performance optimization
• Serve as the primary interface between the data organization and the IT data platform team
• Translate engineering requirements into platform specification across Azure and Databricks for the IT data platform team to deliver
• Conduct structured code reviews that are explicitly developmental for data engineers
• Build a library of engineering standards, patterns, and examples
• Own the technical planning for contractor-staffed project delivery pods
• Design and maintain the employee and contractor onboarding process
• Provide ongoing technical oversight for all data engineers
• Own technical compliance enforcement in the Databricks environment: data lineage tracking via Unity Catalog, access audit logs, encryption verification, and contractor access governance
• Hire, mentor, develop, and assess a team of data engineers and contractors

Benefits

• In addition to paid time off for company holidays, vacation, sick and personal days, Ryan offers paid parental leave, mental health services and more.
• For additional information on Ryan Specialty Total Rewards, visit our website https://benefits.ryansg.com/.
• We provide individuals with disabilities reasonable accommodations to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment in accordance with applicable law.

What This Job Offers

• Job Type: Full-time
• Career Level: Manager
• Number of Employees: 1,001-5,000 employees