Principal Data Architect
About The Position
Berkshire Hathaway Specialty Insurance (BHSI) is modernizing its data ecosystem from a monolithic, reporting-centric infrastructure to a Lakehouse platform supporting insight-driven decisions and AI-enabled outcomes. The Principal Data Architect will set technical direction and own the architecture of BHSI’s Databricks-based Lakehouse across data engineering, analytics, and AI/ML, ensuring the platform is secure, performant, cost-efficient, and ready for Generative AI. This is a hands-on senior individual-contributor role, not a people-management position. The role reports to the Global Head of Data Analytics Solutions and partners with Enterprise Technology portfolio leaders and business stakeholders. BHSI describes itself as a values-based organization centered on respect, integrity, excellence, collaboration, and passion, and seeks to foster an inclusive environment in which teammates can contribute and be recognized.
Requirements
- 15+ years in data engineering, data architecture, or software architecture, including 5+ years architecting on the Databricks Lakehouse Platform.
- Deep expertise in Apache Spark, Delta Lake, Unity Catalog, and MLflow.
- Proven experience designing infrastructure for traditional machine-learning and Generative AI workloads, including RAG, fine-tuning, vector search, and agentic frameworks.
- Strong proficiency in Python and SQL, infrastructure-as-code such as Terraform, and software design patterns.
- Hands-on architecture experience on AWS or Microsoft Azure.
- Demonstrated ability to influence senior technical and business stakeholders without direct authority.
Nice To Haves
- Databricks Professional- or Expert-level certification.
- Experience with advanced agentic architectures, DSPy, or graph databases.
- Background in insurance, financial services, fintech, or other regulated environments.
- Track record of optimizing cloud compute costs while improving pipeline throughput.
Responsibilities
- Define and evolve the enterprise Lakehouse reference architecture on Databricks for batch, streaming, and AI workloads at scale across AWS or Azure.
- Lead the design and rollout of Databricks AI capabilities, including vector search, model serving, and LLM orchestration, enabling teams to build, fine-tune, and deploy models securely against enterprise data.
- Implement lineage, access controls, and auditing with Unity Catalog to meet regulatory requirements and ensure data quality across domains.
- Establish CI/CD and infrastructure-as-code standards for data pipelines and ML models.
- Optimize Spark performance, cluster configuration, and compute spend.
- Set architecture standards, conduct design reviews, and mentor engineers and data scientists.
- Translate business strategy into a phased technical roadmap and build alignment with engineering and business leaders.
Benefits
- Comprehensive health, dental, and vision benefits.
- Short-term and long-term disability insurance.
- Life insurance for the employee and family.
- Accidental death and dismemberment insurance for the employee and family.
- Flexible Spending Accounts.
- Health Reimbursement Account.
- Employee Assistance Program.
- Retirement Savings 401(k) Plan with Company Match.
- Generous holiday and Paid Time Off.
- Tuition reimbursement.
- Paid parental leave.
- Annual bonus eligibility.
- Competitive package and career growth opportunities.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
Education Level
No Education Listed
