Principal Data Engineer

Springbrook Holding Company LLC•,

1d•Remote

About The Position

We're hiring a Principal Data Engineer who wants to stay deeply technical while growing into people leadership. This is fundamentally an individual-contributor / architect role and you'll be the primary hands-on builder of our data platform, writing production code in Databricks and Azure on a daily basis. Alongside that, you'll manage and mentor a small team of 3–5 offshore data engineers, giving you a natural runway toward a full-time Data Engineering Manager role as the team and platform scale. You'll be the technical owner of a global data platform that unifies data across 10+ products, supports both transactional and analytical workloads, and is architected from day one to onboard future acquisitions cleanly. You'll also design the platform's orchestration layer to support complex agentic AI workflows, including agent-to-agent (A2A) communication between autonomous systems.

Requirements

8+ years of experience in data engineering or data architecture, with deep, hands-on You should be someone who still enjoys writing and reviewing code, not just reviewing architecture diagrams.
Expert-level, hands-on experience with Databricks ( Delta Lake, Unity Catalog, Workflows/Jobs, cluster/job optimization, MLflow).
Strong hands-on experience with the Azure data ecosystem (ADLS Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Functions, Azure DevOps, Key Vault, Event Hub/Event Grid).
Expert-level SQL and Python (PySpark); comfortable debugging Spark jobs at a deep level.
Experience designing data platforms that serve both transactional and analytical workloads, and that can flexibly onboard new data sources (e.g., from M&A activity or new product lines).
Familiarity with agentic AI orchestration concepts and frameworks (e.g., LangChain, LangGraph, AutoGen, or similar) and an interest in designing infrastructure for agent-to-agent (A2A) communication and multi-agent workflows.
Prior experience leading or mentoring engineers (formal or informal) with genuine interest in growing into a people-management role over time.
Experience working with offshore/distributed engineering teams, including async communication and code review across time zones.
Solid understanding of data modeling, medallion architecture, CI/CD for data workflows (Azure DevOps/GitHub Actions), and Infrastructure-as-Code (Terraform/Bicep).
Excellent communication skills; able to work directly with cross-functional stakeholders, including Corp Dev/M&A teams, on platform strategy.
Big Data & Processing: Apache Spark (PySpark), Databricks (Delta Lake), Azure Data Factory, Azure Synapse, Kafka, Event Hub/Event Grid
Cloud Platforms: Azure (ADLS, AKS, Azure Functions, Cosmos DB, Azure SQL, API Management), with some AWS/GCP exposure
AI/ML & GenAI Tooling: Azure Machine Learning, Azure OpenAI Service, MLflow, LangChain, RAG pipelines, vector search
DevOps/MLOps: Azure DevOps, GitHub Actions, Docker, Kubernetes (AKS), Terraform/Bicep, MLflow, Airflow
Languages & Frameworks: Python (PySpark, FastAPI, Django/Flask), SQL, Java/Spring Boot, Node.js
Monitoring & Observability: Azure Monitor, Application Insights, ELK Stack, Prometheus/Grafana
Security & Compliance: Azure AD B2C, Key Vault, OAuth2/JWT, RBAC, HIPAA-ready configurations

Nice To Haves

Databricks Certified Data Engineer Professional or Azure Data Engineer Associate/Expert certification.
Experience integrating a data platform after a merger or acquisition or building with future M&A integration in mind.
Experience with vector databases (FAISS, Pinecone, Weaviate) and RAG pipelines, in support of agentic AI use cases.
Experience with Kafka, Event Hubs, or other streaming technologies for real-time transactional and analytical pipelines.
Background in a regulated industry (healthcare, finance) with associated compliance requirements (HIPAA, SOC 2).

Responsibilities

Architect and build a unified, multi-tenant data platform on Databricks (Delta Lake, Unity Catalog, Workflows, MLflow) and Azure (ADLS Gen2, Azure Data Factory, Synapse, Azure Functions, Event Hub/Event Grid, Key Vault) that integrates data from 10+ product lines into a single source of truth.
Design the platform to be growth-ready, with modular ingestion patterns, flexible schema onboarding, and governance structures that let new products or acquired companies plug in without re-architecture.
Build and optimize pipelines that serve both transactional (OLTP-adjacent, low-latency) and analytical (OLAP, batch/streaming) use cases from a shared underlying platform, using the medallion architecture (bronze/silver/gold) and appropriate serving layers for each workload type.
Design and implement the orchestration layer for agentic AI workflows, including agent-to-agent (A2A) communication, multi-step task automation, and event-driven coordination between autonomous agents operating on the data platform.
Write and review production-grade PySpark, SQL, and Python code; this role stays in the codebase, not just in design docs.
Own performance, scalability, and cost optimization across the Databricks/Azure environment.
Directly manage and mentor 3–5 offshore data engineers, setting technical direction, reviewing their code, and unblocking their work.
Establish engineering standards, CI/CD practices, and testing/observability norms for the team to follow.
As the platform and team grow, take on expanded management scope, with a clear path toward a Data Engineering Manager.
Partner with Product, Engineering, Security, and M&A/Corp Dev stakeholders to ensure the data platform's architecture supports business growth, including future acquisitions.
Establish data governance, lineage, access control (RBAC/Unity Catalog), and compliance practices across a growing, multi-product data estate.
Evaluate and integrate new tools (including emerging agentic AI orchestration frameworks) to keep the platform modern and extensible.
Manage 3–5 offshore data engineers, setting architectural direction and coding standards, reviewing their Databricks/PySpark work, guiding their approach to pipeline design, and developing their skills toward platform ownership over time.