Lead Data Engineer

TalentOla•Cary, NC

21h

About The Position

Altimetrik is building a centralized, AI-first enterprise Data Hub for a global insurance and financial services client on Azure Databricks. The platform ingests more than 150 inbound data feeds and distributes data to more than 35 downstream systems, using a Bronze/Silver/Gold medallion architecture. AI is embedded throughout ingestion, canonical mapping, data quality, reconciliation, and business-user access. This is a hands-on technical leadership role responsible for end-to-end design of the data and AI layers, developing reference implementations, and regularly delivering production-grade Python, Scala, and PySpark code. The Lead Data Engineer will architect and build the lakehouse and its ingestion, transformation, quality, governance, semantic access, and AI capabilities; establish engineering standards; and mentor the team. The role also includes presenting designs to architecture and AI governance forums and producing documentation and reusable components that enable the client team to operate the platform independently.

Requirements

12–18 years of experience in data engineering, data platform delivery, or related disciplines.
Expert-level proficiency in Python, Scala, and PySpark, with a track record designing and delivering production-ready, modular, well-tested solutions.
Experience developing and troubleshooting Spark workloads and optimizing large-scale batch and streaming pipelines using Delta Lake and Spark technologies.
Strong SQL and data modelling skills, including dimensional and normalized modelling, schema design, and data contract definition.
Databricks expertise, including Delta Lake, Unity Catalog, Jobs and Workflows, cluster and pool management, performance tuning, and Model Serving.
Azure data stack experience with ADLS Gen2 zone design, ACLs and lifecycle; Azure Data Factory parameterized and metadata-driven frameworks and error handling; and Azure Event Hubs.
At least 3 years designing and shipping LLM-based systems in production, including RAG pipelines, agentic or tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.
Evaluation experience using golden datasets, regression suites, accuracy and hallucination tracking, and human-in-the-loop feedback loops.
Hands-on experience with LangChain, LlamaIndex, or LangGraph, and at least one provider stack: Azure OpenAI, OpenAI, or Databricks Model Serving.
Metadata-driven experience or capability, including schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks for onboarding sources without new code.
Proven delivery of an enterprise-scale medallion or lakehouse architecture.
Azure security and governance experience with Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.
CI/CD and infrastructure-as-code experience with Azure DevOps, Terraform, Databricks Asset Bundles, and automated data-pipeline testing.
Clear technical writing skills and ability to present and defend designs to engineers and non-technical stakeholders.
Able to work with US-based client stakeholders and explain technical trade-offs to finance business owners without jargon.

Nice To Haves

Knowledge graphs and ontologies, including RDF/SPARQL, property graphs such as Neo4j, or graph modelling over a lakehouse.
Enterprise-scale text-to-SQL or semantic-layer-backed natural-language query systems, including access control and ambiguity handling.
ML-based anomaly detection on time-series or transactional financial data.
Financial services or insurance domain knowledge, such as finance close, general ledger, subledger, reconciliation, or actuarial data.
LLMOps and MLOps experience, including model and prompt versioning, cost governance, and observability tooling.
Databricks Data Engineer Professional, Azure DP-203 or DP-700, or AZ-305 certification.
Experience with dbt, Great Expectations, or similar data-quality and transformation tooling.
Workday, Prism, or Accounting Center exposure.

Responsibilities

Own the end-to-end lakehouse architecture, including medallion layer contracts, ADLS Gen2 zone layout, Delta Lake table design, partitioning, schema evolution, and retention strategy.
Design and build metadata-driven, parameterized ingestion frameworks that onboard new data sources without bespoke pipeline code for every feed.
Develop canonical PySpark and Scala Spark transformation jobs, establish coding and testing standards, review pull requests, and debug production incidents.
Tune Spark clusters and pools, apply partition pruning and caching strategies, and set cost guardrails as data volumes grow.
Build and automate CI/CD for Databricks and Azure Data Factory pipelines using Azure DevOps, Databricks Asset Bundles, and Terraform; maintain observability with Azure Monitor and Log Analytics.
Design repeatable ingestion patterns for batch files, database extracts, CDC feeds, and streaming using Azure Event Hubs or Kafka and Spark Structured Streaming.
Build an AI-augmented metadata ingestion framework that generates bridge documents, DML statements, canonical table definitions, and control metadata from source schemas.
Build AI-assisted source-to-canonical attribute mapping with schema reasoning, data profiling, confidence scoring, and human review and approval before production.
Generate file-level and record-level validation rules from historical data and metadata analysis, supporting a configurable rules-engine-backed data quality framework.
Develop AI-assisted data quality that analyses patterns across Bronze, Silver, and Gold layers and proposes rules beyond predefined checks.
Deliver anomaly detection for outliers, data drift, schema drift, volume shifts, and reconciliation breaks, with actionable alerting.
Build AI-assisted automated reconciliation and test-data generation for the automated testing framework.
Produce synthetic, privacy-preserving lower-environment datasets using differential privacy, format-preserving masking, and referential-integrity-safe generation.
Design and build an ontology-driven semantic layer and knowledge graph to model finance data relationships and support data discovery, semantic integration, and AI/BI tooling.
Build a GPT-powered natural-language querying interface for financial data using text-to-SQL or knowledge-graph-grounded semantic retrieval, enforcing row- and column-level security and source traceability.
Own AI architecture decisions covering model selection, RAG and retrieval design, prompt strategy, evaluation harnesses, guardrails, cost and latency budgets, and observability.
Implement Unity Catalog for cataloguing, lineage, and fine-grained access control; define PII classification, masking, tokenization, and encryption standards.
Present designs to Architecture Review Boards and AI governance forums, addressing responsible AI, data residency, model approval, auditability, and human-in-the-loop controls.
Mentor data engineers, run design reviews, and establish engineering patterns without becoming a bottleneck.
Produce documentation and reusable components that enable the client's team to operate the platform independently at the end of the engagement.