Data Engineer Azure Databricks Snowflake

TekWissen•Frisco, TX

21h•Hybrid

About The Position

We are seeking a Data Engineer to join a data engineering team responsible for a third-party data enrichment platform. This platform augments first-party datasets with external identity and attribute data to support analytics, activation, and research. The enriched datasets are consumed by multiple downstream systems and teams, including the Customer Data Platform (CDP) and other analytics/research stakeholders. The platform is Azure-native and built primarily on Databricks (processing + some ML workloads) and Snowflake (analytics/warehouse). A major focus is building reliable, governed, vendor-agnostic datasets while ensuring privacy/compliance, data governance, and cost efficiency.

Requirements

Strong coding: PySpark + SQL (hands-on, not only orchestration)
Databricks: notebooks/jobs, performance tuning fundamentals, medallion patterns
Spark fundamentals: partitioning, skew/shuffle optimization, understanding failures via logs
Snowflake: data modeling/usage for analytics/warehousing workloads
Azure ecosystem: Azure Data Factory (ADF) (orchestration)
Azure-native integrations and services exposure
Data engineering reliability patterns: validation, idempotency, replay/backfills, dedup, auditability
Data governance: Unity Catalog (preferred), lineage, access control patterns, PII handling
Ownership mindset: can execute independently without constant approvals/check-ins

Nice To Haves

Event-driven/streaming ingestion exposure
Delta/Databricks patterns such as Delta Live Tables (DLT)
Experience building config-driven export frameworks for multiple downstream consumers/vendors
Exposure/interest in identity resolution concepts
Familiarity with CAPI integrations / marketing tech data signals
Experience implementing operational telemetry: dashboards, alerts, SLA monitoring

Responsibilities

Build and enhance ingestion pipelines for large batch and event-driven paths.
Integrate data from third-party enrichment vendors, digital platforms via Conversion API (CAPI) integrations, and Rewards/Promotions systems.
Implement strong data validation, idempotency, replay/backfill strategies, and deduplication to prevent quality drift.
Own monitoring, alerting, dashboarding, and operational readiness.
Troubleshoot failures with root cause analysis, interpret Spark logs, and diagnose performance issues.
Apply privacy, compliance, and governance requirements across pipelines and datasets.
Support governance standards such as Unity Catalog, lineage, access controls, and managing PII vs non-PII access.
Design pipelines with cost awareness, including cluster sizing, workload tuning, and efficient compute/storage usage.
Work in a small, fast-moving team, be self-driven and ownership-oriented.
Raise and manage data quality escalations when issues are detected.
Contribute to evolving architecture.