Azure Databricks Engineer (Pyspark)
About The Position
We are seeking a Tech Lead to drive the end-to-end migration of enterprise data integration workloads from Informatica to Azure Databricks and provide thought leadership for the Databricks platform within the supply chain ecosystem. This role combines hands-on technical leadership, architecture guidance, delivery ownership, stakeholder alignment, and platform enablement. The ideal candidate will lead modernization efforts that improve scalability, reliability, performance, and business value across supply chain data products and analytics capabilities. The current environment uses Informatica PowerCenter 10.4.1 for batch ETL workloads, with Oracle as both the source and target database. The target platform is Azure Databricks Runtime 14.4 LTS with Unity Catalog, Delta Live Tables (DLT), and ADLS Gen2.
Responsibilities
- Lead the end-to-end migration of data pipelines, workflows, and integration patterns from Informatica to Azure Databricks.
- Assess current-state Informatica assets and define target-state architecture, migration strategy, and phased execution plans.
- Modernize ETL patterns into scalable ELT and data engineering approaches aligned with Azure Databricks best practices.
- Analyze complex Informatica PowerCenter mappings and build equivalent PySpark-based pipelines in Azure Databricks.
- Assess Oracle stored procedures, functions, and packages invoked by or tightly coupled with Informatica workflows, and determine whether to migrate, refactor, or recreate the logic in Databricks using PySpark, Python, or Spark SQL.
- Provide technical leadership across solution design, engineering standards, code quality, testing strategy, deployment, and production readiness.
- Partner with business, product, architecture, infrastructure, security, and operations stakeholders to align migration outcomes with enterprise priorities.
- Establish migration success criteria covering data quality, reconciliation, performance, resilience, observability, and business continuity.
- Identify delivery risks, dependencies, and technical constraints early, and drive mitigation plans through execution.
- Guide cutover, stabilization, and post-migration optimization to support sustainable platform adoption and operational excellence.
- Mentor engineers and technical teams on Databricks design patterns, reusable frameworks, and modern data engineering practices.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Manager
Education Level
No Education Listed
