Sr Software Engineer - Enterprise Data Platform & Customer Data Platform

Five Below-posted 2 months ago
Full-time • Senior
Hybrid • Philadelphia, PA
General Merchandise Retailers
Resume
Match Score
Upload and Match ResumeTrack Jobs with Teal

We are seeking a Senior Software Data Engineer to lead the design, development, and optimization of our Enterprise Data Platform (EDP) and Customer Data Platform (CDP) using Azure and Azure Databricks. This role will architect metadata-driven data solutions, lead a team of data engineers, and collaborate with cross-functional stakeholders to deliver scalable, secure, and efficient data infrastructure. The Senior Data Engineer will take a holistic approach, ensuring data pipelines, governance, and analytics align with enterprise and customer-centric objectives in an Azure environment.

  • Mentor and lead a team of data engineers, fostering innovation, collaboration, and technical excellence while overseeing project delivery and resource allocation.
  • Architect and maintain a robust EDP on Azure to centralize data assets, enabling enterprise-wide analytics, reporting, and insights generation.
  • Design and implement a CDP on Azure to unify customer data, enabling personalized experiences, segmentation, and advanced analytics.
  • Build scalable, metadata-driven frameworks in Azure Databricks for data ingestion, transformation, and orchestration, ensuring flexibility and reusability.
  • Develop and optimize end-to-end data pipelines using Azure Databricks, Spark, and Apache Kafka for batch and real-time processing to support EDP and CDP use cases.
  • Deploy and manage cloud-native data solutions on Azure, integrating seamlessly with Azure Databricks, Azure Data Lake Storage and other Azure services.
  • Implement metadata management, data cataloging, and lineage tracking using Databricks Catalog to ensure data quality, security, and regulatory compliance (e.g., GDPR, CCPA).
  • Design and manage scalable databases (e.g., lakebase) for efficient storage, retrieval, and querying of enterprise and customer data.
  • Leverage Apache Airflow Workflows to orchestrate complex data pipelines reliably and at scale.
  • Enhance pipeline performance, query efficiency, and cost optimization for large-scale data processing in Azure environments.
  • Partner with product managers, data scientists, analysts, and business stakeholders to align data solutions with enterprise and customer goals.
  • Stay abreast of emerging Azure data technologies and trends, driving adoption of best practices and tools to enhance EDP and CDP capabilities.
  • Design and implement ad-hoc and automated data pipelines with a strong focus on encryption and decryption requirements.
  • Collaborate with external vendors to ensure secure data exchange by verifying keys, service accounts, and repository credentials (e.g., SFTP, S3 buckets).
  • Manage high-volume data transmissions to advertising vendors, including encryption, file parsing, and ensuring end-to-end delivery success.
  • Provide technical guidance to vendors, often coaching them through secure transmission protocols despite their role as data brokers.
  • Rapidly deploy and harden transmission solutions to meet compliance standards under tight timelines.
  • Demonstrate expertise or quick learning ability in file-level and record-level encryption/decryption, as well as hashing algorithms at the data element level.
  • 8 -10+ years in data engineering, with 4+ years focused on enterprise-scale data platforms and/or customer data platforms, including 3+ years in a lead or senior role managing teams.
  • Proven ability to lead, mentor, and inspire data engineering teams, with experience in agile project management and cross-functional collaboration.
  • Advanced proficiency in Azure Databricks, Delta Lake, Spark, and Databricks Workflows for building scalable data solutions.
  • Deep expertise in Spark, Apache Kafka, and related big data technologies for large-scale and real-time data processing.
  • Strong skills in designing and managing MongoDB, Delta Lake, or similar databases for enterprise and customer data use cases.
  • Hands-on experience with Apache Airflow workflows for pipeline orchestration.
  • Expertise in metadata management, data cataloging, and lineage tools to ensure data quality and compliance.
  • Proficient in Python, PySpark, Spark SQL, and SQL for developing robust data pipelines and transformations.
  • Advanced skills in query optimization and performance tuning for large-scale datasets in Azure data lakes and warehouses.
  • Strong knowledge of data modeling techniques, including star schema, Snowflake schema, and data lake/warehouse architectures.
  • Exceptional analytical skills with a solution-oriented mindset to tackle complex data challenges.
  • Outstanding verbal and written communication skills to collaborate with technical and non-technical stakeholders.
  • Experience building or managing Customer Data Platforms (CDP) for marketing, personalization, or customer analytics use cases on Azure.
  • Familiarity with data privacy frameworks and compliance standards (e.g., GDPR, CCPA).
  • Exposure to machine learning pipelines or data science workflows integrated with EDP/CDP on Azure.
  • Knowledge of hybrid or multi-cloud environments involving Azure and other platforms.
Track Jobs with Teal

Job Search Resources

•
AI Resume Builder
•
Platform Engineer Resume Examples
•
Cover Letter Examples

Tools

Career Hubs

Guides

Company

© 2024 Teal Labs, Inc
Privacy PolicyTerms of Service