Data Engineering Lead

MediaRadar

9d • Remote

About The Position

We are seeking a visionary and hands-on Data Engineering Lead to spearhead the design, development, and optimization of our next-generation data platform. As a Lead, you will balance technical excellence with people leadership, ensuring our data architecture is scalable, resilient, and perfectly aligned with our business goals - while collaborating cross-functionally to support analytics, reporting, and operational data needs..The ideal candidate should be a PySpark expert who thrives in the Azure ecosystem and has a deep appreciation for clean, modular code and robust ETL patterns.This is an exciting opportunity to work along with a great team of data engineers, demanding technologies and an engaging work environment to help shape our data engineering best practices.

Requirements

• Azure Databricks: Expert-level experience managing workspaces, clusters, and job scheduling.
• Solid understanding of data lakehouse architectures and Delta Lake.
• Proven experience in Performance Tuning, Spark Optimization and Cost Reduction.
• PySpark: Advanced proficiency in Spark DataFrame APIs and Spark SQL for large-scale data processing involving various data formats.
• SQL Mastery: Exceptional ability to write, tune, and troubleshoot complex queries.
• PostgreSQL: Hands-on experience with relational database design, indexing, and performance optimization.
• ETL/ELT Frameworks: Proven track record of building scalable data pipelines from scratch.
• 10+ years of experience in Data Engineering or Software Engineering.
• 3+ years as a formal technical Lead managing an agile team and implementing E2E solutions.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Strong communication skills with the ability to explain complex technical concepts to non-technical stakeholders.
• Strong problem-solving skills and attention to detail.

Nice To Haves

• Workflow Orchestration: Experience with Apache Airflow for managing complex task dependencies.
• Containerization: Familiarity with Azure Kubernetes Service (AKS) for deploying containerized data services.
• Infrastructure as Code (IaC): Knowledge of Terraform or Bicep for managing Azure resources.

Responsibilities

• Design and supervise the implementation of comprehensive data pipelines utilizing Azure Databricks and PySpark.
• Direct a team of data engineers, performing code reviews, offering technical expertise, and cultivating a culture of ongoing learning.
• Develop high-performance schemas in PostgreSQL and refine complex SQL queries for large datasets.
• Establish and apply optimal practices for data ingestion, transformation, and storage (Delta Lake/Lakehouse patterns).
• Collaborate closely with Data Analysts, Architects, and Product Managers to convert business requirements into technical specifications.
• Promote the implementation of CI/CD, unit testing, and automated monitoring to achieve 99.9% data reliability.
• Ensure data quality, governance, and compliance through validation, documentation, and secure practices.
• Continuously improve data systems for enhanced performance, reliability, and scalability.
• Effectively engage within an agile, cross-functional project team.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level