Data Engineer

Waste Connections

14d

About The Position

We are seeking a highly skilled and experienced Data Engineer to join our data engineering team. The ideal candidate will have deep expertise in building scalable data pipelines, optimizing big data workflows, and integrating Databricks with AWS services. You will play a key role in designing and implementing cloud-native data solutions that drive business insights and innovation. Key Responsibilities: Proficient in Databricks SQL for executing complex ad-hoc queries on large-scale lake house datasets, enabling rapid data exploration, trend identification, and actionable insights for business decision-making. Skilled in leveraging Databricks notebooks to design and implement scalable data transformation workflows, integrating PySpark and SQL to cleanse, enrich, and prepare large datasets for downstream analytics and reporting. Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Architect and implement ETL/ELT workflows leveraging AWS services such as S3, Glue, Lambda, Redshift, and EMR. Optimize Spark jobs for performance and cost-efficiency in a cloud environment. Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust solutions. Implement CI/CD pipelines for Databricks notebooks and jobs using tools like GitHub Actions, Azure DevOps, or Jenkins. Ensure data quality, security, and governance using tools like Unity Catalog, Delta Lake, and AWS Lake Formation. Monitor and troubleshoot production data pipelines and jobs. Mentor junior engineers and contribute to best practices and standards.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in software/data engineering with at least 2 years working with Databricks and Apache Spark.
• Strong proficiency in Python, SQL, and PySpark.
• Deep understanding of AWS cloud architecture and services (especially S3, Glue, Lambda, IAM, Redshift, and CloudWatch).
• Experience with Delta Lake, Databricks Workflows, and Databricks SQL.
• Familiarity with data modeling, data warehousing, and data lakehouse architectures.
• Experience with infrastructure-as-code tools like Terraform or CloudFormation.
• Strong problem-solving skills and ability to work in a fast-paced, agile environment.

Nice To Haves

• Databricks Certified Data Engineer or AWS Certified Solutions Architect.
• Experience with streaming data using Kafka, Kinesis, or Structured Streaming.
• Knowledge of MLflow, feature stores, or MLOps practices.
• Familiarity with data governance and compliance frameworks (e.g., GDPR, HIPAA).

Responsibilities

• Proficient in Databricks SQL for executing complex ad-hoc queries on large-scale lake house datasets, enabling rapid data exploration, trend identification, and actionable insights for business decision-making.
• Skilled in leveraging Databricks notebooks to design and implement scalable data transformation workflows, integrating PySpark and SQL to cleanse, enrich, and prepare large datasets for downstream analytics and reporting.
• Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks.
• Architect and implement ETL/ELT workflows leveraging AWS services such as S3, Glue, Lambda, Redshift, and EMR.
• Optimize Spark jobs for performance and cost-efficiency in a cloud environment.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
• Implement CI/CD pipelines for Databricks notebooks and jobs using tools like GitHub Actions, Azure DevOps, or Jenkins.
• Ensure data quality, security, and governance using tools like Unity Catalog, Delta Lake, and AWS Lake Formation.
• Monitor and troubleshoot production data pipelines and jobs.
• Mentor junior engineers and contribute to best practices and standards.

Benefits

• Competitive compensation package includes base salary plus annual cash bonus and equity grant (restricted share units) eligibility.
• We also offer excellent family benefits including: medical, dental, vision, flexible spending account, long term and short term disability, life insurance, 401(k) retirement and unlimited opportunities to "Connect with Your Future."

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 5,001-10,000 employees