Big Data Engineer

TMX Group•Toronto, ON

6d•CA$100,000 - CA$120,000•Hybrid

About The Position

TMX is seeking a seasoned and motivated Big Data Engineer to join their Enterprise Data and Advanced Analytics Platform team. The ideal candidate is an expert data pipeline builder and data wrangler who thrives on optimizing data systems and architecting them from the ground up. This role requires a self-directed professional, comfortable supporting the diverse data needs of multiple teams, systems, and products across a full spectrum of cutting-edge technologies. This role reports to the Senior Manager, Data & Analytics & Enterprise Data Architect.

Requirements

University degree, College diploma, or equivalent experience in Computer Science, Statistics, or Mathematics.
5-8 years of overall IT experience, with 4+ years specifically focused on Big Data technologies (Spark, Hive, Presto, etc.).
Strong technical proficiency in AWS cloud technologies, specifically S3, EC2, and EMR.
Proven experience working with Snowflake.
Proficiency in Python, Scala, Shell Scripting, or Java.
Solid understanding of data warehousing and data modeling techniques.
Hands-on experience in performance tuning for Spark jobs and Hive tables.
Experience working within Agile development methodologies.
Meticulous attention to detail, excellent analytical problem-solving skills, and superior interpersonal communication abilities.
Must be authorized to work in the country where the position is located.

Nice To Haves

Good understanding of the Equity Markets business.
Experience with Terraform or Ansible.
Experience with Kafka.
A strategic mindset with the ability to multitask in a fast-paced environment.

Responsibilities

Design, construct, and manage robust, large-scale, and efficient data pipeline architectures to meet complex functional and non-functional business requirements.
Drive continuous internal process improvements by automating manual tasks, optimizing data delivery, and re-designing infrastructure for enhanced scalability and performance.
Administer and manage a comprehensive suite of AWS services, including S3, RDS, EC2, EMR, VPC, Security Groups, CloudFormation, CloudWatch, Managed Airflow, and SNS.
Develop optimal infrastructure for Extract, Transform, and Load (ETL) operations from disparate data sources using SQL and AWS ‘big data’ technologies, with a focus on both bulk and streaming data.
Adhere to development best practices, ensuring meticulous code management, branching, and merging within GIT repositories.
Provide on-call and off-business hours support on a rotational basis to ensure system reliability.
Act as an effective team player and mentor to junior staff through technical reviews and brainstorming sessions, while expertly managing stakeholder expectations and inquiries.