RQ11099 - DataOps/Cloud Data Engineer - Senior

Maarut • Toronto, ON

1d

About The Position

This role involves designing, developing, testing, implementing, and troubleshooting data pipelines and complex data transformation procedures. The engineer will also be responsible for creating data models for data warehouses and lakehouses.

Requirements

• Data engineering (pandas, pyspark)
• Data lakehouse solutions such as Databricks (Delta Lake)
• Working with XLSX, CSV, JSON files, relational databases, cloud storage, structured and unstructured data
• Experience using AWS Services such as Glue, StepFunctions, Lambda, S3
• Extract/Transform/Load data using tools such as Informatica IDMC
• Programming & scripting: Python, SQL, Linux shell, PowerShell
• Data manipulation/analysis using pandas and pyspark
• Working with XLSX, CSV, JSON files, relational databases, cloud storage, structured and unstructured data
• AWS and/or Azure services
• Cloud data warehouse solutions such as AWS Redshift
• Data lakehouse solutions such as Databricks (Delta Lake)
• Data processing orchestration/automation
• ETL tools (such as Azure Data Factory, AWS Glue) as well as cloud agnostic tools such as Informatica IDMC
• Data modeling & architecture: relational and dimensional modeling
• Data reporting/visualization
• Full SDLC from requirements gathering, design, implementation, testing to deployment and production support
• Project management (agile/scrum and waterfall)
• Change and Incident management
• Communication, presentation and negotiation skills
• Consulting, problem-solving and decision-making skills
• Experience working in OPS and/or public sector in general

Responsibilities

• Design, develop, test, implement, and troubleshoot data pipelines using various tools such as Databricks, Azure Data Factory, AWS Glue
• Design, develop, test, implement, and troubleshoot complex data transformation procedures
• Design, develop, test, implement, and troubleshoot data models for data warehouses and lakehouses

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed