Data Platform Engineer

Scotiabank•Dallas, TX

About The Position

The Platform Engineer will work under the guidance of Sr Engineers/Platform Leads to build, tune, manage infrastructure, DevOps, monitor troubleshoot, enhance, enable new features & provide Ops Support on Data & AI platform(s) as per banks Data & AI strategy. This consists of working with cross functional teams like IAM, Network, Cloud Ops, Security, Client partners etc for integration, process automation, platform enhancement and delivery of new projects.

Requirements

5+ years of IT experience in big organizations operating in various geographies/regulations.
3–5 years of hands on cloud engineering experience (Azure preferred).
Experience with Azure networking, IAM/RBAC, Key Vault, Storage Accounts, and related services.
Practical experience with Databricks (clusters, jobs, Unity Catalog basics, notebooks).
Solid understanding of Terraform, IaC workflows, and Git based development.
Experience building pipelines with Azure DevOps or GitHub Actions.
Good skills in scripting languages (Python, Bash, PowerShell).
Hands on experience with containers (Docker) and basic Kubernetes concepts.
Familiarity with monitoring tools (Azure Monitor, Log Analytics).
Ability to troubleshoot platform issues and collaborate with cross functional teams.
Strong communication skills, proactive attitude, and willingness to learn.

Nice To Haves

Degree in Computer Science, Engineering, or related field is preferred.

Responsibilities

Guidance and Collaboration: Work closely with senior engineers and platform leads to understand goals, deliver assigned tasks, and support team objectives. Contribute to platform improvements and provide regular updates on progress.
Technical Execution: Support the design, deployment, and maintenance of Azure and Databricks platform components. Work with Azure networking/security configurations (VNets, NSGs, route tables, private endpoints, RBAC).Assist in implementing and maintaining Databricks resources including clusters, SQL warehouses, and Unity Catalog objects.
Quality Assurance: Ensure high quality delivery by following established coding standards, Terraform best practices, and platform validation procedures. Participate in code reviews, automated testing, and CI/CD validation steps.
Process Improvements: Contribute to automation and operational efficiency by developing reusable Terraform modules and standardizing configurations. Build and maintain IaC modules for Azure native resources and Databricks. Participate in creating SOPs, technical documentation, and onboarding guides.
Monitoring and Troubleshooting: Assist in diagnosing platform issues, analyzing logs/metrics, and supporting stability efforts. Help maintain observability using Azure Monitor and Log Analytics Monitor Databricks jobs, clusters, and infrastructure health.
Platform Reliability Support: Participate in troubleshooting sessions and contribute to meeting platform SLOs. Support incident response efforts under the guidance of senior engineers.
Incident Management Support: Assist during incidents by gathering diagnostic data, executing remediation steps, and communicating updates to internal teams as directed by senior or on call leads.
Observability Tooling: Help develop monitoring dashboards and alert rules for Databricks and Azure resources. Contribute to log ingestion pipelines and basic automation scripts.
Cloud Managed Application Development: Build and support Azure resource deployments (Storage Accounts, Key Vault, EventHub, Azure SQL) and Databricks assets (jobs, workflows, notebooks) using Terraform and Databricks Asset Bundles.
Release Management Support: Participate in maintaining CI/CD pipelines using GitHub Actions and Azure DevOps. Implement Terraform plan/apply pipelines. Help automate Databricks Bundle deployments and quality checks.
Client Pipeline Support: Support the implementation of CI/CD workflows for infrastructure and data workloads. Maintain Docker images, build pipelines, artifact repositories, and deployment templates.
Operational Support: Assist in provide operational support, patching, remediation risk/audit items, platform hardening & troubleshoot tenants tickets, which include working after hours on rotational basis and if required on weekends.