Data Engineer III (US)

TD Bank•Mount Laurel, NJ

About The Position

The Data Engineer III provides a broad range of data engineering functions, including data modeling, data quality, data profiling, data acquisition and ingestion, ETL, metadata enrichment and management, data provenance and lineage, and other specialized data management functions. The role performs highly complex technical tasks independently and applies expert knowledge of data engineering frameworks, technologies, tools, processes, patterns, and procedures, including their integration with areas such as architecture and infrastructure. The engineer designs and develops solutions to business and technical requirements; analyzes, adapts, integrates, codes, tests, debugs, and executes; and uses or develops patterns to solve complex problems. The position is recognized as a subject matter expert and consults with peers and partners across teams on research, analysis, design, and support. The role analyzes data and data-management requirements for a platform or journey; supports data acquisition and ingestion; develops complex data models and ETL jobs; and ensures data complies with enterprise standards, policies, and guidelines. It works with business partners, engineering teams, QA, architecture, and other stakeholders to resolve data issues, support testing and deployments, and lead data engineering and governance initiatives. The role also documents technical decisions and risks, supports release and change management, and mentors team members while promoting engineering best practices.

Requirements

Degree, postgraduate degree, technical certificate in Data Management or a related discipline such as Computer Science or Engineering, or equivalent practical experience.
At least 5 years of relevant experience.
At least 5 years of experience with Azure Databricks, including designing, developing, optimizing, and supporting notebooks, transformations, and scalable data-processing workloads in enterprise environments.
At least 5 years of experience with PySpark, including distributed data transformations, large-scale data processing, performance optimization, and troubleshooting.
At least 5 years of experience with SQL, including complex queries, transformations, data-validation logic, and performance-intensive workloads.
At least 5 years of experience with Azure Data Factory, including designing, developing, scheduling, monitoring, and troubleshooting enterprise ingestion and orchestration pipelines.
At least 5 years of experience with big data, ETL, and data engineering, including integration patterns, batch and streaming processing, pipeline design, performance, scalability, reliability, and production support.
At least 5 years of experience using GitHub or Bitbucket for source control, branching, pull requests, code reviews, versioning, and collaborative engineering.
At least 5 years of experience applying analytical and problem-solving skills to investigate complex data and application issues, perform root-cause analysis, evaluate solutions, and implement sustainable fixes.
At least 3 years of experience with Delta Lake, including lakehouse solutions, table design, schema management, data versioning, performance optimization, and scalable data operations.
At least 3 years of experience with Unity Catalog, including centralized governance, access controls, metadata management, lineage, and data discovery across Databricks environments.
At least 3 years of experience with CI/CD and DevOps practices for automated build, test, deployment, and promotion of data pipelines, notebooks, and related components.
At least 3 years of experience with data governance and lineage, including metadata management, data catalogs, lineage, data-quality controls, critical data elements, and enterprise governance standards.
At least 3 years of experience delivering cloud migration and modernization initiatives for legacy data platforms, pipelines, and workloads.
At least 2 years of technical leadership experience in a Lead Engineer, Technical Lead, Senior Engineer, or equivalent senior technical role, providing technical direction and guidance.
Experience leading solution design, technical implementation, code reviews, engineering standards, troubleshooting, and production-readiness activities.
Ability to mentor engineers, promote engineering best practices, coordinate across technical and business teams, and own complex deliverables from design through production.

Nice To Haves

At least 2 years of Data Product development experience, including designing, building, and supporting reusable, governed, production-grade Data Products for analytics, reporting, applications, or downstream consumers.
Experience defining and implementing data contracts, quality expectations, metadata, lineage, SLAs/SLOs, and ownership models for Data Products.
Experience building Data Products with modern cloud and lakehouse technologies such as Azure Databricks, Delta Lake, Unity Catalog, PySpark, and Azure Data Factory.
Graduate degree.

Responsibilities

Analyze data and assess data-management requirements for a platform or journey, including complex analysis involving multiple pods or products.
Maintain expert knowledge of upstream data through profiling, data-quality reporting, and metadata production.
Support data acquisition and ingestion.
Explain complex, large-scale, and high-impact technical design and development details to non-technical business partners.
Elicit, analyze, and understand business and data requirements to develop complete solutions, including data models, ETL and business rules, data-lifecycle management, governance, lineage, and metadata.
Ensure data complies with enterprise data standards, policies, and guidelines.
Develop and maintain complex data models using industry-standard modeling tools.
Develop and maintain complex ETL jobs and frameworks using the bank’s standard tools.
Support development and testing teams in resolving data issues, including escalated complex issues.
Support partners and stakeholders in interpreting and analyzing data.
Coordinate with technology work teams, including ITS, ARE, Architecture, and Enterprise Protect, to support successful delivery.
Support QA data analysis and investigations for complex issues and test cases during SIT, UAT, and PAT testing.
Provide oversight of post-implementation activities during the warranty period.
Execute and approve code check-in and check-out in the source-code repository.
Work with ITS and ARE teams on code packaging and deployment into higher environments.
Lead or participate in design and architecture reviews for the application.
Raise ServiceNow requests and work with change management to support release-management activities.
Lead data engineering initiatives and capabilities, including applying data-governance principles across the organization.
Ensure metadata and data lineage are captured and compatible with enterprise metadata and data-management tools and processes.
Follow and contribute to secure coding practices to help prevent common coding vulnerabilities.
Identify and document technical decisions, risks, and lessons learned, and implement enhancements accordingly.
Identify and manage organizational risks and escalate non-standard, high-risk activities as necessary.
Follow internal policies, procedures, and applicable regulatory guidelines; keep current with emerging trends and developments.
Share knowledge and engineering best practices, mentor and guide team members, and support their development.
Keep stakeholders informed of project status, progress, and relevant day-to-day information.
Build productive relationships with internal and external business partners and stakeholders.
Contribute to a fair, positive, equitable, and diverse work environment and represent the business area and bank as a brand ambassador.