[Expression of Interest] Data Engineer

Experian

7h•Hybrid

About The Position

Join Experian’s Ascend Sandbox team, which develops a cloud-based analytical platform that enables clients and data scientists to explore, model, and derive insights from data in a secure, governed environment. The engineer will help build, maintain, and scale the platform for internal teams and external clients across the APAC region. The role covers the design, development, testing, deployment, and operation of scalable batch and, where applicable, streaming data pipelines. It involves building Python, PySpark, and SQL solutions; creating analytical data models and reusable datasets; and engineering AWS-based data lake and lakehouse solutions. The engineer will also implement orchestration, data quality controls, monitoring, governance, security, and operational practices, while optimizing distributed processing for performance, scalability, and cost. The position works with clients, business users, data scientists, and technical teams to turn requirements into data-processing solutions. It includes investigating production issues, conducting root-cause analysis, improving reliability, and following established architecture, coding, testing, security, and quality standards. The posting describes the opportunity as a paid internship, while its qualifications seek a degree or advanced diploma and at least six years of relevant experience; seniority is commensurate with experience and qualifications.

Requirements

Degree or Advanced Diploma in Computer Science, Data Engineering, Software Engineering, or equivalent.
At least 6 years of experience in data engineering or developing data-intensive systems, including designing, building, and supporting production data pipelines.
Strong hands-on experience with Python and PySpark for large-scale and distributed data processing.
Strong SQL skills, including complex transformations, joins, window functions, query optimization, and performance tuning.
Hands-on experience designing and operating scalable ETL/ELT pipelines, including incremental loads, retries, backfills, dependency management, and failure recovery.
Experience with analytical data modeling, including dimensional modeling, facts and dimensions, normalization and denormalization, and handling historical data.
Hands-on experience with relevant AWS data services, such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, and Amazon EMR.
Experience with data lake or lakehouse concepts, storage layout, partitioning, schema evolution, and efficient data formats such as Parquet.
Strong understanding of Spark performance engineering, including partitioning, shuffles, joins, data skew, parallelism, memory and resource optimization, and troubleshooting.
Experience with pipeline orchestration and workflow scheduling using Airflow/MWAA, AWS Step Functions, AWS Glue Workflows, or equivalent.
Experience implementing automated data quality checks and data observability, including monitoring pipeline health, data freshness, volumes, failures, and quality anomalies.
Practical knowledge of data governance, metadata management, data cataloguing, lineage, and schema management.
Practical understanding of AWS data security, including IAM and least privilege, encryption/KMS, secrets management, S3 security, and data-access controls.
Understanding of cloud cost and performance optimization for storage, compute, Spark workloads, and query patterns.
Experience with Git-based workflows, branching strategies, code reviews, CI/CD pipelines, and automated deployment.
Experience writing automated unit, integration, and data-pipeline tests for Python/PySpark transformations and data quality.
Strong testing, debugging, troubleshooting, and root-cause analysis skills in production data environments.
Good understanding of Agile, SCRUM, SDLC, and software engineering practices.
Strong communication and stakeholder management skills, including the ability to engage clients, business users, data scientists, and technical teams.
Strong analytical thinking, problem-solving, teamwork, willingness to learn, and ability to work independently when required and deliver under tight deadlines.

Nice To Haves

Exposure to AWS Lambda, Kinesis/MSK, DMS, or Lake Formation, where relevant.
Familiarity with JSON, CSV, and Avro data formats.
Familiarity with Infrastructure as Code tools such as Terraform, AWS CloudFormation, or AWS CDK.
Familiarity with Jira, Bitbucket, Jenkins, and Checkmarx.
Familiarity with AI tools for coding assistance, documentation, or automation.

Responsibilities

Design, build, test, deploy, and operate scalable batch and, where applicable, streaming data pipelines for the Ascend Sandbox platform.
Develop Python, PySpark, and SQL solutions to ingest, transform, and process large-scale datasets accurately, reliably, and efficiently.
Design data models and reusable, trusted datasets or data products for analytics, data science, and downstream applications.
Engineer and optimize AWS-based data solutions using services such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, and Amazon EMR, in line with platform architecture.
Apply data lake and lakehouse engineering practices, including efficient storage formats, partitioning, schema evolution, and storage and query optimization.
Design robust ETL/ELT patterns for incremental processing, dependency management, idempotency, retries, backfills, and failure recovery.
Implement pipeline workflow orchestration and scheduling using approved tooling such as Airflow/MWAA, AWS Step Functions, or AWS Glue Workflows.
Build automated data quality controls and monitoring for completeness, validity, reconciliation, schema compliance, freshness, pipeline health, and processing SLAs/SLOs.
Optimize Spark workloads and distributed processing for performance, scalability, and cost, including partitioning, joins, shuffles, skew, and resource utilization.
Implement data governance, metadata, lineage, cataloguing, security, and access-control requirements across pipelines and datasets.
Collaborate with clients, business users, data scientists, and technical teams to translate requirements into scalable data-processing solutions and integrations.
Investigate and resolve production data and pipeline issues, perform root-cause analysis, and implement preventive improvements to reliability and operational efficiency.
Ensure solutions comply with architecture, coding, testing, security, and quality standards, and contribute to code reviews, CI/CD, and automated deployment practices.