Data Engineer GCP AWS & Databricks

iShare Inc

8h•$45 - $50•Remote

About The Position

We are seeking an experienced Data Engineer to design, develop, and maintain scalable cloud-based data pipelines and data platforms. The ideal candidate will have strong hands-on experience with GCP, AWS, Databricks, Apache Spark, PySpark, Python, and SQL. This role requires expertise in building reliable ETL/ELT pipelines, integrating data from multiple sources, and optimizing cloud data solutions for performance, security, scalability, and cost.

Requirements

5+ years of professional data engineering experience.
Strong hands-on experience with both GCP and AWS.
Expertise in Databricks, Apache Spark, and PySpark.
Strong programming skills in Python and SQL.
Proven experience developing ETL/ELT pipelines and cloud data platforms.
Experience with data warehouses, data lakes, and dimensional data modeling.
Experience with orchestration tools such as Apache Airflow or Google Cloud Composer.
Understanding of data security, governance, monitoring, and quality frameworks.
Strong analytical, troubleshooting, and problem-solving skills.
Excellent communication and cross-functional collaboration skills.

Nice To Haves

Experience with GCP services such as BigQuery, Cloud Storage, Dataflow, Dataproc, Pub/Sub, and Cloud Composer.
Experience with AWS services such as S3, Glue, EMR, Redshift, Lambda, and Kinesis.
Familiarity with Delta Lake and Databricks Lakehouse architecture.
Experience with streaming technologies such as Apache Kafka.
Familiarity with CI/CD, Git, Terraform, and cloud infrastructure automation.
Experience working in Agile development environments.

Responsibilities

Design, develop, and maintain scalable ETL/ELT data pipelines.
Build cloud-based data solutions using GCP and AWS services.
Use Databricks, Apache Spark, and PySpark for large-scale data processing.
Integrate structured, semi-structured, and unstructured data from multiple sources.
Develop and optimize batch and real-time data-processing workflows.
Improve pipeline performance, reliability, scalability, and cost efficiency.
Implement data-quality checks, monitoring, security, and governance standards.
Design and support cloud data warehouses and data lakes.
Troubleshoot production issues and perform root-cause analysis.
Collaborate with data architects, analysts, application teams, and business stakeholders.
Create and maintain technical documentation for pipelines, data models, and workflows.