Data Engineer

APN Consulting, Inc.

3d • Remote

About The Position

APN Consulting, Inc. is a progressive IT staffing and services company offering innovative business solutions to improve client business outcomes. We focus on high impact technology solutions in ServiceNow, Fullstack, Cloud & Data, and AI / ML. Due to our globally expanding service offerings we are seeking top-talent to join our teams and grow with us. This is a direct client requirement for a Data Engineer position, which is a full-time, remote role.

Requirements

• 6+ years in data engineering, ETL/ELT, analytics engineering, data warehouse development, or big data pipelines.
• Minimum 4 years on GCP cloud data platforms preferred.
• Strong SQL and Python.
• Data modelling, warehousing concepts, partitions, clustering, indexing, optimisation.
• Cloud storage and data lake concepts.
• Ability to explain scale scenarios, not just tool definitions.
• Python and advanced SQL.
• Data quality checks, reconciliation, alerting, retries, and backfills.
• Mandatory GCP Skills: BigQuery: datasets, tables, views, partitioning, clustering, SQL optimisation, cost awareness.
• Option 1: Dataflow / Apache Beam or equivalent streaming/batch framework.
• Option 2: Dataproc / PySpark or strong Spark experience.
• Pub/Sub or equivalent event streaming.
• Cloud Composer / Airflow DAG development.

Nice To Haves

• Dataform, dbt, CI/CD for data.
• BigQuery ML.
• Dataflow templates.
• Looker / LookML basics.
• Terraform for data infrastructure.

Responsibilities

• Design and production support for ETL/ELT pipelines.
• Batch and streaming pipeline fundamentals.
• Orchestration using Airflow, Composer, Control-M, ADF, dbt, or equivalent.
• Big data processing using Spark, PySpark, Beam, Dataflow, Dataproc, Databricks, EMR, or equivalent.
• Data quality, reconciliation, schema validation, lineage, monitoring, and alerting.
• CI/CD for data pipelines and code versioning.
• Production pipeline monitoring and troubleshooting.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed