Big Data Engineer

TP-Link Systems Inc.

25d • $100,000 - $120,000

About The Position

TP-Link Systems Inc. is hiring a Big Data Engineer to develop and operate production data pipelines and datasets for analysts, dashboards, and business stakeholders. The role owns ETL pipelines running on Spark on EMR and orchestrated with Airflow, including monitoring, alerting, and recovery. It involves building data models with fact and dimension tables according to team layering conventions, ensuring data quality, and improving job efficiency and cost. The engineer will also use and extend shared patterns and templates, automate or reuse recurring work, and collaborate with analysts and business stakeholders to deliver useful datasets.

Requirements

• 2–4 years of hands-on data development in a production environment, including responsibility for scheduled pipelines with downstream consumers and handling failures.
• Strong SQL skills, including window functions, complex multi-table joins, incremental loads, and diagnosing slow queries.
• Python for production ETL and tooling, including PySpark, pandas, and boto3; experience with scheduled code rather than only notebooks.
• Hands-on experience with Spark on a cloud platform such as AWS EMR, Databricks, Glue, or equivalent.
• Production experience with a scheduler; Airflow is preferred, including DAG design, dependencies, retries, and safe reruns.
• Working understanding of dimensional modeling, including fact and dimension tables, star schema, and warehouse layering.
• Comfort working with AWS, including S3 with Parquet or ORC, Linux, and Git, and willingness to learn new tools as the platform evolves.
• Effective use of AI to solve data problems, with judgment to identify incorrect outputs.
• Bachelor's degree in Computer Science, Information Systems, or a related field, or equivalent practical experience.

Nice To Haves

• Experience with data ingestion or CDC tools such as DataX, Sqoop, Debezium, or Fivetran.
• Experience with an OLAP or MPP engine such as StarRocks, Doris, ClickHouse, or Redshift.
• Spark performance experience, including partitioning, shuffle, skew, and memory tuning.
• AWS cost optimization experience, including EMR instance sizing, Spot strategy, and S3 lifecycle policies.
• Experience with lakehouse formats such as Iceberg, Hudi, or Delta; streaming tools such as Kafka or Flink; dbt; or a data quality framework.
• Experience with QuickSight or another BI tool, including dataset and permission design.
• Self-directed learning demonstrated through a project, open-source contribution, or a tool independently learned and put to practical use.

Responsibilities

• Build, run, and own ETL pipelines on EMR (Spark) orchestrated in Airflow, including monitoring, alerting, and recovery.
• Write SQL and Python for data ingestion, transformation, and datasets used by analysts and dashboards.
• Run data quality checks before shipping work, add checks where needed, and confirm results are correct.
• Build and extend fact and dimension data models according to the team's layering conventions.
• Monitor job runtime and cost, and raise slow or expensive jobs for improvement.
• Use and extend shared patterns and templates, and automate or make recurring work reusable.
• Work with analysts and business stakeholders to turn requests into datasets that are used.

Benefits

• Fully paid medical, dental, and vision insurance; partial coverage for dependents.
• Contributions to 401k funds.
• Over four weeks of PTO per year.
• Bi-annual pay increases.
• Health and wellness benefits, including a free gym membership.
• Quarterly team-building events.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level