Senior Data Engineer
About The Position
The core mission of the Senior Data Engineer is to own and evolve data pipelines on GCP, building new ones, hardening existing ones, improving data quality, and making clean, trustworthy data available across the organization. This role involves end-to-end work on streaming and batch pipelines, from CDC and event ingestion through transformation, serving, and the feature layer for ML and AI products. Daily tasks include designing ELT/ETL processes on BigQuery and ClickHouse, building real-time pipelines on Pub/Sub and Kafka with Dataflow (and potentially Flink/Spark), orchestrating workflows with Airflow, and ensuring data is cleaned, modeled, and served for analytics, ML training, and online inference. The role also involves partnering with ML engineers on feature pipelines, monitoring data drift, keeping models fed and retrained, consuming and building REST APIs, integrating with third-party SaaS sources, and treating infrastructure as code.
Requirements
- Strong data modelling and warehouse architecture skills (dimensional modelling, event-driven, lakehouse patterns)
- Hands-on experience with GCP data services — BigQuery is a must; Pub/Sub, Dataflow, Bigtable, Cloud Composer are strong pluses
- Production experience with streaming pipelines on Dataflow/Beam, Flink, or Spark Structured Streaming, ingesting from Kafka and/or Pub/Sub
- Solid SQL and strong Python — you write production-quality code, not just notebooks
- Experience with ClickHouse or another columnar OLAP engine in production
- Workflow orchestration experience with Airflow (or Prefect/Dagster)
- Comfortable with dbt or equivalent transformation frameworks
- Docker & Kubernetes experience
- CI/CD mindset, infrastructure-as-code sensibility, and a bias for simple, observable systems
Nice To Haves
- Experience migrating off legacy warehouses (Snowflake, Redshift, Synapse) onto cloud-native stacks is a plus
- Working knowledge of ML in production — feature engineering, feature stores, model deployment, drift monitoring, retraining
- CDC tooling (Datastream, Debezium)
- Vertex AI / Feature Store
Responsibilities
- Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery, Bigtable, and ClickHouse
- Help drive the migration off Snowflake onto our GCP-native stack — and retire shadow pipelines along the way
- Own the orchestration layer in Airflow, including SLAs, retries, and data quality gates
- Model data for analytics and for ML — including feature pipelines that serve both training and low-latency online inference
- Partner with ML engineers on feature stores, drift monitoring, and retraining workflows
- Capture requirements from stakeholders and translate them into pragmatic, well-scoped data products
- Continuously improve data quality, reliability, observability, and cost efficiency
- Identify new data sources worth acquiring and integrate them cleanly
Benefits
- Medical, Dental & Vision (employee coverage 100% paid for by Hack The Box)
- 401K w/ employer match
- Employer-paid Life and AD&D Insurance
- Supplemental Life Insurance
- Short-term and Long-term Disability
- Healthcare and Dependent Care FSA
- Paid parental leave
- 25 annual leave days
- Home Office Allowance
- Dedicated budget for training and professional development, participation in conferences
- State-of-the-art equipment
- Full access to the Hack The Box lab offerings
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
