Data Engineer

Chartmetric•San Mateo, CA

1d•$110,000 - $150,000•Hybrid

About The Position

Build and operate data ingestion pipelines using Python and Apache Airflow to collect data from streaming platforms, social networks, and third-party data vendors. Work with datasets covering more than 10 million artists, 100 million tracks, and 15 million playlists, as well as growing datasets for creators, brands, and athletes. Model and serve data in ClickHouse and Snowflake, and help migrate pipelines from Snowflake to S3 and ClickHouse. Build and maintain data shares and deliveries for enterprise customers, including major labels and publishers. Monitor data freshness and correctness through data monitoring and quality checks. Collaborate with senior engineers, product, and data science on new data products, using AI coding tools as a regular part of the workflow. The position is hybrid, with work from the San Mateo or New York office three to four days per week. The Data Engineering team works collaboratively to remove friction from development and enable other teams. It emphasizes automation, measurement, and continuous improvement to improve the engineering experience and ensure data quality for insights.

Requirements

2–4 years of experience building production data pipelines, preferably in high-volume consumer or entertainment industries.
Strong Python and SQL skills.
Hands-on experience with Apache Airflow or a similar orchestrator, a columnar warehouse such as ClickHouse or Snowflake, and AWS.
Experience building and running pipelines that handle millions of records a day, including messy third-party APIs with rate limits, retries, and changing schemas.
Strong communication skills and ability to translate complex technical concepts to cross-functional stakeholders.
Bachelor's degree in Computer Science, Data Engineering, or equivalent practical experience.
Ability to thrive in dynamic startup environments with rapidly evolving requirements and tight delivery timelines.

Nice To Haves

Experience with music industry data, streaming APIs, or social media analytics platforms.

Responsibilities

Build and run Python ingestion pipelines for data from streaming platforms, social networks, and third-party data vendors, orchestrated in Apache Airflow.
Work with data covering 10M+ artists, 100M+ tracks, 15M+ playlists, and growing datasets for creators, brands, and athletes.
Model and serve data in ClickHouse and Snowflake.
Help move pipelines from Snowflake to S3 and ClickHouse.
Build and maintain data shares and deliveries for enterprise customers, including major labels and publishers.
Monitor data freshness and correctness, including data monitoring and quality checks.
Collaborate with senior engineers, product, and data science on new data products.
Use AI coding tools as a normal part of the workflow.