Data Engineer II

Magnite•Boston, MA

13d•$120,000 - $135,000•Remote

About The Position

Data Engineer II US - East Coast (Remote) At Magnite, we cultivate an environment of continuous growth and collaboration. Our work impacts what millions of people read, watch, and buy, and we’re looking for people to help us tackle that responsibility with creativity and focus. Magnite (NASDAQ: MGNI) is the world’s largest independent sell-side advertising platform. Publishers use our technology to monetize their content across all screens and formats including CTV / streaming, online video, display, and audio. Our tech fuels billions of transactions per day! About this team: You will join the Innovative Services Group (part of the wider Audience team). Innovative services build solutions across the Audience group (and beyond) that are leveraged by other teams. Examples include probabilistic services able to handle over 500k requests a second, and optimising how compute is deployed in our data pipelines yielding very high cost savings across the teams. The wider Audience team is responsible for managing and delivering the first and third-party data that powers all of the platforms that Magnite operates. We’re a globally distributed team of software and data engineers who build and operate huge scale data pipelines and real-time scaled services to enable Magnite and Magnite’s customers to reach their goals.

Requirements

More than anything we expect you to love learning and see problems as things to overcome rather than blockers. You are part of a team that moves fast, makes incremental improvements towards a vision and solves problems other teams are struggling with. You should look at this list and our tech stack and think “wow i want to learn that”, rather than being an expert. We expect you to have expertise with 1-2 items and familiarity with a couple more.
Most candidates will have 2+ years of experience.
Hands-on experience with Apache Spark (mainly PySpark). Ideally, you will have written a variety of pipelines that you have deployed in production.
Strong Python skills, particularly in the context of Databricks and Spark.
Data pipeline design knowledge, from sourcing event data to creating queryable data layers.
Familiarity with event-based data architecture (e.g., Kafka).
SQL experience for data analysis and transformations.
Hands-on experience with AWS and Terraform for IaaS solutions.
Experience writing Jenkins or Github actions pipelines to run build and integration actions.
GoLang for distributed service development
Kubernetes experience

Responsibilities

Build and optimize data pipelines using Apache Spark and Python, processing billions of events in both streaming and batch modes. We work at TB scale.
Collaborate with senior data engineers to improve Magnite’s Audience data pipelines by identifying and implementing innovative solutions.
Own the operational performance of your builds, ensuring uptime and implementing recovery strategies.
Collaborate with SRE teams to provision infrastructure using Terraform and AWS (future Kubernetes on-premise).
Proactively identify and help solve impactful problems.