Senior Big Data Engineer

LiveRamp • San Francisco, CA

13h

About The Position

LiveRamp is the data collaboration platform of choice for the world’s most innovative companies. A groundbreaking leader in consumer privacy, data ethics, and foundational identity, LiveRamp is setting the new standard for building a connected customer view with unmatched clarity and context while protecting precious brand and consumer trust. LiveRamp offers complete flexibility to collaborate wherever data lives to support the widest range of data collaboration use cases—within organizations, between brands, and across its premier global network of top-quality partners. Hundreds of global innovators, from iconic consumer brands and tech giants to banks, retailers, and healthcare leaders turn to LiveRamp to build enduring brand and business value by deepening customer engagement and loyalty, activating new partnerships, and maximizing the value of their first-party data while staying on the forefront of rapidly evolving compliance and privacy requirements.

You will: Design, build, and optimize scalable data processing pipelines using Spark, Airflow, and related big data technologies to support both batch and real-time use cases. Lead and collaborate with technical, application, and security stakeholders to deliver reliable, secure Big Data infrastructure leveraging tools and platforms such as Spark, Dataproc, Redpanda, and Temporal. Own and participate in on-call responsibilities for Big Data platforms, triaging and resolving incidents, responding to tickets, and ensuring systems consistently meet defined SLAs for availability, performance, and data quality. Design, develop, and automate large-scale infrastructure on Kubernetes using Terraform and other infrastructure-as-code patterns to support high‑performance data processing, analytics, and AI/ML workloads. Define and implement monitoring, alerting, and runbooks to provide end‑to‑end observability and drive continuous improvement in reliability and operational excellence using tools like Grafana. Onboard, train, and mentor vendor teams and external partners so they can effectively support, operate, and extend the solutions owned by the Big Data Infrastructure (BDI) team. Drive the technical roadmap for emerging data and infrastructure technologies by evaluating options, building proofs of concept (POCs), and authoring solution selection and design documents. Champion engineering best practices (code reviews, testing strategies, CI/CD, security and compliance standards) across the Big Data ecosystem to ensure maintainable, resilient, and cost‑efficient systems.

Your team will: Build and operate the core Big Data Infrastructure (BDI) platform that powers large‑scale data processing, analytics, and AI/ML workloads across LiveRamp using tools like Dataproc, Redpanda, Airflow and Temporal. Provide secure, self-service, and cost‑efficient data environments that enable product and data teams to experiment, ship, and scale data collaboration applications quickly. Partner closely with application, security, SRE, and platform teams to ensure data systems meet LiveRamp’s standards for privacy, compliance, reliability, and performance. Evolve common patterns, tooling, and reference architectures that reduce friction for teams adopting new big data, streaming, and AI/ML capabilities. Continuously improve platform reliability and developer experience by automating operations, simplifying workflows, and incorporating feedback from internal customers.

Requirements

• 7+ years of experience in software engineering or data engineering, with a focus on large-scale data processing and distributed systems.
• Hands-on experience building and operating pipelines with Apache Spark (batch and/or streaming) and at least one orchestration framework such as Airflow or Temporal.
• Practical experience running Big Data workloads on cloud platforms (for example, using Dataproc or similar managed compute services).
• Proficiency with Kubernetes and infrastructure-as-code tools such as Terraform to provision, configure, and manage production services.
• Experience with modern streaming and messaging technologies (for example, Redpanda, Kafka, or similar) and real-time data processing patterns.
• Solid understanding of systems design, scalability, reliability, and observability for data-intensive workloads.
• Experience participating in or leading on-call rotations and incident management processes for production systems.
• Strong collaboration and communication skills, including working with cross-functional partners (security, application owners, vendors) and documenting designs and decisions clearly.
• Ability to evaluate new technologies, build POCs, and make data-informed recommendations that influence team and platform roadmaps.
• Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related technical field, or equivalent practical experience.

Nice To Haves

• Experience working on Big Data Infrastructure, data platform, or core infrastructure teams in a cloud-native environment.
• 3 years of Software Development experience in one of the languages - Python, Go, Java or Scala.
• 3 years of experience with Linux Operating Systems
• 3 years of experience in setting up, managing and optimizing Big Data ecosystems such as Kafka , airflow, Redpanda and temporal.
• 3 years of experience using cloud-based platforms such as AWS, GCP, Azure or similar.
• 3 years of experience with a Version Control System (GIT, Subversion, or similar)
• Experience in using AI tools, NLP and agent creation
• Exposure to security and compliance considerations for data platforms (for example, access control, encryption, data governance, and auditing).
• Prior experience mentoring engineers, leading technical initiatives, or coordinating work with vendors and external partners.

Responsibilities

• Design, build, and optimize scalable data processing pipelines using Spark, Airflow, and related big data technologies to support both batch and real-time use cases.
• Lead and collaborate with technical, application, and security stakeholders to deliver reliable, secure Big Data infrastructure leveraging tools and platforms such as Spark, Dataproc, Redpanda, and Temporal.
• Own and participate in on-call responsibilities for Big Data platforms, triaging and resolving incidents, responding to tickets, and ensuring systems consistently meet defined SLAs for availability, performance, and data quality.
• Design, develop, and automate large-scale infrastructure on Kubernetes using Terraform and other infrastructure-as-code patterns to support high‑performance data processing, analytics, and AI/ML workloads.
• Define and implement monitoring, alerting, and runbooks to provide end‑to‑end observability and drive continuous improvement in reliability and operational excellence using tools like Grafana.
• Onboard, train, and mentor vendor teams and external partners so they can effectively support, operate, and extend the solutions owned by the Big Data Infrastructure (BDI) team.
• Drive the technical roadmap for emerging data and infrastructure technologies by evaluating options, building proofs of concept (POCs), and authoring solution selection and design documents.
• Champion engineering best practices (code reviews, testing strategies, CI/CD, security and compliance standards) across the Big Data ecosystem to ensure maintainable, resilient, and cost‑efficient systems.

Benefits

• Work with talented, collaborative, and friendly people who love what they do.
• We host in-person and virtual events such as game nights, happy hours, camping trips, and sports leagues.
• Flexible paid time off, paid holidays, options for working from home, and paid parental leave.
• Our benefits package offers medical, dental, vision, life and disability, an employee assistance program, voluntary benefits as well as perks programs for your healthy lifestyle, career growth and more.
• Our 401K matching plan—1:1 match up to 6% of salary—helps you plan ahead.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior