Software Engineer, Big Data, Tech Enablement

ZipRecruiter • Santa Monica, CA

2d • Hybrid

About The Position

ZipRecruiter’s Tech Enablement team builds the scalable, reliable, and efficient infrastructure, tools, and processes that help engineering teams deliver products and services. The team provides foundational technology for data infrastructure and systems, including tools for data processing, dataset ingestion and discoverability, pipeline orchestration, documentation, analysis, ML model pipelines, and experimentation. It also enables AI adoption through an agentic data exploration and analysis tool and an LLM Gateway for customer-facing products.

In this role, you will design, build, deploy, and maintain data pipelines and systems that make web events available in the data lake and event streams for downstream applications. You will work with large, evolving datasets using batch and streaming approaches, develop frameworks that accelerate ML development, improve data documentation, and contribute to Sift, an agentic data exploration and analysis tool. The position is offered in a hybrid work environment, with most US-based positions also eligible for remote work.

Requirements

• At least 2 years of professional software development experience, including building and operating data pipelines that feed into a customer-facing product.
• Hands-on Apache Spark experience writing, tuning, and debugging batch and/or streaming jobs at scale.
• Experience deploying and operating containerized services and batch jobs.
• Solid computer science fundamentals, including data structures, algorithms, distributed systems, and testing.
• Professional software development experience building and operating data pipelines and infrastructure that feed into a customer-facing product.

Nice To Haves

• At least 4 years of professional software development experience focused on data-intensive systems.
• Working knowledge of AWS services such as S3, IAM, EC2, and MSK, or equivalents, and experience instrumenting jobs with metrics, logs, and alerts.
• Experience with a data lake stack, including Parquet on object storage, a data catalog such as AWS Glue, and an open table format such as Apache Iceberg or Delta Lake.
• Streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.
• Experience with a workflow orchestrator such as Apache Airflow, including authoring DAGs, backfills, dependency management, and on-call for scheduled pipelines.
• Experience with open table format internals, including schema evolution, partitioning, and compaction.
• Experience deploying services on Kubernetes with Docker.
• Experience building and maintaining a data platform that powers a user-facing product, including infrastructure and tools for streaming and batch ingestion, ETL pipelines, data permissions management, and data discovery.
• BS, MS, or PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.

Responsibilities

• Design, build, deploy, and maintain data pipelines and systems that make web events available in the data lake and through event streams for downstream applications.
• Process large, evolving datasets using batch and streaming techniques, balancing speed, efficiency, and reliability.
• Design, build, deploy, and maintain frameworks and systems that accelerate machine-learning development and improve its velocity, quality, observability, and reliability.
• Design, build, deploy, and maintain frameworks and systems that improve the comprehensiveness and quality of data documentation and make documentation easier to add, maintain, and discover.
• Design, build, deploy, and maintain Sift, an agentic data exploration and analysis tool that supports data access, collaboration, and data-driven decision-making.

Benefits

• Competitive compensation
• Exceptional benefits package
• Flexible Vacation & Paid Time Off
• Employer-matched 401(k) plan
• Medical, financial, and other benefits
• Equity, bonuses, commission, or other compensation may be offered depending on the position.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed