Copy of Senior Software Engineer, Data Acquisition

People Data Labs

4d•$160,000 - $200,000•Remote

About The Position

People Data Labs is hiring a Senior Software Engineer for its Data Engineering & Acquisition Team. The team ensures customers have standardized, high-quality data and supports the development of standalone data products for data teams and independent developers. The role contributes to improving existing datasets and pursuing new ones, with autonomy and opportunities to make direct contributions. The engineer will help architect and improve the data acquisition and processing platform; develop and use web crawling technologies; and build and operate large-scale distributed systems that collect, process, and deliver data from the web. The role also involves backend services for distributed job orchestration, data pipelines, and asynchronous workloads, as well as structuring and modeling captured data to maintain dataset quality and consistency. The engineer will collaborate with data product and engineering teams to develop new products and improve existing ones.

Requirements

7+ years of professional experience building or operating backend or infrastructure systems at scale.
Solid programming experience in Python, Go, Rust, or similar, including async/await, coroutines, or concurrency frameworks.
Strong understanding of software architecture and backend fundamentals, including concurrency, scalability, and fault tolerance.
Understanding of the browser rendering pipeline and web application architecture, including authentication, cookies, and HTTP request/response.
Familiarity with network architecture and debugging, including HTTP, DNS, proxies, packet capture, and analysis.
Understanding of distributed systems concepts, including parallelism, asynchronous programming, backpressure, and message-driven design.
Experience designing or maintaining resilient data ingestion, API integration, or ETL systems.
Proficiency with Linux/Unix command-line tools and system resource management.
Familiarity with message queues, orchestration, and distributed task systems such as Kafka, SQS, or Airflow.
Experience evaluating and monitoring data quality to ensure consistency, completeness, and reliability across releases.
Ability to work independently in a fast-paced, remote-first environment, proactively unblocking oneself and collaborating asynchronously.
Clear, thoughtful written communication, including in Slack, documentation, and design proposals.
Ability to write and maintain technical design documents, including pipeline designs, schema designs, and data flow diagrams.
Ability to scope complex projects into deliverable milestones and communicate progress, risks, and blockers effectively.
Ability to balance pragmatism with craftsmanship and deliver reliable systems while continuously improving them.

Nice To Haves

Degree in a quantitative field such as computer science, mathematics, or engineering.
Experience as a Red Teamer.
Experience working on large-scale data ingestion, crawling, or indexing systems.
Experience with Apache Spark, Databricks, or other distributed data platforms.
Experience with streaming data systems such as Kafka, Pub/Sub, or Spark Streaming.
Proficiency with SQL and data warehousing, such as Snowflake, Redshift, or BigQuery.
Experience with cloud platforms; AWS is preferred, and GCP or Azure are also relevant.
Understanding of modern data storage and design patterns, including Parquet, Delta Lake, partitioning, and incremental updates.
Knowledge of data design and storage patterns such as incremental updating, partitioning and segmentation, rebuilds, and backfills.
Experience building and maintaining data pipelines on modern big-data or cloud platforms such as Databricks or Spark.

Responsibilities

Contribute to the architecture and improvement of the data acquisition and processing platform, increasing reliability, throughput, and observability.
Use and develop web crawling technologies to capture and catalog data on the internet.
Build, operate, and evolve large-scale distributed systems that collect, process, and deliver data from across the web.
Design and develop backend services for distributed job orchestration, data pipelines, and large-scale asynchronous workloads.
Structure and model captured data to ensure quality and consistency across datasets.
Improve the speed, scalability, and fault tolerance of ingestion systems.
Partner with data product and engineering teams to design and implement new data products and enhance existing products.
Learn and apply domain-specific knowledge in web crawling and data acquisition.