Data Collection Engineer

Two Six Technologies

6h•$160,000 - $200,000•Remote

About The Position

At Two Six Technologies, we build, deploy, and implement innovative products that solve the world’s most complex challenges today. Through unrivaled collaboration and unwavering trust, we push the boundaries of what’s possible to empower our team and support our customers in building a safer global future. Two Six Technologies is seeking a highly skilled Data Collection Engineer to design, scale, and maintain our distributed web scraping and data extraction infrastructure. In this role, you will be responsible for building resilient data pipelines that harvest data from complex web ecosystems, ensuring strict data quality through automated validation, and managing containerized workloads at scale. If you thrive on reverse-engineering web applications, overcoming anti-bot barriers, and orchestrating distributed systems, we want you on our team.

Requirements

7+ years of professional software engineering experience, with a heavy focus on web scraping, data engineering, or distributed systems.
Excellent reverse-engineering skills, with the ability to dissect network traffic, unearth hidden APIs, and bypass complex web barriers.
A strong commitment to data integrity, system monitoring, and building self-healing scraping systems.
Expert-level proficiency in Python.
Deep experience with Scrapy and distributed scraping architectures (e.g., handling distributed queues, broad vs. deep crawling).
Proven experience with browser automation tools (Playwright, Selenium, or Puppeteer).
Mastery of JSON, JSON Schema, and data validation using Pydantic.
Hands-on experience indexing, querying, and optimizing Elasticsearch clusters.
Strong proficiency in managing and scaling applications within Kubernetes environments.
Experience building structured pipeline workflows to handle complex, multi-stage data extraction tasks.
Bachelor’s degree in Computer Science, Engineering

Nice To Haves

Experience leveraging LLMs or Computer Vision for adaptive scraping, parsing unstructured HTML, or bypassing CAPTCHAs (AI in data collection).
Strong hands-on experience with AWS ecosystems (e.g., EKS, EC2, S3, RDS).
Proficiency in SQL for querying, schema design, and storing structured relational data.
Experience with Redis (specifically for caching, deduplication, or as a Scrapy distributed queue back-end).
Familiarity with Apache Kafka for real-time data streaming and decoupled pipeline architectures.
Strong foundation in Docker for local development and containerizing scraping microservices.
Experience with CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins) for automated testing and deployment of crawlers.

Responsibilities

Design and deploy high-performance, distributed web scrapers using Python and Scrapy to extract massive datasets efficiently.
Utilize Browser Scripting tools to navigate, interact with, and extract data from modern, dynamic, and JavaScript-heavy websites.
Deploy, scale, and manage scraping workloads on Kubernetes, ensuring optimal resource allocation and fault tolerance.
Define strict JSON Schemas and leverage Pydantic to enforce data types, validate incoming payloads, and catch data drift early.
Build and optimize search and storage pipelines using Elasticsearch, transforming raw web dumps into highly structured, searchable data.
Architect robust pipeline workflows to manage the end-to-end data lifecycle—from discovery and extraction to validation and storage.
Manage complex proxy rotation, session handling, and browser fingerprinting to maintain high success rates against advanced anti-scraping systems.