Data Engineer II

SpyCloud•Austin, TX

10d

About The Position

SpyCloud is hiring a Data Engineer II to join its Engineering team and report to the Senior Director of Data Engineering. This hands-on individual contributor role owns complex data engineering initiatives that support the scalability, reliability, and continued evolution of SpyCloud’s data platform. The work includes evolving the platform with technologies such as Databricks and Spark, building scalable data pipelines, and serving customers through purpose-built data stores including RDS, Elasticsearch, and DynamoDB. The Data Engineer II designs and improves scalable data systems, prepares and serves high-quality data, and builds and maintains real-time and batch pipelines using AWS technologies and the Databricks ecosystem. The role also contributes to architectural decisions, evaluates modern tools for data processing, ML/AI workflows, and observability, and works across Product Management, SpyCloud Labs, and Engineering. The position includes on-call support for troubleshooting and remediating pipeline failures outside standard business hours. SpyCloud transforms recaptured darknet data to disrupt cybercrime and protect workforce, consumer, and supplier identities. The company is headquartered in Austin, Texas, and has more than 250 cybersecurity experts. It describes its culture as innovative, collaborative, inclusive, and driven by the mission to make the internet safer by disrupting the criminal underground.

Requirements

At least 5 years of professional experience in data engineering, software engineering, or a closely related discipline.
Experience building and operating Lakehouse, Databricks, or large-scale distributed data architectures in AWS.
Strong understanding of distributed systems as they relate to data storage, processing, and computing.
Experience building and managing Elasticsearch/OpenSearch data stores.
Hands-on experience designing, building, and maintaining ETL and data-processing pipelines.
Experience preparing and managing data for machine learning and LLM workflows.
Strong Python development skills, particularly for data-intensive systems.
Practical experience with DynamoDB and other AWS-native data services.
Experience with infrastructure-as-code tools such as Terraform.
Experience troubleshooting complex data pipeline, performance, and data quality issues in production environments.
Ability to independently own complex technical projects from design and development through deployment and production support.
Ability to evaluate technical trade-offs and contribute effectively to architectural and design decisions.
Strong communication and collaboration skills across technical and non-technical stakeholders.

Nice To Haves

Familiarity with AI/ML tools and technologies such as SageMaker, LangChain, or vector databases.
Understanding of data governance, privacy, and compliance considerations within AI and ML environments.
Experience with observability and monitoring tools for large-scale data platforms.
Experience with Go or C programming languages.
Experience working with cybersecurity or security-related datasets and technologies.
Comfort working in fast-paced environments with evolving product and technical requirements.
Bachelor’s or Master’s degree in Computer Science, Engineering, or a related STEM field.

Responsibilities

Design, build, and improve scalable data systems with a focus on performance and reliability.
Use Databricks and Spark to prepare, transform, and serve high-quality data to customers and downstream systems.
Build, optimize, and maintain real-time and batch data pipelines using AWS technologies, including Lambda, S3, DynamoDB, ECS, EKS, and Terraform, and develop and maintain pipelines within the Databricks ecosystem.
Identify and implement improvements to the cost and performance of data ingestion, transformation, and storage workflows.
Provide on-call support to troubleshoot and remediate data pipeline failures outside standard business hours.
Partner with Product Management, SpyCloud Labs, and Engineering teams to understand requirements and develop technical solutions.
Evaluate and implement modern tooling for data processing, ML/AI data workflows, and observability.
Contribute to architectural and technical design decisions and implement scalable solutions aligned with engineering standards.
Promote engineering best practices through collaboration, code reviews, documentation, and knowledge sharing.
Identify and resolve complex data quality, performance, security, and reliability issues across production systems.
Own complex data engineering initiatives from design and development through deployment and production support.
Contribute to the continued evolution of SpyCloud’s data platform as product, customer, and AI requirements grow.

Benefits

401(k) with employer contribution (U.S.-based employees).
Health, vision, and dental insurance (U.S.-based employees).
Health Savings Account (HSA) available with employer contribution (U.S.-based employees).
Employer-paid life, short-term, and long-term disability insurance (U.S.-based employees).
Generous PTO plan and 16 paid holidays per year (U.S.-based employees).
Retirement Savings Plan with employer contribution (U.K.-based employees).
Employer-provided private health insurance and healthcare cash plan (U.K.-based employees).
Employer-paid life insurance, income replacement, and critical illness protection (U.K.-based employees).
Employee Assistance Program (U.K.-based employees).
25 days annual accrued holiday, 8 paid bank holidays, and a paid company shutdown during winter holidays (U.K.-based employees).
Flexible and remote-friendly work options.
Competitive salary package.