Lead Software Engineer, Full Stack
About The Position
Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. About the Team: We are seeking a Lead Software Engineer to own the lifecycle of a massive, high-velocity data ecosystem. We currently manage a large Cloudera-based stack (HBase, Kafka, NiFi) hosted on AWS EC2 and over the next 24 months, we are migrating this entire infrastructure to a modern, AWS-native serverless/managed architecture stack.
Requirements
- Bachelor’s Degree
- At least 4 years of experience in software engineering (Internship experience does not apply)
- At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)
Nice To Haves
- Master's Degree
- 5+ years of experience with the Hadoop ecosystem, specifically HBase (RegionServer tuning, schema design) and NiFi.
- 3+ years of experience architecting and deploying AWS DocumentDB or DynamoDB at a multi-billion record scale.
- 3+ years of experience with Apache Kafka (both self-managed and AWS MSK) and AWS Lambda.
- 3+ years of experience maintaining live production systems.
- 1+ years of experience leveraging interactive AI tooling to accelerate productivity, utilizing capabilities beyond basic code completion
- AWS Certified Data Engineer (Associate)
- AWS Certified Solutions Architect (Professional)
- AWS Certified CloudOps Engineer (Associate) or SysOps Administrator.
Responsibilities
- Maintain & Optimize: Ensure the current Cloudera/HBase/Kafka production environment remains performant and stable during the transition.
- Lead the Migration: Design and execute the phased migration of billions of records to AWS Stack with zero data loss and minimal downtime.
- Enable Application Teams: Work directly with internal application developers to refactor data consumers, ensuring a smooth cutover from HBase APIs to new AWS-native access patterns.
- Enforce Compliance: Implement and maintain strict data governance and security protocols (encryption, auditing, and lineage) to ensure the system remains compliant throughout the migration.
- Modernize Ingestion: Transition legacy NiFi and Kafka workflows into streamlined, cloud-native pipelines using AWS MSK, Lambda, and Glue.
Benefits
- comprehensive, competitive, and inclusive set of health, financial and other benefits that support your total well-being
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
