Staff Engineer - Kafka Specialist
About The Position
The Venmo Data Infrastructure team supports Venmo's internal and customer-impacting data movement and message streaming infrastructure. We are actively engaged in partnering with development groups, platform, and infrastructure teams — providing deep subject matter expertise on Kafka-based streaming architecture, and acting as the escalation point for analyzing and resolving complex, large-scale streaming and performance issues. This is a senior technical leadership role for an engineer who will serve as the Kafka subject matter expert and technical authority for Venmo's real-time data and message streaming platform. You will act as a project or system lead, setting technical direction for other engineers, driving Kafka architecture and operational excellence at scale, and representing the team's engineering standards both internally and externally. This role goes beyond hands-on Kafka operations — it requires shaping how the broader organization designs, scales, and reasons about streaming infrastructure.
Requirements
- 1+ years relevant experience and a Bachelor’s degree OR Any equivalent combination of education and experience.
- Deep, hands-on expertise operating and scaling Apache Kafka in production at high throughput/high availability, including internals such as partitions, replication, ISR, controller quorum/KRaft, and consumer group rebalancing
- Proven experience architecting Kafka topologies for multi-region resilience, disaster recovery, and zero/low-downtime upgrades
- Strong experience securing Kafka with IAM-based authentication/authorization (e.g., AWS IAM/MSK IAM), including fine-grained IAM policy design for producers/consumers
- Experience with Schema Registry, schema governance, and Kafka Connect/Kafka Streams patterns
- Hands-on experience with IaaS and PaaS solutions from AWS (or similar cloud provider), including: Infrastructure as code (Terraform, AWS CloudFormation), Programming and scripting (Python, Java, Lambda)
- Hands-on experience with related big data infrastructure — AWS S3, AWS DMS, AWS EKS, Apache Spark, AWS Glue — and how they integrate with a streaming backbone
- Track record of leading technical initiatives end-to-end and influencing engineering standards across teams, not just within one system
- Excellent communication skills, with the ability to explain complex Kafka/streaming trade-offs to both technical and non-technical stakeholders, including leadership
- Demonstrated ability to mentor engineers and raise the technical bar of a team
- 5+ years relevant experience and a Bachelor's degree, OR any equivalent combination of education and experience.
Nice To Haves
- AWS Solutions Architect certification or Confluent Kafka certification is a plus
Responsibilities
- Implements tasks within the Software Development Lifecycle (SDLC), receiving structure and oversight from more experienced staff
- Follows well-established internal conventions and standard procedures
- Understands internal standards & processes an applies them to make technical decisions
- Collaborates with peers, manager, and project lead to gain understanding of tasks and review solutions
- May contribute to code & design reviews
- Own the technical roadmap and architecture for Venmo's Kafka streaming platform, ensuring it is secure, scalable, reliable, and performant for Venmo product, analytics, and downstream infrastructure teams
- Set the technical direction and design patterns for Kafka cluster architecture, topic design, partitioning strategy, and multi-tenant governance across the organization
- Lead the design and rollout of Kafka lifecycle management, including cluster upgrades, capacity planning, disaster recovery, and multi-region/multi-cluster replication strategies
- Drive deep root-cause analysis and resolution for the most complex, large-scale Kafka performance, reliability, and data-loss incidents
- Define and evangelize best practices for producer/consumer design, schema governance, and Kafka Connect/Kafka Streams usage across engineering teams
- Establish and improve engineering standards, runbooks, and operational rigor for streaming infrastructure in partnership with management
- Mentor and provide technical guidance to senior and junior engineers working on Kafka and adjacent streaming/data infrastructure
- Partner with Executive, Product, Engineering, and Analytics stakeholders to translate business needs into scalable streaming architecture decisions
Benefits
- generous paid time off
- healthcare coverage for you and your family
- resources to create financial security
- support your mental health
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
