Staff Software Engineer, Search & Distributed Systems

ACV • Buffalo, NY

1d

About The Position

We are looking for a Staff Software Engineer who would thrive on being accountable for our Search infrastructure: its scalability, reliability, and data resiliency. This role requires a battle-scarred Distributed Systems expert who understands the deep internals of Elasticsearch and possesses a comprehensive toolbox for analyzing, monitoring, alerting, and quickly resolving critical issues. The ideal candidate knows how Elasticsearch fails under load and how to architect a topology to prevent such failures. Additionally, this role involves owning the architectural connective tissue, ensuring seamless interaction between service layers, and the event-based ecosystem with Search. As a Staff Engineer, you will be responsible for setting technical standards, driving systemic reliability, and mentoring senior engineers across the organization.

Requirements

• 8+ years of software engineering experience, with at least 3+ years operating at a Senior or Staff level focusing on distributed systems and high-throughput platforms.
• Deep, authoritative knowledge of Elasticsearch internals. You have managed large-scale clusters and deeply understand mapping, analysis, query optimization, cluster state management, and split-brain mitigation.
• Proficiency in the systems upstream and downstream of Search. You have hands-on experience with Kubernetes (EKS/GKE), API Gateway/BFF architectures, and event streams (Kafka).
• A proven track record of implementing fault-tolerant patterns (retries, rate limiting, circuit breaking, dead letter queues) in microservice architectures.
• Expert-level ability to instrument systems and diagnose complex performance issues using modern observability stacks (Datadog, Prometheus, Grafana, OpenTelemetry).
• Strong communication skills with a proven ability to influence cross-functional teams, build consensus around architectural decisions (the Knoster model!), and mentor mid-level and senior engineers.

Nice To Haves

• Experience with Infrastructure as Code (Terraform, Helm) for stateful applications.
• Familiarity with FinOps practices, specifically optimizing Elasticsearch compute and storage costs.
• Experience integrating AI-assisted development tools into your daily workflow.

Responsibilities

• Architect for Scale: Design, configure, and scale our Elasticsearch clusters. You will define our global strategies for shard routing, Index Lifecycle Management (ILM), heap tuning, and data tiering to support massive auction throughput.
• Master the Failure Modes: Anticipate and engineer away points of failure. You will design circuit breakers, implement backpressure mechanisms, and tune asymmetric timeouts to prevent retry storms between our BFFs, K8s services, and the Search layer.
• Expert Troubleshooting & IR: Act as the ultimate technical escalation point for complex, cross-system performance degradation. You will dive deep into JVM metrics, Garbage Collection pauses, K8s network bottlenecks, and slow logs to uncover and remediate root causes.
• Holistic System Ownership: Manage the entire data lifecycle. You will optimize the ingestion pipelines syncing our event datastreams driven by producers and consumers (Kafka) to Elasticsearch, ensuring eventual consistency and data integrity at scale.
• Drive Engineering Excellence: Draft authoritative architectural Blueprints, SOPs, and Runbooks. You will elevate the surrounding engineering culture by coaching teams on distributed systems design, observability best practices, and incident management.
• Modernize & Innovate: Scan the horizon for emerging technologies. You will help evaluate and integrate next-generation search capabilities (e.g., Vector Search, RAG architectures) to support our broader AI and machine learning initiatives.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed