Staff Software Engineer, Data & Scalability Platform

Cisco•San Francisco, CA

8d•$186,900 - $307,800•Hybrid

About The Position

The Splunk Agent Resilience team is developing AI resilience capabilities through scalable, cost-effective evaluation and guardrails that help AI agents behave as intended. Its unified approach is designed to improve reliability and reduce risk, enabling customers to deploy and operate AI-powered applications with greater observability and control. The Staff Software Engineer on the Data & Scalability Platform team provides technical leadership for the architecture, scalability, and cost of the data plane, including data stores, streaming and compute components, and tooling for evaluating behavior under load. The role defines the data plane's multi-quarter technical direction, establishes standards used by other teams, leads initiatives to keep the platform fast and correct as it scales, and influences platform direction in partnership with engineering leadership and product teams.

Requirements

Bachelor's degree and 8+ years of related experience, or master's degree and 6+ years, or PhD and 3+ years, in Computer Science, Software Engineering, or a related field.
At least 6 years building and/or operating data-intensive backend services and/or platforms.
Technical leadership experience, including setting technical direction, driving multi-quarter initiatives, and/or influencing decisions beyond one's immediate scope.
Experience designing, operating, and/or delivering highly scalable, reliable, production-grade data-intensive platforms.
Experience designing large-scale distributed systems and solving scalability, availability, performance, reliability, and/or fault-tolerance problems.
Backend software engineering experience with one or more of Go, Java, Rust, C++, or Python.
Experience with cloud-native technologies, containerized environments such as Kubernetes, and public cloud platforms such as AWS or GCP.

Nice To Haves

Performance engineering ability, including profiling, benchmarking, and load testing real systems and using measurements to make capacity, design, and cost decisions.
Production-scale experience operating and tuning both an OLTP database such as PostgreSQL or MySQL and an analytical, columnar, or time-series store such as ClickHouse, Druid, BigQuery, or Snowflake, including schema design, indexing, query tuning, and migration safety.
Deep production experience with streaming or queueing systems such as Kafka, RabbitMQ, Pulsar, or Kinesis, including partitioning, consumer-group semantics, delivery guarantees, backlog handling, and dead-letter handling.
Experience scaling data-processing compute such as queue consumers, asynchronous task workers, or streaming jobs, including concurrency tuning, batching, backpressure, and sustained-load throughput behavior.
Deep experience running ClickHouse or a comparable columnar store at scale, including sharding and replication topology, materialized views, merge and mutation behavior, and retention and TTL mechanics.
Experience with a platform-scope capacity model or FinOps practice, including cost per unit of telemetry, cardinality governance, and quota or rate-limit design.
Experience designing multi-tenant isolation and noisy-neighbor mitigation in shared data systems.
Experience leading a significant production data migration or storage-engine transition without customer-visible impact.
Experience defining SLOs for data services and driving an organization to meet them.
Experience with backup and restore, tested disaster-recovery drills against stated RPO/RTO, and data-retention or deletion compliance such as GDPR/DSR hard delete.
Experience building observability instrumentation and pipelines, including OpenTelemetry, collectors, and metrics or tracing backends, rather than only consuming dashboards.
Experience building or operating performance tooling for other engineers, such as load-test harnesses, continuous profiling and flame-graph workflows (e.g., Pyroscope), or benchmarking suites.
Experience with distributed task frameworks and worker orchestration such as Celery, Temporal, Flink, or custom consumer fleets at production scale.
Familiarity with streaming databases, stream-processing frameworks, or distributed query engines such as RisingWave, Flink, Spark Streaming, or Trino/Presto, and with model-serving infrastructure.
Experience supporting stateful systems in both cloud/SaaS and air-gapped or on-premises deployments.
Experience mentoring senior engineers and raising engineering standards across multiple teams.
Strong communication skills and ability to translate performance, capacity, and cost findings into clear recommendations for engineering leadership and product partners.

Responsibilities

Define and drive the multi-quarter technical roadmap for the data platform, including storage, streaming, compute, and measurement tooling.
Lead data-plane architecture and evolution, including storage engine selection, sharding and tenancy models, and control-plane/data-plane boundaries.
Establish platform-wide standards for schema design, migration safety, and data retention, and lead the review process that enforces them.
Lead capacity and performance modeling and define service-level objectives (SLOs) for correctness and performance.
Lead major scalability and performance initiatives across data stores, streaming pipelines, and distributed worker fleets.
Drive cost efficiency, including cost per unit of telemetry, cost attribution, and quota and rate-limit design.
Set the direction for multi-tenant isolation so that one tenant cannot degrade another.
Design and build internal frameworks and performance tooling, including load testing, profiling, and benchmarking, for engineering design and capacity decisions.
Establish platform observability standards so signals are correct, complete, and affordable.
Lead production investigations and drive systemic remediation rather than point fixes.
Make and document technology-selection and build-versus-buy decisions, including explicit cost, risk, and migration analysis.
Partner with engineering leadership to influence platform direction and investment, and represent the data platform to product teams building on it.
Mentor engineers and raise engineering standards through design reviews, technical standards, and documentation.

Benefits

Medical, dental, and vision insurance, subject to plan eligibility rules.
401(k) plan with a Cisco matching contribution.
Paid parental leave.
Short- and long-term disability coverage.
Basic life insurance.
Paid holidays and a floating holiday for non-exempt employees.
Paid birthday day off, paid year-end holiday shutdown, and personal wellness days.
Vacation time for non-exempt employees; exempt employees participate in a flexible vacation time-off program with no defined limit, subject to availability and business limitations.
Sick time off, with eligible unused hours carried forward.
Additional paid time away may be requested for critical or emergency family issues.
Optional paid volunteer days.
Annual bonuses for eligible non-sales roles, subject to Cisco policies.
Eligible employees may receive Cisco restricted stock unit grants, subject to vesting conditions.