Distinguished Engineer - Spark Infrastructure Architecture
About The Position
Splunk, a Cisco company, is building a safer and more resilient digital world with an end-to-end full stack platform made for a hybrid, multi-cloud world. Leading enterprises use our unified security and observability platform to keep their digital systems secure and reliable. Come help organizations be their best, while you reach new heights with a team that has your back. Meet the Team: Our Distinguished Engineer team drives architecture and technical direction for Splunk's $3.5B platform, ingesting petabytes of data for over 95% of the Fortune 100. As a tight-knit group of DEs reporting to the VP of Architecture, we operate autonomously while coordinating closely on practical, durable solutions that evolve the platform for future requirements. Each DE owns cross-organizational domains that shift fluidly as needs arise. Work alongside exceptionally smart engineers who value rigorous thinking and friendly collaboration-this is where the technical future gets defined! Impact: Architect and operationalize Spark at scale as a core component of Cisco Data Fabric, enabling customers to access and analyze exponentially more data across diverse organizational sources. Build resilient, always-on infrastructure with automatic retry logic, fault tolerance, and seamless integration across myriad data sources and formats including Parquet, Iceberg, and modern table formats. Design the operational backbone-monitoring, observability, error handling, and horizontal scalability-that enables hundreds of engineers to build features confidently on solid foundations. Establish architectural patterns and support mechanisms that ensure high availability while maintaining performance at petabyte scale. Success is measured by customer adoption of data fabric capabilities and operational reliability metrics-uptime, ingestion performance, and incident reduction-that directly impact customer value.
Requirements
- Bachelor's in Computer Science (or equivalent) with 15+ years of related experience; or Master's with 12+ years; or PhD with 8+ years or equivalent experience
- Designed and deployed production Spark deployments at scale in cloud environments
- Experience with modern data formats including Parquet, Iceberg, and related table/columnar formats
- Production experience with AWS cloud platform
- Led technical decisions and architectural direction across engineering organizations of 50+ engineers
Nice To Haves
- Production experience with Azure and/or GCP cloud platforms
- Experience integrating with diverse data platforms such as Snowflake, Pinot, Databricks, Trino, or similar systems
- Extensive Kubernetes experience in production environments
- Proven track record mentoring and growing engineers, with strong collaboration skills across sponsors, design, product management, and engineering teams
Responsibilities
- Architect and operationalize Spark at scale as a core component of Cisco Data Fabric
- Build resilient, always-on infrastructure with automatic retry logic, fault tolerance, and seamless integration across myriad data sources and formats including Parquet, Iceberg, and modern table formats
- Design the operational backbone-monitoring, observability, error handling, and horizontal scalability
- Establish architectural patterns and support mechanisms that ensure high availability while maintaining performance at petabyte scale
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Industry
Professional, Scientific, and Technical Services
Number of Employees
5,001-10,000 employees
