Senior Staff Engineer, Software

Celestica International LP•Bothell, WA

1d•Remote

About The Position

We are seeking an experienced full-stack software developer to design, develop, and test the next-generation AI-enabled automation infrastructure for switches and integrated data center racks. This is not a traditional "scripting" role; you will be building a full-stack, SDLC-integrated, test-driven engineering ecosystem centered around products running SONiC (Software for Open Networking in the Cloud) and OpenBMC within hyperscaler racks. Your mission, collaborating with the larger DevTestOps Team and software development teams, is to build an immersive, intuitive, robust, and full-featured control center for managing world-wide network automation test infrastructure. The Test Automation Control Center will consist of a set of cloud-hosted applications controlling multiple physical on-prem systems distributed worldwide. Control capabilities cover scheduling, monitoring, analyzing, and reporting tests from the initial developer Pull Requests to the final release regression validation. Managed equipment includes the entire range of Celestica switches, high-performance IXIA traffic generators, and a variety of equipment commonly found in data center racks. You will use your knowledge of AI agents to integrate intelligent workflows that select the best automation test cases to run, diagnose test failures for rapid resolution, and analyze test coverage to provide predictive product quality assessment.

Requirements

Architect a CI/CD Pipeline: Design the integration between Git-based workflows and physical hardware labs, ensuring code changes trigger automated builds and deployments to SONiC-based switches.
Cloud-to-On-Prem Connectivity: Lead the development of a cloud-hosted GUI and backend services that securely manage and command on-premise physical test beds.
Hardware Abstraction: Oversee the management of physical test beds, ensuring consistent state and availability for automated testing.
Framework Leadership: Standardize automated testing using SPyTest, ensuring robust coverage for NOS (Network Operating System) features.
Traffic Emulation: Integrate IXIA traffic generators into the automated suite to perform high-scale performance, stress, and regression testing.
Regression Management: Own the final validation gate, ensuring that no code reaches production without passing a rigorous, automated physical battery.
Failure Analysis Agents: Build and deploy AI/LLM-based agents to parse complex log files and SPyTest results to identify the "root cause" of test failures automatically.
Self-Healing Test Beds: Develop agents capable of test bed failure recovery (e.g., automatically power-cycling hung PDUs, re-flashing corrupted ONIE images, or re-seating virtual links).
Quality Insights: Leverage AI to analyze long-term software quality trends and predict potential regressions before they occur.
Networking: Deep expertise in SONiC, SAI (Switch Abstraction Interface), and standard protocols (BGP, EVPN, VXLAN).
Automation Frameworks: Expert-level knowledge of SPyTest and Python-based automation.
Hardware: Experience with IXIA (IxNetwork/IxLoad) and physical switch hardware (Mellanox/NVIDIA, Broadcom-based whitebox).
Software Development: Strong proficiency in Python, C/C++, Rust, or Java; experience building RESTful APIs and cloud-native backends (GCP/Azure).
AI/ML: Familiarity with integrating LLM APIs (like Google Gemini) for text/log analysis.
DevOps: Advanced experience with GitHub Actions, Azure DevOps or Jenkins, and containerization (Docker/Kubernetes).

Nice To Haves

Active contributor to the Azure/SONiC open-source community.
Experience building custom dashboards using React or Vue.js.
Knowledge of deploying and operating software within GCP.
Background in developing "Self-Healing" infrastructure or AIOps.

Responsibilities

Lead the design, development and implementation of technical solutions for complex projects, involving multiple domains. Participate in project planning and scheduling.
Global SME with comprehensive knowledge and industry recognition. Provides technical leadership and direction to a global team of engineers.
Take responsibility for non-technical elements of an engineering project (people, financials etc.)
Review and interpret customer specifications and may act as primary customer contact.
Analyze trade-offs in complex systems and recommend solutions. Develops deployment strategies and plans.
Lead the deployment of strategic complex programs and coordinate site-wide deployment efforts.
May manage relationships with key vendors/partners.
Analyze, design and develop tests and test-automation suites.
Design and develop a processing platform using various configuration management technologies.
Test software development methodology (may be done in agile environment)
Provide ongoing maintenance, support and enhancements in existing systems and platforms.
Collaborate cross-functionally with customers, users, project managers and other engineers including Peer-Reviews to achieve elegant solutions.
Provide recommendations for continuous improvement.
Work alongside other engineers on the team to elevate technology and consistently apply best practices.
Keep up to date with relevant industry knowledge and regulations