Principal System Engineering

AT&T•Plano, TX

1d•Onsite

About The Position

The Principal System Engineering role is responsible for partnering with various Consumer Technology Experience (CTx), Development, Operations/Infrastructure, and Solution Architect teams in designing a comprehensive Performance Engineering framework based on business needs. This role involves planning, designing, and rolling out Performance Engineering Test solutions to prevent disruption and re-engineering costs by detecting defects early and building Performance into projects from the onset. The position is directly responsible for cloud platform planning, creation, and maintenance of applications on public cloud instances. The ideal candidate will possess expert-level working knowledge of all key methodologies, strategies, technology, and experience of a performance and cloud engineer. This role also includes mentoring junior resources and guiding peer team members in test engineering and quality strategic organizations. The Principal System Engineer will design robust Performance Test Strategies to validate the impact of changes on individual applications and the consumer technology ecosystem, participate in design discussions to ensure Performance is integrated, and collaborate with Vendor/Partner teams for load test execution. They will review performance test results, evaluate infrastructure and application responsiveness for production readiness, and provide SME leadership within the Consumer Quality Engineering (CQE) team on Transformative Efforts, implementing best practices in Performance Testing. Additionally, the role involves designing and implementing automated Performance Testing in the CI/CD pipeline, defining and implementing post-mortem/root-cause analysis processes, and developing improved testing scenarios. Responsibilities include performing workload and future growth analysis, application endurance certification using automated client scripts and performance assurance tools, and implementing automated utilities for capturing application transaction traces, sessions, and browser network logs. The role requires understanding applications and technologies, onboarding new applications by testing them in lower environments, identifying gaps/anomalies, and providing root causes. Collaboration with vendor support for issue resolution and enhancements, and working with application teams for code changes and metric capture are also key. The role involves developing documentation, ensuring systems meet user requirements, and participating in issue identification, analysis, and resolution. Building service simulations using Broadcom Dev Test Service Virtualization and Mock Server, creating hypotheses based on production outages, and designing/executing chaos scenarios using Gremlin for system resources, state, and network resilience testing are also part of the role. Recommendations will be provided based on chaos experiment analysis. Collaboration with developers, operations, and security teams to understand system architecture and identify weaknesses is essential. A detailed volumetric analysis and derivation of a Workload Model are also required.

Requirements

Requires a Bachelor’s degree, or foreign equivalent degree in Electronic Engineering and 5 Years of progressive, post-baccalaureate experience in the job offered or 5 Years of progressive, post-baccalaureate experience in a related occupation.
Utilizing experience for creating and maintaining applications in cloud engineering.
Proficient experience in automated Performance validation in the software delivery through the CI/CD pipeline.
Microservices architecture and containerization technologies like Docker and Kubernetes.
Analysis, design, estimation, project planning and development of CQE solutions for CTX applications in Java along with stakeholder interaction.
Gather client’s business requirements and check the feasibility of the same with proper gap analysis.
Performance testing using automated testing tools which generates and executes the test cases and test data.
Prepare Technical/Business reports and support the bug fix and issues reported in performance testing Phase of a particular release and weekly status reports.

Nice To Haves

Expert level working knowledge of all key methodologies, strategies, technology, and experience of a performance & cloud engineer.
Experience with Broadcom Dev Test Service Virtualization and Mock Server.
Experience with Gremlin tool for chaos engineering.
Experience with ELK and Dynatrace.
Experience with performance assurance tools for automating applications user flows.
Experience with capturing application transaction traces, sessions, browser network logs, response time.

Responsibilities

Partnering with CTx, Development, Operations/Infrastructure, and Solution Architect teams in designing a comprehensive Performance Engineering framework.
Planning, designing, and rolling out Performance Engineering Test solutions.
Directly responsible for cloud platform planning, creation, & maintenance of applications residing on public cloud instances.
Mentoring junior level resources and serving as a guide for peer team members.
Designing robust Performance Test Strategies to validate impact of changes to performance.
Participating and providing recommendations to ensure Performance is built into the design of a solution.
Working closely with Vendor/Partner teams for execution of load tests.
Reviewing performance test results and ensuring all aspects of infrastructure and application responsiveness are evaluated.
Providing key SME leadership within CQE team on Transformative Efforts and implementing best industry practices.
Designing and implementing automated Performance Testing in the CI/CD pipeline.
Defining and implementing post-mortem / root-cause analysis processes.
Performing workload, future growth analysis and application endurance certification.
Implementing an automated utility to capturing application transaction traces, sessions, browser network logs, response time.
Understanding the application, technologies and onboarding a new application by testing and validating them in lower environments.
Identifying the gaps/anomalies and providing root cause for the issues identified.
Working with vendor support and providing required details for resolution of issues or enhancements needed for application.
Working with the application team to capture additional metrics, meta data from the application pages.
Working with the application team for code changes where manual injections are needed.
Developing documentation (as required) on new or existing systems.
Ensuring systems meet documented user requirements.
Participating in identification, analysis, and resolution of identified issues.
Building service simulations of project blocking and overhead services on private premises cloud.
Creating hypothesis based on production outages in reactive approach.
Designing chaos scenarios for critical applications in a proactive approach.
Executing various chaos scenarios using Gremlin tool.
Providing recommendations based on chaos experiments analysis.
Collaborating with other teams, including developers, operations, and security, to understand the system's architecture and identify potential weaknesses.
Conducting a detailed volumetric analysis and deriving a Workload Model.

Benefits

Medical/Dental/Vision coverage
401(k) plan
Tuition reimbursement program
Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
Paid Parental Leave
Paid Caregiver Leave
Additional sick leave beyond what state and local law require may be available but is unprotected
Adoption Reimbursement
Disability Benefits (short term and long term)
Life and Accidental Death Insurance
Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
Employee Assistance Programs (EAP)
Extensive employee wellness programs
Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone