Principal Member of Technical Staff

AT&T•Plano, TX

6h•Onsite

About The Position

This position requires office presence of a minimum of 5 days per week and is only located in the location(s) posted. No relocation is offered. Join AT&T and help shape the future of communications and technology that connect the world. We value innovators who seek to explore the unknown and challenge the status quo. Bring your bold ideas and fearless spirit to redefine connectivity and transform how people share stories and experiences. At AT&T, you won’t just imagine the future—you’ll build it. The AT&T Network Technology Support Team is seeking a Principal Member of Technical Staff engineer with expertise in technical design, planning and advanced technical support for all layers of AT&T's network, including IP, Data, Transport, and Switching for all types of networks included in the AT&T Family. This includes planning and support for voice, data, video and IP based services in both the wireline and wireless environments. This position will work with our vendor partners to design, implement and support our new Artemis DAP (Drivenets Automation Platform) network. DAP is a hybrid-cloud platform used to implement infrastructure and services for OSS flows. The support of the Artemis DAP network requires the highest level (Tier-3) of network technical support within AT&T’s technical support structure requiring 24x7 on call from escalating organizations including but not limited to Tier-2 Operations and NPE. Provides the technical troubleshooting and leadership skills required to recover from the most complex network problems in real time, directing the activities of technical resources from Tier-2, Labs, and the vendor(s) to restore or implement new services. Responsible for the investigation and management of all follow-up activities needed to determine root cause, attempt problem recreation in the Tier Support lab, identify workarounds when feasible, coordinate with Labs for delivery and certification of vendor fixes or design changes, and plan for eventual implementation within the network. Develop scripts to proactively identify known defect signatures and define the least intrusive recovery processes. Develop scripts to extract forensic data from network elements and analyze them to identify potential patterns of unexpected behavior needed to understand root cause. Ultimately responsible for the reliability of the end-to-end network and the quality of the delivered service tracking and analyzing outage patterns leading to identification of chronic issues. Leverage a data-driven approach to influence business decisions to provide the best technical decisions.

Requirements

8-10 years of related experience.
SUN / Oracle hardware and OS knowledge
Operating system sysadmin (Unix, Linux, Solaris)
Extensive expertise in designing solutions with Kubernetes
Extensive expertise in System Design
Core principles of high availability, including hardware and software watchdog mechanisms, redundancy management, real-time replication, and related practices
Extensive expertise in Microservices Architecture
Extensive expertise in Python and Go programming languages
Extensive expertise to create bash Scripts
Knowledge of networking protocols like IP, TCP, UDP, LACP, LLDP, BFD, SNMP, BGP, VXLAN,ACL
Knowledge of Specification like REST API, RPC, gRPC
Knowledge of Software Defined Networks (SDN) including but not limited to Calico/Flannel/Cilium
Experience in deploying and managing cloud resources on Azure/AWS/OpenStack & Kubernetes
Extensive knowledge of OS patch, OS upgrades, and system hardware support
Experience in managing the key/value datastores like Etcd/Zookeeper
Proven expertise in provisioning hypervisors and virtual machines across on-premises environments, utilizing technologies such as KVM and QEMU, and public cloud environments such as Azure/AWS
Experience in management of RDBMS like Postgres/MySQL/MariaDB
Experience in management of NoSQL databases like Cassandra/MongoDB
Advanced troubleshooting - (Real-time isolation and recovery, Root Cause Analysis, Trouble Investigation for Network Elements
Excellent written and verbal communication skills.
Excellent organizational skills.
PhD in an approved field from an approved university with a minimum of 4 years of relevant experience or Masters in an approved field from an approved university with a minimum of 6 years of relevant experience or a Bachelors in an approved field from an approved university with a minimum of 8 years of relevant experience.

Nice To Haves

Intermediate knowledge of Element Management Systems (EMS) functions
Veritas Incident/Outage Management
MS Office Suite

Responsibilities

Design, develop, and support hybrid cloud infrastructure
Facilitate collaborative architecture design discussions that incorporate diverse perspectives, and apply a balanced, data-driven methodology when making strategic decisions.
Resolve critical outages, evaluating failure modes, and leading root cause analysis investigations.
Provide care for the overall health of the end-to-end network via the development of Element Management Systems (EMS) performance metrics; advanced data analytics; and identification, tracking, and resolution of chronic issues.
Requires 24x7 on call from escalating organizations.
Provides the technical troubleshooting and leadership skills required to recover from the most complex Networking problems in real-time, directing the activities of technical resources from Tier 2, CTO, and the vendors to restore service.
Drive problem management initiatives to solve complex issues that span multiple organizations which may include problem recreation, identification of fixes, certification of fixes, and/or design changes.
Responsible for all production equipment and software vendor management relationships ensuring objectives are met, including vendor performance feedback during contract negotiations.
Will lead cross-functional teams in support of technology insertion and root cause analysis investigations and will be expected to be the technical subject matter expert on special projects.
May interact with customers virtually or in-person to outline or recommend technical solutions for their business or explain the root cause of a network event.

Benefits

Medical/Dental/Vision coverage
401(k) plan
Tuition reimbursement program
Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
Paid Parental Leave
Paid Caregiver Leave
Additional sick leave beyond what state and local law require may be available but is unprotected
Adoption Reimbursement
Disability Benefits (short term and long term)
Life and Accidental Death Insurance
Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
Employee Assistance Programs (EAP)
Extensive employee wellness programs
Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone