Principal Software Engineering - Reliability and FinOps Engineer

AT&T • Dallas, TX

8d • Onsite

About The Position

AT&T Technology Services is modernizing its technology ecosystem through secure, scalable, cloud-enabled, and AI-powered platforms. Within Reliability & FinOps Engineering, the organization strengthens the reliability and resiliency of critical systems, maximizes the value of technology investments, and accelerates responsible automation. It brings together reliability engineering, FinOps and infrastructure optimization, AI and automation enablement, and customer-centric business intelligence.

As a Principal System Engineer, you will provide senior technical leadership across FinOps, cloud optimization, system reliability, and AI-driven automation. You will shape and execute enterprise strategies to improve the reliability of critical services, reduce cloud waste, strengthen financial accountability, and automate operational workflows across Azure, AWS, Kubernetes, and hybrid-cloud environments. You will translate business priorities into durable technical solutions, influence cross-organizational roadmaps, and guide optimization opportunities from insight through governed execution and measurable value realization.

This is a full-time, onsite office role performed at the designated work location five days per week. Locations include Alpharetta, Georgia; Atlanta, Georgia; Bothell, Washington; Middletown, New Jersey; and Plano/Dallas, Texas. No relocation is offered. AT&T will not hire applicants who require employer sponsorship now or in the future. The position has a 40-hour work week and a stated annual salary range of $155,400–$261,100; individual starting salary may depend on geography, experience, expertise, and education or training.

Requirements

• 7+ years of progressive experience in systems engineering, cloud engineering, site reliability engineering, platform engineering, DevOps, FinOps, or enterprise automation.
• Demonstrated ability to lead complex, cross-functional engineering initiatives and influence technical direction without relying solely on direct authority.
• Deep hands-on experience with Azure and/or AWS architecture, networking, compute, storage, identity, monitoring, governance, and cost-management services.
• Strong knowledge of FinOps practices, including allocation, tagging, showback or chargeback, budgeting, forecasting, unit economics, commitment management, anomaly detection, optimization, and savings validation.
• Experience implementing cloud optimization for virtual machines, storage, databases, platform services, Kubernetes, and idle or underutilized resources.
• Advanced scripting and automation experience using Python, PowerShell, Bash, REST APIs, JSON, and event-driven integration patterns.
• Strong Infrastructure as Code and CI/CD experience using Terraform, GitHub Actions, Azure DevOps, or comparable enterprise tooling.
• Hands-on experience with Kubernetes, AKS, EKS, or OpenShift, including workload rightsizing, Helm, observability, node lifecycle management, and container cost allocation.
• Experience applying generative AI, prompt engineering, AI assistants, AI-fueled coding, or agentic automation to engineering and operational use cases.
• Strong understanding of reliability engineering, observability, incident management, capacity planning, performance optimization, recovery, change control, and operational readiness.
• Experience designing secure automation using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, secrets management, and auditable approval controls.
• Ability to analyze complex cloud cost, utilization, telemetry, and operational data and convert findings into prioritized actions and clear executive insights.
• Excellent written and verbal communication skills, including presenting complex technical and financial information to engineers, application owners, finance partners, vendors, and senior leaders.
• AT&T will not hire applicants who require employer sponsorship now or in the future.

Nice To Haves

• Bachelor's or master's degree in Computer Science, Computer Engineering, Information Systems, Data Science, or Finance (preferred).
• FinOps Certified Practitioner certification or certifications in Azure, AWS, Kubernetes, Terraform, ITIL, site reliability engineering, security, data, or artificial intelligence.
• Experience with IBM Turbonomic, IBM Cloudability, Kubecost, Flexera SPOT/Ocean, Flexera ElasticGroup, Azure Cost Management, AWS Cost Explorer, or comparable FinOps and optimization platforms.
• Experience building intelligent automation with approved enterprise AI services such as Microsoft 365 Copilot, GitHub Copilot, Azure OpenAI, Azure AI Foundry, Power Platform, or comparable platforms.
• Experience with Power BI, Kusto Query Language, Log Analytics, Snowflake, data engineering, anomaly detection, forecasting, or machine-learning pipelines.
• Knowledge of the FinOps Open Cost and Usage Specification (FOCUS), reservation and savings-plan economics, and enterprise cost-allocation models.
• Experience integrating cloud automation with ServiceNow change management, configuration management databases, approval workflows, audit evidence, and operational controls.
• Experience designing safe remediation workflows with policy validation, human-in-the-loop approvals, post-change monitoring, rollback, evidence capture, and drift detection.
• Knowledge of container security, software supply-chain security, vulnerability management, policy as code, and compliance automation.
• Demonstrated success delivering measurable cost savings, reliability improvements, risk reduction, or productivity gains at enterprise scale.
• Knowledge of AT&T technology governance, public-cloud architecture patterns, internal change-management processes, or application ownership practices.

Responsibilities

• Provide principal-level technical leadership for resilient, secure, scalable, and cost-efficient platforms across Azure, AWS, Kubernetes, and hybrid-cloud environments.
• Lead FinOps and infrastructure optimization capabilities, including cost allocation, showback, anomaly detection, forecasting, rightsizing, workload scheduling, commitment optimization, waste reduction, and validated savings realization.
• Develop policy-driven, closed-loop optimization workflows covering recommendation identification, application-owner engagement, approval, ServiceNow change control, execution, validation, evidence capture, and reporting.
• Apply automation-by-default using Python, PowerShell, Bash, REST APIs, Azure Functions, Power Automate, and event-driven services to reduce manual handoffs and operational toil.
• Design practical generative AI, AI-fueled coding, and agentic automation solutions for engineering operations, cost analysis, anomaly investigation, remediation guidance, executive reporting, knowledge retrieval, and decision support.
• Establish cloud-governance guardrails, policy-as-code controls, exception processes, auditability, and risk-based automation patterns aligned with security, architecture, compliance, and change-management requirements.
• Engineer secure identity and authorization models using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, and time-bound permissions.
• Integrate optimization and automation capabilities with Terraform, GitHub, Azure DevOps, CI/CD pipelines, ServiceNow, monitoring platforms, cost-management tools, and enterprise data sources while preventing configuration drift.
• Advance reliability engineering practices, including observability, service-level objectives, capacity management, performance analysis, incident prevention, operational readiness, post-change validation, rollback planning, and continuous improvement.
• Build executive dashboards and operational scorecards communicating cloud spend, reliability, automation coverage, optimization pipeline health, risk, realized savings, and customer or business outcomes.
• Lead proofs of concept, production readiness, phased adoption, and complex technical issue resolution in partnership with internal teams and strategic vendors.
• Mentor engineers, create reusable reference architectures and standards, and influence priorities across Reliability, FinOps, Security, Architecture, Finance, Operations, and application teams.
• Demonstrate AT&T leadership behaviors by serving customers first, moving faster, acting boldly, winning as one, and developing talent.

Benefits

• Medical, dental, and vision coverage
• 401(k) plan
• Tuition reimbursement program
• Paid time off and holidays, including at least 23 vacation days each year and 9 company-designated holidays (based on date of hire)
• Paid parental leave
• Paid caregiver leave
• Additional sick leave beyond what state and local law require may be available; it is unprotected
• Adoption reimbursement
• Short- and long-term disability benefits
• Life and accidental death insurance
• Supplemental benefit programs, including critical illness, accident, hospital indemnity, and group legal
• Employee Assistance Programs (EAP)
• Employee wellness programs
• Employee discounts up to 50% off eligible AT&T mobility plans and accessories, AT&T internet (including fiber where available), and AT&T phone

What This Job Offers

• Job Type: Full-time
• Career Level: Senior