Principal Software Engineering - Reliability and FinOps Engineer

AT&T•Atlanta, GA

9d•$155,400 - $261,100•Onsite

About The Position

AT&T is hiring a Principal System Engineer for its Reliability & FinOps Engineering organization. The team modernizes AT&T’s technology ecosystem with secure, scalable, cloud-enabled, and AI-powered platforms, strengthening reliability and resiliency, maximizing technology investment value, and accelerating responsible automation. The role provides senior technical leadership across FinOps, cloud optimization, system reliability, and AI-driven automation. The engineer will shape and execute enterprise strategies to improve critical-service reliability, reduce cloud waste, strengthen financial accountability, and automate operations across Azure, AWS, Kubernetes, and hybrid-cloud environments. This includes translating business priorities into durable technical solutions, influencing cross-organizational roadmaps, and moving optimization opportunities from analysis through governed execution and measurable value realization. This is a full-time onsite office role performed at the designated work location five days per week. No relocation is offered. Applicants who require employer sponsorship now or in the future will not be hired for this position. The role is listed as regular, with 40 weekly hours.

Requirements

At least 7 years of progressive experience in systems engineering, cloud engineering, site reliability engineering, platform engineering, DevOps, FinOps, or enterprise automation.
Demonstrated ability to lead complex, cross-functional engineering initiatives and influence technical direction without relying solely on direct authority.
Deep hands-on experience with Azure and/or AWS architecture, networking, compute, storage, identity, monitoring, governance, and cost-management services.
Strong knowledge of FinOps practices, including allocation, tagging, showback or chargeback, budgeting, forecasting, unit economics, commitment management, anomaly detection, optimization, and savings validation.
Experience implementing cloud optimization for virtual machines, storage, databases, platform services, Kubernetes, and idle or underutilized resources.
Advanced scripting and automation experience using Python, PowerShell, Bash, REST APIs, JSON, and event-driven integration patterns.
Strong Infrastructure as Code and CI/CD experience using Terraform, GitHub Actions, Azure DevOps, or comparable enterprise tooling.
Hands-on experience with Kubernetes, AKS, EKS, or OpenShift, including workload rightsizing, Helm, observability, node lifecycle management, and container cost allocation.
Experience applying generative AI, prompt engineering, AI assistants, AI-fueled coding, or agentic automation to engineering and operational use cases.
Strong understanding of reliability engineering, observability, incident management, capacity planning, performance optimization, recovery, change control, and operational readiness.
Experience designing secure automation using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, secrets management, and auditable approval controls.
Ability to analyze complex cloud cost, utilization, telemetry, and operational data and convert findings into prioritized actions and clear executive insights.
Excellent written and verbal communication skills, including the ability to present complex technical and financial information to engineers, application owners, finance partners, vendors, and senior leaders.

Nice To Haves

Bachelor's or master's degree in Computer Science, Computer Engineering, Information Systems, Data Science, or Finance preferred.
FinOps Certified Practitioner certification or certifications in Azure, AWS, Kubernetes, Terraform, ITIL, site reliability engineering, security, data, or artificial intelligence.
Experience with IBM Turbonomic, IBM Cloudability, Kubecost, Flexera SPOT/Ocean, Flexera ElasticGroup, Azure Cost Management, AWS Cost Explorer, or comparable FinOps and optimization platforms.
Experience building intelligent automation with approved enterprise AI services such as Microsoft 365 Copilot, GitHub Copilot, Azure OpenAI, Azure AI Foundry, Power Platform, or comparable platforms.
Experience with Power BI, Kusto Query Language, Log Analytics, Snowflake, data engineering, anomaly detection, forecasting, or machine-learning pipelines.
Knowledge of the FinOps Open Cost and Usage Specification (FOCUS), reservation and savings-plan economics, and enterprise cost-allocation models.
Experience integrating cloud automation with ServiceNow change management, configuration management databases, approval workflows, audit evidence, and operational controls.
Experience designing safe remediation workflows with policy validation, human-in-the-loop approvals, post-change monitoring, rollback, evidence capture, and drift detection.
Knowledge of container security, software supply-chain security, vulnerability management, policy as code, and compliance automation.
Demonstrated success delivering measurable cost savings, reliability improvements, risk reduction, or productivity gains at enterprise scale.
Knowledge of AT&T technology governance, public-cloud architecture patterns, internal change-management processes, or application ownership practices.

Responsibilities

Provide principal-level technical leadership for resilient, secure, scalable, and cost-efficient platforms across Azure, AWS, Kubernetes, and hybrid-cloud environments.
Lead FinOps and infrastructure optimization, including cost allocation, showback, anomaly detection, forecasting, rightsizing, workload scheduling, commitment optimization, waste reduction, and validated savings realization.
Develop policy-driven, closed-loop optimization workflows covering recommendations, application-owner engagement, approvals, ServiceNow change control, execution, validation, evidence capture, and reporting.
Automate operational workflows using Python, PowerShell, Bash, REST APIs, Azure Functions, Power Automate, and event-driven services to reduce manual handoffs and toil.
Design practical generative AI, AI-fueled coding, and agentic automation solutions for engineering operations, cost analysis, anomaly investigation, remediation guidance, executive reporting, knowledge retrieval, and decision support.
Establish cloud-governance guardrails, policy-as-code controls, exception processes, auditability, and risk-based automation aligned with security, architecture, compliance, and change-management requirements.
Engineer secure identity and authorization models using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, and time-bound permissions.
Integrate optimization and automation with Terraform, GitHub, Azure DevOps, CI/CD pipelines, ServiceNow, monitoring platforms, cost-management tools, and enterprise data sources while preventing configuration drift.
Advance reliability engineering practices, including observability, service-level objectives, capacity management, performance analysis, incident prevention, operational readiness, post-change validation, rollback planning, and continuous improvement.
Build executive dashboards and operational scorecards covering cloud spend, reliability, automation coverage, optimization pipeline health, risk, realized savings, and customer or business outcomes.
Lead proofs of concept, production readiness, phased adoption, and resolution of complex technical issues in partnership with internal teams and strategic vendors.
Mentor engineers, create reusable reference architectures and standards, and influence priorities across Reliability, FinOps, Security, Architecture, Finance, Operations, and application teams.
Demonstrate AT&T leadership behaviors by serving customers first, moving faster, acting boldly, winning as one, and developing talent.

Benefits

Medical, dental, and vision coverage
401(k) plan
Tuition reimbursement program
Paid time off and holidays, including at least 23 vacation days each year and 9 company-designated holidays, based on date of hire
Paid parental leave
Paid caregiver leave
Additional sick leave beyond what state and local law require may be available but is unprotected
Adoption reimbursement
Short- and long-term disability benefits
Life and accidental death insurance
Supplemental benefit programs, including critical illness, accident hospital indemnity, and group legal
Employee Assistance Programs (EAP)
Employee wellness programs
Employee discounts up to 50% off eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available), and AT&T phone