Staff Software Engineer

Home Depot•Atlanta, GA

1d•Remote

About The Position

The Staff Software Engineer develops, deploys, operates, and maintains infrastructure platform software for Site Reliability Engineering, focused on observability, availability, and operational efficiency of enterprise retail store systems. The role builds and extends internal reliability platforms on Google Cloud Platform (GCP) and Google Kubernetes Engine (GKE), using Terraform and cdk8s to provision and manage cloud resources and Kubernetes workloads. The engineer deploys, configures, tunes, and maintains open-source observability systems on GKE, including Grafana, Loki, Tempo, Prometheus, Pyroscope, and AlertManager. The role uses PromQL, LogQL, and TraceQL for dashboards, alert rules, and production troubleshooting, and defines and implements SLOs, SLIs, and error budgets with Prometheus-based queries and multi-window, multi-burn-rate alerting. The position also maintains operational automation platform infrastructure, develops integration plugins, manages GitHub-backed access controls, and responds to platform support requests. It builds automation to reduce operational toil, including self-service onboarding tools, GitHub Actions CI/CD pipelines, and AI-assisted workflows using GitHub Copilot. Responsibilities include participating in an on-call rotation, investigating and resolving production incidents, contributing to blameless post-mortems, validating platform resilience through destructive and performance testing, and documenting operations, runbooks, and onboarding guides. Telecommuting is available.

Requirements

Must be at least 18 years of age.
Must be legally permitted to work in the United States.
Bachelor’s degree in Computer Science, Computer Engineering, Information Technology Management, Electronics Engineering, or a related field.
At least 3 years of experience in a related occupation.
At least 3 years of experience deploying, configuring, and troubleshooting containerized workloads on GKE using infrastructure-as-code tools, including cdk8s for TypeScript-based Kubernetes manifest generation and Helm charts for templated deployments.
At least 3 years of experience provisioning and managing GCP infrastructure with Terraform, including GKE clusters, Cloud Storage, BigQuery, Cloud Pub/Sub, Cloud SQL, Cloud Logging sinks, Compute Engine, IAM policies, and Workload Identity federation.
At least 3 years of experience deploying, operating, and tuning Kubernetes observability platforms, specifically Loki with LogQL, Tempo with TraceQL, Prometheus with PromQL, Pyroscope, and AlertManager.
At least 3 years of experience defining and implementing SLOs, SLIs, and error budgets using Prometheus-based SLI queries and multi-window, multi-burn-rate alerting configured through YAML definitions.
At least 3 years of experience building and maintaining operational automation platform infrastructure, including Kubernetes and virtual-machine runners, integration plugins, GitHub-backed access control and runbook configurations, and support request resolution.
At least 3 years of experience programming in Python, TypeScript, or a similar language for infrastructure automation, platform tooling, observability configuration, and Kubernetes manifest generation.
At least 3 years of experience designing and operating synthetic monitoring platforms on Kubernetes, including test orchestration, result collection via Cloud Pub/Sub, and alerting integration.
At least 3 years of experience configuring Kubernetes security controls, including RBAC, namespace isolation, service accounts, and GCP IAM policies including Workload Identity.
At least 3 years of experience building GitHub Actions CI/CD pipelines for infrastructure platform services, including automated testing, builds, deployments, container image builds, Helm chart packaging, and Terraform plan/apply workflows.
At least 3 years of experience building infrastructure-operations automation, including self-service onboarding tools, GitHub Copilot-assisted workflows, and automated support request handling.
At least 3 years of experience troubleshooting production infrastructure issues across distributed Kubernetes systems, including pod failures, resource contention, network policies, storage issues, and observability pipeline bottlenecks using kubectl, log analysis, and tracing data.
At least 3 years of experience managing secrets and credential lifecycles with Google Secret Manager, Kubernetes secrets, and automated credential rotation for platform integrations.

Responsibilities

Develop, deploy, operate, and maintain infrastructure platform software for Site Reliability Engineering, supporting observability, availability, and operational efficiency of enterprise retail store systems.
Build and extend internal reliability platforms on GCP and GKE using Terraform and cdk8s.
Deploy, configure, tune, and maintain Grafana, Loki, Tempo, Prometheus, Pyroscope, and AlertManager on GKE clusters.
Write PromQL, LogQL, and TraceQL queries for dashboards, alert rules, and production infrastructure troubleshooting.
Define and implement SLOs, SLIs, and error budgets using Prometheus-based SLI queries and multi-window, multi-burn-rate alerting.
Develop and maintain operational automation platform infrastructure, including runners on Kubernetes and virtual machines, integration plugins, GitHub-backed access controls, and platform support.
Build toil-reduction automation, including self-service onboarding tools, GitHub Actions CI/CD pipelines, and AI-assisted workflows using GitHub Copilot.
Participate in an on-call rotation for observability and automation infrastructure.
Investigate and resolve production incidents affecting logging, tracing, profiling, and automation platforms.
Contribute to blameless post-mortem reviews and implement systemic improvements.
Validate platform resilience through destructive and performance testing.
Document platform operations, create runbooks, and maintain onboarding guides for internal engineering teams.