Software Development Engineer (US Federal)

Workday•Reston, GA

2d•Hybrid

About The Position

The Data Platform and Observability team, located in Pleasanton, CA; Boston, MA; Boulder, CO, Atlanta, GA, and Dublin, Ireland, empowers real-time insights across Workday's platforms, infrastructure, and applications. Our mission is to develop and operate a large-scale distributed data platform that fuels critically important Workday applications. We provide software solutions for the collection, ingestion, storage, and visualization of critical data assets. We manage hundreds of terabytes of data, encompassing billions of messages generated daily by Workday applications and underlying services. If you're passionate about crafting efficient software or optimizing and scaling large distributed systems, you'll find a fulfilling home with us. This role will support one or more direct or indirect contracts with the U.S. Federal Government which, due to federal government security requirements, mandates that all Workday personnel working on the contracts be United States citizens (naturalized or native). Are you eager to work on highly-scalable distributed systems? Come join our team of competent, personable engineers operating and improving Workday Government’s observability platform.

Requirements

BS in Computer Science or a related technical field
Minimum 5 years of experience building and deploying highly-scalable software
Experience deploying software to public cloud platforms such as AWS, Azure, or GCP
Scripting skills with any programming language
Understanding of system design, particularly distributed systems
Good knowledge and understanding of Kubernetes
Experience with Linux-based system administration
United States citizen (naturalized or native)

Nice To Haves

Demonstrable experience with Prometheus and Grafana Mimir
Demonstrable experience with Fluent Bit, Kafka, and Elasticsearch
Strong DevOps knowledge and hands-on experience with IaC and CI/CD tools

Responsibilities

Use Terraform and Argo CD to deploy services to EKS AWS in production environments
Be responsible for the reliable operation of Metrics, Logging, Tracing, and Alerting services handling terabytes of data in tens of billions of documents across hundreds of indexes and tens of millions of active time series
Collaborate with our cyber defense team to ensure security operations, cybersecurity posture, governance, risk, compliance, and platform security health
Participate in the on-call rotation supporting the observability platform
Resolve issues internal customers may have with our services
Write various forms of technical documentation, such as ADRs, RCAs, and runbooks
Collaborate with managers and engineers at our parent company to ensure healthy engineering practices for both government and non-government environments
Build out test environments, tooling, and automation to implement CI/CD best practices
Construct new metrics, logs, alerts, and dashboards where there are gaps with what we can observe about our system and the other services using it
You may be responsible for observability services deployed to an air-gapped region and you may be required to regularly work from a SCIF