Sr. Lead Software Engineer, Site Reliability & Monitoring

Chick-fil-A•Atlanta, GA

14d•Hybrid

About The Position

Overview At Chick-fil-A, Reliability and Monitoring is a technical function which mixes in influence. Across our 3000+ North American Restaurants, cloud, and private data centers, SREs work with our DevOps teams to introduce and hone SRE principles, establish reliability goals, and develop tooling for operational observability. We are a small team working through many different patterns to bring observability to everyone. SREs at Chick-fil-A collaborate across teams and roles, feed learnings back into the organization, and learn all the ways technology is used in the process. The team is focused on tooling and enablement rather than traditional SRE roles. The BUILD team is transforming the way that observability data is captured within Chick-fil-A. In our pursuit of what's next, the delivery and adoption of OpenTelemetry is at the core of our updated observability tooling. In this role, the ideal candidate will work independently across platform, SDK, and application teams to gather requirements, build, offer support, and troubleshoot our observability pipelines. Our Flexible Future model offers a healthy mix of working in person and virtually, strengthening key elements of the Chick-fil-A culture by fostering collaboration and community. Responsibilities Own solution architecture decisions for the team’s product Lead delivery and operations of the team’s product, including both individual contribution and support as well as delegated tasks and support to your team’s engineers. We desire our lead engineers to be both leads and engineers, spending about half of their time on leading others and half contributing engineering work themselves. Lead, mentor, and assess other staff engineers, exemplifying and teaching best practices, helping to solve complex problems, reviewing code, and sharing stories Interview, select, onboard, and oversee contract engineers Assist in the design and delivery of products outside of your team, develop tools and processes for use outside of your team, and/or mentor other engineers or especially lead engineers outside of your team Guide engineering team in adoption of Chick-fil-A software engineering standards Document learnings to share with the broader engineering team(s) Ensure clear communication around R+M objectives Collaborate broadly across the entire engineering organization Oversee other SRE teams to bring best practices or learnings from across the organization to them Build internal tooling around operational observability Bring a strong mindset of continual improvement An aversion to toil and automatable tasks Advocate for R+M as a part of engineering culture Act as a conduit for Architecture, Security, Tools, and DTT Framework Keep abreast of industry changes and evaluate for implementation Work independently with DevOps teams to refine running production systems Building on-call processes Creating Incident Management and Response procedures Instrumenting for observability and coaching on best practices Monitoring SLIs Work to varying degrees with DevOps teams Provide consultation on SRE best practices Give guidance on specific topics Oversee groups of dedicated engineers Embed directly with teams Work with teams to define SLOs and error budgets Ensure services and systems meet availability needs of customers Design and develop software solutions Serve as a model developer in programming languages like Java, Go, Python, and Python Exercise skills in infrastructure and deployment services like AWS and Kubernetes as well as areas like application security, data analytics, and machine learning Note - Working in a DevOps model, this opportunity includes both building and running solutions that could require off hours support. This support is shared amongst the team members to cover weekends and weeknights.  The goal is to design for failure and, using cloud-native infrastructure patterns, automate responses to possible issues so they can be worked during normal hours.

Requirements

Bachelor’s Degree or the equivalent combination of education, training and experience from which comparable skills can be acquired
4+ years of relevant work experience
Experience designing complex software solutions
Experience mentoring and leading a team, including good interpersonal and team collaboration skills
Broad and deep programming experience in Java, JavaScript, Python, or other comparable languages
Experience with SQL and data modeling
Experience with source control systems like Git or Subversion
Experience implementing application security, software design patterns, and the SDLC
Proven ability to positively influence the engineering culture and practices in a professional environment
Significant experience with: Building and supporting systems Enterprise cloud providers Production containerized environments
Excellent written and verbal communication
Experience with CI/CD pipelines
Ability to build strong relationships, collaborate, and influence diverse groups of engineers and non-technical roles
Ability to influence other engineers without organizational authority

Nice To Haves

Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or similar area of study
9+ years of relevant work experience, with: 4+ years supporting a production system on a Devops team 2+ years in cloud platforms such as amazon web services, google cloud, or Microsoft azure
Experience with test-driven development, continuous integration and deployment, Scrum discipline, or comparable software development practices
Deep understanding of AWS architecture
Familiarity with version control systems and code merging/branching; specific experience with git desirable
Experience working with an agile development methodology featuring sprints, points estimation, and daily standups
Experience in design, data collection, and data analysis
Experience with Unix/Linux
Experience with Kubernetes
Experience with OpenTelemetry, Fluentd, Grafana Alloy, or Datadog Agent (DDOT preferred)

Responsibilities

Own solution architecture decisions for the team’s product
Lead delivery and operations of the team’s product, including both individual contribution and support as well as delegated tasks and support to your team’s engineers.
Lead, mentor, and assess other staff engineers, exemplifying and teaching best practices, helping to solve complex problems, reviewing code, and sharing stories
Interview, select, onboard, and oversee contract engineers
Assist in the design and delivery of products outside of your team, develop tools and processes for use outside of your team, and/or mentor other engineers or especially lead engineers outside of your team
Guide engineering team in adoption of Chick-fil-A software engineering standards
Document learnings to share with the broader engineering team(s)
Ensure clear communication around R+M objectives
Collaborate broadly across the entire engineering organization
Oversee other SRE teams to bring best practices or learnings from across the organization to them
Build internal tooling around operational observability
Bring a strong mindset of continual improvement
An aversion to toil and automatable tasks
Advocate for R+M as a part of engineering culture
Act as a conduit for Architecture, Security, Tools, and DTT Framework
Keep abreast of industry changes and evaluate for implementation
Work independently with DevOps teams to refine running production systems
Building on-call processes
Creating Incident Management and Response procedures
Instrumenting for observability and coaching on best practices
Monitoring SLIs
Work to varying degrees with DevOps teams
Provide consultation on SRE best practices
Give guidance on specific topics
Oversee groups of dedicated engineers
Embed directly with teams
Work with teams to define SLOs and error budgets
Ensure services and systems meet availability needs of customers
Design and develop software solutions
Serve as a model developer in programming languages like Java, Go, Python, and Python
Exercise skills in infrastructure and deployment services like AWS and Kubernetes as well as areas like application security, data analytics, and machine learning