Cloud Senior Site Reliability Engineer

About the position

At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. We do this by driving Responsible Growth and delivering for our clients, teammates, communities and shareholders every day. Being a Great Place to Work is core to how we drive Responsible Growth. This includes our commitment to being a diverse and inclusive workplace, attracting and developing exceptional talent, supporting our teammates' physical, emotional, and financial wellness, recognizing and rewarding performance, and how we make an impact in the communities we serve. At Bank of America, you can build a successful career with opportunities to learn, grow, and make an impact. Join us! Our team designs, builds, and maintains Public Cloud platforms for Bank of America's. We provide our customers an innovative platform with built-in integrations that allow for a faster time-to-market with reduced complexity. We believe in a high-quality engineering culture, a customer focused mindset, and building for scale and resiliency. As part of this team, you will have a large impact on the evolution of next generation Cloud services for Bank of America and explore an extensive list of new technologies that will drive innovation across our company.

Responsibilities

• Collaborates with a diverse set of engineers, architects, and teams to design, develop, test, and implement secure, robust, highly available and scalable solutions for BofA's External Cloud Platform
• Collaborates with other software engineers and teams to design and implement deployment approaches using highly scalable, automated, continuous integration and continuous delivery pipelines
• Responsible for all aspects of reliability, collaborates with technical experts, key stakeholders, and team members to resolve complex problems, owning the issue until you are sure it will not reoccur
• Deep understanding of SRE practices, service level indicators, and service level objectives; proactively utilize them to resolve issues before they impact customers
• Gather, analyze, synthesize, and develop visualizations and reporting from large, diverse data sets in service of continuous improvement of the platform
• Implement infrastructure, configuration, and network as code for the applications and platforms in your remit
• Identify opportunities to eliminate toil and automate the triage of issues to improve overall operational stability
• Collaborate with a global team to identify, analyze, and resolve platform vulnerabilities
• Proactively promotes the adoption of site reliability engineering best practices within the team and organization
• Participate in 24x7 on-call coverage follow the sun model and performs blameless Postmortems (RCAs) as needed
• Designs solutions to visualize key production support metrics enabling Operational Readiness and Site Reliability Engineer teams to identify scenarios requiring intervention
• Develops software solutions and/or improved processes to address work identified as 'toil' by collaborating with key partners to identify, track and remediate processes to free time allocated to reliability
• Partners with Development and Infrastructure teams to create error budget policies prioritizing reliability stories that fall below Service Level Objective (SLO) thresholds and suggests code optimizations, additional instrumentation and/or logging structures to gain service reliability visibility
• Identifies and plans for capacity bottlenecks, vulnerabilities and opportunities for reliability improvement, such as low level error rates and 'noise', and reduces manual support effort and/or improves system reliability
• Assesses monitoring for new changes with development partners and works with monitoring tools team to monitor dashboards and enhance application and system monitoring designs
• Engages as a subject matter expert in incident triage efforts, failure scenario modelling and works with the Problem Manager to diagnose root causes for complex/high impact incident/problem management investigations
• Collaborates with Development and Infrastructure teams to understand technical solutions and develop Service Level Indicators and SLOs to measure/improve the reliability of the services they support

Requirements

• 10+ years of combined experience in either SRE, software development, or infrastructure engineering
• 7+ years of hands-on experience building and maintaining cloud platforms on a major cloud service provider
• Strong experience in implementing, monitoring, and maintaining a highly scalable and resilient Data Services platform on major CSP's like AWS, Azure or GCP
• Strong experience with monitoring tools such as Grafana, Prometheus, Splunk, or Dynatrace, as well as cloud native tools like CloudWatch & CloudTrail, Azure Monitor and Log Analytics
• Proficiency in implementing, monitoring, and maintaining a Databricks, RDS, or OpenAI platform
• Proficient in at least one programming language such as Python, Java/Spring Boot, and .Net; 5+ years applied experience in Python/Java
• Proficiency in implementing CI/CD pipelines with tools such as git and Jenkins, familiarity with using a GitOps model
• Advanced knowledge of networking (firewalls, DNS, Load Balancing, Proxies, etc.)
• Advanced understanding of Linux & Windows operating systems including shell scripting
• Excellent interpersonal, organizational and communication (written, verbal, and presentation) skills are a must
• Proven ability to work independently with minimal supervision and as part of a global team with direct responsibilities and an ability to juggle competing priorities and adapt to changes in project scope

Nice-to-haves

• Strong experience working with a complex IAM infrastructure, including Active Directory, Azure AD Connect, Azure AD, and PingIdentity, Okta, or other SSO solutions
• Proficiency in creating automation using Python, Terraform, or Ansible
• Proficiency in implementing, monitoring, and maintaining a Databricks, CosmosDB, or OpenAI platform
• Experience in implementing, monitoring, and maintaining a highly scalable and resilient enterprise platform on Microsoft Azure using native services related to compute, storage, networking, security, and observability
• Experience with containerization technologies such as EC2, EKS, Fargate, Openshift, or Kubernetes
• Understanding of cost management, inventory management, FinOps model