Operations Support Engineer - Lead

Black Canyon Consulting•Bethesda, MD

About The Position

Black Canyon Consulting (BCC) is seeking a Lead Operations Support Engineer for its DevOps and infrastructure team, supporting application developers and operations at the National Center for Biotechnology Information (NCBI), part of the National Library of Medicine at the National Institutes of Health. The role leads the platform operations team supporting NCBI’s modern DevOps platform, based on GitLab and Kubernetes, and helps internal developers transition legacy software development and deployment to that platform. The team provides scalable solutions for development, deployment, and operations across teams, programming languages, and cloud and on-premises environments. The position involves troubleshooting operational, deployment, and runtime issues in a microservice environment; working with developers and users; and helping maintain application reliability against error budgets and service-level objectives. The lead also documents procedures, contributes to postmortems and outage-prevention actions, participates in hiring recommendations, trains new team members, and provides on-call support to NCBI developers and staff. The opportunity is full-time at NCBI in Bethesda, Maryland, and/or remote.

Requirements

BS degree in STEM or equivalent experience.
Customer-focused, team-oriented disposition.
Good systems debugging skills.
Comfort with the Linux environment or UNIX CLI.
Experience with a programming or scripting language.
Experience creating processes, procedures, and SOP documentation.
General understanding of TCP/IP, HTTP, and related protocols.
Initiative to take ownership of tasks and drive them to completion.
Comfort dealing with users with varying levels of IT knowledge.
Eagerness to learn new technologies.
Strong communication and soft skills to interface with customers, peers, and management.
Good judgment, integrity, and responsibility.

Nice To Haves

Kubernetes, OpenShift, cloud, or Linux experience.
Experience with service reliability engineering.
Linux systems administration experience.
Experience with automated CI servers, especially TeamCity and/or GitLab.
Automation programming or scripting experience in bash, Ruby, Python, Go, Java, Scala, Rust, C++, or Perl.
Experience with automated configuration management tools such as Puppet, Ansible, Chef, bcfg2, or cfengine; Puppet is preferred.
Experience with version control systems, especially git.
Experience with service mesh technologies such as linkerd or Istio.
Experience configuring or using monitoring and alerting technologies such as the TIGK stack, Grafana, Prometheus, or OpsGenie.
Experience with Confluence, Jira, and Microsoft Office suite.
Experience with GitOps tools, especially ArgoCD.
Google Anthos experience.
Understanding of Linux internals, including system calls, file systems, and processes.
Understanding of Linux network configuration.
Understanding of Linux application containerization, especially Docker.
Understanding of attached network storage technologies.
Understanding of cloud computing environments such as AWS, GCP, or Azure.
Understanding of automated CI/CD pipelines.
Understanding of distributed systems design principles.

Responsibilities

Identify and resolve operational problems in a microservice environment.
Work with developers to resolve deployment and runtime problems.
Perform analysis and debugging across multiple technologies.
Prioritize issues to keep applications within error budgets and meeting their SLOs.
Provide technical solutions to a wide range of problems and user requests.
Document processes, procedures, and SOPs by soliciting feedback and suggestions from team members.
Compile postmortems and action items to minimize future outages.
Interview candidates for team member roles and decide which candidates to recommend for hire.
Train new team members and assist them with issues.
Provide on-call support to NCBI's internal developers and other staff.
Lead the platform operations team supporting the DevOps platform and assist developers transitioning legacy software development and deployment to it.