Sr. Software Engineer, Observability - Slack

Salesforce•San Francisco, CA

About The Position

The Monitoring Infrastructure team in the Service Delivery Platform & Reliability group at Slack develops platforms and tools which produce telemetry, provide insights, and improves observability in Slack production services with a focus on performance and reliability. We develop management tools for distributed applications and infrastructure, maintain datasets for performance and system analytics, and build interfaces and backend systems to answer questions and infer behavioral patterns about our users and systems. Our toolset is varied, we work with open-source observability/monitoring technologies like Astra and Prometheus, or cloud providers such as AWS, and develop software using a combination of Go, Python, or Java. As part of the Monitoring Infrastructure team, you will be focused on our log pipelines and work closely with other teams in engineering, product development, and customer experience to provide valuable insights to drive decisions and ensure a positive user experience for our Slack customers. You will also help build and maintain distributed services in an environment that processes millions of data points per second with the ability to self-heal and scale up or down to meet demand. We are an inclusive team with deep empathy for our colleagues and customers.

Requirements

You are a strong communicator. Explaining complex technical concepts to designers, support, and other engineers is no problem for you.
You enjoy helping onboard new team members, mentoring, and teaching others.
You live for unit tests, code review, design documentation, debugging, and solving problems.
You have a deep curiosity about how things work under the hood.
You are motivated by helping others succeed.
When things break — and they will — you are eager and able to help fix things.
You like thinking of ways to improve efficiency or bring delight to your coworkers.
You care about improving the performance of systems through data-informed decisions.
You also know that the internet is a scary place and understand security concepts deeply and can put them into action to protect us and our users.

Nice To Haves

Passionate about data visualization, graphing, and maximizing signal versus noise.
Experience with Elasticsearch, Logstash, and Kibana.

Responsibilities

Build, maintain, and ensure timely delivery of our high-volume event log pipelines.
Create libraries, tools, and automation to help ensure that critical event data gets to the right place.
Encouraging a culture of Observability at Slack - help suss out problem areas and consult on improving visibility into our systems.
Prototyping tooling interfaces or building new features for engineering use cases.
Improving auto-remediation in our logging infrastructure to avoid recurring failures.
Teaching engineers or customer experience agents how to use our tools to introspect their systems.
Collaborating with the rest of Platform to integrate observability solutions, enabling service teams to make data-driven decisions regarding operations, cost optimization, and scaling.
Participating in the Monitoring Infrastructure on-call rotation, triaging, and addressing production issues as they arise.