Cloud Systems Engineer - Site Reliability

TherapyNotes.com•Philadelphia, PA

1d•Remote

About The Position

TherapyNotes is seeking a Site Reliability Engineer to improve the reliability and operability of production services and shared platforms supporting its 24×7 SaaS environment. The role applies software and systems engineering practices to improve availability, performance, scalability, resilience, observability, incident response, and operational automation. The engineer will partner with software development, infrastructure, database, security, and other technology teams to establish measurable reliability goals, reduce operational toil, and ensure services are supportable throughout their lifecycle. The position focuses on operating critical systems, improving Datadog-based observability, defining and advancing reliability practices, participating in production incident response and on-call support, and automating infrastructure and operational work. Supported systems and activities must comply with organizational security, HIPAA, and operating policies.

Requirements

Bachelor of Science degree in Information Systems or Engineering, or equivalent experience.
At least 5 years of engineering experience in Systems Engineering, Cloud or Platform Engineering, DevOps, Software Engineering, and/or SRE.
Experience designing and operating production systems using cloud-based compute, storage, networking, and containerization technologies.
Strong Linux systems and networking fundamentals, including experience troubleshooting complex distributed systems in production.
Expertise with an observability platform.
Experience with scripting and operational automation using tools such as Bash, PowerShell, or Python, as well as infrastructure-as-code and configuration-management practices.
Experience participating in production on-call rotations, incident response, root cause analysis, and post-incident improvement.
Experience working in Agile/DevOps environments and operating production services using ITSM practices where applicable.

Nice To Haves

Azure and Kubernetes experience are preferred.
Datadog experience is strongly preferred.
Experience with Prometheus, Grafana, New Relic, or equivalent observability platforms is valuable.
Prior software development experience, or experience investigating application behavior through code, logs, and distributed traces, is a plus.

Responsibilities

Own and continuously improve Datadog use across metrics, logs, traces, dashboards, monitors, alerts, and service-level views.
Design, implement, and maintain highly available, high-throughput, data- and compute-intensive systems supporting the 24×7 SaaS platform.
Partner with service owners to define and improve SLIs, SLOs, error budgets, actionable alerting, and operational-readiness practices.
Participate in and help lead production incident management as an incident commander or technical responder.
Coordinate incident triage, service restoration, escalation, communication, documentation, root cause analysis, and corrective actions.
Investigate infrastructure- and application-layer issues with development teams using metrics, logs, distributed traces, and code-level context.
Improve deployment safety and service resilience through automated validation, recovery and rollback capabilities, reliability testing, and failure-mode analysis.
Work with technical leaders to ensure newly introduced systems are supportable and maintainable by development and operations teams.
Provide escalated technical guidance and support to technology teams across the organization.
Provide on-call production support and perform other duties as required.
Ensure supported systems and operational activities comply with organizational security, HIPAA, and operating policies.
Identify and eliminate repetitive operational toil using Bash, PowerShell, Python, or Ansible.
Manage infrastructure as code using Terraform/OpenTofu and configuration automation using Ansible.