Senior Software Engineer - SRE

OneTrust•Atlanta, GA

2d•$116,475 - $174,713•Hybrid

About The Position

OneTrust is seeking a Senior Software Engineer to join their R&D Team, focusing on mission-critical applications. This role involves designing, delivering, and maintaining a highly available and performant application platform. The engineer will build and implement observability and monitoring tools, automate processes to eliminate toil, and evaluate new technologies. Collaboration with various teams, including Engineering, Operations, and Product, is key to identifying gaps, resolving issues, and defining/implementing SLIs and SLOs. The position also involves managing error budgets, improving alert quality, reviewing architectures for reliability risks, and sharing knowledge across the organization. Scripting for operational automation and incident response, as well as building and operating AI-assisted incident response systems, are also core responsibilities.

Requirements

Bachelor's degree in computer science, Engineering, or related technical or business field
4+ yrs. of application development experience with Java or other equivalent language
Experience with Spring environment
Experience in cloud-based infrastructure (Azure, AWS, GCP, etc.)
Experience with the factors that affect software application performance at different levels. These factors include database performance, network performance, CPU utilization, JVM tuning, memory analysis, thread management, and query performance.
A knowledge of the importance of centralizing logging, metrics dashboards, and alerting. Able to articulate about some of the tools used for these tasks
A good awareness of databases (ideally SQL/NoSQL)
Hands-on experience with observability tools (Datadog, Prometheus, Grafana, etc.)
Knowledge with CI/CD pipelines and infrastructure-as-code (Terraform, Helm, jenkins, gitlab)
Kubernetes and container orchestration (EKS/AKS/GKE)
Experience with distributed systems at scale
Familiarity with service meshes and microservices architectures

Nice To Haves

Experience with chaos engineering tools (Gremlin, Chaos Monkey)
Background in product-facing services with high traffic scale
Understand how to use incident management platforms. This includes using tools like PagerDuty for alerts. It also includes working with DataDog for monitoring.

Responsibilities

Engage and partner with various Engineering, Operations, and Product teams to design, deliver, and maintain a highly available and performant application platform.
Build and implement application observability and platform monitoring tools to continuously improve the customer experience
Eliminate toil by automating processes, tuning alerts, and improving code where it is most needed
Frequently evaluate new ideas and trends to identify potentially useful tools and techniques
Collaborate with different functional groups to identify gaps, prioritize, and resolve issues
Defining, implementing, and maintaining SLIs and SLOs aligned with customer experience.
Design and instrument SLIs such as latency, error rates, and availability across critical services
Manage and enforce error budgets to balance system reliability with product feature velocity.
Improving alert quality by reducing noise and focusing on actionable, high-signal alerts
Embed with product teams to review architectures and catch reliability risks early
Share your knowledge and experience with the Engineering organization
Share your findings with technical leadership and senior management
Build scripts in python/bash/java or ruby for operational automation and incident response
Build and operate AI-assisted incident response systems (root cause analysis, log summarization, anomaly triage)
Develop or integrate LLM-based tools to reduce MTTR and improve alert quality
Apply machine learning techniques for anomaly detection, capacity prediction, or failure pattern analysis
Experience deploying AI systems in production (not just experimentation)
Knowledge with vector databases, embeddings, or RAG architectures for operational intelligence
Well-developed insight of prompt engineering and evaluation of LLM outputs in the reliability workflow