Observability Engineer

Pennymac • Westlake Village, CA

4d • Onsite

About The Position

PENNYMAC Pennymac (NYSE: PFSI) is a specialty financial services firm with a comprehensive mortgage platform and integrated business focused on the production and servicing of U.S. mortgage loans and the management of investments related to the U.S. mortgage market. At Pennymac, our people are the foundation of our success and at the heart of our dynamic work culture. Together, we work towards a unified goal of helping millions of Americans achieve aspirations of homeownership through the complete mortgage journey. Job Overview We're looking for an experienced, forward-thinking Observability Engineer to expand our Observability team in Core Services Engineering, and strengthen our observability capabilities across Pennymac environments. In this role, you will be responsible for designing, implementing, and maintaining our observability platform, with a strong focus on New Relic. You will leverage your expertise in Infrastructure as Code (IaC) to automate and manage our monitoring and alerting infrastructure, ensuring our systems are reliable, performant, and transparent. You will work closely with Core Services, DevOps, Development and Operation teams to foster a culture of proactive monitoring and data-driven decision-making. If you're passionate about automation, cloud-native patterns, and making systems run smarter and safer, we want to hear from you.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).
• 7+ years of experience in a Cloud Engineering role (Observability, DevOps, SRE, etc).
• Proven New Relic Expertise: 3+ years of hands-on experience with the New Relic platform, including deep knowledge of Dashboards, NRQL, APM and setting up effective alerting.
• Strong IaC Proficiency: 3+ years of experience managing infrastructure and configurations with IaC tools like Terraform/OpenTofu (preferred), AWS CDK, CloudFormation, Chef or Ansible.
• Cloud Platform Experience: Extensive hands-on experience working with major cloud providers such as AWS (preferred), GCP , or Azure .
• Scripting Skills: Proficiency in a scripting language such as Python , Go, or Bash for automation and tooling.
• System Knowledge: Strong understanding of cloud architecture, networking principles, Windows/Linux Server administration, microservices in a SaaS context, containerization ( Docker , Kubernetes ), and CI/CD principles.
• Security & Compliance experience : Deep understanding of security best practices and their implementation in cloud infrastructure and CI/CD pipelines. Experience working in complexly regulated environments.
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills.

Responsibilities

• Design & Implement Observability Solutions: Architect, build, and scale comprehensive monitoring solutions using the New Relic platform , including APM, Infrastructure, Logs, Synthetics, and custom instrumentation (NRQL).
• Automate with IaC: Develop, manage, and maintain observability configurations—including alerts, dashboards, and synthetic checks—using Infrastructure as Code (IaC) tools such as Terraform/OpenTofu .
• Develop Dashboards & Alerts: Create and refine insightful dashboards and actionable alerting policies in New Relic to provide real-time visibility into infrastructure and application health.
• Promote Best Practices: Act as a subject matter expert on observability, guiding teams on best practices for logging, metrics, and tracing to improve system reliability and reduce mean time to resolution (MTTR).
• Troubleshoot & Optimize: Analyze performance data and telemetry to identify bottlenecks, troubleshoot production issues, and drive performance optimization efforts across the stack.

Benefits

• Comprehensive Medical, Dental, and Vision
• Paid Time Off Programs including vacation, holidays, illness, and parental leave
• Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations)
• Retirement benefits, life insurance, 401k match, and tuition reimbursement
• Philanthropy Programs including matching gifts, volunteer grants, charitable grants and corporate sponsorships

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 5,001-10,000 employees