Principal Software Engineer – Guest Experience Site Reliability

Choice Hotels International • Scottsdale, AZ

5d • Hybrid

About The Position

Choice Hotels is hiring a Principal Software Engineer for its Digital Site Reliability team in Digital Experience. The team maintains guest-facing platforms used by millions of travelers, including choicehotels.com, the Choice Hotels Android and iOS apps, and call center technologies. These platforms support bookings, reservation management, and loyalty program engagement.

The engineer will investigate complex production problems across front-end, API, cloud, and mobile systems; analyze logs and telemetry; build dashboards; and develop lasting reliability solutions. The role combines hands-on production coding with technical leadership during incidents, root-cause analysis, cross-functional decision-making, and pragmatic use of AI-assisted tools to improve diagnostics and reduce operational toil. The position is an individual contributor role reporting to the Head of Digital & Loyalty Engineering. It is a four-days-onsite hybrid role at the N. Scottsdale office and is not eligible for sponsorship.

Requirements

• Bachelor’s degree in Computer Science or a related technical field, or equivalent practical experience building and operating production systems.
• 10+ years of relevant software development experience, including distributed systems, cloud platforms, API-based architecture, and technical leadership or architecture experience.
• Strong hands-on experience with AWS public cloud services and understanding of how managed services affect reliability, scalability, and operational behavior.
• Hands-on experience building and maintaining CI/CD pipelines in Harness; Git-based source control and sound branching and collaboration practices.
• Strong Java experience building and operating Java-based services; Spring Boot is preferred.
• Proficiency building front-end applications with React and modern JavaScript or TypeScript.
• Experience designing, consuming, and troubleshooting GraphQL APIs, and familiarity with RESTful APIs and OpenAPI specifications.
• Experience diagnosing complex production issues with observability platforms and centralized logging. Hands-on Grafana experience, including log exploration and dashboard creation, is required.
• Demonstrated use of AI-assisted tools to improve engineering effectiveness or system reliability.
• Understanding of Site Reliability Engineering principles, including SLOs, error budgets, incident management, and blameless postmortems.
• Experience participating in an on-call rotation and leading critical incident response for high-traffic, consumer-facing systems.
• Working knowledge of relational and non-relational data stores and how persistence, availability, and failure characteristics affect system resiliency.
• Strong knowledge of application security best practices and awareness of compliance requirements such as PCI, OAuth, and OWASP.
• Excellent written and verbal communication, including the ability to explain complex technical issues to engineers and senior leaders and facilitate discussion under time pressure.
• Ability to deal with ambiguity, take ownership of highly complex problems from inception to completion, and lead others without formal positional power.

Nice To Haves

• Master’s degree or equivalent experience.
• Experience with Jenkins, GitHub Actions, or similar CI/CD tools.
• Experience with AppDynamics, OpenSearch, Amazon CloudWatch, or similar observability and logging tools.
• Kotlin experience and experience supporting Android and/or iOS applications, including mobile release, crash, and performance monitoring.
• AWS certifications or equivalent technology certifications.

Responsibilities

• Troubleshoot and resolve complex production issues across choicehotels.com, Choice’s mobile apps, and call center technologies by analyzing logs, metrics, traces, and runtime signals, using Grafana as a primary tool.
• Lead cross-functional conversations during incidents and time-sensitive technical discussions, aligning engineering, product, and business partners on root cause, options, and next steps.
• Participate in the Guest platforms on-call rotation as a senior escalation point and drive critical incidents to resolution.
• Write secure, stable, maintainable code to fix reliability defects, build tooling and automation, and strengthen application resiliency.
• Apply expertise in CDN, DNS, WAF, security, and vulnerability management, including solutions such as Akamai.
• Improve CI/CD pipelines and release safety in Harness, including automated deployment verification, rollback strategies, and pipeline guardrails.
• Identify systemic risks and failure modes in AWS-hosted services and translate them into well-scoped improvements.
• Apply AI-assisted development, log analysis, and incident tools where they measurably improve root-cause analysis, engineering productivity, and reliability outcomes.
• Conduct trade-off analysis of technical approaches and present clear pros, cons, and recommendations to peers and leadership.
• Mentor and coach engineers through code reviews, pairing, and knowledge sharing, and influence engineering practices across teams without formal authority.
• Partner with Solution Architects, Engineering Managers, and Mobile, Web, and Contact Center teams to define non-functional requirements for scalability, availability, and security.
• Evaluate and introduce tools and technologies that improve observability, platform safety, and developer productivity, prioritizing simplicity, adoption, and long-term impact.

Benefits

• Competitive compensation and benefits, including medical, dental, and vision coverage.
• Leave and paid time off for holidays, vacation, personal, family, volunteer, sick, jury duty, bereavement, military, and religious observance.
• Financial benefits for retirement and health savings.
• Employee recognition programs.
• Discounts at Choice hotels worldwide.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal