Principal Engineer - Resiliency

ForeFlight•Austin, TX

10d•$208,000 - $244,000•Hybrid

About The Position

ForeFlight is hiring a Principal Engineer to own resiliency strategy and reference architecture for its AWS estate. The role establishes availability targets and standard multi-AZ and multi-region patterns, determines suitable failover topologies, and guides services through resiliency improvements. It leads business continuity and disaster recovery end to end, including RTO/RPO targets, automated failover, recovery exercises, runbooks, and follow-up architectural changes. The engineer will conduct AWS Well-Architected Reviews and drive remediation, improve resiliency across container and data platforms, encode recovery capabilities in infrastructure automation, and own reliability signals such as SLOs, error budgets, health checks, dependency maps, and blast-radius analysis. The role also strengthens incident response and postmortems and translates continuity requirements from engineering, product, and business stakeholders into AWS designs and validated recovery capabilities. The position is preferred as hybrid in Austin, Texas; Houston, Texas; or Denver, Colorado. Virtual work may be considered for the right candidate.

Requirements

12 or more years of engineering experience, including at least 7 years as the primary architect or technical owner of infrastructure resiliency and high-availability architecture at scale.
Deep hands-on AWS production experience in large multi-account, multi-region environments, including VPC and networking design, Route 53, ALB/NLB, Global Accelerator, IAM, and account-boundary decisions affecting failover.
Demonstrated ownership of highly available architectures, with multi-AZ as a baseline and real multi-region capabilities for critical workloads, and sound judgment about failover topology tradeoffs.
Track record owning business continuity and disaster recovery strategy end to end, including RTO/RPO targets, automated regional failover, and genuine recovery exercises.
Expert command of the AWS Well-Architected Framework, particularly the Reliability pillar, including formal reviews and completing resulting remediation.
Production depth with AWS container platforms (ECS/Fargate, EKS, or both), including scaling, placement, failure domains, deployment safety, and degradation under failure.
Strong data-tier resiliency experience with Aurora or RDS and at least one of DynamoDB global tables, ElastiCache replication, or S3 cross-region replication.
Deep production experience designing and operating infrastructure as code at scale; Terraform is preferred and CDK is acceptable. Strong knowledge of module strategy, state management, and guardrail enforcement across accounts and environments.
Expert command of infrastructure CI/CD, including pipeline design, drift detection, plan/apply workflows, and safe self-service patterns.
Hands-on experience with chaos engineering and production fault injection, including tooling and game-day operations.
Strong infrastructure observability skills, including SLO and error-budget design, health-check strategy, dependency mapping, and fleet-wide visibility.
Cross-functional leadership and ability to work with senior engineering, product, and finance leaders on availability, redundancy costs, and recovery capabilities.
Comfort coordinating across divergent AWS estates and in-flight unification, including consolidation-related organizational challenges.

Nice To Haves

Hands-on experience with AWS resiliency tools such as Resilience Hub, Route 53 Application Recovery Controller, Elastic Disaster Recovery, or Fault Injection Service.
AWS Solutions Architect Professional certification or equivalent demonstrable depth.
Experience with infrastructure orchestration tools beyond raw Terraform, such as Terragrunt, Atlantis, Spacelift, env0, or Crossplane.
SRE or platform reliability background, with strong instincts for SLO design, error-budget policy, and toil reduction.
Experience modeling redundancy costs and explaining tradeoffs to business stakeholders.
Experience integrating infrastructure and resiliency practices across two or more legacy AWS estates after an M&A.
Experience in regulated industries or other domains with mission-critical workloads and strict continuity requirements.

Responsibilities

Own the resiliency strategy and reference architecture for the AWS estate, including availability targets and standard multi-AZ and multi-region patterns.
Define decision-making frameworks for active-active, active-passive, warm-standby, and pilot-light architectures, including tradeoffs and migration paths.
Lead business continuity and disaster recovery, including RTO/RPO target-setting, automated failover, recovery exercises, runbooks, and incorporation of exercise findings.
Conduct AWS Well-Architected Reviews focused on Reliability and Operational Excellence and drive prioritized remediation through completion.
Improve resiliency across ECS/Fargate and EKS workloads, including placement, capacity headroom, failure-domain isolation, graceful degradation, and recovery during partial-region impairment.
Improve data-tier recovery across Aurora, RDS, DynamoDB, ElastiCache, and S3, setting expectations for failover time, replica lag, and data-loss boundaries.
Encode resilience in automation through infrastructure-as-code failover paths, self-healing, auto-remediation, drift detection, and continuous fault injection.
Own reliability signals for core infrastructure, including SLOs, error budgets, health checks, failover triggers, dependency mapping, and blast-radius analysis.
Improve incident response for availability events, lead resiliency-focused postmortems, and turn recurring failure modes into architectural fixes and platform guardrails.
Translate continuity and availability needs from engineering, product, and business stakeholders into AWS designs and validated recovery capabilities.