Principal Infrastructure Engineer

Sezzle

12d•Remote

About The Position

We are seeking an exceptional Principal Infrastructure Engineer to design, build, operate, and scale the platform that powers Sezzle. Your focus will be the hardest infrastructure problems: increasing throughput, reducing latency, removing capacity bottlenecks, strengthening resilience, and making production operations more automated and predictable as the business grows. You will own complex technical initiatives from architecture and prototyping through implementation, production rollout, and ongoing operation. Your impact will come from the systems you build, the problems you solve, and measurable improvements in reliability, performance, and cost efficiency. Our stack runs on AWS, with workloads orchestrated on Kubernetes and data anchored in Aurora RDS (MySQL and Postgres). You should know these technologies deeply and be comfortable moving between cloud architecture, networking, cluster internals, database performance, and application behavior to understand how the entire system scales. You will write code and infrastructure-as-code, debug production systems, and deliver changes that hold up under real traffic and failure conditions. Operational ownership is part of the job. You will participate in the on-call rotation and take a hands-on role in recovering from major incidents, including full outages. We need someone who can form and test hypotheses using logs, metrics, and traces, make sound mitigation decisions with incomplete information, and turn incident findings into lasting engineering fixes. You will also build and apply AI-assisted infrastructure and SRE tooling for incident investigation, capacity analysis, runbook automation, and toil reduction. You will evaluate these tools through practical results and apply appropriate access controls, validation, and auditability to their use in production. This role reports to engineering leadership and works closely with application engineers, Security, and Compliance. You will develop a deep understanding of how Sezzle's business operates and how customer journeys, transaction patterns, and product decisions shape infrastructure demand and behavior. You will use that understanding to identify and deliver improvements across teams and technical domains, connecting infrastructure decisions to better customer outcomes and business performance.

Requirements

Bachelor's degree in Computer Science or a similar technical field (required)
12+ years of experience across infrastructure, platform, site reliability, software development, or related engineering disciplines, with substantial hands-on depth designing and operating production infrastructure at scale.
Deep expertise with AWS: production experience across compute, IAM, multi-account architectures, and networking, including VPC design and private connectivity.
Deep expertise with Kubernetes in production: cluster lifecycle, scheduling, resource management, autoscaling, networking, and troubleshooting business-critical workloads. EKS experience is strongly preferred.
Deep expertise with relational databases at scale, specifically RDS/Aurora (MySQL and/or Postgres): query performance, indexing, connection management, replication, high availability, failover, and verified backup and recovery.
A track record of personally delivering infrastructure scaling improvements: identifying constraints, measuring baseline behavior, implementing changes, and demonstrating gains in capacity, latency, reliability, or cost efficiency.
Strong coding and automation skills, using Golang, Python, or similar languages to build production tooling and eliminate operational toil, alongside infrastructure-as-code experience with Terraform or equivalent.
Strong systems fundamentals: Linux, networking, DNS, TLS, storage, concurrency, and distributed system failure modes, with the ability to debug problems across infrastructure and application boundaries.
Experience operating a 24/7, high-availability platform where downtime has direct customer or revenue impact, including hands-on incident response and postmortem remediation.
Willingness to participate in an on-call rotation and demonstrated ability to recover production systems under pressure using evidence-based triage, safe mitigation, and clear technical communication.
Experience implementing and testing disaster recovery against defined recovery objectives, including restoring data and validating service recovery.
Practical experience with observability, load testing, capacity planning, and safe CI/CD practices for shared production infrastructure.
Active use of AI tooling in engineering or operations, with practical judgment about its limitations and how to verify generated code, recommendations, and operational actions.
Ability to carry ambiguous technical problems from investigation through production delivery and collaborate across engineering disciplines to resolve system-wide constraints.

Nice To Haves

Experience in fintech, payments, or banking, operating infrastructure with demanding reliability, security, and audit requirements.
Experience with multi-region architectures, chaos engineering, and failure testing, including the consistency and recovery tradeoffs of distributed data systems.
Proficiency with Prometheus, Grafana, Loki, Tempo, or comparable observability systems.
Experience building internal platform capabilities and self-service tooling, including deployment automation, progressive delivery, and reusable infrastructure components.
Experience building AI-assisted incident investigation or operational automation with restricted access, auditable execution, and clear human review points.

Responsibilities

Own the technical architecture and evolution of core infrastructure: identify system limits, prioritize technical improvements, and implement changes that support increasing traffic, data volume, and workload complexity.
Connect business understanding to improvements across the system: learn how key business workflows behave in production, trace their impact across applications, data, and infrastructure, and partner across teams to improve performance, reliability, and cost efficiency beyond any single service or team's scope.
Engineer for scale and performance: build capacity models, run load and stress tests, diagnose bottlenecks across compute, networking, Kubernetes, and databases, and validate improvements against throughput, latency, saturation, and cost per workload.
Design and build AWS infrastructure: implement resilient account, IAM, network, and service architectures; address service quotas, fault isolation, and multi-AZ or multi-region requirements as workloads grow.
Build and operate the Kubernetes platform: improve cluster architecture, lifecycle automation, workload isolation, resource allocation, autoscaling, safe upgrades, and deployment reliability.
Scale and optimize Aurora RDS for MySQL and Postgres: tune queries and indexes, address connection and replication bottlenecks, plan capacity, improve failover behavior, and implement safe schema changes and database migrations with application engineers.
Improve reliability through engineering: define and instrument service-level objectives and error budgets with service owners; implement failure isolation, backpressure, load shedding, and safe retry behavior where needed to prevent cascading failures.
Participate in on-call and drive technical recovery during serious incidents: use evidence-based triage, execute mitigations, communicate findings, and implement corrective actions from postmortems.
Implement and test disaster recovery: design backup, restore, and failover mechanisms against agreed recovery time and recovery point objectives; run recovery exercises and document measured results.
Build infrastructure-as-code and operational automation: make provisioning, configuration, deployments, upgrades, and recovery reproducible, reviewed, testable, and recoverable. Eliminate recurring manual work through code.
Build observability that makes production diagnosable: improve metrics, logs, traces, dashboards, and actionable alerts, with visibility into service health, scaling limits, and customer impact.
Deliver safe infrastructure migrations: design phased rollouts, compatibility checks, validation, and rollback paths for changes to shared production systems.
Improve cloud cost efficiency through technical changes: right-size resources, improve utilization, tune autoscaling and storage, and quantify savings while maintaining reliability and performance targets.
Build and evaluate AI-assisted operational tooling: apply AI to investigation, runbooks, anomaly analysis, and repetitive operations, with bounded permissions, reviewable actions, and measurable improvements in accuracy or toil.
Make technical decisions clear and executable: write architecture proposals, evaluate technology tradeoffs through prototypes and benchmarks, review changes affecting shared infrastructure, and document how systems operate and fail.