Principal Software Engineer

Crisis Prevention Institute•Milwaukee, WI

10d

About The Position

The Principal Software Engineer is the senior-most individual contributor in CPI's Product Engineering organization. The role owns the technical architecture and long-term health of the Integrated Training and Learning (ITL) platform, a cloud-native learning platform comprising more than 30 C#/.NET microservices using CQRS and event-driven patterns, Azure Service Bus, Azure SQL, and a React single-page application. The platform serves CPI learners, Certified Instructors, and customer administrators worldwide. This is a hands-on principal role responsible for setting the platform's technical vision, making cross-cutting architectural decisions, and working directly with product squads and the Platform team through prototyping, pairing, code review, and delivery. Success is measured by platform outcomes such as deployment frequency, reliability, cost efficiency, and feature-delivery speed. The role also models curiosity, ownership, drive, experimentation, learning from failure, and continuous improvement. The Principal Software Engineer owns the ITL target architecture and roadmap, including service boundaries, event and API contracts, data ownership, integration patterns, and front-end architecture. The role leads efforts to decouple ITL domain data from upstream enterprise systems, evolve the event-driven backbone, and establish practical standards for CQRS, domain-driven design, APIs, data modeling, security, and observability. It evaluates technical and platform options using cost, risk, and time-to-value analysis, documents decisions in lightweight ADRs, and uses operational and cost data to guide architectural investments. The engineer contributes production code, prototypes patterns, builds reference implementations, pairs on difficult problems, leads technical design reviews, and conducts time-boxed experiments and proofs of concept. The role is the final technical escalation point for complex production incidents and leads blameless reviews and architectural remediation. It also leads the strategy for AI-assisted software development across Product Engineering and identifies opportunities to apply AI in the ITL product. The role mentors Lead and Senior engineers, facilitates architecture guilds and technical talks, partners with the VP of Engineering, engineering managers, product leadership, and the Platform team, and aligns technical strategy with product and business strategy. It provides direction through influence and evidence, communicates architecture in business and engineering terms, stays current with relevant technologies, and makes responsible, accountable, and defensible technical and financial decisions. Other position-related duties may be assigned.

Requirements

Bachelor's degree in computer engineering, computer science, or a related field.
At least 12 years of professional software engineering experience, including at least 5 years in a principal, staff, or architect role responsible for a multi-team product platform's architecture.
Deep hands-on expertise in C# and .NET, including ASP.NET Core and .NET 6+, designing and building production services at scale.
Demonstrated experience designing and operating production event-driven microservice architectures, including message brokers such as Azure Service Bus or equivalent, event contracts, sagas, idempotency, back-pressure, and high-volume failure handling.
Demonstrated experience applying CQRS, event sourcing, and domain-driven design to products, including defining bounded contexts and decomposing or redrawing service boundaries.
Experience leading the decoupling or migration of a product platform from a monolithic system of record, such as an ERP, CRM, or legacy application, without a big-bang cutover.
Deep Azure architecture experience, including App Service, Functions, Service Bus, Azure SQL, Azure Monitor/Application Insights, Key Vault, networking and identity, and cost management.
Yearly personal development demonstrated by successfully completing an accredited software engineering course, acquiring a software engineering-related certification, or presenting a completed learning project to the team.
Curiosity: asks why before how, explores unfamiliar code and systems, and understands the learner, instructor, and customer problem behind requests.
Ownership: takes responsibility for the health of services and features from design through production support and proactively solves problems.
Drive: ships working software in small, frequent increments and follows through to achieve outcomes.
Experimentation: proposes and runs small, time-boxed experiments and uses evidence rather than opinion to resolve technical debates.
Learning from failure: participates in blameless retrospectives and incident reviews, shares lessons, and turns them into guardrails, tests, or runbooks.
Continuous improvement: improves codebases, processes, and documentation.
Ability to set technical direction across multiple teams and gain support through influence, evidence, and demonstrated results rather than authority.
Ability to maintain a long-term architectural vision while delivering pragmatic incremental steps alongside ongoing feature work.
Ability to explain architecture in business terms such as cost, risk, speed, and reliability for executive stakeholders, and as concrete guidance for engineers.
Excellent analytical and problem-solving skills, comfort with ambiguity, and ability to make sound decisions with incomplete information.
Well-developed interpersonal, negotiation, writing, speaking, and listening skills.
Strong business acumen and strategic thinking ability.

Nice To Haves

Experience architecting learning management, EdTech, or other B2B SaaS platforms serving end users globally.
Experience with Microsoft Dynamics 365 integration patterns and Dataverse.
Experience applying large language models or other AI capabilities within a product.
Experience with infrastructure as code such as Bicep or Terraform, containers, and Azure Kubernetes Service.
Azure Solutions Architect Expert certification or equivalent.
Public speaking, technical writing, or open-source contributions in distributed systems or .NET.

Responsibilities

Own the target architecture for the ITL platform and the roadmap for service boundaries, bounded contexts, event and API contracts, data ownership, integration patterns, and front-end architecture.
Lead the decoupling of ITL domain data from upstream enterprise systems so ITL can own production data, deploy independently, and release multiple times per day.
Design and evolve the platform's event-driven backbone, including Azure Service Bus event taxonomy and routing, filtering, prioritization, idempotency, saga and retry strategies, throughput, and cost optimization.
Define and enable standards for CQRS, domain-driven design, API design, data modeling, security, and observability through templates, reference implementations, and tooling.
Evaluate build-versus-buy and platform decisions using cost, risk, and time-to-value analysis, and document durable decisions in lightweight architecture decision records.
Partner with the Platform team on distributed tracing, service-level dashboards, and cost visibility, and use the results to prioritize architectural investments.
Write production code, prototype patterns, build reference implementations, pair with engineers on difficult problems, and contribute to critical initiatives.
Lead technical design reviews across squads, review code for architectural consistency, and teach through code review.
Run time-boxed experiments and proofs of concept to reduce risk around major decisions, and discontinue ideas unsupported by evidence.
Serve as the final technical escalation point for complex production incidents, lead blameless reviews of systemic failures, and drive architectural remediation.
Lead technical strategy for AI-assisted software development, including tooling selection, guardrails, quality and productivity evaluation, and shared practices.
Identify and architect opportunities to apply AI in the ITL product that create measurable value for learners, instructors, and customers.
Mentor Lead and Senior engineers, run architecture guilds and tech talks, and develop future technical leaders.
Partner with engineering and product leaders and the Platform team to align technical strategy with product and business strategy and communicate business outcomes to executive stakeholders.
Provide direction to peers and shape decisions through evidence, customer knowledge, and CPI business context.
Stay current on .NET, Azure, distributed systems, and AI, and apply relevant developments pragmatically.
Make responsible, accountable, justifiable, and defensible technical and financial decisions in accordance with organizational policies and procedures.
Perform other position-related duties as assigned.