Senior Principal Performance Engineer
About The Position
The Information Technology group delivers secure, reliable technology solutions that enable DTCC to be the trusted infrastructure of the global capital markets. The team delivers high-quality information through activities that include development of essential, building infrastructure capabilities to meet client needs and implementing data standards and governance. As a Senior Principal Performance & Resiliency Engineer, you will play a critical role in ensuring the reliability, scalability, and operational excellence of DTCC's most business-critical platforms. You will lead the strategy, architecture, and execution of performance engineering, resilience testing, capacity planning, and operational readiness initiatives across complex distributed systems and cloud-native environments. In this role, you will partner closely with engineering, infrastructure, architecture, and product teams to proactively identify performance bottlenecks, eliminate systemic risks, and enhance platform resilience. Your leadership will help ensure that critical financial market infrastructure systems can meet evolving business demands, regulatory expectations, and client service commitments while maintaining the highest standards of availability and reliability. You will drive a culture of engineering excellence by championing observability, automation, chaos engineering, disaster recovery preparedness, and continuous performance optimization. Your contributions will directly strengthen DTCC's ability to deliver secure, resilient, and highly available services that support global financial markets.
Requirements
- Minimum of 10 years of related experience
- Bachelor's degree preferred or equivalent experience
- 10+ years of hands-on performance engineering and reliability experience in latency-sensitive or high-volume transaction systems, with continued depth in profiling, dump analysis, and test harness development.
- Strong performance engineering discipline across load, stress, soak, and spike testing, using workload models grounded in real production traffic and analyzing results through p99 and p99.9 behavior rather than averages.
- Deep diagnostic capability across JVM tuning, heap and thread dump analysis, flame graphs, async profiling, and bottleneck isolation across application, database, network, and storage layers—even when standard dashboards appear healthy.
- Resiliency and chaos engineering expertise including fault injection, failure-mode cataloging, game day design, and DR exercises supported by measured RTO/RPO evidence.
- Distributed systems and data platform performance depth spanning Kafka throughput, consumer lag, rebalancing, partition skew, backpressure, exactly-once tradeoffs, object storage, lakehouse query performance, and Snowflake warehouse tuning.
- Observability architecture experience covering OpenTelemetry, distributed tracing, metrics taxonomy, SLI/SLO and error budget design, actionable alerting, and disciplined control of cardinality and telemetry cost.
- Automation embedded into delivery pipelines through CI performance gates, automated baselining, regression detection, and practical test data and environment strategies for constrained end-to-end environments.
- Broad tooling proficiency with engineering depth across JMeter, Gatling or k6; Prometheus and Grafana; Dynatrace, Datadog or Splunk; and the ability to build custom harnesses when standard tools do not fit the workload.
- Regulated environment fluency including DR obligations, production readiness reviews, audit evidence, and disciplined incident and postmortem practices.
- Influence without authority with the credibility to set and enforce NFR standards across delivery squads and support release-readiness decisions with clear, evidence-based judgment.
Responsibilities
- Make NFRs measurable and release-gated. Define clear targets for latency, throughput, availability, recovery, and capacity, and validate them under production-like volume before release.
- Embed continuous performance and resilience validation. Automate regression testing and failure-injection scenarios in CI so reliability issues are identified before production impact.
- Establish estate-wide observability and readiness standards. Create a common instrumentation, SLI, and capacity-planning baseline so risk platforms can scale, recover, and be operated consistently across the estate.
Benefits
- Competitive compensation, including base pay and annual incentive
- Comprehensive health and life insurance and well-being benefits, based on location
- Pension / Retirement benefits
- Paid Time Off and Personal/Family Care, and other leaves of absence when needed to support your physical, financial, and emotional well-being.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
Associate degree
