Principal Software Quality Engineer

Weekday AI

5d•Remote

About The Position

This is a senior technical individual-contributor role defining and driving quality strategy for a distributed database platform. The role requires hands-on expertise in distributed database testing, systems engineering, debugging, and test automation. The engineer will work across database server components, client SDKs, operators, backup tooling, and management interfaces, collaborating with QE, client, and server engineering teams to improve product reliability. The role also leads integration of AI agents into testing frameworks for test generation, failure triage, and evaluation of agent reliability. It is not a manual QA or people-management position; the engineer is expected to write framework code, debug server processes, investigate low-level system behavior, evaluate AI-agent performance, and build reliable testing infrastructure.

Requirements

10–18 years of experience in Software Quality Engineering, SDET, or Systems Engineering, focused on quality systems and with demonstrated Staff- or Principal-level technical impact.
Deep hands-on experience testing distributed databases or storage systems, preferably NoSQL platforms; distributed relational database experience is also relevant.
Strong experience testing replication, partitioning, sharding, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication.
Solid understanding of distributed-systems fundamentals, including replication protocols, consensus, quorum reads and writes, strong and eventual consistency, split-brain scenarios, clock skew, and failure/recovery semantics.
Ability to read and debug server-side code written in C, C++, Go, Rust, or Java and trace defects to specific functions and root causes.
Understanding of protocol and serialization edge cases, including wire formats, MessagePack, Protocol Buffers, Thrift, and compression codecs.
Strong Python software engineering and automation skills, with proficiency in at least one systems programming language.
Practical experience with fault injection, GDB, chaos-style testing, process termination, network partitions, latency and packet-loss injection, disk faults, corrupted inputs, version skew, and boundary-value testing.
Experience across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and soak testing.
Ability to lead debugging and root-cause investigations across QE, client, and server teams, provide actionable technical analysis, and follow defects through resolution.
Strong CI/CD experience with GitHub Actions, Jenkins, or equivalent tools, and experience managing shared test infrastructure.
Practical experience using AI agents for testing, including agent workflows, tool integration, prompt and evaluation design, and integration with real-world testing frameworks.
Excellent written and verbal communication skills, including the ability to produce clear technical tickets, investigation reports, Slack discussions, and RFCs.

Nice To Haves

Experience testing client SDKs across multiple programming languages, including Python, Java, Go, C, Node.js, and .NET.
Knowledge of compatibility matrices, upgrade/downgrade scenarios, and behavioral consistency across client SDKs.
Experience testing backup and restore, checkpointing, snapshotting, and WAL/journal correctness, including performance at terabyte scale.
Expertise in query-engine and secondary-index testing, including query planning, index selection, pagination, boundary values, and encoding/collation correctness.
Kubernetes and database operator testing experience using tools such as KUTTL, Chainsaw, or envtest.
Cloud infrastructure experience across AWS, GCP, or Azure, including TLS/mTLS, IAM, secrets management, and air-gapped installations.
Security-focused testing experience, including fuzzing, CVE reproduction, and identifying memory-safety issues such as out-of-bounds reads and use-after-free defects in native code.

Responsibilities

Own the quality strategy for the distributed database platform, including test architecture, coverage models, release gates, and risk-based prioritization.
Drive quality across server components, client SDKs, operator and backup tooling, and management interfaces.
Design testing strategies for replication, rebalancing, partitioning, sharding, consistency, transactions, isolation, secondary indexes, query planning, compaction, backup and restore, snapshots, and cross-datacenter replication.
Establish reliable coverage for critical database operations and incorporate quality requirements early in development.
Validate cluster correctness under production-like conditions, including node failures, network partitions, clock skew, disk-full scenarios, slow disks, rolling upgrades, mixed-version deployments, and soak tests.
Design and execute fault-injection scenarios involving process crashes, GDB-driven state manipulation, packet drops, latency injection, corrupted wire data, oversized inputs, and version skew.
Investigate complex system defects, reproduce them at source-code level, identify suspect functions, and provide minimal reproducible cases and root-cause analysis.
Build and maintain scalable testing frameworks using Python, pytest, custom harnesses, API testing, and end-to-end automation.
Ensure test suites run reliably across shared test infrastructure and CI/CD environments, and improve test-farm reliability by reducing flaky tests and strengthening regression coverage.
Develop strategies across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and long-running regression testing.
Lead integration of AI agents into testing frameworks, including workflows for planning and executing tests, interacting with tools, inspecting logs, traces, metrics, and database state, and generating defect reports.
Develop evaluation frameworks using golden datasets, graders, and regression suites to measure the reliability and accuracy of AI-generated outputs.
Integrate AI agents with pytest, custom testing harnesses, and MCP-style tool servers; enable assistance with edge-case generation, flaky-test maintenance, and failure triage while retaining human oversight where required.
Participate in feature reviews, product requirements discussions, design reviews, and client specifications to define testability requirements before implementation.
Create test plans for new features and collaborate with development teams to validate technical designs.
Turn production and pre-release incidents into regression tests, monitoring checks, and preventive measures.
Lead debugging and defect-resolution efforts across QE, client, and server engineering teams.
Mentor senior QEs and SDETs in debugging techniques, test architecture, and framework development.
Establish engineering practices and quality standards that teams can adopt without direct people-management responsibilities.