Senior Data Platform Engineer — Kafka & PostgreSQL

Mirantis•Remote, REMOTE

About The Position

We are looking for an experienced Senior Data Platform Engineer to own the event-streaming, transactional database, and custom connectivity backbone driving the k0rdent-ai platform — our multi-tenant control plane for enterprise GPU infrastructure. Every cluster provisioned, every GPU-hour consumed, and every tenant action produces high-cardinality metadata that must be transactionally recorded and securely isolated. You will architect the core pipeline connecting our streaming data plane (Apache Kafka via Strimzi) to our relational data tier (Open Source PostgreSQL via CloudNativePG), which runs across both elastic Kubernetes clusters and high-performance bare metal hardware. Beyond standard administration, a substantial focus of this role is custom ecosystem engineering: deep Go (Golang) systems programming to build custom open-source clients and connectors, design specialized integration libraries, and develop proprietary middleware that bridges our database multi-tenancy with enterprise identity planes like Active Directory (AD)/LDAP. Because this system is the transactional record of every tenant action across the platform, this role also owns its disaster-recovery posture — designing how the platform survives a full region or cluster loss, not just a single-node failure.

Requirements

10+ years in systems, software, data, or platform engineering, with a track record of owning large-scale, business-critical data infrastructure.
Go (Golang) or similar engineering skills — comfortable writing clean, concurrent systems code, building custom client libraries, and interacting directly with database/network APIs.
5+ years managing native PostgreSQL. Deep knowledge of database internals (WAL streams, logical replication, publications/subscriptions, and connection architectures) across both Kubernetes stateful environments and bare metal physical nodes.
Hands-on experience running PostgreSQL on Kubernetes via a mature Postgres operator — CloudNativePG (CNPG) preferred, but equivalent experience with Zalando's postgres-operator or a comparable operator is acceptable.
Strong experience managing Apache Kafka at scale on Kubernetes via a declarative operator — Strimzi preferred, but equivalent experience with Confluent for Kubernetes, Koperator, or a comparable operator is acceptable.
Proven experience tuning Kafka Connect and Debezium pipelines. Experience writing or contributing to open-source database connectors, Kafka client libraries, or integration frameworks.
Hands-on experience building integrations connecting distributed systems to an enterprise IAM/SSO provider — Active Directory, LDAP, Okta, Azure AD, or a generic OIDC provider all count; the specific product matters less than real experience integrating application-level authorization with an enterprise identity system.
Strong grasp of K8s primitives (Pod lifecycles, Operators, CSI storage layers, and CNI overlay routing), backed by production experience in AWS or GCP.
Bachelor's degree in Computer Science & Engineering (or a related field), or 10+ years of equivalent related experience.

Nice To Haves

Experience designing and operating cross-region disaster recovery and failover orchestration for stateful systems (Postgres and/or Kafka), including split-brain prevention and RPO/RTO trade-off decisions.
Active contributions to open-source Kafka connectors, PostgreSQL operators (e.g., CloudNativePG, Zalando), or Go-based database libraries.
Experience with schema governance for event streams (e.g., Karapace or Confluent Schema Registry) and compatibility-mode management for evolving event contracts.
Familiarity with high-cardinality telemetry ingestion using Apache Flink or Kafka Streams.
Familiarity with US enterprise compliance benchmarks (SOC 2, ISO 27001).
Hands-on knowledge of open-source Redis (caching, pub/sub, or as a lightweight data store) — a huge plus.
Hands-on knowledge of open-source MongoDB (document modeling, replication, sharding) — a huge plus.

Responsibilities

Design, deploy, and operate high-availability Open Source PostgreSQL topologies (via CloudNativePG on Kubernetes and native Patroni-style topologies on bare metal) and distributed Apache Kafka clusters (via Strimzi), across both containerized Kubernetes environments and high-performance bare metal hardware.
Write custom, enterprise-grade open-source Kafka clients, standalone connector services, and utility libraries from scratch using preferably Go (Golang) to extend data capabilities where off-the-shelf tooling falls short.
Architect and build system integrations connecting PostgreSQL authentication and row-level security (RLS) policy evaluation to enterprise Active Directory (AD), LDAP, and OIDC identity providers — via role-mapping and session-context layers (e.g., mapped Postgres roles, JWT claims consumed by RLS policies) rather than a direct connection.
Scale high-throughput Change Data Capture (CDC) pipelines via Debezium and Kafka Connect. Implement resilient architectural patterns to maintain absolute data integrity between databases and topics without dual-write risk.
Design and operate cross-region failover and disaster-recovery orchestration for both PostgreSQL and Kafka — including replication topology (sync vs. async trade-offs), split-brain prevention via quorum/witness mechanisms, and explicit RPO/RTO targets for a full region or cluster loss, not just single-node HA.
Optimize PostgreSQL instances for heavy ingestion and zero-downtime operations. Tune Write-Ahead Logs (WAL), logical replication streams, connection pooling (PgBouncer), and configure underlying Kubernetes infrastructure primitives (CSI storage volumes and CNI network paths) to eliminate replication lag and unnecessary cross-node latency.
Maintain a strictly declarative infrastructure-as-code (IaC) culture using Terraform and ArgoCD, creating self-service workflows so internal product teams can securely provision databases, topics, schemas, and ACLs through code.
Implement strict multi-tenant isolation, combining database-level row-level security (RLS) with CNI network policies, mTLS, and Kafka topic-level RBAC.