Kafka Platform Engineer

Smart IMS•Austin, TX

3d•$60 - $80•Hybrid

About The Position

As a Kafka Platform Engineer, you will be responsible for designing, implementing, securing, and optimizing enterprise-scale Kafka environments across both on-premises and cloud platforms. This role requires deep expertise with the Confluent Kafka Platform, distributed systems, event-driven architectures, and platform engineering best practices to support mission-critical data streaming and integration initiatives. You will serve as a trusted technical advisor for Kafka architecture, automation, DevOps enablement, AI-driven engineering practices, and enterprise integration strategies. The ideal candidate will combine strong platform engineering skills with hands-on experience leveraging AI-powered development tools and building AI-driven agents to improve operational efficiency, automation, and software delivery outcomes.

Requirements

10+ years of experience in distributed systems, messaging technologies, platform engineering, or enterprise infrastructure environments.
Extensive hands-on experience with the Confluent Kafka Platform in both on-premises and cloud deployments.
Strong expertise in Kafka architecture, administration, security, performance tuning, troubleshooting, and operations.
Experience designing secure Kafka environments with authentication, authorization, encryption, and network security controls.
Strong knowledge of event-driven architecture, streaming platforms, data ingestion, and enterprise integration patterns.
Experience with schema management technologies including Avro, Protobuf, and JSON Schema.
Strong background in Linux/Unix systems administration, operating system performance, networking, and security.
Proven experience with automation technologies including Python, Ansible, shell scripting, and operational tooling.
Experience integrating Kafka platforms with DevOps and CI/CD tools such as Azure DevOps, Jenkins, Puppet, Chef, or similar platforms.
Hands-on experience leveraging AI-powered development tools and technologies to enhance engineering productivity and operational efficiency.
Experience building AI-driven agents, intelligent workflows, automation frameworks, and agentic orchestration solutions.
Strong understanding of GitHub Copilot, prompt engineering, AI-assisted software development, and AI-enabled workflow optimization.
Proven ability to lead production incident management, outage response, root cause investigations, and remediation efforts.
Excellent written, verbal, presentation, and stakeholder management skills.
Strong organizational, collaboration, leadership, and relationship management abilities.

Nice To Haves

Experience supporting large-scale enterprise platforms in regulated environments is preferred.
Confluent Kafka certifications, cloud platform certifications, or related industry certifications are highly desirable.

Responsibilities

Design and implement secure Kafka architectures, including authentication, authorization, encryption, network isolation, and access controls.
Define and support high availability, disaster recovery, replication, backup, and resiliency strategies for Kafka environments.
Ensure Kafka platforms comply with enterprise security, governance, and regulatory standards.
Architect and support enterprise event streaming, data ingestion, batch processing, and real-time streaming solutions.
Provide guidance on data modeling, schema governance, and serialization technologies including Avro, Protobuf, and JSON Schema.
Resolve complex Kafka integration challenges involving multiple enterprise platforms and systems.
Design and develop AI agents, custom skills, and intelligent automation frameworks to support engineering and operational workflows.
Leverage AI-powered development tools such as GitHub Copilot and similar technologies to accelerate software delivery, testing, automation, and code quality improvements.
Design multi-agent orchestration frameworks and agentic workflows to address distributed systems and operational challenges.
Drive automation-first operations using Python, Ansible, scripting, and infrastructure automation tools.
Build and maintain self-service operational platforms and tools to reduce administrative overhead.
Integrate Kafka platform operations with CI/CD pipelines and DevOps ecosystems.
Lead investigations into complex production incidents and platform outages.
Perform root cause analysis, document findings, implement corrective actions, and drive issue resolution through closure.
Collaborate with engineering, operations, architecture, and leadership teams to provide technical guidance and strategic recommendations.
Communicate complex technical concepts effectively to technical and non-technical stakeholders.
Partner with internal teams and external vendors to deliver scalable, secure, and reliable platform solutions.