Kafka Lead Engineer

i4DM•Millersville, MD

15h

About The Position

We are seeking a highly skilled Kafka Lead Engineer to join our team and work closely with the Army on mission-critical data stream engineering projects. In this role, you will play a key part in delivering essential data solutions in an artificial intelligence (AI) and low code no code (LCNC) architecture, helping the Army transform and optimize their data infrastructure. The ideal candidate will have a deep technical background in Kafka development and a strong focus on customer engagement, especially in adapting solutions to fit the Army's unique needs and working in an agile environment. As the Kafka Lead, you will design, develop, and implement data pipelines, streams, and connectors that organize and correlate disparate data, enabling real-time access for business consumers. Your ability to integrate Kafka technology with the Army’s mission and operational requirements is critical to the success of the project.

Requirements

Proven experience as a Kafka Engineer or Kafka Lead, with a strong background in architecting, designing and implementing Kafka-based data pipelines and real-time data streaming solutions.
Expertise in Kafka architecture, streaming technologies, and data pipeline development (including Kafka Streams, Kafka Connect, and related tools).
Expertise in Kafka Performance tuning, data tagging and data aggregation
Expertise in building DataOps practices, primarily as it relates to automating testing and deployments of Kafka based data products through integrated CI/CD pipelines.
Agile methodology experience, with the ability to adapt to fast-paced, changing environments and work closely with teams using agile frameworks (Scrum, Kanban, etc.).
Strong customer focus with the ability to engage directly with clients, understand their needs, and deliver technical solutions that align with their business goals.
Experience working in government or defense sectors is a plus, with a strong understanding of how to work in highly regulated and mission-critical environments.
Proficiency in Java, Scala, Python or other relevant programming languages for Kafka development.
Experience with Kafka-related tools (e.g., Kafka Streams, Kafka Connect, Schema Registry).
Strong understanding of data integration, ETL processes, and real-time analytics.
Excellent problem-solving, communication, and collaboration skills.
Security clearance or the ability to obtain one is preferred (depending on project requirements).

Nice To Haves

Familiarity with Army-specific systems, workflows, and organizational culture.
Experience integrating Kafka with Appian Data fabric
Experience using Kafka to automate data tagging
Experience with cloud-based solutions (e.g., AWS, Azure) and deploying Kafka on cloud platforms.
Strong experience in data modeling, distributed systems, and microservices architecture.

Responsibilities

Kafka Architecture and Engineering: Lead the design and development of Kafka-based data pipelines and streaming solutions, ensuring they meet the high availability, scalability, disconnected operations, and performance needs of the Army's mission-critical systems.
Data Integration: Develop connectors, transformations, and integration patterns that make disparate data accessible in real-time for business consumers, ensuring consistency, reliability, and speed.
Agile Collaboration: Work closely with cross-functional teams, including project managers, architects, data engineers, and Army stakeholders, to deliver solutions that align with their goals and are adaptable in an agile environment.
Customer-Focused Solutions: Engage with Army personnel to understand their workflows, operational needs, and mission goals. Tailor Kafka-based solutions to meet specific Army requirements, ensuring that the data infrastructure aligns with their unique operational context.
Mentorship and Leadership: Lead and mentor associate Kafka developers, guiding them through best practices, code reviews, and technical challenges. Provide direction to ensure all Kafka solutions are implemented efficiently and effectively.
Optimization and Troubleshooting: Monitor Kafka systems for performance, troubleshoot issues, and optimize pipeline efficiency, ensuring minimal downtime and continuous data flow.
Documentation and Best Practices: Create clear, detailed documentation of Kafka architectures, processes, and solutions. Ensure adherence to best practices, coding standards, and operational processes.