Data Flow Engineer, Warsaw (Near site) – EU Public Organisations

The White Team•Capon Bridge, WV

13h•Hybrid

About The Position

The Data Flow Engineer will be responsible for defining, designing, implementing, and maintaining complex data flows, primarily using Cloudera DataFlow (Apache NiFi). This role involves developing ingestion, transformation, routing, and egress pipelines, as well as building and optimizing real-time and near-real-time CDC pipelines. The engineer will integrate external systems, manage data schemas, and ensure reliable data delivery. Additionally, the position requires configuring and managing data governance and security using Apache Atlas and Apache Ranger, monitoring pipeline performance, and collaborating with stakeholders. The role also includes creating technical documentation and participating in system upgrades.

Requirements

Minimum level of education: Level 6.
Minimum English language skills (CEFR): B2.
Minimum IT relevant professional experience (years): 8.
Minimum experience at similar position (years): 6.
Security Clearance.
At least one of the following certifications: Cloudera Certified Developer for Apache NiFi or equivalent certification, or Cloudera DataFlow (CFM) related certification or equivalent certification.
Expert knowledge in defining, designing, implementing, and maintaining complex data flows in Apache NiFi (Cloudera DataFlow).
Advanced Python programming skills for data processing, NiFi custom logic, automation, and integrations.
Advanced experience in REST API–based integrations, including authentication (OAuth/JWT), rate limiting, and error handling.
Hands‑on experience in building CDC‑based data flows using native NiFi processors, connectors, and SQL Builder.
Good knowledge of Apache Iceberg (tables, schema evolution, partitioning).
Knowledge of data governance and cataloging in CDP, including Apache Atlas (metadata, lineage, tagging) and Apache Ranger (authorization, security policies).
Experience with Apache Kafka as messaging backbone (topics, producers/consumers, schema registry, NiFi integration).
Practical knowledge of Apache Avro as serialization standard, including schema evolution and compatibility.
Minimum 2–3 years hands‑on daily experience with Apache NiFi, preferably in a Cloudera Data Platform (CDP) environment (design, deployment, monitoring, troubleshooting of advanced flows).
Documented experience delivering at least one large‑scale integration project using NiFi as the central integration tool.
Practical experience with Apache Iceberg in CDP environments (table management, integration with NiFi / Spark / Flink).
Proven experience implementing CDC pipelines to and from relational databases.
Practical knowledge of configuring Apache Atlas and Ranger in the context of NiFi flows (tagging, policies, auditing).
Experience working with Kafka in CDP ecosystems, including schema management with Avro and downstream integrations.

Responsibilities

Design, implement, test, and maintain complex data flows in Cloudera DataFlow (Apache NiFi).
Develop ingestion, transformation, enrichment, routing, and egress pipelines.
Build and optimize real-time and near-real-time CDC pipelines using NiFi, Kafka, and Debezium / SQL CDC connectors.
Integrate external systems using REST APIs, JDBC, Kafka, and other protocols.
Manage and evolve data schemas using Apache Avro.
Ensure reliable delivery to downstream consumers and analytical platforms.
Configure and manage metadata, lineage, and governance using Apache Atlas.
Define and maintain security and authorization policies using Apache Ranger.
Monitor, alert, and troubleshoot performance, reliability, and data quality of pipelines.
Collaborate with data engineers, architects, and business stakeholders on requirements and data flow architecture.
Create and maintain technical documentation, SOPs, and operational runbooks.
Participate in CDP, NiFi, and Kafka upgrades and migration activities.
Perform other duties as assigned by the team leader.