Data Flow Engineer (Warsaw, 40% remote/60% onsite) – EU Public Institutions

The White Team • Capon Bridge, WV

6h • Hybrid

About The Position

The Data Flow Engineer will be responsible for defining, designing, implementing, and maintaining complex data flows using Apache NiFi (Cloudera DataFlow). This role involves working with Big Data Streaming and Flow Management, Python programming for data processing and automation, building integrations via REST APIs, and implementing Change Data Capture (CDC) based data flows. The position also requires knowledge of Apache Iceberg for data tables and schema evolution, and data governance tools like Apache Atlas and Apache Ranger. Experience with Apache Kafka as a message broker and Apache Avro for serialization is also essential. The role is based in Warsaw at Frontex Headquarters, with a hybrid work model (60% onsite, 40% remote) and a mission duration of 48 months. A security clearance of Restricted / EU Restricted is required.

Requirements

• Minimum level of education: Level 6.
• Minimum English language skills: B2.
• Minimum IT relevant experience: 8 years (6 years in relevant roles).
• Expert knowledge in defining, designing, implementing and maintaining complex data flows in Apache NiFi (Cloudera DataFlow).
• Advanced Python programming skills (data processing, NiFi custom logic, flow automation, integrations).
• Advanced in building integrations based on REST API (endpoints' calling, OAuth/JWT authentication, rate limiting, error recovery).
• Hands-on experience in building CDC-based data flows (Change Data Capture) - using native NiFi processes/connectors as well as with SQL Builder.
• Good knowledge of Apache Iceberg (tables, schema evolution, partitioning).
• Knowledge on data governance and catalog in CDP: Apache Atlas (metadata, lineage, tagging) and Apache Ranger (security policies, authorization).
• Experience with Apache Kafka as message broker (topics, producers/consumers, schema registry, NiFi integration) and Apache Avro as serialization standard (schema evolution, compatibility etc.).
• Hands-on experience in daily working with Apache NiFi, preferably in CDP environment - design, deployment, monitoring, troubleshooting of advanced flows - min. 2-3 years of hands-on.
• Documented experience in realisation of at least one big integration project using NiFi as central tool (API calling, db integrations, transformations, data routing and delivery).
• Practical knowledge and experience with Apache Iceberg, preferably in CDP environment - table creation/management, NiFi/Spark/Flink integration etc.
• Experience in implementation of CDC pipelines to/from relational databases.
• Practical knowledge of configuring and managing governance/lineage in Apache Atlas + Ranger in the context of NiFi flows (tagging, policies, audit).
• Experience in working with Apache Kafka in CDP ecosystem (integration NiFi -> Kafka -> downstream consumers, schema management with Avro).
• At least 1 certification among: Cloudera Certified Developer for Apache NiFi or an equivalent certification, Cloudera Data Flow (CFM) related certification or an equivalent certification. Each equivalent certification must be recognized internationally (subject to acceptance by the Contracting Authority).

Responsibilities

• Design, implementation, testing and maintenance of complex data flows in Cloudera DataFlow (Apache NiFi) - ingest, transform, enrich, route, egress.
• Building and optimisation of CDC-based pipelines (real-time / near-real-time) with usage of NiFi, Kafka, Debezium / SQL CDC connectors.
• Integration with external systems via REST API, JDBC, Kafka and other protocols.
• Managing data schemas (Avro), metadata and lineage in Apache Atlas.
• Configuration of security and governance (Ranger policies) for data flows.
• Monitoring, alerting and troubleshooting performance / reliability of data pipelines.
• Collaboration with data engineers, architects and business stakeholders on defining requirements and architecture of data flows.
• Creation and maintenance of SOPs and documentation for data flows, and runbooks.
• Participation in CDP / NiFi / Kafka upgrades and migrations.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Associate degree