Big Data Platform Engineer

ZipStaff•Jersey City, NJ

19h•Onsite

About The Position

ZipStaff is hiring a Big Data Platform Engineer for a leading global financial services firm. This 12-month contract role focuses on managing, supporting, and optimizing enterprise-grade big data platforms, with primary emphasis on Apache Spark, HDFS, and Kafka. The engineer will administer production environments and infrastructure, including monitoring, performance tuning, security, and lifecycle operations. Cloudera environment administration is preferred but secondary to strong expertise in the underlying platforms. The position is fully onsite in Jersey City, New Jersey, and candidates must be local to the area.

Requirements

At least 3 years of hands-on production administration experience with Apache Spark, HDFS, and Kafka.
Strong demonstrable expertise in Spark administration, tuning, and troubleshooting.
Strong demonstrable expertise in HDFS administration, including storage management, security, and performance.
Strong demonstrable expertise in Kafka administration, including topic management, producer/consumer architecture, performance, and security.
Strong Linux and command-line skills.
Experience with Grafana and Splunk for monitoring and performance analysis.
Solid knowledge of Kerberos, SSL/TLS, and enterprise IAM/LDAP.
Excellent problem-solving, analytical, and communication skills.
Must be local to the Jersey City, NJ area for the fully onsite role.

Nice To Haves

Experience administering Spark, HDFS, and Kafka within a Cloudera Hadoop/CDP environment, including Cloudera Manager.
Cloudera or equivalent certifications.
Automation experience with Python, Bash, or Perl.

Responsibilities

Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads, including job performance, resource management, capacity planning, and production operations.
Administer HDFS capacity, performance, availability, permissions, replication, and troubleshooting, applying a strong understanding of HDFS architecture and its integration with distributed compute platforms.
Deploy, configure, manage, monitor, and troubleshoot Kafka clusters, topics, producers, and consumers, addressing performance, capacity, availability, security, and production operations.
Monitor and tune platform performance using Grafana, Splunk, and related tools, and resolve bottlenecks across Spark, HDFS, and Kafka.
Implement and maintain security and compliance controls, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integration.
Automate operational tasks using Python, Bash, or Perl.
Provide expert troubleshooting and production support, including participation in an on-call rotation.
Perform platform administration, including installs, patching, upgrades, migrations, and lifecycle management with minimal downtime; support Cloudera Hadoop/CDP administration as a secondary focus.