Big Data Platform Engineer – Jersey City, NJ

Codetech • Jersey City, NJ

8h

About The Position

Seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage, support, and optimize enterprise-grade big data platforms. Deep hands-on expertise with Apache Spark, HDFS, and Kafka is required and is the primary focus of the role. Experience administering these technologies within a Cloudera environment is preferred but secondary to expertise in the underlying technologies. The role also includes Cloudera Hadoop/CDP administration as a secondary responsibility.

Requirements

• Minimum 3 years of hands-on production experience administering and supporting Apache Spark, HDFS, and Kafka.
• Strong, demonstrable expertise in Spark administration, configuration, performance tuning, and troubleshooting.
• Strong, demonstrable expertise in HDFS administration, storage management, security, performance, and troubleshooting.
• Strong, demonstrable expertise in Kafka administration, topic management, producer/consumer architecture, performance, security, and troubleshooting.
• Experience with Linux operating systems and command-line tools.
• Experience with enterprise monitoring and operational tooling such as Grafana and Splunk.
• Solid understanding of Kerberos, SSL/TLS, IAM/LDAP, and security concepts for distributed data platforms.
• Excellent problem-solving, analytical, and communication skills.

Nice To Haves

• Experience with Cloudera Hadoop/CDP and Cloudera Manager.
• Cloudera certifications or equivalent big data certifications.

Responsibilities

• Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads; support job performance, resource management, capacity planning, and production operations.
• Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting.
• Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers; support performance, capacity, availability, security, and production operations.
• Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and relevant platform management tools; identify and resolve performance bottlenecks across Spark, HDFS, and Kafka.
• Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.
• Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.
• Provide expert-level troubleshooting and technical support for Spark, HDFS, Kafka, and related big data technologies, including participation in an on-call rotation.
• Support installations, patching, upgrades, migrations, and platform lifecycle activities while minimizing downtime.
• Perform Cloudera Hadoop/CDP administration.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed