Hadoop Solutions Developer
About The Position
Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data processing pipelines and analytics platforms using Hadoop and related big-data technologies. The role involves ingesting, transforming, and analyzing structured and unstructured data to support enterprise analytics, machine learning, and reporting workloads, while delivering reliable, performant, and cost-effective production data platforms. This is a full-time, direct W-2 position with Bright Vision Technologies, working remotely within the continental United States on the company’s in-house Statement of Work engagement. The position does not involve a third-party client, vendor, or implementation partner. Candidates must work directly as Bright Vision Technologies employees; C2C, 1099, and third-party arrangements are not accepted. New H-1B sponsorship is unavailable, though H-1B transfers are supported for qualified candidates. A technical coding assessment is mandatory.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
- At least five years of professional experience designing and operating big-data pipelines on Hadoop.
- Strong production experience with Apache Spark using Scala, Python, or Java.
- Experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
- Hands-on experience with streaming platforms such as Kafka, Spark Streaming, or Flink.
- Strong SQL skills and experience with relational and NoSQL data stores.
- Experience with workflow orchestration tools such as Airflow or Oozie.
- Understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
- Strong Python or Shell scripting skills.
- Strong troubleshooting, debugging, and documentation skills.
- Willingness to work directly as a full-time W-2 employee of Bright Vision Technologies; C2C, 1099, and third-party arrangements are not accepted.
- Ability and willingness to complete a mandatory technical coding assessment.
Nice To Haves
- Experience operating Hadoop on cloud platforms such as AWS EMR, Azure HDInsight, or Databricks.
- Familiarity with modern lakehouse formats such as Delta, Iceberg, or Hudi.
- Exposure to data governance tools such as Apache Atlas or Collibra.
- Experience with Kubernetes-based data platforms such as Spark-on-K8s or Trino.
- Hands-on experience with CI/CD and infrastructure-as-code in data engineering workflows.
Responsibilities
- Design, develop, and operate end-to-end big-data pipelines on Hadoop using relational, file-based, streaming, and API-driven data sources.
- Build ETL/ELT workflows with Apache Spark, Hive, Pig, and Sqoop, focusing on data quality, idempotency, error handling, and recoverability.
- Develop high-throughput streaming pipelines using Kafka, Spark Streaming, or Flink and integrate them with analytical and operational systems.
- Tune Spark and MapReduce jobs, including partitioning, memory, serialization, and skew handling, to meet SLAs efficiently.
- Design and maintain data models and storage layouts on HDFS, Hive, HBase, and lakehouse formats such as Parquet, ORC, Delta, Iceberg, and Hudi.
- Implement data governance, lineage, and quality controls in collaboration with governance and security teams.
- Create monitoring, alerting, and logging strategies for pipelines, including job-level SLAs and proactive failure detection.
- Partner with data scientists and analysts to provide curated, reliable, and documented datasets.
- Automate pipeline orchestration with Airflow, Oozie, or similar workflow engines.
- Evaluate big-data and cloud technologies, including EMR, Databricks, Snowflake, and BigQuery, when they offer meaningful improvements.
- Lead performance reviews and architecture audits, and propose pipeline refactoring and optimization initiatives.
- Document data architectures, schemas, pipeline behavior, and operational runbooks.
- Mentor junior engineers and contribute to engineering standards and best practices.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
