Hadoop Solutions Developer

Bright Vision Technologies • Sugar Land, TX

1d • $100,000 - $150,000 • Remote

About The Position

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data processing pipelines and analytics platforms using Hadoop and related big-data technologies. The role involves ingesting, transforming, and analyzing structured and unstructured data to support enterprise analytics, machine learning, and reporting. The developer will help deliver reliable, performant, and cost-effective production data platforms.

This is a 100% remote, full-time, direct W-2 role with Bright Vision Technologies, based in the continental United States. It is an in-house Statement of Work engagement; Bright Vision Technologies is the employer, with no third-party client, vendor, or implementation partner. C2C, 1099, and third-party arrangements are not accepted. New H-1B sponsorship is unavailable, but qualified candidates with a valid H-1B visa may apply for a transfer. A technical coding assessment is mandatory.

The role offers a salary of $100,000–$150,000 and requires at least five years of relevant experience. A bachelor's degree in Computer Science, Engineering, or a related technical discipline is required.

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
• At least five years of professional experience designing and operating big-data pipelines on Hadoop.
• Strong hands-on expertise with Apache Spark using Scala, Python, or Java in production environments.
• Experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
• Hands-on experience with streaming platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills and experience with relational and NoSQL data stores.
• Experience with workflow orchestration tools such as Airflow or Oozie.
• Understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
• Strong scripting skills in Python or Shell.
• Excellent troubleshooting, debugging, and documentation skills.
• Must work directly as a full-time W-2 employee of Bright Vision Technologies; C2C, 1099, and third-party arrangements are not accepted.
• A technical coding assessment is mandatory.
• No new H-1B sponsorship is available; qualified candidates with a valid H-1B visa may apply for a transfer.

Nice To Haves

• Experience operating Hadoop on cloud platforms such as AWS EMR, Azure HDInsight, or Databricks.
• Familiarity with lakehouse formats such as Delta, Iceberg, or Hudi.
• Exposure to data governance tools such as Apache Atlas or Collibra.
• Experience with Kubernetes-based data platforms such as Spark-on-K8s or Trino.
• Hands-on experience with CI/CD and infrastructure-as-code in data engineering workflows.

Responsibilities

• Design, develop, and operate end-to-end Hadoop data pipelines that ingest data from relational, file-based, streaming, and API-driven sources.
• Build ETL/ELT workflows using Apache Spark, Hive, Pig, and Sqoop, with attention to data quality, idempotency, error handling, and recoverability.
• Develop streaming data pipelines using Kafka, Spark Streaming, or Flink and integrate them with downstream analytical and operational systems.
• Tune Spark and MapReduce jobs, including partitioning, memory, serialization, and skew handling, to meet SLAs while controlling costs.
• Design and maintain data models and storage layouts on HDFS, Hive, HBase, and lakehouse formats such as Parquet, ORC, Delta, Iceberg, and Hudi.
• Implement data governance, lineage, and quality controls in collaboration with governance and security teams.
• Build monitoring, alerting, and logging strategies for pipelines, including job-level SLAs and proactive failure detection.
• Partner with data scientists and analysts to provide curated, reliable, well-documented datasets.
• Automate pipeline orchestration with Airflow, Oozie, or similar workflow engines, with clear dependency management and ownership boundaries.
• Evaluate and adopt big-data and cloud technologies, including EMR, Databricks, Snowflake, and BigQuery, when they provide meaningful improvements.
• Lead performance reviews and architecture audits of existing pipelines and propose refactoring and optimization initiatives.
• Document data architectures, schemas, pipeline behaviors, and operational runbooks.
• Mentor junior engineers and contribute to engineering standards and best practices.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level