Hadoop Solutions Developer
About The Position
Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data processing pipelines and analytics platforms using Hadoop and related big-data technologies. The role focuses on ingesting, transforming, and analyzing structured and unstructured data for enterprise analytics, machine learning, and reporting, while delivering reliable, performant, and cost-effective production data platforms. This is a full-time, direct W2 position with Bright Vision Technologies on an in-house Statement of Work engagement. It is 100% remote within the continental United States, with no third-party client, vendor, or implementation partner. C2C, 1099, and third-party arrangements are not accepted. New H1B sponsorship is unavailable, although qualified candidates who need a transfer of a currently valid H1B visa may apply. A technical coding assessment is mandatory.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
- At least five years of professional experience designing and operating big-data pipelines on Hadoop.
- Strong production experience with Apache Spark using Scala, Python, or Java.
- Experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
- Hands-on experience with streaming platforms such as Kafka, Spark Streaming, or Flink.
- Strong SQL skills and experience with relational and NoSQL data stores.
- Experience with workflow orchestration tools such as Airflow or Oozie.
- Understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
- Strong scripting skills in Python or Shell.
- Strong troubleshooting, debugging, and documentation skills.
- Willingness to work directly as a full-time W2 employee and contribute to in-house SOW deliverables.
- Confidence in technical abilities and hands-on experience to complete the mandatory technical coding assessment.
Nice To Haves
- Experience operating Hadoop on cloud platforms such as AWS EMR, Azure HDInsight, or Databricks.
- Familiarity with modern lakehouse formats such as Delta, Iceberg, and Hudi.
- Exposure to data governance tools such as Apache Atlas or Collibra.
- Experience with Kubernetes-based data platforms such as Spark-on-K8s or Trino.
- Hands-on experience with CI/CD and infrastructure-as-code in data engineering workflows.
Responsibilities
- Design, develop, and operate end-to-end Hadoop big-data pipelines that ingest data from relational, file-based, streaming, and API-driven sources.
- Build ETL/ELT workflows using Apache Spark, Hive, Pig, and Sqoop, emphasizing data quality, idempotency, error handling, and recoverability.
- Develop high-throughput streaming pipelines using Kafka, Spark Streaming, or Flink and integrate them with analytical and operational systems.
- Tune Spark and MapReduce jobs, including partitioning, memory, serialization, and skew handling, to meet SLAs cost-effectively.
- Design and maintain data models and storage layouts on HDFS, Hive, HBase, and lakehouse formats such as Parquet, ORC, Delta, Iceberg, and Hudi.
- Implement data governance, lineage, and quality controls in collaboration with governance and security teams.
- Build monitoring, alerting, and logging strategies for pipelines, including job-level SLAs and proactive failure detection.
- Partner with data scientists and analysts to provide curated, reliable, well-documented datasets.
- Automate pipeline orchestration using Airflow, Oozie, or similar workflow engines, with clear dependency management and ownership boundaries.
- Evaluate and adopt big-data and cloud technologies such as EMR, Databricks, Snowflake, and BigQuery when they offer meaningful improvements.
- Lead performance reviews and architecture audits of existing pipelines and propose refactoring and optimization initiatives.
- Document data architectures, schemas, pipeline behavior, and operational runbooks.
- Mentor junior engineers and contribute to engineering standards and best practices.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
