Apache Spark Developer

Bright Vision Technologies•Overland Park, KS

1d•$125,000 - $185,000•Remote

About The Position

Bright Vision Technologies is seeking an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting enterprise analytics, machine learning, real-time reporting, and cloud-based data platforms. The role involves building high-performance Spark applications that process large volumes of structured and semi-structured data while delivering scalable, reliable, and cost-efficient data pipelines. The developer will collaborate with data architects, data engineers, cloud platform teams, machine learning engineers, and business intelligence developers to create modern data processing solutions using Apache Spark, cloud-native technologies, and distributed computing frameworks. The role is a hands-on engineering position contributing to distributed system architecture, Spark application development, cloud migration, performance optimization, production support, and ongoing improvement of enterprise-scale data platforms. The team works on cloud-native big data platforms supporting enterprise analytics, AI, and business intelligence. Project areas include lakehouse implementation, real-time streaming analytics, customer analytics, financial risk and fraud detection, healthcare analytics, cloud migration, machine learning pipelines, and enterprise reporting. The position is 100% remote in the U.S., full-time, direct W2, with an annual salary range of $125,000–$185,000. The employer cannot sponsor new H-1B visa petitions; U.S. citizens, Green Card holders, EAD holders, and H-1B transfer candidates are encouraged to apply.

Requirements

Six or more years of professional software or data engineering experience.
Four or more years of hands-on Apache Spark development experience in enterprise production environments.
Strong proficiency in PySpark, Scala, or Spark SQL for distributed data processing.
Deep understanding of Apache Spark architecture, including RDDs, DataFrames, Datasets, Catalyst Optimizer, DAG execution, and Tungsten engine.
Strong experience with distributed computing concepts, including partitioning, shuffling, caching, broadcast joins, and fault tolerance.
Advanced SQL skills with databases such as SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.
Experience with Hadoop ecosystem technologies including Hive, HDFS, YARN, and Parquet.
Experience processing streaming data using Spark Structured Streaming, Apache Kafka, or Event Hubs.
Hands-on experience with cloud platforms including Azure Databricks, AWS EMR, AWS Glue, Azure Synapse Analytics, or Google Dataproc.
Experience integrating Spark applications with Delta Lake, Apache Iceberg, or Apache Hudi.
Strong understanding of data warehousing concepts, dimensional modeling, and data lake architecture.
Experience using Git, CI/CD pipelines, Azure DevOps, GitHub Actions, or Jenkins.
Strong debugging, troubleshooting, and Spark performance tuning skills.
Experience working in Agile Scrum development environments.

Nice To Haves

Experience building enterprise Lakehouse architectures using Databricks or Delta Lake.
Familiarity with Apache Airflow, Azure Data Factory, AWS Step Functions, or Control-M for workflow orchestration.
Experience with machine learning workflows using Spark MLlib, MLflow, or feature engineering pipelines.
Knowledge of Kubernetes, Docker, and containerized Spark deployments.
Experience implementing data quality frameworks using Great Expectations or Deequ.
Familiarity with Apache NiFi, Apache Flink, Trino, or Presto.
Experience working with cloud object storage including Amazon S3, Azure Data Lake Storage (ADLS Gen2), or Google Cloud Storage.
Knowledge of Infrastructure as Code using Terraform or ARM templates.
Experience with enterprise monitoring tools including Prometheus, Grafana, Datadog, or OpenTelemetry.
Cloud certifications in Azure, AWS, Databricks, or Apache Spark-related technologies.

Responsibilities

Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
Build scalable batch and real-time ETL/ELT pipelines that process large volumes of enterprise data.
Develop Spark applications using PySpark, Scala, or Spark SQL for data transformation, aggregation, and analytics.
Optimize Spark jobs for memory utilization, partitioning, shuffle performance, and execution efficiency.
Process structured, semi-structured, and streaming data from enterprise databases, APIs, Kafka, cloud storage, and data lakes.
Develop reusable Spark libraries, data processing frameworks, and metadata-driven ingestion pipelines.
Collaborate with cloud engineering teams to deploy Spark workloads on Databricks, EMR, Azure Synapse, or Kubernetes.
Implement data quality validation, reconciliation, monitoring, and automated error handling across distributed pipelines.
Integrate Spark applications with enterprise data warehouses, lakehouses, and reporting platforms.
Participate in architecture reviews, code reviews, technical design discussions, and Agile development activities.
Troubleshoot production issues involving distributed processing, cluster performance, resource utilization, and data quality.
Support cloud migration initiatives by modernizing legacy ETL workloads into Spark-based architectures.