Spark / Scala Senior Engineer

TalentOla • Jacksonville, FL

4d • Onsite

About The Position

We are looking for a highly skilled Spark/Scala Senior Engineer with 8 to 10 years of experience in designing, developing, and optimizing large-scale batch and real-time data processing solutions. The ideal candidate will possess strong expertise in Scala, Apache Spark, Snowflake, Kafka/NiFi, and modern Data Engineering frameworks. This role requires hands-on technical leadership, performance optimization expertise, and the ability to mentor junior team members while driving engineering best practices.

Requirements

• 8 to 10 years of experience in designing, developing, and optimizing large-scale batch and real-time data processing solutions.
• Strong expertise in Scala, Apache Spark, Snowflake, Kafka/NiFi, and modern Data Engineering frameworks.
• Hands-on technical leadership.
• Performance optimization expertise.
• Ability to mentor junior team members.
• Driving engineering best practices.
• Strong hands-on experience with Scala development.
• Deep understanding of Scala: Case Classes, Traits, Pattern Matching, Collections Framework, Implicits.
• Strong knowledge of Functional Programming concepts: Immutability, Higher-Order Functions (HOFs), Monads.
• Experience developing and maintaining production-grade Scala applications.
• Strong expertise with Apache Spark: Spark SQL, DataFrames, Datasets.
• Extensive experience with Spark Structured Streaming.
• Deep understanding of Spark internals: DAG Execution, Lazy Evaluation, Shuffle Operations, Stage Execution.
• Experience optimizing large-scale Spark workloads.
• Hands-on experience with Parquet, Avro, ORC, Delta Lake.
• Hands-on experience with Apache Kafka, Apache NiFi.
• Hands-on experience with Snowflake.
• Experience designing and querying Snowflake data warehouse schemas.
• Familiarity with Snowflake Stages, File Formats, Streams, Tasks.
• Strong SQL and query optimization skills.
• Experience with Git-based version control workflows.
• Familiarity with Docker and containerized application development.
• Experience working within Agile development environments.
• 9+ Years of experience.

Nice To Haves

• Knowledge of Snowpark for Scala is preferred.
• Technical certification in multiple technologies is desirable.
• Pharmacy Domain.

Responsibilities

• Design, develop, and optimize large-scale batch and streaming data pipelines using Scala and Apache Spark.
• Build robust data processing solutions using Spark SQL, DataFrames, and Datasets.
• Develop and maintain Spark Structured Streaming applications for real-time data ingestion and processing.
• Integrate data pipelines with enterprise data platforms such as Snowflake.
• Implement scalable data ingestion frameworks utilizing Apache Kafka and/or Apache NiFi.
• Perform Spark performance tuning, including: Partition optimization, Caching strategies, Shuffle optimization, Broadcast joins, Resource allocation tuning.
• Develop clean, maintainable, and testable Scala code following functional programming principles.
• Participate in code reviews and establish coding standards and best practices.
• Create automated tests and support CI/CD pipeline improvements.
• Monitor production data pipelines and troubleshoot performance or data quality issues.
• Conduct root cause analysis and implement preventive measures.
• Lead technical and architectural discussions across teams.
• Mentor junior engineers and provide technical guidance.
• Collaborate with cross-functional stakeholders to deliver scalable data solutions.

Benefits

• Comprehensive set of benefits to ensure that you have a long and rewarding career with us.
• Unique benefits and opportunities we offer to support your growth, well-being, and success throughout your journey with us and beyond.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Associate degree