Databricks Data Engineer
Innosystech•Chicago, TX
•Onsite
About The Position
We are seeking a Senior Databricks Data Engineer with expertise in Spark, Scala, and AWS to join our team. This role involves designing, developing, and maintaining scalable enterprise data pipelines and solutions. You will work with cutting-edge technologies to process large-scale datasets and optimize data processing performance. The position requires strong collaboration with various teams to translate business requirements into robust data solutions.
Requirements
- 8+ years of overall experience in Data Engineering, Big Data, ETL, or Data Platform development.
- 4+ years of strong hands-on experience with Databricks.
- Strong hands-on development experience with Apache Spark and PySpark.
- Strong programming experience with Scala.
- Hands-on experience building data solutions on AWS.
- Strong experience developing scalable ETL/ELT and batch data-processing pipelines.
- Experience processing large-scale datasets using distributed computing technologies.
- Strong SQL skills for data transformation, validation, and performance optimization.
- Experience with Delta Lake / Lakehouse architectures.
- Experience troubleshooting and optimizing Spark and Databricks workloads.
- Strong understanding of data warehousing, data modeling, and data integration concepts.
Nice To Haves
- AWS S3, Glue, EMR, Lambda, Redshift, Delta Lake, Unity Catalog, Airflow, Git and CI/CD experience is preferred.
Responsibilities
- Design, develop, and maintain scalable enterprise data pipelines using Databricks, Spark, PySpark, Scala, and AWS.
- Develop Databricks notebooks and Spark applications for large-scale data transformation and processing.
- Build and optimize ETL/ELT pipelines integrating data from multiple enterprise sources.
- Develop reusable data-processing frameworks using Scala and PySpark.
- Optimize Spark jobs, cluster configurations, partitioning, caching, and data-processing performance.
- Implement data-quality checks, monitoring, logging, and exception-handling mechanisms.
- Develop and maintain Delta Lake-based data solutions.
- Troubleshoot data pipeline failures and production performance issues.
- Collaborate with architects, analysts, and engineering teams to translate business requirements into scalable data solutions.
- Participate in code reviews, testing, deployments, and production support.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
