Information Technology_USA - USA_Developer
About The Position
We are seeking experienced Data Engineers with at least 10 years of work experience, including a minimum of 5 years specifically as a data engineer. The role involves designing and developing scalable ETL/ELT pipelines using Databricks and PySpark, building batch and real-time streaming ingestion frameworks, and developing reusable ingestion and transformation frameworks. You will implement the Medallion architecture (Bronze, Silver, Gold layers), develop incremental and CDC-based ingestion pipelines, and design and implement real-time streaming pipelines using Kafka and Structured Streaming. Optimization of Spark jobs, SQL queries, and streaming pipelines is crucial, as is the implementation of Delta Lake-based ingestion and transformation frameworks. Tuning partitioning, caching, and Spark execution strategies are key responsibilities. Strong SQL and data modeling skills are required, along with experience in cloud platforms and distributed systems. Familiarity with CI/CD pipelines and DevOps practices is also expected.
Requirements
- At least 10 years of work experience.
- Minimum of 5 years of experience as a data engineer.
- 8-10+ years experience with MySQL.
- 8-10+ years experience with Databricks.
- 8-10+ years experience with Pyspark.
- Strong SQL and data modeling skills.
- Experience with cloud platforms and distributed systems.
- Familiarity with CI/CD pipelines and DevOps practices.
Responsibilities
- Design and develop scalable ETL/ELT pipelines using Databricks and PySpark.
- Build batch and real-time streaming ingestion frameworks.
- Develop reusable ingestion and transformation frameworks.
- Implement Medallion architecture (Bronze, Silver, Gold layers).
- Develop incremental and CDC-based ingestion pipelines.
- Design and implement real-time streaming pipelines using Kafka and Structured Streaming.
- Optimize Spark jobs, SQL queries, and streaming pipelines.
- Implement Delta Lake-based ingestion and transformation frameworks.
- Tune partitioning, caching, and Spark execution strategies.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
