Backend Engineer (Data pipeline)
About The Position
One of our direct clients is urgently looking for a Backend Engineer specializing in data pipelines. This role involves developing innovation strategies, processes, and best practices, driving the execution of business plans and projects, and ensuring business objectives are met. The engineer will lead the discovery phase of medium to large projects, including high-level design, and will lead a small group of 3 to 4 engineers for assigned projects. Key responsibilities include end-to-end service implementations, data feeds ingestion, orchestration, and designing/implementing low-latency services. A significant part of the role will be accelerating the migration of legacy on-premise Hive/Spark/Airflow pipelines and workloads to the public cloud, specifically leveraging GCP, BigQuery, and DataProc to achieve cost efficiencies, faster execution, improved SLAs, and enhanced data quality. The position also requires troubleshooting business and production issues and promoting company policies, procedures, and values.
Requirements
- Experience with data pipelines
- Experience with GCP, BigQuery, and DataProc
- Experience migrating legacy on-prem Hive/Spark/Airflow pipelines to the public cloud
- Experience designing and implementing low latency services
- Experience with data feeds ingestion and orchestration
- Troubleshooting business and production issues
Responsibilities
- Develops innovation strategies, processes, and best practices.
- Drives the execution of multiple business plans and projects.
- Supports and ensures business objectives being met.
- Leads the discovery phase of medium to large projects to come up with high level design.
- Leads the work of other small group of 3 to 4 engineers for assigned Engineering projects.
- Leads and participates in end to end service implementations, data feeds ingestion, and orchestration.
- Designs and implements low latency services.
- Promotes and supports company policies, procedures, mission, values, and standards of ethics and integrity.
- Troubleshoots business and production issues.
- Accelerates migration of legacy on-prem Hive/Spark/Airflow pipelines and workloads to public cloud, leveraging GCP, BigQuery and DataProc to achieve cost efficiencies and acceleration of execution along with improved SLAs and Data Quality.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
