Manager, Data Engineering
Brainlabs
•$50,000 - $105,000•Onsite
About The Position
We are looking for a motivated and detail-oriented Data Engineer with 2 to 5 years of experience in designing, building, and managing scalable data solutions on Google Cloud Platform (GCP). The ideal candidate will have a strong background in data engineering, cloud-based architectures, and proficiency in implementing data pipelines to transform raw data into actionable insights. Experience building or supporting AI and GenAI data workflows, including pipelines for LLM applications and AI/ML model training, is a strong plus.
Requirements
- 2 to 5 years of experience in designing, building, and managing scalable data solutions on Google Cloud Platform (GCP).
- Strong background in data engineering, cloud-based architectures.
- Proficiency in implementing data pipelines to transform raw data into actionable insights.
- Hands-on experience with GCP services like CloudFunctions, CloudRun, Schedular, BigQuery, Dataflow, Pub/Sub, and Cloud Storage.
- Strong programming skills in Python, SQL.
- Knowledge of data modelling, schema design, and query optimization techniques.
- Experience in building batch and streaming data pipelines.
- Excellent communication and collaboration skills.
- Ability to work in a fast-paced and dynamic environment.
- Must be legally entitled to work in the United States.
Nice To Haves
- Experience building or supporting AI and GenAI data workflows, including pipelines for LLM applications and AI/ML model training.
- Familiarity with orchestration tools like Apache Airflow, Cloud Composer, or similar.
- Working experience on other cloud stack for ETL(AWS or Azure) is a plus.
- Experience with GCP’s AI/ML platform (Vertex AI, BigQuery ML, or AutoML) for building, evaluating, or serving models is a strong advantage.
- Hands-on experience building or supporting LLM/GenAI pipelines using frameworks such as LangChain, LlamaIndex, or Vertex AI Agent Builder.
- Familiarity with AI/ML data preparation practices, including feature engineering, dataset curation, and data versioning for model training workflows.
- Knowledge of CI/CD practices and tools like Git, Jenkins, or Terraform for pipeline deployments.
- Understanding of data security, governance, and compliance practices on GCP.
- Bachelor’s degree in Computer Science, Engineering.
- GCP Data Engineer or Associate Cloud Engineer certification (preferred but not mandatory).
Responsibilities
- Design, develop, and maintain ETL/ELT pipelines using GCP tools like CloudFunctions, CloudRun, Dataflow, Dataproc, or Cloud Data Fusion.
- Ensure data pipelines are scalable, efficient, and optimised for performance.
- Build and manage data pipelines that support LLM and GenAI applications, including Retrieval-Augmented Generation (RAG) architectures, vector data stores, and prompt context assembly workflows.
- Curate and prepare datasets for AI/ML model training, covering feature engineering, labeling pipeline oversight, and data versioning using tools like Vertex AI Feature Store or DVC.
- Integrate data from various sources into GCP services such as BigQuery, Cloud Storage, and Cloud SQL.
- Design and implement data warehouse/mart solutions using BigQuery for analytics and reporting.
- Build transformation logic using SQL, Python, or Spark for preparing clean and structured data.
- Optimise query performance and storage cost in BigQuery or other GCP storage systems.
- Develop processes to ensure data quality, integrity, and consistency across the pipeline.
- Implement monitoring and logging systems using tools like Stackdriver or Looker.
- Understand and interpret business and technical requirements to support data development tasks.
- Assist in building, testing, and maintaining data pipelines while ensuring alignment with project objectives and stakeholder needs.
- Work closely with cross-functional teams, including data analysts, data scientists, and business stakeholders, to understand requirements.
- Provide technical guidance on GCP best practices and tools.
- Maintain clear documentation of processes, workflows, and data architecture.
- Ensure regular maintenance and version control of pipelines and scripts.
Benefits
- We are open to hiring candidates in our various office locations across the United States.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
