Data Engineer

Huge is seeking a Data Engineer to join their Data Products team. The role involves monitoring data pipelines and ETL processes, resolving any issues or bottlenecks, and collaborating with the operations team. The Data Engineer will also be responsible for developing and maintaining data transformation and processing jobs using cloud-native technologies. Ensuring data quality and adherence to governance policies, as well as managing and optimizing cloud-native infrastructure resources, are additional responsibilities of the role. Expertise in Google Cloud Platform (GCP) and strong programming skills in languages such as Python, Java, or Scala are desired qualifications for this position.

• Monitor the health and performance of existing data pipelines and ETL processes
• Use monitoring tools and dashboards to identify and resolve data processing issues, bottlenecks, or failures
• Collaborate with the operations team or developers to troubleshoot and resolve data processing issues
• Develop, test, and maintain data transformation and processing jobs
• Use cloud-native technologies and frameworks to transform raw data into usable formats
• Write and optimize SQL queries or code for data transformations
• Ensure data quality and adherence to data governance policies and standards
• Collaborate with data analysts, data scientists, and stakeholders to provide clean, reliable, and well-structured data
• Implement data validation checks, data profiling, and data cleansing processes
• Collaborate with the DevOps or infrastructure team to manage and optimize cloud-native infrastructure resources
• Provision and configure data storage systems
• Manage compute resources for big data processing
• Demonstrate expertise in Google Cloud Platform (GCP) services such as BigQuery, Dataflow, Pub/Sub, Cloud Storage, Dataproc, Data Catalog, etc.
• Apply data engineering concepts, data modeling, data warehousing, ETL/ELT processes, data integration techniques, and best practices
• Design and build scalable and efficient data pipelines using GCP tools and technologies
• Utilize programming skills in Python, Java, or Scala for data engineering tasks, scripting, and automation
• Write complex SQL queries, optimize query performance, and utilize query tuning techniques
• Design and optimize data storage systems on GCP, including data partitioning, schema design, and performance tuning.

• Data Pipeline Monitoring: Monitor the health and performance of existing data pipelines and ETL processes. Use monitoring tools and dashboards to identify and resolve any data processing issues, bottlenecks, or failures. Collaborate with the operations team or developers to troubleshoot and resolve these issues.
• Data Transformation and Processing: Develop, test, and maintain data transformation and processing jobs. Use cloud-native technologies and frameworks such as Apache Spark, Apache Beam, or cloud-specific tools like AWS Glue or Google Cloud Dataflow to transform raw data into usable formats for downstream applications. Write and optimize SQL queries or code for data transformations.
• Data Quality and Governance: Ensure data quality and adherence to data governance policies and standards. Collaborate with data analysts, data scientists, and other stakeholders to understand their data requirements and provide them with clean, reliable, and well-structured data. Implement data validation checks, data profiling, and data cleansing processes as necessary.
• Infrastructure Management: Collaborate with the DevOps or infrastructure team to manage and optimize cloud-native infrastructure resources. This includes provisioning and configuring data storage systems like AWS S3, Google Cloud Storage, or Azure Data Lake, as well as managing compute resources such as AWS EMR, Google Cloud Dataproc, or Azure HDInsight for big data processing.
• Data Engineer Skills: Google Cloud Platform (GCP) Expertise: In-depth knowledge and hands-on experience with GCP services such as BigQuery, Dataflow, Pub/Sub, Cloud Storage, Dataproc, Data Catalog, and others.
• Strong understanding of data engineering concepts, data modeling, data warehousing, ETL/ELT processes, data integration techniques, and best practices.
• Proficiency in designing and building scalable and efficient data pipelines using GCP tools and technologies, ensuring optimized data ingestion, transformation, and loading processes.
• Strong programming skills in languages such as Python, Java, or Scala for data engineering tasks, scripting, and automation.
• Proficient in writing complex SQL queries, optimizing query performance, and utilizing query tuning techniques for efficient data retrieval and manipulation.
• Expertise in designing and optimizing data storage systems on GCP, including data partitioning, schema design, and performance tuning.

• Remote work opportunity within Colombia
• Competitive salary and benefits package
• Opportunity to work with a global creative consultancy
• Collaborative and inclusive work environment
• Access to cutting-edge cloud-native technologies and frameworks
• Opportunity to work on data transformation and processing projects
• Chance to collaborate with data analysts, data scientists, and other stakeholders
• Focus on data quality and adherence to data governance policies
• Opportunity to optimize cloud-native infrastructure resources
• Chance to work with big data processing tools and technologies
• Opportunity to design and optimize data storage systems
• Strong support for diversity and inclusion in the workforce