Cloud BigData Engineer

Delan Associates, Inc•Charlotte, NC

22d

About The Position

We are seeking a skilled Cloud BigData Engineer with a BS or MS degree in computer science, computer engineering, or a related technical discipline, or equivalent work experience. The role involves 5+ years of hands-on software development experience with Big Data & Analytics solutions, focusing on GCP Cloud technologies such as Big Query, Airflow, DataProc, PubSub, Hadoop Hive, Spark, and Python. You will be responsible for designing, developing, and optimizing data pipelines for large-scale data processing, transformation, and analysis. Proficiency in SQL, database systems, event processing pipelines (Kafka or GCP PubSub), and Object-Oriented Programming is essential. The position also requires experience with CICD pipelines, automated test frameworks, source code management tools, and configuration management tools like GitHub. The ideal candidate will be able to analyze complex data engineering problems, propose and implement effective solutions, and continuously improve engineering practices through automation. Strong communication skills and a willingness to learn new technologies are also key.

Requirements

BS or MS degree in computer science, computer engineering, or other technical discipline, or equivalent work experience.
5+ years of hands-on software development experience with Big Data & Analytics solutions.
GCP Cloud - Big Query, Airflow, DataProc, PubSub, Hadoop Hive, Spark, Python, shell scripting.
Strong experience in designing, developing, and optimizing data pipelines for large-scale data processing, transformation, and analysis using Big Data and GCP technologies.
Proficiency in SQL and database systems, with experience in designing and optimizing data models for performance and scalability.
Experience of building event processing pipelines with Kafka or GCP PubSub.
Knowledge of distributed (multi-tiered) systems, algorithms & relational databases.
Strong Object-Oriented Programming skills and design patterns.
Experience with CICD pipelines, Automated test frameworks, and source code management tools (XLR, Jenkins, Git).
Good knowledge and experience with configuration management tools like GitHub.
Ability to analyze complex data engineering problems, propose effective solutions, and implement them effectively.
Ability and willingness to learn, adopt and build solutions using the Enterprise frameworks established for Big Data.
Communicates effectively with product and cross functional team.
Willingness to learn new technologies and leverage them to their optimal potential.
Understanding of various SDLC methodologies, familiarity with Agile & scrum ceremonies.

Nice To Haves

Design and development experience with Airflow, PubSub Kafka, Git, Jenkins is desirable.
Certifications in cloud platform (GCP Professional Data Engineer) is a plus.

Responsibilities

Design, develop, and optimize data pipelines for large-scale data processing, transformation, and analysis using Big Data and GCP technologies.
Build event processing pipelines with Kafka or GCP PubSub.
Analyze complex data engineering problems, propose effective solutions, and implement them effectively.
Learn, adopt, and build solutions using Enterprise frameworks established for Big Data.
Proactively look beyond the obvious for continuous improvement opportunities.
Communicate effectively with product and cross-functional teams.
Learn new technologies and leverage them to their optimal potential.
Provide engineering expertise to the team for developing features for Banking including Financial Crimes, Anti-money Laundering platforms.
Improve engineering practices, including driving resilience, predictability, and end efficiency through automation.
Partner with product teams to envision and deliver solutions which transform Financial Crimes Compliance using technology.