Tech Lead - Data Engineering

CGI•Pittsburgh, PA

4d•Onsite

About The Position

CGI is looking for a Lead Data Engineer. This role will require someone at our client site in Pittsburgh, PA. . Work on projects for one of the largest financial institutions in the US. . Work in a challenging technical environment and gain exposure to advanced tools and technology suites . Should be able to work independently and manage the work on their own . You will work closely with your team and Product Owner, and peer Scrum Masters to support Sprint and Release Planning, foster team self-awareness, encourage cross-training, decision-making, and continuous improvement CGI is looking for a Lead Data Engineer to join our on-site team. This candidate should have extensive hands-on experience working with Python, Spark, SQL, Elastic Search (ELK), and Reporting (ELK) . Manage and coordinate with the offshore team to ensure deliverables are met. . Interact with project stakeholders on a day-to-day basis and report on project status to the Client Management team . Build and optimize Spark Streaming applications for real-time ingestion, transformation, and event-driven processing using Kafka or other messaging systems. . Excellent communication skills and work independently . Provide insights to the team on value adds and process efficiencies . The candidate should have hands-on experience with the Kafka streaming platform and the ELK (Elastic) stack. . Build CI/CD workflows using Git, Jenkins, and Bitbucket for automated deployments and version control. . Collaborate with cross-functional teams to troubleshoot, monitor, and improve data pipelines in production environments. . Ensure compliance with data security, governance, and access control practices. . The ideal candidate will have a can-do attitude, passion for technology, extensive analysis coding. This candidate will serve as a technical expert of a large-scale digital development project consisting of both on-shore and off-shore developers.

Requirements

8+ yrs of professional experience in software development and/or data engineering.
5+ yrs of hands-on experience working with relational databases, including Oracle, Teradata, and MySQL.
6+ yrs of hands-on experience with Python and PySpark development. (Mandatory)
3+ yrs Hands-on Experience with Kafka Streaming Platform (Mandatory)
3+ yrs Hands-on Experience with Elastic Search (ELK Stack) and building reporting dashboards (ELK) (Mandatory)
Experience in Data Analysis
Experience with Hadoop ecosystem, i.e., Hadoop, Hive, Spark, Scala, Sqoop, Flume,
Knowledge on Banking Domain/Products
Experience with modern workflow/orchestration tools (e.g., Apache Airflow, Oozie, etc.)
Knowledge of CI/CD tools like Git, Jenkins, and Bitbucket.
Understanding of job monitoring, logging, and performance tuning for both batch and streaming workloads.
Strong analytical abilities.
Agile experience is required with Jira tool expertise
Good knowledge of CI/CD and DevOps principles
Learning knowledge in GenAI, Azure OpenAI, and GPT
Excellent communication and thought leadership skills
Excellent interpersonal skills, including the ability to work with diverse personality types and understand technical issues.
Knowledge of Banking Domain
Knowledge of streaming platforms

Nice To Haves

Experience working with Data Observability within a streaming platform would be a plus.
Experience with containerization and orchestration (e.g., Docker, Kubernetes) is an advantage.

Responsibilities

Manage and coordinate with the offshore team to ensure deliverables are met.
Interact with project stakeholders on a day-to-day basis and report on project status to the Client Management team
Build and optimize Spark Streaming applications for real-time ingestion, transformation, and event-driven processing using Kafka or other messaging systems.
Provide insights to the team on value adds and process efficiencies
Build CI/CD workflows using Git, Jenkins, and Bitbucket for automated deployments and version control.
Collaborate with cross-functional teams to troubleshoot, monitor, and improve data pipelines in production environments.
Ensure compliance with data security, governance, and access control practices.
Provide hands-on technical solutions to business challenges & translate them into process/ technical solutions
Deliver high-quality work and adapt to new challenges, as an individual or as part of a team