Senior Data Integration Engineer

The Sr. Data Integration Engineer at Syndigo is responsible for architecting and implementing data ingestion, validation, and transformation pipelines. They will collaborate with various teams to design and maintain batch and streaming integrations across different data domains and platforms. The ideal candidate should have experience in big data and cloud architecture, as well as the ability to effectively communicate ideas and lead projects. They will also be responsible for troubleshooting data pipelines and resolving issues, while estimating and tracking the status of assigned items.

- Take ownership in building solutions and proposing architectural designs related to building efficient and timely data ingestion and transformation processes geared towards analytics workloads - Manage code deployment to various environments - Be proficient at positively critiquing and suggesting improvements via code reviews - Work with stakeholders to define and develop data ingest, validation, and transform pipelines - Troubleshoot data pipelines and resolve issues in alignment with SDLC - Ability to diagnose and troubleshoot data issues, recognizing common data integration and transformation patterns - Estimate, track, and communicate the status of assigned items to a diverse group of stakeholders

- 5+ years of experience in developing and architecting large scale data pipelines in a cloud environment - Demonstrated expertise in Scala (Object Oriented Programming) / Python (Scala preferred), SPARK SQL - Experience with Databricks, including Delta Lake - Experience with Azure and cloud environments, including Azure Data Lake Storage (Gen2), Azure Blob Storage, Azure Tables, Azure SQL Database, Azure Data Factory - Experience with ETL/ELT patterns, preferably using Azure Data Factory and Databricks jobs - Fundamental knowledge of data ingestion, validation, and transformation processes geared towards analytics workloads - Ability to troubleshoot data pipelines and resolve issues in alignment with SDLC - Ability to diagnose and troubleshoot data issues, recognizing common data integration and transformation patterns - Strong communication skills and ability to effectively communicate ideas and concepts to peers - Experience leading projects that support business objectives and goals - Proficient in code deployment to various environments - Proficient in positively critiquing and suggesting improvements via code reviews - Ability to estimate, track, and communicate status of assigned items to a diverse group of stakeholders

- Competitive salary and benefits package - Opportunity to work with a rapidly growing company - Collaborative and innovative work environment - Chance to challenge the status quo and make a difference - Ownership in building solutions and proposing architectural designs - Code deployment management - Positive critique and suggestion of improvements through code reviews - Development of data ingest, validation, and transform pipelines - Troubleshooting and issue resolution in alignment with SDLC - Ability to diagnose and troubleshoot data issues - Estimation, tracking, and communication of project status - 5+ years of experience in developing and architecting large scale data pipelines - Expertise in Scala/Python, SPARK SQL - Experience with Databricks and Delta Lake - Experience with Azure and cloud environments - Experience with ETL/ELT patterns using Azure Data Factory and Databricks jobs - Fundamental knowledge of distributed data processing and storage - Fundamental knowledge of working with structured, unstructured, and semi-structured data - Excellent analytical and problem-solving skills - Effective time management and ability to adjust to changing priorities - Remote job location (Bangalore) - Commitment to diversity, equity, and inclusion