Python Engineer

TATA Consulting Services•Irving, TX

49d

About The Position

Design , build and maintain scalable pipelines using Python/Databricks Leverage Spark and SQL to process and transform large scale datasets Develop and optimize ELT/ETL processes for high volume of data workflows ETL: Hands on experience of building data pipelines. Proficiency in data integration platforms such as Apache Spark or Talend Big Data: Experience of 'big data' platforms such as Hadoop, Hive or Snowflake for data storage and processing Data Warehousing & Database Management: Understanding of Data Warehousing concepts, Relational Oracle database design Data Modeling & Design: Good exposure to data modeling techniques; design, optimization and maintenance of data models and data structures Languages: Proficient in any programming languages commonly used in data engineering such as Python or Scala DevOps: Exposure to concepts and enablers - CI/CD platforms, version control systems (e.g. GIT), automated quality control management Data Quality & Controls: Exposure to data validation, cleansing, enrichment and data controls

Requirements

Design , build and maintain scalable pipelines using Python/Databricks
Leverage Spark and SQL to process and transform large scale datasets
Develop and optimize ELT/ETL processes for high volume of data workflows
ETL: Hands on experience of building data pipelines. Proficiency in data integration platforms such as Apache Spark or Talend
Big Data: Experience of 'big data' platforms such as Hadoop, Hive or Snowflake for data storage and processing
Data Warehousing & Database Management: Understanding of Data Warehousing concepts, Relational Oracle database design
Data Modeling & Design: Good exposure to data modeling techniques; design, optimization and maintenance of data models and data structures
Languages: Proficient in any programming languages commonly used in data engineering such as Python or Scala
DevOps: Exposure to concepts and enablers - CI/CD platforms, version control systems (e.g. GIT), automated quality control management
Data Quality & Controls: Exposure to data validation, cleansing, enrichment and data controls

Responsibilities

Design , build and maintain scalable pipelines using Python/Databricks
Leverage Spark and SQL to process and transform large scale datasets
Develop and optimize ELT/ETL processes for high volume of data workflows
ETL: Hands on experience of building data pipelines. Proficiency in data integration platforms such as Apache Spark or Talend
Design, develop, and maintain ETL processes to extract, transform, and load data from various sources into our data warehouse.
Write complex SQL queries and PL/SQL scripts to perform data manipulation, validation, and transformation.
Develop and maintain data pipelines using Python and related libraries.
Optimize ETL processes and data pipelines for performance and scalability.
Collaborate with data analysts and other stakeholders to understand data requirements and develop solutions to meet their needs.
Implement data quality checks and monitoring to ensure data accuracy and consistency.
Troubleshoot and resolve data-related issues.
Create and maintain technical documentation for ETL processes, data pipelines, and database solutions.
Stay up-to-date with the latest trends and technologies in data management and analytics.