DataStage Engineer

ZENITH INFOTEK LLC

1d•$120 - $140•Remote

About The Position

We are seeking an experienced DataStage Migration Validation & ETL Test Engineer to support the migration of legacy IBM InfoSphere DataStage workloads to the Databricks platform. The ideal candidate will possess strong expertise in DataStage architecture, ETL validation, automated testing, SQL, and Python, with the ability to analyze legacy ETL logic and validate equivalent PySpark implementations. This role is critical in ensuring data integrity, accuracy, and successful migration to modern cloud-based data platforms.

Requirements

Strong experience with IBM InfoSphere DataStage (Parallel Jobs, Server Jobs, Sequences).
Experience analyzing legacy ETL workflows and operational metadata.
Expertise in ETL testing and data validation methodologies.
Strong SQL skills for complex data analysis and reconciliation.
Proficiency in Python for automation and validation scripting.
Experience with XML and JSON parsing.
Knowledge of automated testing frameworks such as: PyTest, Great Expectations, Other ETL/Data Validation tools.
Experience performing regression and reconciliation testing for enterprise data migrations.

Nice To Haves

Exposure to Databricks and PySpark.
Experience working with modern cloud data warehouse platforms.
Knowledge of data migration best practices and cloud modernization initiatives.
Familiarity with Agile development methodologies and CI/CD processes.

Responsibilities

Analyze and interpret complex IBM InfoSphere DataStage Parallel and Server jobs, sequences, and workflows.
Translate legacy ETL business logic into detailed functional documentation for the PySpark development team.
Design, develop, and execute automated testing frameworks for large-scale ETL migration validation.
Perform comprehensive data reconciliation between DataStage and Databricks environments.
Develop automated validation scripts to verify: Data schemas, Row counts, Data quality, Business rule transformations, End-to-end ETL processing.
Execute regression testing on migrated PySpark pipelines using historical production datasets.
Identify, analyze, and resolve data discrepancies throughout migration testing.
Document validation metrics, testing results, and migration KPIs.
Collaborate with Data Engineers, Architects, QA teams, and business stakeholders to ensure migration quality.
Provide formal migration validation sign-off prior to production cutover.