Senior Data Engineer
About The Position
Designs and implements scalable data pipelines, optimizes workflows for performance and reliability, and ensures compliance with data governance policies. Programs testable and maintainable software solutions using Object Oriented (OO) Python programming and Machine Learning (ML) libraries, including Pandas, NumPy, Scikit-learn, and TensorFlow. Develops and designs functional programming, emerging technologies, and messaging frameworks, using Kafka. Implements business rule management systems in Python or Java with Drools, Pyke, and Nools. Leverages quantitative, statistics, and econometrics (including probability, linear regression, time series data analysis, and optimizations) techniques and methods. Programs testable and maintainable software solutions using Splunk, Snowflake, YugabyteDB, Aerospike, and S3 database management systems. Employs Agile development lifecycle methodologies (Kanban and SCRUM).
Requirements
- Object Oriented (OO) Python programming and Machine Learning (ML) libraries, including Pandas, NumPy, Scikit-learn, and TensorFlow.
- Functional programming, emerging technologies, and messaging frameworks, using Kafka.
- Business rule management systems in Python or Java with Drools, Pyke, and Nools.
- Quantitative, statistics, and econometrics (including probability, linear regression, time series data analysis, and optimizations) techniques and methods.
- Splunk, Snowflake, YugabyteDB, Aerospike, and S3 database management systems.
- Agile development lifecycle methodologies (Kanban and SCRUM).
- Cloud-native, event-driven, data platform architecture using Amazon Web Services (AWS) (Lambda, Glue, EC2, EMR, Athena, and Crawler), PySpark, Kafka, and Airflow in enterprise environments.
- Building and optimizing data lakes, warehouses, and models, using Snowflake, Redshift, SQL, and Denodo to access federated databases (Oracle, DB2, Teradata, MongoDB, PostgreSQL, MSSQL, Yugabyte, and Aerospike); and enabling scalable transformations, performance tuning, and cross-platform insights.
- Continuous Integration and Continuous Delivery (CI/CD) pipeline development and data workflow orchestration, using Airflow, Control-M, Jenkins, SQL, Python, Terraform, and Docker within the Software Development Life Cycle (SDLC), to automate ingestion, transformation, validation, and monitoring for data integrity, Agile delivery, and operational efficiency.
- Extract Transform Load /Extract Load Transform (ETL/ELT) for predictive modeling and analytics, using Python, Pandas, Scikit-learn, TensorFlow, and Snowpark; generating insights using PowerBI, Tableau, or Quicksight; and developing scalable solutions using Java, Shell, PL/SQL, Talend, Alteryx, Informatica, Unix, and Linux.
Nice To Haves
- Experience in a financial services environment.
Responsibilities
- Develops software system testing and validation procedures, programming, and documentation.
- Develops original and creative technical solutions to on-going development efforts.
- Designs applications or subsystems on major projects and for/in multiple platforms.
- Performs technical and functional analysis for data engineering projects.
- Supports and performs all phases of testing leading to implementation.
- Develops comprehensive documentation for multiple applications supporting several corporate initiatives.
- Responsible for post-installation testing of any problems.
- Establishes project plans for projects of moderate scope.
- Works on complex assignments and often multiple phases of a project.
- Collaborates with teams to support data-centric initiatives.
- Performs independent and complex technical and functional analysis for multiple projects supporting several initiatives.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
