Data Engineer II
About The Position
Responsible for designing, building, and maintaining data platform that supports data integrations for Enterprise Data Warehouse, Operational Data Store or Data Marts etc. with appropriate data access, data security, data privacy and data governance. Establish enterprise-scale data integration procedures, data pipelines and frameworks across the data development life cycle. Suggest and implement appropriate technologies to deliver resilient, scalable, and future-proof data solutions. Create high performing data ingestion pipelines. Create scheduled as well as trigger-based ingestion patterns using scheduling tools. Create performance optimized DDLs for any row-based or columnar databases. Conduct performance tuning of complex data pipelines and SQL queries. Perform impact analysis of proposed changes on existing architecture, capabilities, system priorities, and technology solutions. Enhance data processing systems and procedures through system analysis, python coding, testing, and other activities that facilitate technical business objectives. Oversee data deliverables and coordinate design reviews and release management activities. Estimate and provide timelines for project activities. Identify, document, and communicate technical risks, issues and alternative solutions discovered during project. Drive automation, identify inefficiencies, optimize processes and data flows, and recommend improvements. Use agile engineering practices and various data development technologies to rapidly develop and implement efficient data products. Collaborate with Product Owners to understand PI goals, PI planning, requirement clarification, and delivery coordination. Provide technical support for production incidents and failures.
Requirements
- Bachelor’s degree in Computer Science, Information Science, or a related field plus five (5) years of experience in the job offered or five (5) years of experience as a Systems Engineer, Business Intelligence Engineer, ETL Developer, or related occupation. Or a Master’s degree in Computer Science, Information Science, or a related field plus three (3) years of experience in the job offered or three (3) years as a Systems Engineer, Business Intelligence Engineer, ETL Developer, or related occupation.
- Experience writing and optimizing using SQL queries in a business environment with large datasets.
- Experience with big data systems with distributed storage and serverless compute technologies (AWS or Hadoop).
- Programming proficiency in modern programming language, Python or PySpark.
- Experience designing data schema.
- Experience operating internal data warehouses and SQL/NoSQL database systems.
- Experience with traditional ETL, and cloud based ETL tools, Informatica Intelligent Cloud Services or Streamsets.
- Experience with scheduling tools (Control-M, Redwood or Autosys).
- Experience with reporting tools (Tableau or MicroStrategy).
- Experience with traditional and cloud based databases (Oracle, Sqlserver, Teradata and Hive).
- Experience with Tortoise SVN or similar code repository system.
- Knowledge of System Life Cycle Development methodology (Waterfall and Agile).
Responsibilities
- Designing, building, and maintaining data platform that supports data integrations for Enterprise Data Warehouse, Operational Data Store or Data Marts etc. with appropriate data access, data security, data privacy and data governance.
- Establish enterprise-scale data integration procedures, data pipelines and frameworks across the data development life cycle.
- Suggest and implement appropriate technologies to deliver resilient, scalable, and future-proof data solutions.
- Create high performing data ingestion pipelines.
- Create scheduled as well as trigger-based ingestion patterns using scheduling tools.
- Create performance optimized DDLs for any row-based or columnar databases.
- Conduct performance tuning of complex data pipelines and SQL queries.
- Perform impact analysis of proposed changes on existing architecture, capabilities, system priorities, and technology solutions.
- Enhance data processing systems and procedures through system analysis, python coding, testing, and other activities that facilitate technical business objectives.
- Oversee data deliverables and coordinate design reviews and release management activities.
- Estimate and provide timelines for project activities.
- Identify, document, and communicate technical risks, issues and alternative solutions discovered during project.
- Drive automation, identify inefficiencies, optimize processes and data flows, and recommend improvements.
- Use agile engineering practices and various data development technologies to rapidly develop and implement efficient data products.
- Collaborate with Product Owners to understand PI goals, PI planning, requirement clarification, and delivery coordination.
- Provide technical support for production incidents and failures.
Benefits
- Competitive, thoughtfully designed and quality benefits program
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Number of Employees
5,001-10,000 employees
