Data Engineer Automation Controls

Redolent•Irving, TX

4d•Onsite

About The Position

We are seeking a Data Engineer with a strong interest in big data technologies and a background in data engineering. This role involves building data pipeline frameworks for high-volume and real-time data delivery, transforming analytical models into production-ready solutions, and supporting an advanced anomaly detection machine learning platform. You will continuously integrate and ship code into cloud production environments and develop cloud-based applications from scratch using a modern technology stack. Collaboration with Product Owners and customers in an agile environment is key to delivering data products.

Requirements

BS degree in Computer Science, Data Engineering or similar.
Intermediate to senior level experience in an Apps Development role.
Demonstrated strong execution capabilities.
5+ years of experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Java or Scala or Python.
At least 4 years of experience in the following Big Data frameworks: File Format (Parquet, AVRO, ORC), Resource Management, Distributed Processing and RDBMS.
5+ years of developing applications with Monitoring, Build Tools, Version Control, Unit Test, TDD, Change Management to support DevOps.
At least 2 years of experience with SQL and Shell Scripting experience.
Experience of designing, building, and deploying production-level data pipelines using tools from Hadoop stack (HDFS, Hive, Spark, HBase, Kafka, NiFi, Oozie, Apache Beam, Apache Airflow etc.).
Experience with Spark programming (pyspark or scala or java).
Experience troubleshooting JVM-related issues.
Experience and strategies to deal with mutable data in Hadoop.
Experience with Stream sets.
Familiarity with machine learning implementation using PySpark.

Nice To Haves

Angular.JS 4 Development and React.JS Development expertise in a up to date Java Development Environment with Cloud Technologies.
1+ years’ experience with Amazon Web Services (AWS), Google Compute or another public cloud service.
2+ years of experience working with Streaming using Spark or Flink or Kafka or NoSQL.
2+ years of experience working with Dimensional Data Model and pipelines in relation with the same.
Intermediate level experience/knowledge in at least one scripting language (Python, Perl, JavaScript).
Hands on design experience with data pipelines, joining data between structured and unstructured data.
Familiarity of SAS programming will be a plus.
Experience implementing open source frameworks & exposure to various open source & package software architectures (AngularJS, ReactJS, Node, Elastic Search, Spark, Scala, Splunk, Apigee, and Jenkins etc.).
Experience with various noSQL databases (Hive, MongoDB, Couchbase, Cassandra, and Neo4j) will be a plus.
Experience in Ab Initio technologies including, but not limited to Ab Initio graph development, EME, Co-Op, BRE, Continuous flow.
Successfully complete assessment tests offered in Pluralsight, Udemy, etc. or complete certifications to demonstrate technical expertise on more than one development platform.

Responsibilities

Build data pipeline frameworks to automate high-volume and real-time data delivery for our Spark and streaming data hub.
Transform complex analytical models in scalable, production-ready solutions.
Provide support and enhancements for an advanced anomaly detection machine learning platform.
Continuously integrate and ship code into our cloud production environments.
Develop cloud based applications from the ground up using a modern technology stack.
Work directly with Product Owners and customers to deliver data products in a collaborative and agile environment.
Develop sustainable data driven solutions with current new generation data technologies to drive our business and technology strategies.
Build data APIs and data delivery services to support critical operational and analytical applications.
Contribute to the design of robust systems with an eye on the long-term maintenance and support of the application.
Leverage reusable code modules to solve problems across the team and organization.
Handle multiple functions and roles for the projects and Agile teams.
Define, execute and continuously improve our internal software architecture processes.
Be a technology thought leader and strategist.