Data Engineer

As a Senior Data Engineer, you will be responsible for developing data engineering solutions to support the analytical needs of the business. Your role will involve designing and building data pipelines, improving reporting and analysis processes, and automating end-to-end data pipelines. You will also be involved in optimizing the data pipeline to support machine learning workloads and collaborating with various stakeholders to translate data into meaningful insights. Additionally, you will have the opportunity to mentor and coach other development team members.

• Design and build mission critical data pipelines with a highly scalable distributed architecture
• Help continually improve ongoing reporting and analysis processes, simplifying self-service support for business stakeholders
• Build and support reusable framework to ingest, integration and provision data
• Automation of end to end data pipeline with metadata, data quality checks and audit
• Build and support a big data platform on the cloud
• Define and implement automation of jobs and testing
• Optimize the data pipeline to support ML workloads and use cases
• Support mission critical applications and near real time data needs from the data platform
• Capture and publish metadata and new data to subscribed users
• Work collaboratively with business analysts, product managers, data scientists as well as business partners and actively participate in design thinking session
• Participate in design and code reviews
• Motivate, coach, and serve as a role model and mentor for other development team associates/members that leverage the platform

• Minimum of 3 years' experience in distributed systems like Data warehouse / Data lake technical architecture
• 3+ years of experience in using programming languages (Python / Scala / Java / C#) to build data pipelines
• Minimum 3 years of Big Data and Big Data tools in one or more of the following: Batch Processing (e.g. Hadoop distributions, Spark, Databricks), Real-time processing (e.g. Kafka, Flink/Spark Streaming)
• Minimum of 2 years' experience with AWS or engineering in other cloud environments
• Advanced proficiency in SQL and familiarity with DBT
• Experience with Database Architecture, Schema design
• Strong familiarity with batch processing and workflow tools such as AirFlow, NiFi
• Ability to work independently with business partners and management to understand their needs and exceed expectations in delivering tools/solutions
• Strong interpersonal, verbal and written communication skills and ability to present complex technical/analytical concepts to executive audience
• Strong business mindset with customer obsession; ability to collaborate with business partners to identify needs and opportunities for improved data management

• Empowered to leverage data to drive amazing customer experiences and business results
• Opportunity to own the end-to-end development of data engineering solutions
• Passionate about working with disparate datasets
• Ability to bring data together to answer business questions at speed
• Deep expertise in the creation and management of datasets
• Collaboration with analysts, data scientists, and business stakeholders
• Design and build mission-critical data pipelines with a highly scalable distributed architecture
• Simplifying self-service support for business stakeholders
• Building and supporting a big data platform on the cloud
• Automation of end-to-end data pipeline with metadata, data quality checks, and audit
• Optimizing the data pipeline to support ML workloads and use cases
• Supporting mission-critical applications and near real-time data needs
• Motivating, coaching, and serving as a role model and mentor for other development team associates/members
• Opportunity to work with distributed systems like Data warehouse/Data lake technical architecture
• Experience in using programming languages (Python/Scala/Java/C#) to build data pipelines
• Familiarity with Big Data and Big Data tools (Hadoop distributions, Spark, Databricks, Kafka, Flink/Spark Streaming)
• Experience with AWS or engineering in other cloud environments
• Advanced proficiency in SQL and familiarity with DBT
• Strong familiarity with batch processing and workflow tools such as AirFlow, NiFi
• Ability to work independently with business partners and management
• Strong interpersonal, verbal, and written communication skills
• Opportunity to collaborate with business partners to identify needs and opportunities for improved data management and delivery
• Experience providing technical leadership and mentoring other engineers
• Equal opportunity employer that values diversity