Data Engineer

A.M. Best • Oldwick, NJ

67d • Hybrid

About The Position

The Data Engineer will build, orchestrate and maintain data pipelines to populate our future single version of truth data warehouse. They will discuss data requirements with other teams in an agile environment and will have the ability to develop proof-of-concepts to evaluate new technologies and suggest improved and better ways to re-architect existing data loading processes and data structures.

Requirements

• Bachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field
• 4 plus years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar roles
• Must hold a current/active Databricks Data Engineer/Analyst certification

Nice To Haves

• ETL / ELT tools: Spark, Kafka, Azure Data Factory (ADF)
• Languages: R, Java, Scala
• Databases: Redis, Elasticsearch

Responsibilities

• Assist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etc
• Design and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lakehouse
• Optimize data pipelines and data processing workflows for performance, scalability, and efficiency
• Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data
• Help create and maintain documentation for data mappings, data definitions, architecture and data flow diagrams
• Build proof-of-concepts to determine viability of possible new processes and technologies
• Deploy and manage code in non-prod and prod environments
• Investigate and troubleshoot data related issues and fix or provide solutions to fix defects
• Identify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performance

Benefits

• Flexible and hybrid work arrangements
• Paid time off/Paid company holidays
• Medical plan options/prescription drug plan
• Dental plan/vision plan options
• Flexible spending and health savings accounts
• 401(k) retirement savings plan with a Roth savings option and company matching contributions
• Educational assistance program

What This Job Offers

• Career Level: Mid Level
• Industry: Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services
• Education Level: Bachelor's degree
• Number of Employees: 501-1,000 employees