Senior Data Engineer

Hudson Valley Credit Union•Poughkeepsie, NY

About The Position

Hudson Valley Credit Union is currently recruiting for the position of Senior Data Engineer. Primary Functions: Design, build, and optimize our data infrastructure, leveraging the power and scalability of Databricks within our Azure environment. Maintain a deep expertise in Databricks and its ecosystem in order to build high-performance data pipelines, ensure data quality, and enable advanced analytics and machine learning initiatives.

Requirements

Bachelor’s Degree or equivalent in Computer Science or related field required.
Minimum 5 years of experience as Data Engineer required.
Minimum 5 years of experience Data Wharehousing, Data Lakes and big date platforms required.
Minimum 5 years Python experience required.
Minimum 3 years Databricks experience required.

Nice To Haves

Master's Degree or equivalent in Computer Science or related field preferred.
License/Certification Databrick Data Engineer Associate required.
License/Certification Databrick Data Engineer Professional preferred.
License/Certification Mircrosoft Azure Data Engineer preferred.

Responsibilities

Design, develop, and maintain robust and scalable data pipelines primarily using Databricks, leveraging Spark (PySpark or Scala), Delta Lake, and Databricks Workflows
Build and optimize data lakehouse solutions on Azure Data Lake Storage Gen2 integrated with Databricks
Implement and enforce data quality frameworks and data governance policies within Databricks using tools and features like Delta Lake's ACID properties and data validation techniques
Identify and resolve performance bottlenecks in Databricks-based data pipelines and Spark applications
Collaborate closely with analysts to understand their data requirements for advanced analytics, providing them with curated and performant datasets within Databricks
Automate data ingestion, transformation, and deployment processes, potentially integrating with CI/CD tools and Databricks MLOps
Provide technical leadership and mentorship to junior data engineers on Databricks best practices and performance tuning
Document Databricks notebooks, data pipelines, data models, and configurations
Ensure data security and compliance within the Databricks environment, adhering to relevant security standards and best practices
Troubleshoot and resolve complex data-related issues within the Databricks ecosystem
Work independently and contribute to immediate and cross-functional teams
Participate in design discussions
Analyze problem statements, break down problems, and provide solutions
Follow an agile-based approach to deliver data products
Take ownership of large tasks, delivering on time, and mentoring team members
Oversee the maintenance and enforcement of SDLC and ITSM processes, change management procedures, departmental standards, style requirements and all other procedures necessary for data and system integrity
Adhere to all Credit Union policies, procedures, and regulatory agency requirements. Participate in all required and recommended training and development including, but not limited to, Bank Secrecy Act training (BSA) and demonstrate attained knowledge. Participate on Credit Union teams, projects and strategic initiatives when the opportunity arises. Perform additional duties and special projects as assigned.
Embrace and apply HVCU’s guiding principles to all activities and responsibilities. This includes the Credit Union’s Mission, Vision, Core Values, Employee and Member Value Propositions, Sales and Service Model, and commitment to Lean Six Sigma practices. Support the Credit Union’s initiatives by demonstrating teamwork and professionalism.
Responsible for regular and predictable attendance including punctuality.