Developer - Cloud Data Engineer

Duke Careers•Durham, NC

47d

About The Position

At Duke Health, we're driven by a commitment to compassionate care that changes the lives of patients, their loved ones, and the greater community. No matter where your talents lie, join us and discover how we can advance health together. About Duke Health Technology Solutions Pursue your passion for caring and innovation with Duke Heath Technology Solutions, which is dedicated to the transformation, development, and management of enterprise information technology solutions across Duke Health. By harnessing the power of innovative technologies like cloud computing and artificial intelligence — and pairing them with a forward-thinking approach — Duke Health Technology Solutions is revolutionizing the future of health care at Duke Health and beyond. Primary Role The Data Engineer will be responsible for designing, developing, and optimizing data pipelines, manage servers, and integrate APIs. They will collaborate closely with data scientists and analysts to meet data requirements and ensure robust data management solutions.

Requirements

Basic understanding of local databases (Clarity; Caboodle) and underlying tables pertaining to role/domain
Basic understanding of ETL technologies and scripting languages and terminology related to role
Basic understanding of cloud data engineering technologies, including storage accounts, data pipelines, data lakes, structured vs. unstructured data, etc.
Basic knowledge of SQL and relational databases
Follows all policies and procedures set forth by DUHS/DHTS/ACE as pertains to role
Maintains ongoing communications (timely, clear, concise, and accurate) with manager keeping them apprised of request status
Maintains appropriate project documentation as assigned such as GITLAB or data dictionary
Advanced knowledge of the assigned domain/role. Understands data, workflows, and tools related to role.
Advanced knowledge of, and experience with, cloud data engineering concepts, including (but not limited to) Microsoft Azure Lakehouses and Delta Tables, Azure Data Factory, Azure Synapse, and medallion-style data architecture.
Ability to work with stakeholders to understand business needs and perform business analysis independently >75% of the time.
Develop technical requirements from business requirements independently >75% of the time.
Can write/modify accurate SQL queries independently > 90% of the time
Ability to optimize SQL queries, stored procedures and scripts for performance reliability
Can write complex SQL inclusion/exclusion criteria for cohort development
Ability to locate data elements independently > 75% of the time
Knowledge/utilization of basic data design and infrastructure
Ability to manage multiple projects to completion in a timely manor
Ability to develop and present topics as assigned/needed by customer and/or ACE team
Expertise in systems and software related to project area to accomplish job duties (e.g. SSIS, Azure Data Factory, etc..)
Expert knowledge and experience with Microsoft Azure Data Engineering tools, concepts, and industry best practices.
Expert locating data elements as required.
Demonstrates ability to work independently to solve problems and complete assigned work >95% of the time
Demonstrates ability to work with customers to understand business need or problem to address independently >95% of the time
Bachelor's degree in a related field, or four years of equivalent technical experience required.

Nice To Haves

Preferred certification in development technologies, as appropriate for the position.

Responsibilities

Designing, developing, and optimizing data pipelines
Manage servers
Integrate APIs
Collaborate closely with data scientists and analysts to meet data requirements and ensure robust data management solutions
Design, construct and test data management systems under guidance of senior team members
Assist in server administration tasks, ensuring system performance and availability
Optimize SQL queries, stored procedures and scripts for performance reliability
Manage and optimizes server infrastructure for performance and reliability
Troubleshoot and resolve complex data pipeline, server and API issues
Lead the development of complex data pipelines and API integration, ensuring scalability and efficiency
Define standards and best practices for data pipeline design, development and maintenance
Collaborates with data architect on design and implementation of data solutions (lakehouse, data warehouse, etc.)