Staff Data Engineer - Data Engineering

Arine • San Francisco, CA

179d • $165,000 - $180,000 • Remote

About The Position

As a key technical leader and team architect working in a fast-paced environment, you will drive the design, development, and optimization of scalable data ingestion pipelines within the Arine platform. Leveraging expert-level proficiency in Python and AWS, you will architect solutions that handle diverse file types and large-scale healthcare datasets. You will have a direct impact on building reusable, configurable tools set for handling data needs for the entire company.

Requirements

• 10+ years of professional experience in data engineering with focus on large-scale data ingestion and infrastructure
• Deep expertise in Python programming and modern data engineering tools
• Experience creating an automated production grade ETL process using Python and SQL
• Strong understanding of ETL/ELT frameworks and distributed data processing
• Experience with data processing, validation, cleaning and debugging data sets
• Experience with API integration for seamless data exchange between systems
• Proven experience handling and processing various file types and formats, including specialized healthcare standards such as HL7, 834, 837, and NCPDP
• Experience integrating and consolidating data from diverse source systems into a unified repository, including data from EHR and claim systems, as well as from file-based and API integrations
• Experience with processing large data sets (over 10GB)
• Experience with incremental data processing and change data capture (CDC) methodologies
• Strong experience designing scalable data architectures in AWS environment
• Deep understanding of software engineering principles including test-driven development, loose coupling, single responsibility, and modular design
• Experience with containerization technologies (Docker, Kubernetes) and building configuration-driven, maintainable systems
• Proven ability to build tools and systems that can be operated by diverse engineering profiles through configuration rather than code changes
• Passion for building new and improving existing data infrastructure with robust, maintainable, and operationally excellent data systems
• Familiarity with healthcare data and regulatory environments (HIPAA compliance) is a plus
• Strong collaboration and communication skills; comfortable working with diverse technical and non-technical stakeholders
• Excellent verbal and written communication skills with ability to explain technical infrastructure concepts to diverse audiences

Nice To Haves

• Familiarity with healthcare data and regulatory environments (HIPAA compliance)

Responsibilities

• Act as the team architect by leading system design reviews, offering recommendations, conducting comprehensive peer reviews, and demonstrating expert-level proficiency in Python and AWS services
• Architecting and implementing scalable data ingestion pipelines handling different file types into Arine platform
• Develop reusable components that can be integrated into data pipelines to enhance efficiency and minimize future implementation time
• Creating configuration-driven, containerized toolsets that can be easily used and maintained by diverse engineering profiles
• Work collaboratively with cross-functional teams to ensure their data requirements are met through ETL components
• Implementing incremental data ingestion strategies for large-scale healthcare datasets
• Building monitoring and alerting systems for data ingestion processes and pipeline health
• Applying software engineering best practices including test-driven development and modular design to data infrastructure
• Refactoring and rebuilding existing data ingestion processes to improve scalability and operational efficiency
• Working with containerization technologies (Docker, Kubernetes) to create portable and maintainable data solutions
• Identify and escalate inefficiencies within and across teams
• Provide technical guidance, mentorship to junior engineers, and promote best practices and coding standards
• Author and support high-quality technical documentation, assisting junior engineers in doing the same

Benefits

• Dynamic role with opportunities for learning and growth
• Collaboration with experienced Clinicians, Engineers, Software Architects, Data Scientists, and Digital Health Entrepreneurs

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Industry: Hospitals
• Number of Employees: 101-250 employees