Senior Data Engineer

Codametrix - Boston, MA

Full-time - Mid Level

Boston, MA

Professional, Scientific, and Technical Services

About the position

The Senior Data Engineer at CodaMetrix plays a crucial role in the Data & Analytics team, focusing on designing and maintaining robust data architectures on the Databricks platform. This position is responsible for managing data pipelines, ensuring seamless data flow from various sources, and integrating data into a unified Data Lake to enhance decision-making and analytical insights. The Data Engineer will support analytics and customer onboarding teams, ensuring optimal data access across the organization while optimizing the data platform architecture to support AI-driven products and solutions.

Responsibilities

• Create, maintain, populate and optimize the CodaMetrix data platform and analytics architecture
• Design, build and maintain robust and scalable infrastructure for optimal extraction, transformation, and loading of data from various sources
• Assemble large, complex data sets that meet functional and non-functional business requirements
• Identify, design, and implement internal process improvements such as automating manual processes and optimizing data delivery
• Collaborate with Analytics, Machine Learning, and Product teams to address data-related technical issues
• Optimize existing data systems and pipelines for performance and scalability
• Provide accurate and relevant data for standard and ad hoc data requests
• Leverage advanced technical skills to support development of the CodaMetrix data lake and business intelligence solutions
• Foster a culture of continuous improvement and learning within the team
• Evaluate and recommend new data technologies and methodologies
• Create and maintain comprehensive documentation of data processes, systems, and architectures
• Provide regular updates to management on data engineering initiatives
• Participate and lead Agile-based ceremonial activities
• Provide technical consulting to users of data warehouse and dashboarding tools

Requirements

• Bachelor's or Master's degree in Computer Science, Data Science, Information Technology or a related field
• 5+ years of experience with Big Data technologies for data processing using Apache Spark, Kafka, Cassandra
• 5+ years of experience in AWS Cloud infrastructure & Databases services such as RDS, PostgreSQL, Aurora or Redshift
• 5+ years of programming experience in ingesting, processing and reading large volumes of data using pyspark or Scala
• 5+ years of experience in writing SQL and performance tuning
• Advanced understanding of structured data in a healthcare setting
• Experience in building Data lake/Data Warehousing with structured and unstructured datasets
• Experience with ETL processes to implement data pipelines and workflows
• Experience with Databricks platform using Unity catalog and DLT pipeline
• Advanced knowledge of data modeling, data architecture and data integration
• Ability to manage multiple tasks or projects simultaneously
• Proven analytical and problem-solving skills
• Effective verbal and written communication skills

Nice-to-haves

• Common BI Tools; Tableau is a huge plus
• Knowledge of HIPAA compliance requirements and other security/compliance practices
• Experience with Streaming workloads and integrating Spark with Apache Kafka
• Experience with consuming or authoring REST and/or SOAP web service APIs
• Understanding of Infrastructure as Code (IaC) and experience with common tools

Benefits

• Health Insurance: 80% coverage of medical and dental insurance and vision insurance
• 401(k) plan available after one month of employment
• Generous Paid Time Off policy
• 7-week Onboarding Program for new hires
• Annual performance evaluations and individual growth prioritization
• Recognition through annual company awards
• Modern open plan workspace in Back Bay, Boston
• Employer-paid life insurance and short-term and long-term disability insurance