AWS/Databricks Cloud Data Engineer

Moser Consulting • Indianapolis, IN

1d • Hybrid

About The Position

We are seeking a Senior Cloud Data Engineer to join our consulting team. In this role, you will design, build, and maintain scalable data pipelines and acquisition workflows across cloud and on-premises environments, contributing to the full spectrum of data engineering – from ingestion through transformation and delivery. You will collaborate with cross-functional teams to translate business requirements into robust data solutions, while upholding best practices in architecture, cloud infrastructure, documentation, and continuous improvement. This position sits within our genomics organization, providing senior-level, strategic technical leadership alongside two existing engineers to build out AWS-based data pipelines that support genomic data science models — hands-on, production AWS pipeline-building experience is the top priority for this role.

Requirements

• Top priority: proven, hands-on experience designing and building AWS-based data pipelines end-to-end (API ingestion through storage and delivery)
• Must-have: hands-on experience working with High Performance Computing (HPC) clusters
• Experience with dimensional modeling, SCDs, and medallion architecture
• Previous cloud experience required; AWS preferred with hands-on exposure to Lambda, S3, and Docker
• Experience implementing and managing role-based access control (RBAC), including group/permissions management for secure system access
• Strong Python experience across pipeline development, ETL, and data engineering tasks, with exposure to complex datasets from disparate batch and streaming sources and an emphasis on low latency, high throughput, and cost-efficiency; pandas experience a plus
• Leveraging SQL and python data tooling develop scalable, optimized ETL/ELT pipelines to ingest, transform, and load large, complex datasets from disparate sources (batch and streaming). Streamlining for low latency, high throughput and cost-efficiency.
• Experience with API development in both consumer and producer capacities (REST APIs)
• CI/CD experience with modern DevOps tooling and deployment pipelines
• Experience with API-based access patterns for retrieving and integrating data from external systems
• Lambda functions must be developed and deployed via code/infrastructure-as-code (e.g., CloudFormation, Terraform, CDK, SAM) — GUI/console-based Lambda creation is not acceptable
• Hands-on experience with Amazon ECS and/or EKS for container orchestration
• Demonstrated ability to learn new technologies quickly and apply them to solve problems
• Strong attention to detail and commitment to producing high-quality work
• Excellent oral and written communication skills
• Professionalism and ethics including trust, positive attitude, commitment, honesty, collaboration, and approachability
• Proven ability to adapt in a fast-paced consulting environment, including managing time effectively and handling multiple tasks simultaneously

Nice To Haves

• Experience with version control systems (Git) and agile development frameworks
• Design and document end-to-end testing plans for Python-based applications
• PySpark experience for large-scale distributed data processing
• Experience with Datadog or similar observability and monitoring platforms
• Familiarity with Dagster or other modern data orchestration frameworks
• Terraform or infrastructure-as-code (IaC) experience for cloud resource provisioning
• Database experience with databases such as AWS RDS, MySQL, or PostgreSQL
• Familiarity with D3.js for building custom data visualizations

Responsibilities

• Participate in team stand-ups and other agile ceremonies to support project delivery
• Assist in building and maintaining data acquisition pipelines from diverse sources including databases, flat-files, and REST APIs
• Leverage development design patterns and best practices across workloads.
• Support data modeling efforts including dimensional models and semantic layers for analytics under senior guidance
• Proactively troubleshoot and resolve pipeline failures, query performance issues, data quality concerns, notebook timeouts, data model inconsistencies, semantic model refresh errors, and capacity throttling using monitoring tools and query optimization techniques
• Participate in requirement gathering meetings and translate business needs into technical specifications
• Engage in ongoing learning through online trainings, certifications, and mentorship opportunities
• Create and maintain technical documentation, operational documentation, data flow diagrams, and data mapping documents
• Collaborate effectively with cross-functional teams and communicate progress to technical and non-technical stakeholders
• Build and maintain AWS-native pipelines (Lambda, ECS, EKS, S3) that ingest and store genomic data retrieved via API calls
• Generate and manage sequencing and post-sequencing data processing workflows in support of genomics data science models

Benefits

• Training Opportunities
• Fully Invested 401K Plan
• PPO and HDHP Medical Plans
• Employer-Paid Dental and Vision Plans
• Onsite Fitness Center
• Wellness Program
• Catered Lunches

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed