AWS/Databricks Cloud Data Engineer

Moser Consulting•Indianapolis, IN

2d•Hybrid

About The Position

Moser Consulting is seeking a Senior Cloud Data Engineer for its genomics organization. The role involves designing, building, and maintaining scalable data pipelines and acquisition workflows across cloud and on-premises environments, covering data ingestion, transformation, and delivery. The engineer will work with cross-functional teams to convert business requirements into reliable data solutions and follow best practices in architecture, cloud infrastructure, documentation, and continuous improvement. The position provides senior-level strategic technical leadership alongside two existing engineers, building AWS-based pipelines that support genomic data science models. Hands-on production AWS pipeline-building experience is the top priority. The work includes AWS-native pipelines using Lambda, ECS, EKS, and S3 to ingest and store genomic data retrieved through API calls, as well as sequencing and post-sequencing processing workflows. The role is based at Moser Consulting's two Indianapolis, Indiana offices, with a hybrid/onsite schedule that allows time at either office. The listed salary range is $85,000–$125,000.

Requirements

Proven, hands-on experience designing and building AWS-based data pipelines end-to-end, from API ingestion through storage and delivery.
Hands-on experience working with High Performance Computing (HPC) clusters.
Experience with dimensional modeling, slowly changing dimensions (SCDs), and medallion architecture.
Previous cloud experience, preferably AWS, with hands-on exposure to Lambda, S3, and Docker.
Experience implementing and managing role-based access control (RBAC), including group and permissions management for secure system access.
Strong Python experience for pipeline development, ETL, and data engineering, including work with complex datasets from disparate batch and streaming sources, with emphasis on low latency, high throughput, and cost-efficiency.
Experience using SQL and Python data tooling to develop scalable, optimized ETL/ELT pipelines for large, complex datasets from disparate batch and streaming sources.
Experience developing APIs in both consumer and producer capacities, including REST APIs.
CI/CD experience with modern DevOps tooling and deployment pipelines.
Experience with API-based access patterns for retrieving and integrating data from external systems.
Ability to develop and deploy Lambda functions through code or infrastructure as code, such as CloudFormation, Terraform, CDK, or SAM; GUI- or console-based Lambda creation is not acceptable.
Hands-on experience with Amazon ECS and/or EKS for container orchestration.
Demonstrated ability to learn new technologies quickly and apply them to solve problems.
Strong attention to detail and commitment to producing high-quality work.
Excellent oral and written communication skills.
Professionalism and ethics, including trust, positive attitude, commitment, honesty, collaboration, and approachability.
Ability to adapt in a fast-paced consulting environment, manage time effectively, and handle multiple tasks simultaneously.

Nice To Haves

Experience with version control systems such as Git and agile development frameworks.
Ability to design and document end-to-end testing plans for Python-based applications.
PySpark experience for large-scale distributed data processing.
Experience with Datadog or similar observability and monitoring platforms.
Familiarity with Dagster or other modern data orchestration frameworks.
Terraform or other infrastructure-as-code experience for cloud resource provisioning.
Database experience with AWS RDS, MySQL, or PostgreSQL.
Familiarity with D3.js for building custom data visualizations.
Pandas experience.

Responsibilities

Participate in team stand-ups and other agile ceremonies to support project delivery.
Build and maintain data acquisition pipelines from databases, flat files, REST APIs, and other sources.
Apply development design patterns and best practices across workloads.
Support data modeling, including dimensional models and semantic layers for analytics, under senior guidance.
Troubleshoot and resolve pipeline failures, query performance issues, data quality concerns, notebook timeouts, data model inconsistencies, semantic model refresh errors, and capacity throttling using monitoring tools and query optimization techniques.
Participate in requirements-gathering meetings and translate business needs into technical specifications.
Pursue ongoing learning through online training, certifications, and mentorship opportunities.
Create and maintain technical and operational documentation, data flow diagrams, and data mapping documents.
Collaborate with cross-functional teams and communicate progress to technical and non-technical stakeholders.
Build and maintain AWS-native pipelines using Lambda, ECS, EKS, and S3 to ingest and store genomic data retrieved via API calls.
Generate and manage sequencing and post-sequencing data processing workflows in support of genomics data science models.