Senior Data Engineer

Zifo

10d

About The Position

Design, build, and operate the data infrastructure for a clinical trial design platform that extracts insights from clinical protocols, regulatory documents, and published articles to accelerate trial design decisions. This role owns the pipelines that ingest, parse, transform, and serve clinical data - from raw PDF extraction through structured storage in Aurora and GraphDB, to serving curated knowledge for LangGraph-based agentic AI workflows.

Requirements

Strong Python development experience, including PDF/document parsing libraries such as PyMuPDF, pdfplumber, unstructured.io, or similar.
Advanced PostgreSQL-compatible SQL, including Amazon Aurora; experience with schema design, migrations, query optimization, and indexing strategies for large clinical datasets.
Hands-on experience with Neptune, Neo4j, or similar graph databases; proficiency in SPARQL or Cypher; experience with ontology and knowledge graph modeling for biomedical entities.
Experience with AWS services including Aurora PostgreSQL, S3, Lambda, Step Functions, SQS/SNS, and IAM, particularly for data pipeline orchestration.
Experience with PDF text extraction, document section classification, and named entity recognition (NER) for clinical/biomedical text; familiarity with embedding models and vector stores such as OpenSearch, pgvector, or Pinecone.
Experience building data-serving APIs using FastAPI, including asynchronous programming patterns and backend integration.
Experience preparing data for LangChain/LangGraph applications and designing RAG pipelines, including chunking, retrieval, reranking, and prompt-data integration.
Experience with Airflow, Prefect, AWS Step Functions, Temporal, or similar workflow orchestration tools; ability to design multi-stage DAGs with dependency management, retry logic, monitoring, and error handling.
Experience with Terraform or AWS CDK, Docker, and Git, including automated pipeline testing and deployment on AWS.

Nice To Haves

Experience with biomedical knowledge graphs (e.g., linking drugs -> targets -> diseases > trials)
Prior work with PubMed/MEDLINE data, ClinicalTrials.gov API, or EMA/CTIS data.
Apache Spark or Databricks for batch processing of large document corpora
dbt for transformation layer management over Aurora

Responsibilities

Build ingestion pipelines for clinical trial protocols, ICF documents, SmPCs, CSRs, and published articles (PubMed, CTIS, ClinicalTrials.gov) - handling PDF parsing, text extraction, and structured data normalization
Design and implement data models in Amazon Aurora (relational) and GraphDB (knowledge graph) to represent trial design entities: endpoints, eligibility criteria, study arms, interventions, therapeutic areas, and their relationships
Develop embedding and vectorization pipelines to prepare extracted clinical text for RAG-based retrieval in LangGraph agentic workflows - chunking strategies, metadata enrichment, and vector store population
Build and maintain ETL/ELT workflows that transform unstructured clinical content into queryable, linked data across both relational and graph stores
Implement data quality validation specific to clinical data - protocol section classification accuracy, entity extraction completeness, cross-reference integrity (NCT IDs, EudraCT numbers, MeSH terms)
Build data serving APIs (Python/FastAPI) that expose curated datasets to the Angular frontend and LangGraph agent layer
Set up data lineage tracking and audit trails to support regulatory traceability of AI-generated trial design recommendation