Machine Learning Engineer

Children's Hospital of Philadelphia • Philadelphia, PA

33d • $104,600 - $138,600 • Onsite

About The Position

The Campbell Laboratory at the Children’s Hospital of Philadelphia (CHOP) is seeking a Machine Learning Engineer to contribute to their mission of diagnosing rare genetic diseases more quickly and accurately. This role involves developing and training large language models (LLMs) to interpret clinical data from electronic health records (EHRs) and to facilitate accurate, equitable diagnoses for all children, particularly those from historically marginalized backgrounds. The Machine Learning Engineer will collaborate with data scientists, clinicians, and researchers to design, implement, and scale advanced machine learning workflows. They will utilize on-premises GPU/SLURM clusters and cloud-based TPU instances (Google Cloud) for training and deploying LLMs using tools like Hugging Face Transformers, PyTorch, and JAX. This position emphasizes robust software engineering practices, advanced machine learning and natural language processing (NLP) techniques, with a strong focus on reproducibility and high-quality code within an innovative and interdisciplinary research environment that values diversity, professional growth, and impactful research.

Requirements

• Bachelor's Degree in Analytics, Data Science, Statistics, Mathematics, Computer Science or a related field.
• At least three (3) years experience with progressively more complex data science, applied statistics, machine learning, or mathematical modeling projects.
• Proven software engineering experience, including structured development methods, testing, and version control.
• Hands-on experience with Python and at least one deep learning framework (e.g., PyTorch, TensorFlow, or JAX).
• Familiarity with relational databases (e.g., Snowflake, BigQuery, Oracle SQL, MySQL).
• Experience with Linux/Unix environments, shell scripting, and cluster computing systems (e.g., SLURM).

Nice To Haves

• Masters or PhD in Analytics, Data Science, Statistics, Mathematics, Computer Science or a related field.
• At least four (4) years with progressively more complex data science, applied statistics, machine learning, or mathematical modeling projects.
• At least one year of experience with complex data science, applied statistics, machine learning, or mathematical modeling projects.
• Natural language processing experience, particularly in the biological and medical domains.
• Experience with transformer architecture and associated software (e.g., PyTorch, Tensorflow, JAX).
• Experience using distributed computing technologies.
• Experience with cloud virtual machine environments.
• Experience implementing distributed training on GPUs or TPUs in cloud platforms (e.g., Google Cloud, AWS, Azure).
• Experience with prompt engineering, semantic search, or retrieval-augmented generation (RAG) in a research or production environment.
• Familiarity with MLOps pipelines (CI/CD, containerization, monitoring, and logging frameworks).
• Exposure to healthcare or biomedical data and associated privacy/security regulations (e.g., HIPAA) is a plus.
• Experience with advanced NLP techniques or LLMs in a research or production environment.

Responsibilities

• Configure and utilize on-premises SLURM cluster with GPU resources for efficient and reliable job scheduling for large-scale model training.
• Manage and optimize cloud-based infrastructures (e.g., TPU Pods on Google Cloud) for distributed model training and evaluation.
• Collaborate with data scientists to implement and fine-tune LLMs (e.g., Transformer architectures in PyTorch, TensorFlow, or JAX) for clinical and biomedical NLP tasks.
• Develop efficient training pipelines, including data loading, preprocessing, feature extraction, and model deployment.
• Evaluate model performance and optimize hyperparameters, GPU/TPU utilization, and distributed training strategies.
• Collaborate cross-functionally with clinicians, data scientists, analysts, and IT teams to support and enhance machine learning operations (MLOps).
• Work with relational databases (e.g., Snowflake, BigQuery, Oracle SQL, MySQL) and distributed storage systems to access and manage EHR data.
• Partner with data scientists and domain experts to design data pipelines that integrate with existing hospital systems.
• Write clean, well-documented, and maintainable code following best practices.
• Contribute to shared code repositories using Git, ensuring reproducibility and version control for collaborative projects.
• Develop CI/CD workflows to automate model testing, containerization, and deployment to production environments.
• Monitor deployed models for performance drift, latency, and reliability, and implement automated alerts and feedback loops to refine model behavior.
• Produce clear technical documentation, including system architecture diagrams, training procedures, and user guides for internal stakeholders.
• Present engineering best practices, findings, and process updates to clinicians, researchers, and other non-technical audiences as needed.

Benefits

• CHOP offers countless ways to change lives.
• CHOP’s Commitment to Diversity, Equity, and Inclusion.
• CHOP is committed to building an inclusive culture where employees feel a sense of belonging, connection, and community within their workplace.
• We are a team dedicated to fostering an environment that allows for all to be their authentic selves.
• We are focused on attracting, cultivating, and retaining diverse talent who can help us deliver on our mission to be a world leader in the advancement of healthcare for children.
• We strongly encourage all candidates of diverse backgrounds and lived experiences to apply.
• Our innovative and interdisciplinary environment values diversity, fosters professional growth, and drives impactful research that benefits children worldwide.
• CHOP is committed to supporting the health of our patients, families, workforce, and global community.
• As a condition of employment, CHOP employees who work in patient care buildings or who have patient facing responsibilities must be fully vaccinated against COVID-19 and receive an annual influenza vaccine.
• Employees may request exemptions for valid religious and medical reasons.
• Start dates may be delayed until candidates are immunized or exemption requests are reviewed.
• EEO / VEVRAA Federal Contractor | Tobacco Statement
• Click here for more information regarding CHOP's Compensation and Benefits.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level