Machine Learning Engineer

Children’s Hospital of Philadelphia • Philadelphia, PA

28d • $104,600 - $138,600 • Onsite

About The Position

The Campbell Laboratory at the Children’s Hospital of Philadelphia (CHOP) is seeking a Machine Learning Engineer to contribute to their mission of diagnosing rare genetic diseases more quickly and accurately. The role involves developing and training large language models (LLMs) to interpret clinical data from electronic health records (EHR) and to facilitate accurate, equitable diagnoses, particularly for marginalized communities. The Machine Learning Engineer will collaborate with data scientists, clinicians, and researchers to design, implement, and scale machine learning workflows. This will involve utilizing on-premises GPU/SLURM clusters and cloud-based TPU instances (Google Cloud) for training and deploying LLMs using Hugging Face Transformers, PyTorch, and JAX. The position emphasizes robust software engineering practices, advanced machine learning and natural language processing (NLP) techniques, with a focus on reproducibility and high-quality code. CHOP's innovative and interdisciplinary environment supports professional growth and impactful research for children globally. The ideal candidate is passionate about building robust machine learning systems, enjoys high-impact problems, and thrives in a collaborative research setting.

Requirements

• Bachelor's Degree Required
• At least three (3) years experience with progressively more complex data science, applied statistics, machine learning, or mathematical modeling projects.
• Proven software engineering experience, including structured development methods, testing, and version control.
• Hands-on experience with Python and at least one deep learning framework (e.g., PyTorch, TensorFlow, or JAX).
• Familiarity with relational databases (e.g., Snowflake, BigQuery, Oracle SQL, MySQL).
• Experience with Linux/Unix environments, shell scripting, and cluster computing systems (e.g., SLURM).

Nice To Haves

• Bachelor's Degree Analytics, Data Science, Statistics, Mathematics, Computer Science or a related field Preferred
• Masters or PhD in Analytics, Data Science, Statistics, Mathematics, Computer Science or a related field Preferred
• At least four (4) years with progressively more complex data science, applied statistics, machine learning, or mathematical modeling projects Preferred
• At least one year of experience with complex data science, applied statistics, machine learning, or mathematical modeling projects Preferred
• Natural language processing experience, particularly in the biological and medical domains Preferred
• Experience with transformer architecture and associated software (e.g., PyTorch, Tensorflow, JAX) is Preferred
• Experience using distributed computing technologies Preferred
• Experience with cloud virtual machine environments Preferred
• Experience implementing distributed training on GPUs or TPUs in cloud platforms (e.g., Google Cloud, AWS, Azure).
• Experience with prompt engineering, semantic search, or retrieval-augmented generation (RAG) in a research or production environment.
• Familiarity with MLOps pipelines (CI/CD, containerization, monitoring, and logging frameworks).
• Exposure to healthcare or biomedical data and associated privacy/security regulations (e.g., HIPAA) is a plus.
• Experience with advanced NLP techniques or LLMs in a research or production environment.

Responsibilities

• Configure and utilize on-premises SLURM cluster with GPU resources to ensure efficient and reliable job scheduling for large-scale model training.
• Manage and optimize cloud-based infrastructures (e.g., TPU Pods on Google Cloud) for distributed model training and evaluation.
• Collaborate with data scientists to implement and fine-tune LLMs (e.g., Transformer architectures in PyTorch, TensorFlow, or JAX) for clinical and biomedical NLP tasks.
• Develop efficient training pipelines, including data loading, preprocessing, feature extraction, and model deployment.
• Evaluate model performance and optimize hyperparameters, GPU/TPU utilization, and distributed training strategies.
• Collaborate cross-functionally with clinicians, data scientists, analysts, and IT teams to support and enhance machine learning operations (MLOps).
• Work with relational databases (e.g., Snowflake, BigQuery, Oracle SQL, MySQL) and distributed storage systems to access and manage EHR data.
• Partner with data scientists and domain experts to design data pipelines that integrate with existing hospital systems.
• Write clean, well-documented, and maintainable code following best practices.
• Contribute to shared code repositories using Git, ensuring reproducibility and version control for collaborative projects.
• Develop CI/CD workflows to automate model testing, containerization, and deployment to production environments.
• Monitor deployed models for performance drift, latency, and reliability, and implement automated alerts and feedback loops to refine model behavior.
• Produce clear technical documentation, including system architecture diagrams, training procedures, and user guides for internal stakeholders.
• Present engineering best practices, findings, and process updates to clinicians, researchers, and other non-technical audiences as needed.

Benefits

• CHOP offers countless ways to change lives. Our diverse community of more than 20,000 Breakthrough Makers will inspire you to pursue passions, develop expertise, and drive innovation.
• At CHOP, your experience is valued; your voice is heard; and your contributions make a difference for patients and families.
• CHOP is committed to building an inclusive culture where employees feel a sense of belonging, connection, and community within their workplace.
• We are a team dedicated to fostering an environment that allows for all to be their authentic selves.
• We are focused on attracting, cultivating, and retaining diverse talent who can help us deliver on our mission to be a world leader in the advancement of healthcare for children.
• Our innovative and interdisciplinary environment values diversity, fosters professional growth, and drives impactful research that benefits children worldwide.
• As a condition of employment, CHOP employees who work in patient care buildings or who have patient facing responsibilities must be fully vaccinated against COVID-19 and receive an annual influenza vaccine.
• Employees may request exemptions for valid religious and medical reasons.
• Start dates may be delayed until candidates are immunized or exemption requests are reviewed.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level