Principal Machine Learning Engineer

Medical Guardian•Philadelphia, PA

About The Position

We are looking for a Principal Machine Learning Engineer to serve as a hands-on technical leader for machine learning, predictive modeling, scoring, decisioning, and applied AI initiatives. This role will primarily focus on building, validating, deploying, and improving machine learning models, while also bringing principal-level judgment to problem definition, model design, stakeholder engagement, and production readiness. This is a hands-on model-building role first. The ideal candidate should be comfortable spending most of their time working directly with data, features, models, scoring logic, validation methods, production workflows, and model improvement. They should also be able to operate with the maturity of a principal-level engineer: shaping unclear problems, making pragmatic technical decisions, mentoring others, and driving work forward without waiting for perfect requirements.

Requirements

8+ years of professional experience in machine learning, data science, software engineering, analytics engineering, applied AI, or related technical fields.
5+ years of hands-on machine learning model development experience, including feature engineering, model training, validation, evaluation, and iteration.
3+ years of experience deploying, operationalizing, or supporting models in production or business-critical environments.
Strong hands-on experience with Python and SQL.
Experience with modern ML and data platforms such as Databricks, Spark, MLflow, Snowflake, Azure, AWS, or similar technologies.
Strong understanding of model evaluation, calibration, thresholding, score interpretation, monitoring, drift, retraining, and production ML lifecycle management.
Experience translating ambiguous business problems into concrete ML designs, model requirements, validation plans, and measurable outcomes.
Ability to explain model behavior, model performance, assumptions, limitations, and tradeoffs to both technical and non-technical stakeholders.
Strong engineering discipline, including clean code, reproducibility, versioning, testing, documentation, and maintainability.
Ability to work independently as a senior hands-on contributor while also providing technical leadership and modeling judgment.

Nice To Haves

10+ years of relevant professional experience in ML, data science, applied AI, software engineering, decisioning systems, commercial software, or production analytics.
Experience building scorecards, risk scores, health scores, engagement scores, churn scores, fraud scores, credit-style models, prioritization models, or operational decision-support models.
Experience with transparent or interpretable models such as logistic regression, GLMs, GAMs, decision trees, monotonic models, calibrated models, scorecard-based models, or Explainable Boosting Machines.
Experience designing score bands, thresholds, risk tiers, intervention rules, recommended actions, or decision logic based on model outputs.
Experience working in commercial software, SaaS, digital products, gaming, fintech, healthtech, consumer technology, marketplace, or other product-driven environments.
Experience building ML, AI, analytics, or decisioning capabilities embedded into customer-facing products, operational workflows, commercial platforms, or revenue-impacting systems.
Experience in startup, scale-up, innovation lab, new product development, or rapid-build environments where the candidate had to operate with ambiguity and drive work forward independently.
Experience partnering with product managers, designers, software engineers, business leaders, and operational teams to turn ML models into usable product capabilities.
Experience building MVPs, validating assumptions, iterating based on feedback, and maturing prototypes into production systems.
Experience with GenAI, LLMs, RAG, AI agents, prompt engineering, model evaluation, conversational AI, summarization, document intelligence, or AI-enabled workflow automation.
Experience combining traditional ML models with GenAI-enabled workflows, such as using predictive scores to trigger outreach, summarize customer/member context, recommend next actions, or support human decision-making.
Experience in healthcare, population health, remote patient monitoring, insurance, financial services, safety, operations, or other domains where model trust and explainability are important.
Experience with MLOps practices including model registries, deployment pipelines, monitoring, drift detection, retraining strategies, and model governance.

Responsibilities

Build, test, validate, and improve machine learning models for scoring, prediction, prioritization, risk detection, engagement, intervention targeting, and decision support.
Perform exploratory data analysis, data quality assessment, feature engineering, model training, model selection, and performance evaluation.
Develop practical ML models that balance predictive performance, explainability, stability, maintainability, and business usefulness.
Work with structured, semi-structured, and operational data to create model-ready datasets and reusable features.
Use tools such as Python, SQL, Spark, Databricks, MLflow, scikit-learn, XGBoost, or similar platforms and libraries.
Move quickly from data exploration to prototype to validated model to production-ready capability.
Design and implement predictive scores, risk tiers, score bands, thresholds, cut points, and intervention logic.
Build transparent and interpretable models where explainability is important, including logistic regression, generalized linear models, decision trees, monotonic models, calibrated models, scorecard-style models, or explainable boosting approaches.
Evaluate models for accuracy, calibration, stability, drift, fairness, interpretability, and operational usefulness.
Help stakeholders understand what a score represents, how it should be used, how it should not be used, and how changes in the score should be interpreted.
Document model logic, features, assumptions, limitations, validation results, and recommended usage in a way that business and technical stakeholders can understand.
Define the evidence needed to show that a model or score is valid, stable, explainable, actionable, and useful.
Partner with data engineering, analytics engineering, platform engineering, and application engineering teams to move models from experimentation into reliable production workflows.
Support model deployment, batch scoring, real-time or near-real-time inference, model versioning, monitoring, retraining, and performance tracking.
Help define data pipelines, feature pipelines, inference flows, model outputs, feedback loops, and monitoring requirements.
Ensure models are observable, supportable, secure, scalable, and aligned with enterprise architecture and governance expectations.
Establish practical monitoring and feedback loops to determine whether models continue to perform and create value over time.
Operate effectively in a rapid-build, startup-like environment where speed, ownership, and pragmatic decision-making are important.
Turn early-stage ideas, ambiguous business needs, and rough concepts into working ML products, scores, prototypes, and production capabilities.
Bring a product-engineering mindset to ML development, including user needs, workflow integration, adoption, usability, feedback loops, and measurable outcomes.
Drive work forward without waiting for perfect requirements, while still identifying critical assumptions, risks, dependencies, and evidence needed before scaling.
Partner with business and product stakeholders to define MVPs, iterate quickly, learn from usage, and improve models over time.
Make smart tradeoffs between quick prototypes, durable platforms, transparent models, GenAI-enabled workflows, and longer-term ML architecture.
Support the design and development of GenAI-enabled solutions, including LLM-powered workflows, RAG, summarization, conversational agents, document intelligence, and decision-support tools.
Help evaluate when GenAI is appropriate versus when traditional ML, rules, analytics, or transparent scoring models are a better fit.
Partner with product, engineering, and business stakeholders to integrate predictive models, scores, and GenAI outputs into practical workflows.
Apply appropriate evaluation, guardrails, monitoring, privacy controls, and human-in-the-loop processes for GenAI use cases.
Help the organization balance innovation with explainability, safety, reliability, privacy, and operational usefulness.
Work directly with business, product, analytics, operations, and engineering stakeholders to clarify what a model is intended to predict, explain, recommend, or trigger.
Translate business questions into measurable ML objectives, target variables, features, validation approaches, and success metrics.
Ask practical questions early: who will use the score, what action will it inform, what does a false positive or false negative mean, and how will we know the model is creating value?
Communicate model behavior, tradeoffs, limitations, and recommended usage clearly to both technical and non-technical audiences.
Help the team avoid becoming an AI ticket factory by shaping solutions, not just executing requests.
Provide technical leadership through hands-on example, strong engineering judgment, and clear recommendations.
Proactively identify model risks, data gaps, unclear requirements, design issues, and opportunities for improvement.
Help establish practical standards for model development, validation, documentation, monitoring, and production readiness.
Mentor other engineers and data scientists through code reviews, design reviews, modeling guidance, and shared best practices.
Demonstrate high ownership by driving clarity, execution, and continuous improvement.