Machine Learning Engineer / SME (LLM Eval & RAG Testing) - SME | TS/SCI

Falls Technology•McLean, VA

15h•Onsite

About The Position

Falls Technology is seeking a Machine Learning Engineer / SME to evaluate and validate AI systems, especially large language model (LLM) systems and retrieval-augmented generation (RAG). The role focuses on designing evaluation experiments, benchmarking LLMs against defined measures, tuning models on custom data, and running GPU-accelerated training and evaluation so the team can deliver AI systems it can trust. The position is full-time and on-site in McLean, Virginia, and requires active TS/SCI clearance and U.S. citizenship. The engineer will build AI/ML models and ensembles using Python or R, select libraries, preprocess data, and assess models. The role also involves using deep learning frameworks, querying analytical datasets with SQL, managing reproducible experiments with GitHub and Jenkins-style CI, and communicating methodology and results to technical and non-technical stakeholders.

Requirements

Active TS/SCI clearance.
U.S. citizenship.
Hands-on RAG testing experience.
Experience tuning neural networks or LLMs on custom datasets for real use cases.
Experience building AI/ML models and ensembles in Python or R.
Experience with PyTorch, TensorFlow, or Keras.
GPU-accelerated computing experience.
Experience testing, evaluating, or validating LLM-based AI systems.
Experience designing experiments to benchmark LLMs.
SQL skills, including CTEs, set operations, aggregates, and nested subqueries.
Experience with GitHub and Jenkins, or equivalent version control and CI tools.

Nice To Haves

Experience with the Hugging Face Transformers library and hub.
Experience building benchmarks that measure LLM performance.
Cloud computing development and architecture experience.
Experience with Flask or similar front-end/web frameworks.
Experience with text classification and topic modeling using scikit-learn or deep learning.
Experience with semantic search applications.
Experience creating Tableau dashboards.
Experience with verification/validation test benches, Explainable AI (XAI), or ONNX.

Responsibilities

Test, evaluate, and validate AI systems, with a focus on LLM-based systems and RAG testing.
Design and run experiments to benchmark LLMs against defined measures.
Tune neural networks and LLMs on custom datasets and apply results to practical use cases.
Build AI/ML models and ensembles in Python or R, including library selection, preprocessing, and model assessment.
Use deep learning frameworks such as PyTorch, TensorFlow, or Keras.
Leverage GPUs for accelerated training and evaluation.
Write SQL queries against analytical datasets, including CTEs, set operations, aggregates, and nested subqueries.
Use GitHub and Jenkins-style continuous integration to support reproducible experiments.
Communicate methodological choices and model results to technical and non-technical stakeholders.