Machine Learning Engineer / SME (LLM Eval & RAG Testing) - SME | TS/SCI

Falls Technology • McLean, VA

1d • Onsite

About The Position

Falls Technology is hiring a Machine Learning Engineer / SME focused on LLM evaluation and RAG testing. The role is full-time and on-site in McLean, Virginia, and requires an active TS/SCI clearance and U.S. citizenship. The engineer will design RAG and LLM evaluation experiments, tune models using custom data, and run GPU-accelerated benchmarks to help deliver trustworthy AI systems.

Requirements

• Active TS/SCI clearance.
• U.S. citizenship.
• Hands-on RAG testing experience.
• Experience tuning neural networks or LLMs on custom datasets for real use cases.
• Experience building AI/ML models and ensembles in Python or R.
• Experience with PyTorch, TensorFlow, or Keras.
• GPU-accelerated computing experience.
• Experience testing, evaluating, or validating LLM-based AI systems.
• Experience designing experiments to benchmark LLMs.
• SQL skills, including CTEs, set operations, aggregates, and nested subqueries.
• Experience with GitHub and Jenkins or equivalent version control and CI.

Nice To Haves

• Hugging Face Transformers library and hub.
• Experience building benchmarks that measure LLM performance.
• Cloud computing development and architecture.
• Flask or similar front-end/web frameworks.
• Text classification and topic modeling using scikit-learn or deep learning.
• Semantic search applications.
• Tableau dashboards.
• Verification/validation test benches, Explainable AI (XAI), or ONNX.

Responsibilities

• Test, evaluate, and validate AI systems, with a focus on LLM-based systems, including RAG testing.
• Design and run experiments that benchmark LLMs against defined measures.
• Tune neural networks and LLMs on custom datasets and apply the results to concrete use cases.
• Build AI/ML models and ensembles in Python or R, including library selection, preprocessing, and model assessment.
• Use deep learning frameworks such as PyTorch, TensorFlow, or Keras.
• Leverage GPUs for accelerated training and evaluation.
• Write SQL queries against analytical datasets using CTEs, set operations, aggregates, and nested subqueries.
• Use GitHub and Jenkins-style CI to support reproducible experiments.
• Communicate methodological choices and model results to technical and non-technical stakeholders.

Benefits

• 9% retirement contribution.
• 100% of family medical premiums paid.
• 4 weeks of PTO.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed