AI Engineer

LawPro.ai•,

9d•$150,000 - $170,000•Remote

About The Position

We are looking for an experienced AI Engineer to own the evaluation, selection, and continuous optimization of the large language models and AI processes that power LawPro.ai's data insights and analytics platform. You will ensure our AI systems remain accurate, cost-effective, and resilient as the LLM landscape evolves, proactively managing transitions to new models and technologies. You will build the solutions and processes needed to raise our bar for cost, quality, and resilience. This role blends AI research and production engineering: staying ahead of a fast-moving model landscape, benchmarking new LLMs, techniques, and frameworks against our use cases, and owning both the recommendation and the implementation. We value engineers who bring deep AI and engineering intuition alongside a systematic, process-driven mindset, people who can design evaluation frameworks, interpret model behavior, and carry changes into production without relying on others to finish the work. You'll be a key contributor to a fast-moving team building production-grade AI systems that materially impact how law firms optimize outcomes for their clients.

Requirements

5+ years of AI/ML engineering experience evaluating, fine-tuning, and deploying LLMs in production environments, including cloud infrastructure (AWS or GCP) at scale and writing production-deployed LLM orchestration frameworks and multi-model pipelines.
Hands-on development of multiple RAG solutions.
Hands-on experience with embedding models and vector databases.
Hands-on experience building agentic workflows and implementing EvalOps or Evals-as-a-Service architecture.
Deep familiarity with the LLM ecosystem, able to critically assess model capabilities, limitations, and fit, including heuristic-gated model routing and cost, quality, speed, and capability tradeoffs.
Proven experience designing evaluation frameworks for LLM output quality in high-stakes domains (legal, medical, or similar), including hallucination detection.
Comfort in a fast-paced, high-ambiguity environment, with strong ownership and a bias for systematic process-building over one-off fixes.
Excellent communication skills; able to translate complex model evaluation findings into clear recommendations for technical and non-technical stakeholders.

Nice To Haves

Experience with unstructured medical or legal document processing, or a background in classical ML (statistics, embeddings, RAG).

Responsibilities

Design and operate a systematic process to evaluate new and emerging LLMs across accuracy, relevancy, speed, and cost, benchmarking continuously against tasks in our orchestration pipeline.
Build and maintain evaluation frameworks and pioneer our internal EvalOps culture, measuring output accuracy, relevance, and faithfulness, with a focus on reducing hallucinations in medical record summarization and legal document analysis.
Monitor the LLM landscape for deprecation timelines and replacement models, then own execution end to end, integrating new models into production, adjusting for model behavior, and decommissioning stale or underperforming prompts and endpoints.
Implement optimizations to LLM-based orchestration pipelines for document understanding, medical record summarization, case chronology generation, and drafting support, owning code changes, deployments, and validation with a bias toward surgical execution over wholesale refactors.
Communicate model evaluation findings to product and GTM stakeholders and lead the technical implementation yourself, ensuring clean handoffs from discovery through staging to production.
Implement monitoring and observability for model performance, benchmarking output and cost, detecting drift, and reporting to management on an ongoing basis.
Maintain documentation of evaluation methodologies, model comparisons, transition decisions, and runbooks for systems you own.