Staff Machine Learning Engineer - Agentic AI

ELLKAY, LLC• US,

4d•$170,000 - $190,000•Hybrid

About The Position

ELLKAY is seeking a versatile Full Stack AI/ML Engineer to architect, build, and deploy production-grade AI/ML and Agentic AI systems on AWS. The role combines foundation models, agentic frameworks, healthcare data standards, and serverless cloud architecture to deliver scalable, secure, and compliant AI solutions for healthcare organizations. The engineer will create agentic systems and production foundation-model applications; develop vector search, memory, and classical machine-learning solutions; and build systems that work with healthcare standards and terminology. The role also includes developing APIs and AWS infrastructure, implementing security and observability practices, and establishing rigorous evaluation and continuous-improvement frameworks for models and agents.

Requirements

Proficiency with Python tools and libraries including Pydantic v2, Pandas or Polars, FastAPI, Boto3, Asyncio, and agent/LLM frameworks.
Proven experience designing stateful agents, function/tool-calling interfaces, autonomous reasoning loops, structured output generation, and evaluation harnesses.
Production experience with AWS Lambda, API Gateway, Step Functions, EventBridge, SQS, SNS, ECS Fargate, Aurora PostgreSQL, DynamoDB, S3, and AWS CDK.
Practical knowledge of OpenSearch Serverless, k-NN algorithms, hybrid retrieval, and ranking optimization for RAG and agent memory stores.
Experience with scikit-learn, clustering algorithms, and confidence calibration techniques.
Good understanding of IAM, KMS, secrets management, and OWASP security principles.
10+ years of software engineering experience, including at least 3 years focused on production AI/ML and GenAI/Agentic systems.
Proven track record of deploying production ML and agentic applications at scale.
Demonstrated ability to design and implement end-to-end ML and agent evaluation frameworks.
Well-versed in AI development using tools such as Codex or Claude Code.

Nice To Haves

Working knowledge of FHIR R4, HL7 v2, USCDI, LOINC, SNOMED, and RxNorm.
Experience with healthcare data or regulated industries is strongly preferred.

Responsibilities

Architect, implement, and deploy end-to-end Agentic AI systems using tool calling, stateful multi-agent collaboration, dynamic task routing, reflection loops, and self-correction workflows.
Design and implement foundation-model applications using prompt engineering, structured outputs, function calling, dynamic tool retrieval, and hallucination mitigation.
Build and optimize vector search and long-term memory solutions using OpenSearch Serverless, including hybrid retrieval and multi-stage ranking for RAG and agent context retrieval.
Develop and deploy Amazon Bedrock and Bedrock Agent solutions, including Guardrails for PHI/PII detection and throughput optimization.
Implement classical ML pipelines, including clustering, confidence-score calibration, and model evaluation.
Design systems that work with FHIR R4 resources, HL7 v2 message structures, and USCDI data elements.
Build ontology retrieval and agent tool interfaces using LOINC, SNOMED, and RxNorm terminologies.
Ensure HIPAA compliance and implement security controls for PHI across LLM inputs, tool payloads, and agent memory states.
Build high-performance APIs and agent execution backends using FastAPI and Pydantic v2, asynchronous I/O, retry/backoff mechanisms, state management, and structured error handling.
Design and deploy AWS serverless architectures using Lambda, API Gateway, Step Functions, EventBridge, SQS, and SNS.
Implement container-based agent microservices on ECS Fargate, including autoscaling, service discovery, and Fargate Spot optimization.
Architect data and state solutions using Aurora PostgreSQL Serverless v2, DynamoDB, and S3, with appropriate indexing, partitioning, caching, and lifecycle policies.
Develop production-grade AWS CDK infrastructure supporting multi-tenant provisioning and cross-account deployments.
Implement observability using AWS X-Ray, CloudWatch, agent execution tracing, structured JSON logging, and correlation ID propagation.
Apply security best practices, including least-privilege IAM, KMS envelope encryption, Secrets Manager rotation, JWT validation, tenant isolation, and OWASP API Top 10 awareness.
Design evaluation frameworks for foundation models and AI agents, including tool-selection precision, execution success rates, benchmarks, per-class metrics, held-out test sets, label-leakage prevention, and statistical significance testing.
Implement prompt and agent strategy versioning, human-in-the-loop evaluation pipelines, and A/B testing frameworks.