AI Engineer

KēSTA I.T.•Midvale, UT

8d•Hybrid

About The Position

KēSTA I.T. is seeking an experienced AI Agent Developer & Test Engineer to join a cutting-edge team focused on building, testing, and supporting next-generation AI-powered applications. This role combines hands-on AI development with rigorous testing and quality assurance responsibilities, ensuring intelligent agent workflows perform reliably, securely, and consistently throughout the software development lifecycle. The ideal candidate will possess strong experience with agent-based architectures, Python development, automated testing frameworks, cloud environments, and modern observability practices. This individual will play a key role in designing, validating, and supporting AI-driven solutions operating in production environments.

Requirements

Professional experience developing AI-powered applications using agent frameworks such as LangGraph and/or LangChain.
Strong Python development experience.
Experience with automated testing frameworks including pytest and pytest-asyncio.
Experience designing and executing unit tests for complex workflow-based applications.
Experience testing distributed or orchestrated agent workflows from end to end.
Strong understanding of test isolation, mocking strategies, and simulation techniques.
Experience managing regression testing programs and quality assurance processes.
Hands-on experience with Playwright or similar end-to-end testing frameworks.
Experience producing and interpreting test coverage metrics and reporting.
Familiarity with structured output validation and JSON schema testing.
Experience implementing monitoring and observability solutions for production applications.
Cloud application development and support experience within modern cloud platforms (GCP preferred).
Strong analytical, troubleshooting, and problem-solving skills.
Excellent communication and collaboration abilities.

Nice To Haves

Experience with AI evaluation and observability tools such as LangSmith, RAGAS, or similar platforms.
Experience managing prompt versioning and prompt lifecycle processes.
Knowledge of adversarial testing, red-team testing, and AI security validation methodologies.
Experience monitoring token consumption, model utilization, and AI operational costs.
Experience working with document extraction, OCR, or intelligent document processing solutions.
Experience supporting applications within regulated industries such as financial services, healthcare, or government.
Google Cloud Platform (GCP) Cloud Run
Terraform
Azure DevOps CI/CD pipeline design and automation
Infrastructure-as-Code (IaC) best practices

Responsibilities

Design, develop, and maintain AI agent workflows utilizing modern agent orchestration frameworks.
Build and execute comprehensive testing strategies for AI agent systems, including unit, integration, regression, and end-to-end testing.
Develop node-level test coverage to validate individual workflow components in isolation.
Perform full workflow and graph-level testing to validate complete agent execution paths.
Implement effective LLM mocking and simulation strategies to support deterministic and repeatable testing.
Create and maintain regression test suites to ensure defects remain resolved over time.
Design testing methodologies that address non-deterministic AI behaviors through controlled inputs, seeded testing, and assertion strategies.
Develop and maintain automated end-to-end testing coverage for user-facing applications and workflows.
Monitor test coverage metrics and continuously identify opportunities to improve system quality and reliability.
Validate structured outputs, schema compliance, and data integrity across AI workflows.
Implement production monitoring, logging, tracing, and observability practices for AI systems.
Support cloud-based application deployments and ongoing operational maintenance.
Collaborate closely with software engineers, architects, product owners, and QA teams to deliver high-quality AI solutions.
Participate in troubleshooting, root-cause analysis, and production support activities.