Engineer Lead (FDE) - AI Assurance and Evaluations

Elevance Health•Indianapolis, IN

5d•Hybrid

About The Position

The Engineer Lead (FDE) – AI Assurance and Evaluations is responsible for end-to-end system development and maintenance on medium enterprise-wide technology platforms. The role maintains active relationships with customers to understand business requirements, leads requirements gathering and design reviews, and oversees technical development and delivery. It also owns change requests, application scope and objectives, technical development environments, and system monitoring, while coordinating testing, implementation, and on-call support. The position leads multiple or large projects, facilitates large-group JAD sessions, and conducts vendor evaluation and analysis. The role is hybrid, requiring associates to work in an office 1–2 days per week. Candidates must live within reasonable commuting distance of an office unless an accommodation is granted as required by law. Alternate locations may be considered for candidates within commuting distance.

Requirements

BA/BS degree in Information Technology, Computer Science, or a related field of study, and at least 7 years of related experience; alternatively, an equivalent combination of education and experience.
Multi-platform and multi-dimensional experience.
Expert-level experience with business and technical applications.

Nice To Haves

Proficiency with Python for building evaluation utilities, test harnesses, data pipelines, and automation scripts.
Experience with REST/JSON API testing, including authentication patterns, contract validation, negative testing, and performance or smoke testing.
Hands-on test automation experience across unit, integration, and end-to-end testing, including repeatable regression suites and CI pipeline integration.
Understanding of LLM concepts including tokens and context windows, temperature and top-p sampling, embeddings, RAG fundamentals, and failure modes such as hallucinations, prompt injection, and bias.
Familiarity with prompting and evaluation frameworks, including prompt versioning, test cases, golden datasets, automated scoring, and qualitative and quantitative evaluation approaches.
Data analytics skills for measuring model and application quality and operational performance, including KPI definition, experiment analysis, dashboards, SQL, and basic statistical concepts.
Experience mentoring others and providing troubleshooting support.

Responsibilities

Maintain active customer relationships to determine business requirements.
Lead requirements-gathering meetings and review designs with the business.
Lead efforts with the Web and marketing teams to increase the presence of web products.
Implement improvements in stability, performance, and scalability across major business-critical systems, as needed.
Implement processes to reduce project, service, and process barriers and improve efficiency, as needed.
Own the change request process and coordinate with other teams as necessary.
Develop and own the list of final enhancements.
Define application scope and objectives and supervise preparation of technical and functional specifications.
Perform technical design reviews and code reviews.
Ensure unit testing meets test-plan requirements, system testing is completed, and implementation follows the plan.
Deliver application technology solutions and support data information planning efforts.
Coordinate and manage on-call support and own the system monitoring process.
Own the technical development environment and work on the Enterprise team.
Lead multiple or large projects and facilitate large-group JAD sessions for requirements and modeling across several disciplines.
Lead vendor evaluation and analysis.