Engineer Lead (FDE) - AI Assurance and Evaluations

Elevance Health•Indianapolis, IN

1d•Hybrid

About The Position

The Engineer Lead will be responsible for full delivery of end to end system development and maintenance on medium Enterprise wide technology platforms. This role requires associates to be in-office 1 - 2 days per week, fostering collaboration and connectivity, while providing flexibility to support productivity and work-life balance. This approach combines structured office engagement with the autonomy of virtual work, promoting a dynamic and adaptable workplace. Alternate locations may be considered if candidates reside within a commuting distance from an office. Candidates not within a reasonable commuting distance from the posting location(s) will not be considered for employment, unless an accommodation is granted as required by law.

Requirements

Requires an BA/BS degree in Information Technology, Computer Science or related field of study and a minimum of 7 years related experience; multi-platform, multi-dimensional experience, and expert level experience with business and technical applications; or any combination of education and experience, which would provide an equivalent background.

Nice To Haves

Proficiency with Python for building evaluation utilities, test harnesses, data pipelines, and automation scripts.
Experience with API testing (REST/JSON), including authentication patterns, contract validation, negative testing, and performance/smoke testing.
Hands-on test automation experience (unit/integration/end-to-end), including building repeatable regression suites and integrating tests into CI pipelines.
LLM basics, including core concepts such as tokens/context windows, temperature/top-p sampling, embeddings, retrieval-augmented generation (RAG) fundamentals, and common failure modes (hallucinations, prompt injection, bias).
Familiarity with prompting and evaluation frameworks (prompt versioning, test cases, golden datasets, automated scoring), including qualitative + quantitative evaluation approaches.
Data analytics skills to measure model/application quality and operational performance (e.g., KPI definition, experiment analysis, dashboards; comfort with SQL and basic statistical concepts preferred).
Experience mentoring others and provide troubleshooting support strongly preferred.

Responsibilities

Maintains active relationships with customers to determine business requirements and leads requirements gathering meetings.
Leads requirements gathering meetings and reviews designs with the business.
Leads efforts with Web and marketing team to increase the presence of web products.
May implement improvements in stability, performance, and scalability across major business-critical systems.
May implement process to reduce barriers and roadblocks in projects, services, and processes in order to operate more efficiently.
Owns the change request process and coordinates with other teams as necessary.
Develops and owns list of final enhancements.
Develops and defines application scope and objectives and supervises the preparation of technical and/or functional specifications from with programs will be written.
Performs technical design reviews and code reviews.
Ensures unit test is completed and meets the test plan requirements, system testing is completed and system is implemented according to plan.
Responsible for delivery of application technology solutions and data information planning effort.
Coordinates and manages on-call support and owns the system monitoring process.
Owns the technical development environment and works on the Enterprise team.
Leads multiple or large projects and facilitates large group JAD sessions for requirements, modeling in several disciplines.
Leads vendor evaluation and analysis.