AI-Driven Data Profiling Engineer

American IT Systems•New York City, NY

About The Position

The AI-Driven Data Profiling Engineer will support a proof of concept focused on profiling MGA BDX data through an autonomous agent workflow. The role works closely with data profiling specialists, data consumers, onboarding teams, and enterprise data stakeholders to refine agent outputs and accelerate onboarding of MGA programs into the BDX management platform and enterprise data hub. The engineer will refine the autonomous workflow, improve the quality and usability of profiling outputs, and ensure results meet the needs of downstream data consumers. The work includes profiling a backlog of approximately 80–100 MGA programs, validating results through reviewer-agent and human review, documenting findings and workflow changes, and contributing recommendations about wider adoption of autonomous-agent approaches.

Requirements

Strong background in the insurance industry.
Experience working with Managing General Agents (MGAs) and/or insurance distribution data.
Understanding of BDX data, bordereaux, program data, and insurance data onboarding processes.
Familiarity with insurance data structures, including policy, premium, claims, exposure, and related data concepts.
Experience working with data used for insurance operations, reporting, underwriting, finance, or enterprise data platforms.
Strong data analysis and data profiling experience.
Experience identifying data quality issues, anomalies, inconsistencies, missing values, duplicates, and structural variations.
Ability to interpret complex, semi-structured, and inconsistent data sources.
Experience defining profiling rules, validation logic, business rules, and data quality metrics.
Ability to translate profiling results into clear recommendations for technical and business audiences.
Strong understanding of data lineage, metadata, mappings, data contracts, and onboarding requirements.
Experience working with data consumers to refine requirements and improve the usefulness of data outputs.
Proficiency in agentic engineering and autonomous workflow design.
Hands-on experience designing, configuring, testing, and refining AI-agent workflows.
Experience with the listed Kiro agents: Discovery Agent, PMA Extraction Agent, Planning Agent, Profiling Agent, Reviewer Agent, and Dashboard Updater Agent.
Experience with other generative AI, coding-agent, or orchestration tools.
Ability to design effective prompts, agent instructions, task decomposition, validation steps, and review mechanisms.
Experience creating agent workflows that include planning, execution, review, exception handling, and iterative improvement.
Understanding of human-in-the-loop controls and approaches for ensuring reliable and auditable AI outputs.
Ability to troubleshoot agent behavior and improve consistency, accuracy, and explainability.

Responsibilities

Refine and enhance the autonomous agent workflow for MGA BDX data profiling.
Work with data profiling teams and business data consumers to define profiling requirements and expected outputs.
Review and improve the behavior, prompts, instructions, controls, and outputs of the Kiro agents.
Support discovery and extraction of PMA-related information from MGA BDX data.
Develop and refine profiling plans for individual MGA programs.
Run the profiling workflow across a backlog of approximately 80–100 MGA programs.
Identify recurring data quality issues, structural variations, missing fields, anomalies, and business-rule exceptions.
Continuously improve the workflow based on findings from each profiling cycle.
Validate profiling results through reviewer-agent and human review processes.
Ensure profiling outputs are clear, actionable, traceable, and suitable for downstream teams.
Provide profiling results to teams responsible for onboarding data into the BDX management platform and enterprise data hub.
Maintain documentation of profiling rules, discovered patterns, exceptions, decisions, and workflow changes.
Help define dashboards, metrics, and reporting requirements for profiling progress and data quality.
Collaborate with business, data, engineering, and platform stakeholders to resolve issues and clarify requirements.
Contribute recommendations regarding the potential wider adoption of an autonomous-agent approach.