Data Engineer, AI Support

Insurance Institute for Business & Home Safety•Richburg, SC

About The Position

The Data Engineer, AI Support is a technical partner in IBHS’s responsible adoption and application of artificial intelligence, machine learning, and advanced analytics. The role translates enterprise-wide needs into scalable data solutions that support decision-making, operational effectiveness, and employee capabilities. It combines data engineering, solution development, technical consultation, and employee support, working across business and technical teams to evaluate opportunities, develop and implement solutions, assess performance and risk, and help employees use approved AI-enabled tools effectively. The position supports IBHS’s ability to use data and AI thoughtfully and responsibly by connecting organizational priorities with technical capabilities, identifying useful applications, improving access to actionable information, and building confidence in AI-enabled solutions. It also helps establish consistent practices for solution quality, documentation, data handling, human review, and responsible AI use. The role works with AI researchers and Data Engineering to prepare and improve data for research and machine learning projects; build data workflows; investigate datasets and quality issues; prepare data for language, vision, multimodal, and retrieval-based AI applications; and maintain consistent, traceable, reproducible, well-documented datasets. It also automates recurring data tasks, develops summaries and visualizations, supports integration across research tools and internal systems, protects sensitive information, and contributes to an evolving experimental research environment.

Requirements

Bachelor’s degree in data science, statistics, computer science, mathematics, engineering, or a related quantitative field.
Experience building ETL/data pipelines to clean, transform, integrate, and prepare structured, semi-structured, and unstructured data for AI/ML workflows.
Strong Python data manipulation skills, including efficient use of vectorized libraries for large-scale data processing, exploration, and visualization.
Working knowledge of SQL, NoSQL, data modeling, columnar formats, and modern data storage technologies.
Familiarity with preparing and versioning LLM/VLM training and evaluation datasets, including QA, preference/RL, multimodal, and human-annotated data.
Familiarity with embedding pipelines, vector databases, semantic search, RAG, and metadata-aware retrieval workflows.
Exposure to graph databases, knowledge graphs, and graph-based data modeling for AI applications.
Understanding of data quality, schema validation, dataset versioning, metadata, lineage, and reproducible train/validation/test splits with leakage prevention.
Familiarity with distributed data processing and workflow orchestration concepts such as DAGs, task dependencies, scheduling, and pipeline monitoring.
Comfort working in Linux environments with Bash/shell scripting and basic automation.
Familiarity with experiment tracking and LLM observability tools such as Weights & Biases and Langfuse.
Basic understanding of PII handling, masking, hashing, tokenization, and de-identification within data pipelines.
Familiarity with CI/CD and infrastructure automation tools such as GitHub Actions, GitLab CI, and Terraform.
Comfort working with research datasets that may be incomplete, inconsistent, or evolving, and ability to investigate the data before implementing a solution.
Strong written communication, presentation, and technical-documentation skills.
Ability to build effective working relationships across business and technical functions.
Ability to exercise sound judgment, manage multiple priorities, and work independently while contributing to cross-functional initiatives.

Nice To Haves

Master’s degree in data science, statistics, computer science, artificial intelligence, machine learning, or a related field.
Experience supporting AI/ML research, scientific computing, or other data-intensive research environments.
Hands-on experience with cloud data platforms or services in Azure, AWS, or Google Cloud.
Experience with data orchestration and distributed processing tools such as Airflow, Prefect, Dagster, Spark, or similar technologies.
Familiarity with data annotation and human-in-the-loop platforms such as Label Studio or Prodigy, particularly for machine learning or multimodal datasets.

Responsibilities

Work with AI researchers and Data Engineering to prepare, organize, and improve data used across research and machine learning projects.
Build reliable workflows that transform research data from raw sources into usable datasets for analysis, experimentation, training, and evaluation.
Explore new datasets, identify quality issues or gaps, and help determine how data should be structured and used.
Prepare datasets for language, vision, multimodal, and retrieval-based AI applications.
Help ensure research datasets remain consistent, traceable, reproducible, and well documented as they evolve.
Automate recurring data preparation and processing tasks to make research workflows more efficient and repeatable.
Develop summaries and visualizations that help the team understand datasets, patterns, and potential issues.
Support data integration across research tools, internal systems, and AI platforms.
Protect sensitive information and follow appropriate data handling practices throughout the data lifecycle.
Contribute to an experimental research environment where datasets, methods, and requirements may change as projects develop.
Stay current on developments in artificial intelligence, machine learning, data science, and emerging analytical technologies.