AI/LLM Safety Engineer

Propio • Overland Park, KS

1d

About The Position

The AI/LLM Safety Engineer will join Propio’s AI team and own how safely its models and agents behave in production, focusing on AI Safety, Trust & Safety, and Responsible AI. The role’s primary focus is agent safety: ensuring systems that call tools and take actions operate within well-defined, well-tested boundaries. The engineer will design evaluations to detect unsafe behavior, build guardrails to prevent it, and lead red-teaming to identify gaps before they affect users or are exploited by attackers.

The role includes designing and maintaining safety evaluation frameworks; leading structured red-team exercises; developing guardrails and runtime controls; threat-modeling agent workflows; reviewing reinforcement-learning environments and trajectory data; instrumenting production AI features for safety monitoring; preparing governance evidence; and collaborating with Product, UX, Research, Data, environment, and agent engineering teams.

Propio describes a culture that encourages ownership, innovation, continuous learning, and measurable impact. The position is listed as full-time.

Requirements

• Bachelor's or Master's degree in Computer Science, Software Engineering, Cybersecurity, or a related technical field, or equivalent practical experience.
• At least 4 years building production software, with direct experience working on or securing ML/LLM systems.
• Strong software engineering skills and ability to write production-grade code, primarily in Python, beyond scripting or notebook prototyping.
• Solid understanding of LLMs and ML, including how models work, prompt engineering, and safety implications of fine-tuning and retrieval-augmented generation, such as unsafe retrieval, tool misuse, and data exfiltration.
• Security mindset and demonstrated threat-modeling ability; ability to threat-model AI workflows and familiarity with access control, data retention, and incident-response fundamentals.
• Familiarity with the LLM attack surface, including prompt injection, jailbreaks, data poisoning, and supply-chain risk, and working knowledge of the OWASP LLM Top 10.
• Hands-on experience with at least one of safety evaluation or red teaming, including ability to explain a real finding and how it was remediated.

Nice To Haves

• Hands-on experience with safety tooling such as garak, PyRIT, promptfoo, Giskard, and NeMo Guardrails, and ability to articulate trade-offs between them.
• Visible output in AI safety or security, such as publications at relevant venues, open-source contributions, or responsible disclosures on frontier models with public write-ups.
• Familiarity with AI governance and compliance frameworks such as NIST AI RMF, ISO/IEC 42001, and the EU AI Act, and ability to translate compliance requirements into engineering tasks.
• Engineering experience with agents, reinforcement-learning environments, and/or tool use.
• Practical experience with threat-modeling methodologies such as MITRE ATLAS and STRIDE/PASTA.

Responsibilities

• Design and maintain safety evaluation frameworks, including adversarial prompt sets, scenario-based test suites, and regression suites, to validate model and agent updates before release.
• Lead structured red-teaming exercises covering jailbreaks, prompt injection, tool misuse, and data exfiltration; document findings and drive issues through remediation and closure.
• Build and iterate on guardrail logic, including input/output filtering, tool-boundary constraints, action validation, sensitive-data redaction, and policy prompting.
• Integrate safety checks into CI/CD and runtime so unsafe behavior is intercepted before reaching users.
• Threat-model agent scenarios, including tool-call boundaries, sandbox isolation, and least-privilege access, with attention to data exfiltration and irreversible actions through chained tool calls.
• Review reinforcement-learning environments and trajectory data, partnering with environment and agent engineering teams to embed safety constraints in the environments.
• Instrument AI features with structured logging, tracing, and metrics to detect unsafe patterns and production regressions.
• Prepare governance-review evidence, including test reports, evaluation summaries, and mitigation validation, aligned with internal Responsible AI standards.
• Collaborate with Product and UX to improve safety interactions such as warnings, confirmations, refusal messaging, and feedback collection.
• Align evaluation goals with Research and Data teams.

Benefits

• Opportunities to learn, develop, and expand capabilities while building a meaningful career.
• Opportunity to make a difference and see the connection between work and company success.
• Encouragement of curiosity, creativity, and challenging established approaches.
• Room to grow, pursue leadership opportunities, build expertise, and expand career paths, with support.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level