Software Development Engineer in Test

Sofico • Dallas, TX

1d • Remote

About The Position

SOFICO Intelligence is hiring an SDET to own quality for its autonomous, multi-agent AI platform, Velona. The role focuses on making non-deterministic AI trustworthy by designing and building the evaluation harness that supports AI governance and release confidence. This includes evaluating whether AI findings are accurate and grounded in real data, detecting hallucinations and quality drift, enforcing tenant isolation and agent guard rails, and testing performance under realistic and adversarial load. The position involves production engineering on a real AI system at scale, rather than running scripts against a static application.

Approximately 85% of the role is focused on agentic AI quality, including evaluation datasets and scoring, grounding and faithfulness checks, orchestration testing, security red-teaming, guard-rail validation, performance testing, and automated testing for platform services and APIs. The remaining approximately 15% supports quality engineering across the broader SOFICO Intelligence platform and, where relevant, the wider SOFICO engineering environment. The SDET will partner with AI and data engineers to define quality bars and acceptance criteria and help improve the supervisor review-and-retry loop.

SOFICO Intelligence builds AI and intelligence technology to surface fraud, billing errors, risk, and other insights from complex financial, leasing, fleet, and mobility-management data. It is backed by SOFICO, a global ERP software provider in those domains. The team is remote-first and works with modern Go services, LLM orchestration, proprietary fleet data, and a broad testing and infrastructure stack.

Requirements

• Proven experience spanning software development and quality assurance or quality engineering.
• Strong programming skills in at least one language; the team primarily uses Go and Python.
• Hands-on experience with open-source automated testing tools and frameworks.
• Solid understanding of RESTful APIs and ability to perform HTTP requests using the command line or a tool such as Postman.
• Comfort working from the command line and reasoning about distributed, concurrent systems.
• Hands-on experience testing, evaluating, or building with LLM-based or agentic systems, including direct work with model APIs, tool-calling, or multi-agent workflows.
• Demonstrated ability to evaluate non-deterministic AI output against a measurable quality bar, such as through evaluation harnesses, grounding and faithfulness scoring, regression detection, or red-teaming.
• Working knowledge of LLM quality concepts, including prompting, context windows and token limits, hallucination and grounding, and sources of output non-determinism.
• Fluent in English for interaction with international teams.
• A structured, analytical approach to building scalable evaluation systems, collaborative problem-solving skills, ability to prioritize work, and openness to constructive criticism.

Nice To Haves

• Depth in AI evaluation methodology, such as designing golden datasets, LLM-as-judge scoring, supervisor or critic patterns, or automated regression detection for agentic pipelines.
• Experience with Go, data warehouses or SQL analytics such as Databricks, event-driven systems, or real-time streaming.
• Familiarity with enterprise back-office or ERP environments, including Java back ends and relational databases such as Oracle, MSSQL, or Postgres.
• Exposure to automotive finance, leasing, fleet, or mobility.

Responsibilities

• Design and build the evaluation harness for agentic AI, including golden datasets, automated scoring, faithfulness and grounding checks, and regression detection across prompt, model, and agent changes.
• Verify that every AI insight and number is traceable to real data, and build automated checks to prevent hallucinations from reaching production.
• Test multi-agent orchestration end to end, including tool-calling loops, supervisor-specialist pipelines, autonomous background jobs, and the synthesis of findings into reports.
• Red-team the system for prompt injection, jailbreaks, and tool misuse, and prove multi-tenant data isolation.
• Validate AI guard rails, including iteration limits, agent-recursion depth, and token and cost ceilings.
• Pressure-test performance and scale, including real-time streaming, high-concurrency conversations, latency, and cost under realistic and adversarial loads.
• Build automated tests for Go microservices, data and SQL tools, and APIs, and integrate them into CI/CD for continuous evaluation.
• Partner with AI and data engineers to define quality bars and acceptance criteria, and help evolve the supervisor review-and-retry loop.
• Strengthen automated testing across the broader SOFICO Intelligence platform and, where it intersects, the wider SOFICO engineering environment.
• Help raise quality engineering standards across the combined group.

Benefits

• Health care, including medical, dental, and vision coverage for US employees.
• Flexible PTO policy.
• Conferences, advanced training, and certifications to support professional growth.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed