Production Support Engineer - LLM Platform

Qube Research & Technologies•Chicago, IL

4d•$150,000 - $220,000

About The Position

Qube Research & Technologies (QRT) is hiring a Production Support Engineer to provide day-to-day production support for its platform for accessing internally hosted and external LLMs. Based in the US, the role supports users during US business hours and coordinates with colleagues in other regions for effective incident handovers. This is a hands-on support role focused on incident resolution, user assistance, and established operational procedures. It does not focus on platform architecture, model development, or infrastructure engineering. The position supports engineering and business users, monitors platform health, troubleshoots gateway and model-provider issues, assists with integrations and access queries, and coordinates escalations with platform engineering, other teams, and vendors. It also contributes to operational checks, support documentation, diagnostic utilities, and cross-region handovers.

Requirements

Relevant experience in production support, application support, or platform operations, with the ability to investigate routine incidents independently and recognise when specialist escalation is required.
Practical Linux troubleshooting skills and familiarity with Windows-based developer environments.
Working proficiency in Python, Bash, and/or PowerShell for operational checks, log analysis, and straightforward support automation.
Experience using SQL to investigate application issues and query operational records, ideally in PostgreSQL.
Practical experience interpreting metrics, logs, traces, and dashboards to distinguish a platform-wide incident from a user-, client-, model-, or region-specific issue.
Confidence troubleshooting API-driven services, including HTTP errors, authentication, networking, timeouts, retries, rate limiting, and streaming connections.
Hands-on experience supporting LLM-based platforms or services, with a working understanding of LLM APIs, including prompts, context limits, token usage, model identifiers, and streaming responses.
Familiarity with public-cloud services, ideally AWS, and the ability to inspect containerised workloads using established tools and procedures.
Clear communication with technical and non-technical users, including concise incident updates, realistic expectations, and well-documented escalations.
Awareness of access controls, sensitive-data handling, and safe diagnostic practices, including avoiding exposure of prompts, credentials, or confidential content in support records.
A methodical approach to troubleshooting, prioritisation, and cross-region handovers.

Nice To Haves

Experience supporting AWS Bedrock or other managed model APIs.
Experience supporting AI coding assistants, API clients, or developer workspaces such as Claude Code, Codex CLI, OpenCode, PI, or Coder.
Familiarity with Grafana, Coralogix, or equivalent monitoring and log-analysis tools.
Exposure to LLM gateways or proxies such as LiteLLM, or internally hosted inference services such as vLLM.
Experience troubleshooting containerised services on ECS or Kubernetes.
Familiarity with identity and access provisioning, particularly Okta, SCIM, and application-level permissions.
Experience supporting multi-region services and working with globally distributed support and engineering teams.
Experience coordinating support cases with cloud providers or software vendors during service degradation.

Responsibilities

Provide first- and second-line support for QRT’s LLM platform, investigate issues raised by engineering and business users, maintain support tickets, and follow issues through to resolution or engineering escalation.
Monitor platform health using existing dashboards, logs, and alerts; identify availability, latency, and error-rate problems; and perform approved recovery actions using established runbooks.
Troubleshoot gateway and model-provider issues, including authentication failures, rate limits, timeouts, connection resets, streaming failures, and regional performance differences.
Determine whether incidents originate in a user’s client or configuration, the platform gateway, an internally hosted model, or an external LLM provider; collect diagnostic evidence and coordinate escalation to the appropriate team or vendor.
Support coding assistants, API clients, and developer workspaces connected to QRT’s LLM platform, including Claude Code, Codex CLI, OpenCode, PI, and Coder; investigate compatibility issues following client or vendor updates.
Help users configure supported integrations, select the appropriate regional endpoint, and migrate API access from Open WebUI to the direct gateway.
Investigate access and model-availability queries, check user permissions, and escalate provisioning or entitlement issues through established support and governance processes.
Carry out operational checks for releases and configuration changes, including non-production validation, post-release smoke testing, and approved rollback procedures, in coordination with platform engineering.
Use existing observability tools to investigate time to first token, time per output token, percentile latency, and failure rates; flag recurring issues and monitoring gaps to the platform team.
Maintain runbooks, troubleshooting guides, and known-issue records; provide clear incident updates and reliable handovers between regional support teams.
Maintain small support scripts and diagnostic utilities to reduce repetitive checks and improve evidence collection.