Backend Engineer - AI Gateway (Bilingual Korean)

Cinter Technology Services•NJ at first and then Plano TX, NJ

2d•Onsite

About The Position

We are building an AI Gateway — a high-performance, secure intermediary layer that routes, inspects, and governs all traffic between enterprise clients and LLM/AI service providers. This role requires deep expertise in network programming, async Python, and cloud infrastructure to design and operate a latency-sensitive, policy-driven gateway that handles massive concurrent streaming connections at scale.

Requirements

Async Python & Network Programming (3 + yrs) – Strong hands‑on experience with asyncio, aiohttp/anyio, low‑level TCP/UDP sockets, WebSocket & SSE servers, TLS handshake, connection‑pooling, keep‑alive, and zero‑copy buffering.
Streaming & Large‑Payload Handling – Ability to manage chunked transfer, bidirectional streaming, back‑pressure and high‑throughput data paths for real‑time LLM inference.
Protocol Knowledge – Working familiarity with HTTP/2 & HTTP/3 (multiplexing, HPACK), gRPC, and occasional custom protocol parsers.
FastAPI‑Based Backend (3 + yrs) – Design, develop and test production‑grade REST/Streaming APIs; pragmatic use of Pydantic, dependency injection and OpenAPI docs.
Containerization & Kubernetes – Docker, AKS/EKS/GKE deployment experience; HPA/VPA scaling, rolling updates, and service‑mesh (e.g., Istio/Envoy) basics.
Async Messaging & Event‑Driven Design – Production use of Redis Streams, RabbitMQ or Kafka for decoupled task queues, retries and back‑off logic.
PostgreSQL & Vault – Advanced query tuning, partitioning, PgBouncer pooling, plus HashiCorp Vault (dynamic secrets, transit encryption) for key management.
Cloud Networking & Security – L4/L7 load balancing, TLS termination, reverse‑proxy (NGINX/Envoy), mTLS between services, and automated cert lifecycle (ACME/Let’s Encrypt).
IaC & CI/CD – Terraform modules for multi‑env infra, plus GitHub Actions/GitLab CI pipelines with container image scanning and canary/blue‑green deployments.
Strong communication, ownership, and problem-solving skills.
Ability to collaborate effectively across business and technical teams.
English and Bilingual Korean language proficiency.

Nice To Haves

Enterprise Security Gateways – Integration with CASB/DLP/WSS or similar edge‑security platforms; centralized logging & audit trails.
IAM & SSO – Configuring Azure AD/Entra ID, Okta, SAML 2.0 or OIDC SSO flows; managing Conditional Access & SCIM provisioning.
AI/LLM Pipelines – Building prompt‑engineering workflows, PII anonymization, and monitoring Azure OpenAI (or comparable) usage, rate limits & cost.
API‑Gateway Operations – Custom webhook/REST integrations, circuit‑breaker patterns, rate‑limiting, auto‑retry and health‑checking.
Observability & FinOps – End‑to‑end monitoring with Datadog, Prometheus/Grafana or Azure Monitor; proactive cost‑optimization and 24/7 incident response (SLA/SLO).

Responsibilities

Design and operate a latency-sensitive, policy-driven gateway that handles massive concurrent streaming connections at scale.
Route, inspect, and govern all traffic between enterprise clients and LLM/AI service providers.
Manage chunked transfer, bidirectional streaming, back-pressure and high-throughput data paths for real-time LLM inference.
Design, develop and test production-grade REST/Streaming APIs.
Deploy and manage containerized applications using Docker and Kubernetes (AKS/EKS/GKE).
Implement asynchronous messaging and event-driven designs using Redis Streams, RabbitMQ or Kafka.
Manage PostgreSQL databases, including query tuning, partitioning, and connection pooling.
Utilize HashiCorp Vault for key management, including dynamic secrets and transit encryption.
Configure and manage cloud networking and security components, including load balancing, TLS termination, reverse proxies, and mTLS.
Implement Infrastructure as Code (IaC) using Terraform and manage CI/CD pipelines using GitHub Actions/GitLab CI.
Integrate with enterprise security gateways (CASB/DLP/WSS).
Configure IAM and SSO solutions (Azure AD/Entra ID, Okta, SAML 2.0, OIDC).
Build prompt-engineering workflows, PII anonymization, and monitor AI/LLM service usage.
Implement API Gateway operations such as custom webhook/REST integrations, circuit-breaker patterns, rate-limiting, auto-retry and health-checking.
Implement observability and FinOps practices using tools like Datadog, Prometheus/Grafana or Azure Monitor.
Participate in 24/7 incident response.