Backend Engineer_AI Gateway

SBT Global•Englewood Cliffs, NJ

1d•Onsite

About The Position

We are building an AI Gateway — a high-performance, secure intermediary layer that routes, inspects, and governs all traffic between enterprise clients and LLM/AI service providers. This role requires deep expertise in network programming, async Python, and cloud infrastructure to design and operate a latency-sensitive, policy-driven gateway that handles massive concurrent streaming connections at scale.

Requirements

3+ years of strong hands-on experience with asyncio, aiohttp/anyio, low-level TCP/UDP sockets, WebSocket & SSE servers, TLS handshake, connection-pooling, keep-alive, and zero-copy buffering.
Ability to manage chunked transfer, bidirectional streaming, back-pressure and high-throughput data paths for real-time LLM inference.
Working familiarity with HTTP/2 & HTTP/3 (multiplexing, HPACK), gRPC, and occasional custom protocol parsers.
3+ years of experience designing, developing, and testing production-grade REST/Streaming APIs using FastAPI, Pydantic, dependency injection, and OpenAPI docs.
Experience with Docker, AKS/EKS/GKE deployment, HPA/VPA scaling, rolling updates, and service-mesh (e.g., Istio/Envoy) basics.
Production use of Redis Streams, RabbitMQ or Kafka for decoupled task queues, retries and back-off logic.
Advanced PostgreSQL query tuning, partitioning, PgBouncer pooling.
Experience with HashiCorp Vault (dynamic secrets, transit encryption) for key management.
Experience with L4/L7 load balancing, TLS termination, reverse-proxy (NGINX/Envoy), mTLS between services, and automated cert lifecycle (ACME/Let’s Encrypt).
Experience with Terraform modules for multi-env infra.
Experience with GitHub Actions/GitLab CI pipelines with container image scanning and canary/blue-green deployments.
Integration experience with CASB/DLP/WSS or similar edge-security platforms.
Experience with centralized logging & audit trails.
Configuring Azure AD/Entra ID, Okta, SAML 2.0 or OIDC SSO flows.
Managing Conditional Access & SCIM provisioning.
Building prompt-engineering workflows, PII anonymisation.
Monitoring Azure OpenAI (or comparable) usage, rate limits & cost.
API-Gateway Operations: Custom webhook/REST integrations, circuit-breaker patterns, rate-limiting, auto-retry and health-checking.
End-to-end monitoring with Datadog, Prometheus/Grafana or Azure Monitor.
Proactive cost-optimisation and 24/7 incident response (SLA/SLO).

Nice To Haves

Korean Bilingual Highly Preferred

Responsibilities

Design, develop and test production-grade REST/Streaming APIs using FastAPI.
Manage chunked transfer, bidirectional streaming, back-pressure and high-throughput data paths for real-time LLM inference.
Deploy and manage containerized applications using Docker and Kubernetes (AKS/EKS/GKE), including HPA/VPA scaling, rolling updates, and service mesh basics.
Implement decoupled task queues, retries, and back-off logic using async messaging systems like Redis Streams, RabbitMQ, or Kafka.
Tune PostgreSQL performance, manage partitioning, and configure PgBouncer pooling.
Manage secrets and encryption using HashiCorp Vault.
Configure and manage cloud networking components such as L4/L7 load balancing, TLS termination, reverse proxies (NGINX/Envoy), and mTLS.
Automate certificate lifecycle management using ACME/Let's Encrypt.
Develop Terraform modules for multi-environment infrastructure.
Implement CI/CD pipelines using GitHub Actions/GitLab CI with container image scanning and advanced deployment strategies (canary/blue-green).
Integrate with enterprise security gateways (CASB/DLP/WSS).
Configure centralized logging and audit trails.
Set up and manage IAM and SSO integrations with Azure AD/Entra ID, Okta, SAML 2.0, or OIDC.
Manage Conditional Access and SCIM provisioning.
Build prompt-engineering workflows, PII anonymization, and monitor AI/LLM service usage, rate limits, and costs.
Implement API gateway operations including custom webhook/REST integrations, circuit-breaker patterns, rate-limiting, auto-retry, and health-checking.
Implement end-to-end monitoring using Datadog, Prometheus/Grafana, or Azure Monitor.
Perform proactive cost-optimization and participate in 24/7 incident response with defined SLAs/SLOs.