Software Developer / Engineer

Real Soft • Philadelphia, PA

1d • Hybrid

About The Position

We are seeking a mid-level Software Developer/Engineer with hands-on experience in deploying on-premise LLM solutions and vector databases. The ideal candidate will have strong expertise in Python, RAG pipelines, and enterprise-grade AI system implementation within secure environments.

Requirements

• Strong experience with Python for AI/ML and backend development
• Hands-on experience with open-source LLM deployment (Llama 3, Mistral, Mixtral)
• Experience with CPU-based inference and optimization techniques
• Practical experience with vector databases (Qdrant, Chroma, Milvus, pgvector)
• Proven experience building RAG pipelines
• Knowledge of embeddings, similarity search, and metadata filtering
• Understanding of enterprise security, data privacy, and air-gapped environments

Nice To Haves

• Experience with LangChain or LlamaIndex
• Familiarity with Docker and Kubernetes
• Exposure to Rust, Go, or C++ for high-performance systems
• Experience with LLM inference frameworks (vLLM, llama.cpp, Hugging Face Transformers)
• Prior experience in regulated or enterprise environments

Responsibilities

• Deploy and manage open-source LLMs (e.g., Llama 3, Mistral/Mixtral) in on-prem or private environments
• Develop and optimize LLM inference workflows using Python
• Implement Retrieval-Augmented Generation (RAG) pipelines
• Design and integrate vector database solutions for efficient semantic search
• Perform model quantization and performance tuning for CPU-based inference
• Ensure data privacy, security, and governance compliance in enterprise environments
• Implement access controls, logging, and monitoring mechanisms
• Deliver reference architecture, prototypes, and technical documentation
• Collaborate with internal teams for knowledge transfer and system adoption

What This Job Offers

• Career Level: Mid Level
• Education Level: No Education Listed