Text Generation and Embedding API/Router

FastAPI server that routes LLM chat requests to vLLM and provides local text embeddings.

Quick Start

Prerequisites

• Python 3.8+ with CUDA support
• vLLM
• PM2 process manager

Installation

pip install -r requirements.txt

Running

1. Start vLLM server with:

• Extended context size (to 128k) through YaRN
• Hermes-style tools enabled

pm2 start --name vllm "vllm serve Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
  --dtype half \
  --quantization gptq \
  --gpu-memory-utilization 0.7 \
  --dtype half --quantization gptq \
  --rope-scaling '{\"rope_type\":\"yarn\",\"factor\":4.0,\"original_max_position_embeddings\":32768}' \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --port 8001"

2. Start the router:

pm2 start --name llm_proxy 'uvicorn app.main:app --host 0.0.0.0 --port 4444'

Note on nginx

If Nginx proxy is preferred refer to autodeploy-nginx.sh script. It basicly involves setting up Nginx on 4444 to vLLM and app instances.

Environment Variables

Variable	Description	Default
TG_MODEL	Text generation model	Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
EMBEDDING_MODEL	Embedding model	Qwen/Qwen3-Embedding-0.6B
VLLM_CHAT_URL_BASE	vLLM server base URL	http://localhost:8001
MAX_EMBED_TEXT_SIZE	Max text length for embeddings	16384

API Endpoints

• POST /health - Service health check
• POST /v1/chat/completions - OpenAI-compatible chat completions (proxied to vLLM)
• POST /v1/embeddings - Text embeddings (local inference)

Example Usage

# Chat completion
curl localhost:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Hello!"}]}'

# Text embeddings
curl localhost:8002/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"texts":["Hello world"]}'

Deployment

For production deployment with all dependencies.

To enable daily Vector DB backups, place your SSH private key file for the backup host to /tmp/backup_key and set BACKUP environment variables.

export BACKUP_HOST="backup_host_ip"
export BACKUP_USER="backup_user"
./scripts/autodeploy.sh