Running Inference on Local LLMs with a vLLM Server
This guide explains how to run inference on local LLMs with a vLLM server using GPUs on DCC. The vLLM server supports many concurrent client requests with low latency, making it well suited for HPC environments. Additionally, it provides authentication through a secure key so that only authorized users can access the server.
The code for this example is available at: https://github.com/DukeRC/code/tree/main/Running-Inference-on-Local-LLMs-with-a-vLLM-Server.
As large language models (LLMs) continue to grow in popularity, efficient and scalable inference tools are becoming increasingly important in HPC environments. While Ollama is a strong option for smaller-scale inference, vLLM is better suited for larger-scale HPC workloads allowing for higher throughput, lower latency, and more concurrent requests. For a detailed performance comparison, see Ollama vs. vLLM: A deep dive into performance benchmarking.
This guide walks you through setting up and running inference on local LLMs with a vLLM server on DCC GPUs. For a guide on running inference directly with the vLLM Python API without a server, see Running Inference on Local LLMs with vLLM. The server-based approach allows for more concurrent requests and enable authentication, while the direct Python API approach is simpler for single-user workloads.
Initial setup
Assuming you have a Python environment available (e.g., Miniconda, Miniforge), first create a new environment and activate it.
Then install vLLM and the OpenAI Python API library,
Serving the LLM model
We will serve the Qwen/Qwen2-7B model from Hugging Face on a single NVIDIA RTX 5000 Ada Generation GPU. We will first need to set up some environment variables to ensure that vLLM can access the model directory and connect to Hugging Face to download it if it is not already cached.
First, create a directory in your work directory for this example,
Within this directory, create a .env file to store the environment variables,
echo 'HF_TOKEN=your_hugging_face_api_token' > /work/${USER}/vLLM-server/.env
chmod 600 /work/${USER}/vLLM-server/.env
The 600 permission ensures that only the owner can read and write the file, keeping your Hugging Face API token secure.
Next, add the following to your ~/.bashrc to set the Hugging Face cache directory,
Save the following bash script in the working directory as vllm_server.sh. This script will launch the vLLM server with the specified model and configuration.
#!/bin/bash
# vLLM Server Startup Script
# Usage:
# 1. Set HF_HOME to control where Hugging Face cache is stored.
# export HF_HOME=/path/to/hf_cache
# 2. Set the HF_TOKEN environmental variable in a .env file in the root of the directory.
# 3. Run the script:
# ./vllm_server.sh [ADDITIONAL_VLLM_ARGS...]
#Example:
# ./vllm_server.sh --max-model-len 8192
set -euo pipefail
# Configuration
PORT=8000
MODEL_NAME="Qwen/Qwen2-7B"
SERVED_MODEL_NAME="local-vllm"
API_KEY="your-secret-key"
# Setup Hugging Face parameters
export HF_HOME="${HF_HOME:-/work/${USER}/.huggingface}"
if [[ -f ".env" ]]; then
set -a
source .env
set +a
fi
# Determine tensor parallel size based on CUDA_VISIBLE_DEVICES
if [[ -n "${CUDA_VISIBLE_DEVICES:-}" ]] && [[ "$CUDA_VISIBLE_DEVICES" != "NoDevFiles" ]]; then
IFS=',' read -r -a gpu_ids <<< "$CUDA_VISIBLE_DEVICES"
TENSOR_PARALLEL_SIZE="${#gpu_ids[@]}"
else
TENSOR_PARALLEL_SIZE="1"
fi
# Launch vLLM server
vllm serve \
--host 0.0.0.0 \
--port "$PORT" \
--model "$MODEL_NAME" \
--served-model-name "$SERVED_MODEL_NAME" \
--tensor-parallel-size "$TENSOR_PARALLEL_SIZE" \
--api-key "$API_KEY" \
--trust-remote-code \
"$@" \
> "vllm_server.log" 2>&1 &
# Wait for the vLLM server to become ready by polling its models endpoint,
# and exit if the server process dies during startup.
SERVER_PID=$!
until curl -fsS -H "Authorization: Bearer ${API_KEY}" "http://127.0.0.1:${PORT}/v1/models" >/dev/null 2>&1; do
if ! kill -0 "$SERVER_PID" >/dev/null 2>&1; then
echo "Error: vLLM exited during startup. See vllm_server.log"
exit 1
fi
sleep 2
done
NODE_FQDN="$(hostname -f)"
echo ""
echo "vLLM is serving at: http://${NODE_FQDN}:${PORT}"
echo "Model: ${MODEL_NAME}"
echo "Model Alias: ${SERVED_MODEL_NAME}"
echo "Tensor parallel: ${TENSOR_PARALLEL_SIZE}x GPU"
if (( $# > 0 )); then
echo "Extra vLLM args: $*"
fi
echo ""
echo "Export on client shell:"
echo "export VLLM_HOST=http://${NODE_FQDN}:${PORT}"
echo "export VLLM_API_KEY=${API_KEY}"
echo ""
echo "Stop server:"
echo "kill $SERVER_PID && pkill -f VLLM::"
Feel free to experiment with different models by changing the MODEL_NAME variable. The SERVED_MODEL_NAMEis an alias to the model and can be used to connect the server to an agentic AI coding tool like OpenCode. We will cover this in a future guide. TheAPI_KEY is used to authenticate clients that connect to the server, ensuring that only authorized users can access it.
Let's request a slurm interactive session on an NVIDIA RTX 5000 Ada Generation GPU to serve the model. You may also submit the script as a batch job if you prefer.
Once your allocation is ready, activate the conda environment we created earlier and launch the server,
You should see an output similar to the one below,
vLLM is serving at: http://dcc-core-gpu-ferc-s-aa32-9.rc.duke.edu:8000
Model: Qwen/Qwen2-7B
Model Alias: local-vllm
Tensor parallel: 1x GPU
Export on client shell:
export VLLM_HOST=http://dcc-core-gpu-ferc-s-aa32-9.rc.duke.edu:8000
export VLLM_API_KEY=your-secret-key
Stop server:
kill 567059 && pkill -f VLLM::
Copy the export commands from the output and run them in your client shell to set the VLLM_HOST and VLLM_API_KEY environment variables. These will allow you to connect to the vLLM server from your client applications. The client can be a regular non-GPU node as the inference engine is doing the heavy work on the GPUs of the server node.
Important!
If you will be performing computationally intensive inference tasks, ensure that you are on a compute node requested through a slurm interactive session or submit as a slurm batch job.
Running inference on the model
Now that the client shell knows how to connect to the vLLM server, we can run inference on the served model. The following Python script demonstrates how to connect to the vLLM server and run inference on the served model using the OpenAI-compatible API. Save this script as vllm_client.py in your working directory.
#!/usr/bin/env python
# vLLM client connecting to the OpenAI-compatible API server.
#
# Usage:
# Once you have a vLLM server running, set the VLLM_HOST environment variable to the server's hostname and port.
# Then launch this script with,
# ./vllm_client.py
import os
from openai import OpenAI
HOST = os.environ.get("VLLM_HOST", "http://127.0.0.1:8000")
API_KEY = os.environ.get("VLLM_API_KEY", "your-secret-key")
PROMPT = "Write a Python code that calculates the Fibonacci sequence up to 15."
def main() -> int:
client = OpenAI(base_url=f"{HOST.rstrip('/')}/v1", api_key=API_KEY)
try:
resp = client.models.list()
if not resp.data:
print(f"No models served at {HOST}.")
return 1
except Exception as e:
print(f"Could not list models from {HOST}: {e}")
return 1
model_obj = resp.data[0]
model_id = model_obj.id
model_name = getattr(model_obj, "root", None)
print(f"Connected to {HOST}")
display_model = model_name or model_id
print(f"vLLM model: {display_model}")
print()
try:
stream = client.chat.completions.create(
model=model_id,
messages=[{"role": "user", "content": PROMPT}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print("\n")
except Exception as e:
print(f"Request failed: {e}")
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
Feel free to modify the PROMPT variable in the script to test different prompts. Run the script in your client shell,
If you would like a chat session instead of hard-coded prompts, use the following script,
#!/usr/bin/env python
# vLLM client chat interface connecting to the OpenAI-compatible API server.
#
# Usage:
# Once you have a vLLM server running, set the VLLM_HOST environment variable to the server's hostname and port.
# Then launch this script with,
# ./vllm_client_chat.py
import os
from openai import OpenAI
HOST = os.environ.get("VLLM_HOST", "http://127.0.0.1:8000")
API_KEY = os.environ.get("VLLM_API_KEY", "your-secret-key")
def main() -> int:
client = OpenAI(base_url=f"{HOST.rstrip('/')}/v1", api_key=API_KEY)
try:
resp = client.models.list()
if not resp.data:
print(f"No models served at {HOST}.")
return 1
except Exception as e:
print(f"Could not list models from {HOST}: {e}")
return 1
model_obj = resp.data[0]
model_id = model_obj.id
model_name = getattr(model_obj, "root", None)
display_model = model_name or model_id
messages = [
{
"role": "system",
"content": "You are a helpful assistant for an HPC user.",
}
]
print(f"Connected to {HOST}")
print(f"vLLM model: {display_model}")
print("Type 'exit' or 'quit' to leave.\n")
while True:
try:
user = input("You: ").strip()
except (EOFError, KeyboardInterrupt):
print("\nBye!")
return 0
if not user:
continue
if user.lower() in {"exit", "quit"}:
print("Bye!")
return 0
messages.append({"role": "user", "content": user})
print("Model:", end=" ", flush=True)
assistant_text = ""
try:
stream = client.chat.completions.create(
model=model_id,
messages=messages,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
assistant_text += delta
print("\n")
except Exception as e:
print(f"\nRequest failed: {e}\n")
continue
messages.append({"role": "assistant", "content": assistant_text})
if __name__ == "__main__":
raise SystemExit(main())
and run it with,
Once you are done with your inferencing tasks, make sure to stop the server with the kill command provided in the server startup output. E.g.,