Running an LLM server on DCC with Ollama
This is a guide on setting up an Ollama server to run LLM models on DCC with GPUs. The Ollama server is hosted on a GPU node, while the inferencing can be done from any node on the cluster.
The code for this example can be found at: https://github.com/DukeRC/code/tree/main/Running-an-LLM-server-on-DCC-with-Ollama
Large Language Models (LLMs) have become ever so popular, especially with broader access to computational resources such as HPC clusters and GPUs along with tools like Ollama that allow users to access open-source LLM models. In this guide, I will walk you through setting up an host Ollama server on a GPU node on the DCC cluster which can be used for inference from the same or different node on the cluster.
The setup discussed here was tested on Duke University's DCC cluster consisting of a wide range of GPU models including the powerful NVIDIA H200 GPU.
Starting an Ollama server on the GPU node
For this example, we will host the Ollama server on an NVIDIA RTX A5000 GPU node and run inference from the same node and other nodes on the cluster.
1. Request an interactive slurm session,
Once your session starts, load the ollama module,
At the time of writing, the "latest" module provides Ollama version 0.30.10. Loading the module also sets default environmental variables,
If you have a different storage location for models, export that directory after you load the module,
2. Start the Ollama server,
The PATH for the ollama-serve.sh script is set when the ollama module is loaded. It is a wrapper script that launches the Ollama server with a few options. You can find the script at /opt/apps/rhel9/ollama/bin/ollama-serve.sh.
When run without any optional arguments, ollama-serve.sh by default serves on localhost:11434. You will see something like the following,
$ ollama-serve.sh
🦙 Ollama is now serving at localhost:11434
Run the following command on this shell to connect to the Ollama server:
export OLLAMA_HOST=localhost:11434
Loading the ollama module already sets the OLLAMA_HOST variable to localhost:11434. If you get a different port, then copy and paste the line export OLLAMA_HOST=localhost:<port> on the same shell to connect to the server.
The options for ollama-serve.sh are as follows,
If you do not specify the local or global option, it will default to local which binds the server to localhost:<port> with the default port being 11434 if available. If not, it will automatically assign an unused port. You may also specify a custom port with the second [port] argument.
Starting server for inference from a different node
If you want to run inference on a different node than the one hosting the server, repeat the steps above and pass in the global option to ollama-serve.sh instead,
This will show an output like the following,
$ ollama-serve.sh global
🦙 Ollama is now serving at dcc-vossenlab-gpu-ferc-s-g36-19.rc.duke.edu:11434
Run the following command on the client shell to connect to the Ollama server:
export OLLAMA_HOST=http://dcc-vossenlab-gpu-ferc-s-g36-19.rc.duke.edu:11434
Copy and paste the above line export OLLAMA_HOST=http://<hostname>:<port> on the client shell where you want to run inference to connect to the server. For global inference, the server internally binds to 0.0.0.0 so it can accept connections on all available network interfaces, not just localhost.
Warning
The Ollama server does not provide an authentication system for the client so the server port is open across the cluster. If you require the server-client connection to have authentication, please use a vLLM Server instead.
Running inference sessions
We could run the inference session either by using the ollama app directly, or through the Python API. We will discuss both methods below.
1. Using the Ollama application
Now that the server is hosted, you can download and run LLM models from repositories like Ollama or HuggingFace. To download a model, you can use the ollama pull <model_name> command. To run inference with a model, you can use the ollama run <model_name> command. If the model is not already downloaded, it will be downloaded first and then run.
For example, to run the 35 billion parameter qwen3.6 model,
This will launch a chat session with the model. You can type in your prompts and the model will respond. To exit the chat session, type exit or quit.
Important!
If you will be performing computationally intensive inference tasks, ensure that you are on a compute node requested through a slurm interactive session or submit as a slurm batch job.
2. Using the Python API
This is more suited if you want to incorporate Python code in some workflow.
We will use the ollama API to run some Python code, so let's create a conda environment for that and install the ollama library,
conda create -n ollama-env python=3.13 -y
conda run -n ollama-env python -m pip install --no-input ollama
To run inference for a given prompt following Python script,
#!/usr/bin/env python
from ollama import Client
import os
MODEL = "qwen3.6:latest"
PROMPT = "Write a Python code that calculates the Fibonacci sequence up to 15."
client = Client(host=os.getenv("OLLAMA_HOST", "http://localhost:11434"))
# Check if model already exists
resp = client.list()
models = resp.models
model_names = [m.model for m in models]
if MODEL not in model_names:
try:
print(f"Pulling model '{MODEL}'...")
client.pull(model=MODEL)
except Exception as e:
print(f"Could not pull model: {e}")
# Stream output token by token
for chunk in client.chat(
model=MODEL,
messages=[{"role": "user", "content": PROMPT}],
stream=True,
):
print(chunk.message.content, end="", flush=True)
print("\n")
Run it with,
conda activate ollama-env
module load ollama/latest
ollama-serve.sh
## Export OLLAMA_HOST if needed
# export OLLAMA_HOST=http://<hostname>:<port>
python ollama_client.py
If you want to use the Python API as a chat client, you may use the following code instead,
#!/usr/bin/env python
from ollama import Client
import os
MODEL = "qwen3.6:latest"
def main():
host = os.getenv("OLLAMA_HOST", "http://localhost:11434")
client = Client(host=host)
# Check if model already exists
resp = client.list()
models = resp.models
model_names = [m.model for m in models]
if MODEL not in model_names:
try:
print(f"Pulling model '{MODEL}'...")
client.pull(model=MODEL)
except Exception as e:
print(f"Could not pull model: {e}")
# Start the conversation with an optional system prompt
messages = [
{
"role": "system",
"content": "You are a helpful assistant for an HPC user.",
}
]
print(f"Connected to {host} using model {MODEL}")
print("Type 'exit' or 'quit' to leave.\n")
while True:
try:
user = input("You: ").strip()
except (EOFError, KeyboardInterrupt):
print("\nBye!")
break
if not user:
continue
if user.lower() in {"exit", "quit"}:
print("Bye!")
break
# Add user message to the conversation
messages.append({"role": "user", "content": user})
print("Model:", end=" ", flush=True)
assistant_text = ""
# Stream the response token by token
for chunk in client.chat(
model=MODEL,
messages=messages,
stream=True,
):
if chunk.message.content:
print(chunk.message.content, end="", flush=True)
assistant_text += chunk.message.content
print("\n")
# Add assistant reply to history so the model remembers context
messages.append({"role": "assistant", "content": assistant_text})
if __name__ == "__main__":
main()
Once you are done with your LLM inference session, stop the server we started on the GPU node with,
Slurm batch job workflow
Instead of inferencing through an interactive session, you may also use an AI workflow through a slurm batch job. To perform the above prompt task in a batch job, you can use the following slurm script,
#!/bin/bash
#SBATCH -J ollama_batch_job
#SBATCH -p scavenger-gpu
#SBATCH --gres=gpu:a5000:1
#SBATCH --mem=100G
#SBATCH -t 00:10:00
# Start server
conda activate ollama-env
module load ollama/latest
ollama-serve.sh > server.log
# Extract and export OLLAMA_HOST
eval "$(grep '^export OLLAMA_HOST=' server.log)"
echo $OLLAMA_HOST
# Run inference
python ollama_client.py > output.txt
From your working directory, submit the job with,
Conclusion
In this guide, we have walked through the steps to set up an Ollama server on a GPU node on the DCC cluster and run inference sessions using both the Ollama application and the Python API. This setup allows you to leverage powerful LLM models for various applications while utilizing the computational resources of the DCC cluster effectively. One may take this further by incorporating the Ollama server to run agentic AI tools like OpenCode using local LLM models.