Skip to content

Using Duke's AI Gateway on the DCC

This guide demonstrates querying the OpenAI-compatible API on Duke's AI Gateway.

The code for this example is available at: https://github.com/DukeRC/code/tree/main/Using-DukesAI-Gateway-on-the-DCC

Duke University's AI Gateway provides a central endpoint for various AI models through an OpenAI-compatible API. This allows users to leverage their tokens for research workflows. This tutorial is based on Drew Stinnett's guide, Getting Started with Duke's AI Gateway: A Developer's Guide. Users should refer to it for more detailed information.

Before you start, you need a Duke AI Gateway token (LITELLM_TOKEN). Get it from the AI Dashboard. You should also have a Python installation set up on the DCC.

Warning

If you will be performing computationally intensive inference tasks, ensure that you are on a compute node requested through a slurm interactive session or submit as a slurm batch job.

Initial setup

Create a working directory in your /work space,

mkdir -p /work/${USER}/openai-gateway-example
cd /work/${USER}/openai-gateway-example

Save your LITELLM_TOKEN in a .env file in this directory and set permissions so that it's only user-accessible,

echo 'LITELLM_TOKEN="your_api_token"' > /work/${USER}/openai-gateway-example/.env
chmod 600 /work/${USER}/openai-gateway-example/.env

Create and activate a Python environment,

conda create -n openai-gateway python=3.13 -y
conda activate openai-gateway

Install dependencies,

pip install openai python-dotenv

openai is the API client library and python-dotenv loads secrets from .env.

Listing available models

Use the script, list_models.sh to query the available models,

list_models.sh
#!/usr/bin/env bash
#
# A script to query all available models through Duke's AI Gateway.
# Courtesy of Drew Stinnet:https://ai.colab.duke.edu/colab-ai-blog/all-blogs/getting-started-with-dukes-ai-gateway-a-developers-guide
#
# Usage:
# Get your API token from: https://dashboard.ai.duke.edu/api-keys
# Set your API token in a .env file in this directory with the following content:
# LITELLM_TOKEN="your_api_token_here"
# or export the environment variable directly in your shell. Then run,
# ./list_models.sh

set -e

# This is the base URL for all operations
LITELLM_URL="https://litellm.oit.duke.edu/v1"

# Load environment variables from .env file if it exists
if [[ -f ".env" ]]; then
  set -a
  source .env
  set +a
fi

if [[ -z "$LITELLM_TOKEN" ]]; then
  echo "Error: LITELLM_TOKEN is not set. Please set it to your LiteLLM API token." 1>&2
  exit 1
fi

# Query the API to list all available models
echo "Listing all models available in LiteLLM..."
curl -X GET "${LITELLM_URL}/models" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${LITELLM_TOKEN}" | jq -r .data[].id | sort

Make it executable and run with,

chmod +x list_models.sh
./list_models.sh

Running AI Prompts

Create openai_example.py with the following content,

openai_example.py
#!/usr/bin/env python
#
# A Python script using the OpenAI-compatible API on Duke's AI Gateway to generate a response based on an input prompt.
# Courtesy of Drew Stinnet:https://ai.colab.duke.edu/colab-ai-blog/all-blogs/getting-started-with-dukes-ai-gateway-a-developers-guide
#
# Usage:
# Get your API token from: https://dashboard.ai.duke.edu/api-keys
# Set your API token in a .env file in this directory with the following content:
# LITELLM_TOKEN="your_api_token_here"
# or export the environment variable directly in your shell. Then run,
# ./openai_example.py "Your prompt here"

import os
import sys
from openai import OpenAI
from dotenv import load_dotenv

# Configuration
MODEL = "gpt-5.4"
INSTRUCTIONS = "You are a helpful assistant here to demo the power of AI."


def main():
    # Local .env file content
    load_dotenv()

    # Input arguments
    if len(sys.argv) < 2:
        print("Usage: ./openai_example.py <prompt>")
        sys.exit(1)
    token = os.getenv("LITELLM_TOKEN")
    if not token:
        print("Please set the LITELLM_TOKEN environment variable.")
        sys.exit(1)
    prompt = sys.argv[1]

    # Connect to the OpenAI API
    client = OpenAI(
        api_key=token,
        base_url="https://litellm.oit.duke.edu/v1",
    )

    response = client.responses.create(
        model=MODEL,
        instructions=INSTRUCTIONS,
        input=prompt,
    )

    print(response.output[0].content[0].text)


if __name__ == "__main__":
    main()

Make it executable:

chmod +x openai_example.py

Run the script with a prompt string,

./openai_example.py "Write a short haiku about HPC clusters."

You should see a text response printed to the terminal.

Run as a Slurm batch job (optional)

For repeated prompts or scheduled runs, submit a batch job.

Create openai_batch.slurm,

openai_batch.slurm
#!/bin/bash
#SBATCH -p scavenger
#SBATCH -A rescomp
#SBATCH -t 00:05:00
#SBATCH --mem=2G
#SBATCH -c 4
#SBATCH -J openai-gateway-demo
#SBATCH -o openai-gateway-demo-%j.out

source ~/.bashrc
conda activate openai-gateway

cd $SLURM_SUBMIT_DIR
./openai_example.py "Why is the sky blue?"

Submit with:

sbatch openai_batch.slurm

Check status with squeue -u ${USER} and inspect output in openai-gateway-demo-<jobid>.out.

Customize behavior

You can edit these values in openai_example.py:

  • MODEL: choose a model available to your Duke AI Gateway project.
  • INSTRUCTIONS: define style, tone, or output format.

Example:

MODEL = "gpt-5.4"
INSTRUCTIONS = "You are a concise research assistant. Return bullet points only."