Hands-On Tutorial 5

Building a real app on top

Because every tool in this guide speaks the same OpenAI-compatible API shape, your application code is portable across all of them — swap the base URL and you've swapped your entire inference backend.

Here's a minimal streaming client, useful for a CLI tool or chat UI:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="local")

stream = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Explain the CAP theorem."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

This same snippet works unchanged against Ollama (port 11434), llama-server (port 8080), or vLLM (port 8000) — that's the entire point of the OpenAI-compatible API convention from the Glossary.