Private local inference guide

Run Qwen locally on Windows and use it remotely from any device

Published August 6, 2026 · Updated August 17, 2026 · Tested with Qwen 3.5 9B and Qwen 3.8 27B

Turn a high-RAM Windows PC into your personal AI server. Use Qwen or any other local LLM from every laptop, desktop, and application you already own.

Ollama runs the model, LiteLLM supplies an authenticated OpenAI-compatible endpoint, and Tailscale makes it available to your own devices without placing it on the public internet.

The private inference architecture

1. Ollama (or any other llama.cpp service) loads a local LLM model and listens for API calls only on the computer.
2. LiteLLM Proxy presents one authenticated, OpenAI-compatible API and maps a stable alias to the installed model(s).
3. Tailscale Serve terminates HTTPS and exposes LiteLLM only inside your tailnet. This guide does not enable Tailscale Funnel.

Keeping both application servers on 127.0.0.1 prevents ordinary LAN or public access. Remote clients enter through the authenticated tailnet and still need the LiteLLM master key.

Prerequisites

• Windows 10 or 11 with Ollama installed and an LLM model already downloaded.
• Tailscale installed and signed into the same tailnet as the client devices.
• PowerShell running as Administrator for Tailscale Serve configuration.
• Enough RAM or VRAM for the exact model and quantization you select.

1. Verify Ollama/Llama.cpp works before adding a router

The fastest way to get started with older models is ollama run.

ollama run qwen3.5:9b

ollama list

Invoke-RestMethod http://127.0.0.1:11434/api/tags

Test the exact Qwen 3.5 model directly. This example uses the model from our experiment; substitute your actual model identifier when needed:

$body = @{
    model = "hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M"
    messages = @(
        @{
            role = "user"
            content = "Reply with exactly: I'm working"
        }
    )
    think = $false
    stream = $false
} | ConvertTo-Json -Depth 6

Invoke-RestMethod `
    -Uri "http://127.0.0.1:11434/api/chat" `
    -Method Post `
    -ContentType "application/json" `
    -Body $body

The newer Qwen 3.8 27B model requires optimizations for faster performance. Community members have reported speedups exceeding 66 tok/s by going past default settings. We used llama.cpp instead of Ollama for the newer Qwen model, though Ollama still technically works.

$SERVER = Get-ChildItem "C:\[Llama.cpp folder]" -Recurse -Filter llama-server.exe |
  Select-Object -First 1

$SERVER.FullName

& $SERVER.FullName `
  -m "C:\[path to model]\Qwen3.8-27B-UD-Q4_K_XL.gguf" `
  --alias qwen38 `
  --host 127.0.0.1 `
  --port 8080 `
  --ctx-size 65536 `
  --n-gpu-layers auto `
  --fit on `
  --flash-attn on `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --parallel 1 `
  --jinja `
  --no-mmproj `
  --perf

2. Install a pinned LiteLLM Proxy

winget install --id=astral-sh.uv -e
uv tool install --python 3.12 "litellm[proxy]==1.90.3"

The explicit version makes the tested environment reproducible. Review a later LiteLLM release before changing the pin.

3. Configure a stable model alias

Create a neutral local directory and configuration file:

New-Item -ItemType Directory -Force C:\local-ai-router
notepad C:\local-ai-router\config.yaml

For Ollama, use this configuration:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3.5:9b
      api_base: http://127.0.0.1:11434
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

For the llama.cpp server configured above, use this configuration:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: openai/qwen38
      api_base: http://127.0.0.1:8080/v1
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Generate a master key beginning with "sk-", place it in the process environment, and start LiteLLM on loopback:

$env:LITELLM_MASTER_KEY = "sk-replace-with-a-long-random-secret"

uv tool run litellm `
  --config C:\local-ai-router\config.yaml `
  --host 127.0.0.1 `
  --port 4000

4. Test the authenticated local endpoint

Open a second PowerShell window with the same master-key environment variable, then check health:

Invoke-RestMethod http://127.0.0.1:4000/health/liveliness

Send an OpenAI-compatible chat request through the model alias:

$body = @{
    model = "local-qwen"
    messages = @(
        @{
            role = "user"
            content = "Reply with exactly: router works"
        }
    )
    reasoning_effort = "none"
    stream = $false
    max_tokens = 40
} | ConvertTo-Json -Depth 6

Invoke-RestMethod `
    -Uri "http://127.0.0.1:4000/v1/chat/completions" `
    -Method Post `
    -Headers @{ Authorization = "Bearer $env:LITELLM_MASTER_KEY" } `
    -ContentType "application/json" `
    -Body $body

5. Get Tailscale and put authenticated inference on your private tailnet

Download Tailscale and use the free plan at https://tailscale.com/download/windows. Point Tailscale Serve at authenticated LiteLLM on port 4000. Do not point it at Ollama's unauthenticated port 11434.

& "C:\Program Files\Tailscale\tailscale.exe" serve reset

& "C:\Program Files\Tailscale\tailscale.exe" `
    serve --bg --https=443 http://127.0.0.1:4000

& "C:\Program Files\Tailscale\tailscale.exe" serve status
& "C:\Program Files\Tailscale\tailscale.exe" ip -4

Tailscale prints the private HTTPS hostname. A client must belong to the permitted tailnet and send the LiteLLM bearer token. Use tailscale serve off when the endpoint should no longer be available.

The two internal services remain bound to loopback:

• 127.0.0.1:11434: Ollama, available only on the inference box.
• 127.0.0.1:4000: LiteLLM, available through private Tailscale HTTPS.

6. Test from another tailnet device

On another authorized device, use the same OpenAI chat-completions body and point the request at the private hostname printed by Tailscale:

Powershell version
$headers = @{
    Authorization = "Bearer sk-your-router-key"
}

Invoke-RestMethod `
    -Uri "https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions" `
    -Method Post `
    -Headers $headers `
    -ContentType "application/json" `
    -Body $body

Bash version
cat > request.json <<'JSON'
{
  "model": "local-qwen",
  "messages": [
    {
      "role": "user",
      "content": "Reply with exactly: I'm working"
    }
  ],
  "stream": false,
  "max_tokens": 256
}
JSON

ROUTER_KEY="sk-your-router-key"
URL="https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions"

curl -sS "$URL" \
  -H "Authorization: Bearer $ROUTER_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

OpenAI-compatible applications can use that HTTPS origin as their base URL and the LiteLLM key as their API key. They do not need bespoke Ollama support.

Diagnose one layer at a time

Qwen 3.5, 3.6, and 3.8

Security checklist

• Bind Ollama and LiteLLM to 127.0.0.1, not every network interface.
• Expose LiteLLM through Tailscale Serve; do not enable Funnel for this private endpoint.
• Require a long LiteLLM master key and rotate it if it is copied into logs or shell history.
• Use tailnet access controls to restrict which users and devices can reach the host.
• Run the services under a dedicated non-administrator account for persistent operation.
• Pin software versions and review updates before applying them to an inference host.

Primary references

• Ollama Qwen 3.5 library
• Ollama Qwen 3.6 library
• LiteLLM Proxy documentation
• Tailscale Serve command documentation