Models & on-demand loading

The catalog is curated

Porten doesn't expose every model under the sun — it offers an opinionated, curated catalog. An operator enables the models worth serving (good quality-per-VRAM, current architectures) and disables the rest, so hardware isn't wasted on models that a newer one strictly beats. GET /v1/models returns exactly what's offered.

Each model has a canonical id (e.g. qwen2.5-coder-32b), a context length, capability flags (chat, tools, json, vision), and a VRAM footprint.

Loading on demand

You can request any offered model, loaded or not. The catalog decouples "offered" from "currently in memory":

1. You send a request for a model that no node has loaded.
2. The Hub recognizes it's offered and tells a node that can fit it to load it — for a built-in node, that means downloading the weights (once) and starting the engine.
3. Your request blocks until the model is serving, then completes normally.

For an interactive client like the playground, this shows as a real progress bar — byte-level download percentage, then "loading into memory", then the answer streams. For an API client (OpenCode, a script), there's nothing to handle: the request just takes longer on the first cold call. A first load of a large model can take several minutes while weights download; after that it's warm and fast.

If a load takes longer than the request's warm-up budget, you get a 503 with code model_warming — retry and it'll be ready.

Checking warm-up progress

The portal polls a public endpoint for live load progress:

GET /public/v1/models/warming?model=qwen3-coder-next
→ { "model": "…", "ready": false, "phase": "downloading",
    "done_mb": 22272, "total_mb": 49131, "pct": 45 }

phase moves downloading → loading → ready.

Idle eviction

VRAM is finite, so the fleet unloads models you're not using. A provisioned model that goes a full demand window (default ~10 minutes) with no requests is unloaded to free room — its weights stay on disk, so it reloads quickly next time.

Demand-driven placement and eviction

The Hub continuously measures demand per model (served and unmet requests). When a model is requested and can't fit alongside what's already loaded, the planner evicts a lower-demand model immediately to make room — a just-requested model doesn't have to wait out an anti-thrash delay. This is what lets you "just ask for" a big model and have the fleet rearrange itself to serve it.

Picking a model

• Coding on a consumer GPU: qwen2.5-coder-7b-instruct uses Q4_K_M weights of about 4.7 GB. Porten requires an 8 GB model-memory budget, leaving space beyond the weights for context and working buffers. It is a starting point for C and Python assistance on a 12 GB GPU.
• Coding through the network: qwen3-coder-30b-a3b-instruct is a larger coding option. Check the live catalog for availability; your own GPU does not need to hold a model served by another node.
• General chat: a current 7–14B instruct model is fast and cheap.
• Reasoning: a DeepSeek-R1-distill class model when you want visible chain-of-thought.
• Frontier / huge context: see Build a combined machine — some models only fit on a multi-machine cluster.

What actually fits depends on your hardware — see the Hardware guide for the VRAM-per-model table.

Keeping a model warm on your computer

In Earn → your node, Keep warm requests that the selected model stays loaded on that computer. The toggle records your preference; check the model's status to see whether it is downloading, loading, waiting, or actually loaded. A failed serving check is shown separately. The Porten logs on that computer contain engine errors that are not available in the portal.

VRAM (free / total) shows the GPU's physical memory. Model VRAM budget is the portion Porten can use after reserved memory. A 12 GB GPU can therefore have a 10 GB model budget. The model file's size is only part of its running memory requirement.

Models that exceed the budget are disabled in the picker, with the required memory shown. Keeping multiple models warm divides the budget between them; turn off an existing pin before choosing its replacement. If memory requirements are unknown, the portal warns that loading can still fail. Existing oversized pins can always be turned off.