Local Models Administration

Admin API endpoints for managing local model assignments (cp_cluster_assignments). The operator UI (/cluster/assignments) sits on top of these — but they're also reachable directly via curl / SDK.

All endpoints require admin authentication (Authorization: Bearer <ADMIN_KEY> or JWT). Writes (POST/PUT/DELETE) reject the viewer role.

Core concepts

Concept	Meaning
Assignment	A `(channel_id, model_name, node_id, gpu_index, runtime_key)` 5-tuple. Plans which model runs on which server.
Runtime	A `cp_cluster_runtimes` record (e.g. `vllm`, `blab-tts`). Carries `requires_gpu` + `default_vram_gb` for tier + suggestion behaviour.
Node	A `cp_cluster_nodes` record. Runs qevron-agent at `agent_endpoint`; pipes telemetry to qevron via SSH or agent collector.
Deploy	Actually install the assignment on a node (write systemd unit + enable).
Move	Atomically relocate an assignment from one node to another (5-step pipeline).

Move — relocate an assignment

Atomic 5-step pipeline: deploy_target → wait_target_health → stop_source → undeploy_source → update_assignment. Step 3/4 failures roll back the target and (if source was active) restart it. Step 5 (DB write) failure cleans up the target.

curl

bash

curl -X POST https://app.qevron.ai/api/cluster/assignments/42/move \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_node_id": 2,
    "target_gpu_index": 0,
    "target_channel_id": 7,
    "start_after_move": true
  }'

Request body

Field	Type	Notes
`target_node_id`	int	Required. Target server ID. Cannot equal the source.
`target_gpu_index`	int?	Optional. If absent, the source GPU index is kept.
`start_after_move`	bool?	Optional (default `true`). When `false`, the handler does not wait for `active=active` on the target before stopping the source.
`target_channel_id`	int?	Optional. Promotes / rebinds the routing channel. If absent, the channel is preserved.

Pre-flight validation errors

HTTP	`data.reason`	Meaning
400	—	Target equals source / target node not found
422	`over_subscribed_target`	Target GPU's VRAM reservation sum exceeds capacity (`data.target_gpu_index` returned)
422	`tier_mismatch`	Runtime is `requires_gpu=true` but target node has zero GPUs (`data.target_gpus: 0`)
400	`channel_not_found`	`target_channel_id` does not exist
400	`channel_disabled`	Target channel is disabled
422	`channel_model_drift`	Target channel's `models[]` does not list the assignment's `model_name` (`data.target_channel_models` returned)

Response — `final_state` enum

Value	Meaning
`moved`	All 5 steps clean. Binding updated.
`failed`	Step 1 or 2 failed. Source untouched.
`failed_rolled_back`	Step 3 or step-5 DB write failed; target cleanly rolled back.
`failed_rolled_back_source_restarted`	Step 4 failed; target rolled back, source restarted to prior state.
`failed_rollback_dirty`	Forward step failed AND rollback also failed. Manual inspection required on both nodes.

The response also carries steps[] with per-step ok + ms + active (when relevant) detail.

Bulk Move — relocate multiple assignments at once

Move N assignments to the same target node in one request. Backend runs them sequentially — one failed id never aborts the rest.

curl

bash

curl -X POST https://app.qevron.ai/api/cluster/assignments/bulk-move \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "assignment_ids": [42, 43, 44],
    "target_node_id": 2,
    "target_gpu_index": 0,
    "start_after_move": false
  }'

Response:

json

{
  "data": {
    "total": 3,
    "moved_count": 2,
    "failed_count": 1,
    "items": [
      { "id": 42, "ok": true, "response": { "final_state": "moved", "from_node": "SRV-1", "to_node": "SRV-2", "steps": [...] } },
      { "id": 43, "ok": false, "response": { "final_state": "failed_rolled_back", ... } },
      { "id": 44, "ok": false, "error": { "status": 422, "message": "blab-tts requires a GPU; SRV-3 has none", "details": { "reason": "tier_mismatch", ... } } }
    ]
  },
  "success": true
}

Limit: assignment_ids between 1 and 50.

Control — start / stop / restart

Manage an assignment's systemd unit.

curl

bash

curl -X POST "https://app.qevron.ai/api/cluster/assignments/42/control?action=restart" \
  -H "Authorization: Bearer $ADMIN_KEY"

action ∈ {start, stop, restart}. If the assignment is not deployed, the pre-flight returns 422:

json

{
  "success": false,
  "message": "solab-stt is not deployed on SRV-4. Click Deploy first.",
  "data": {
    "action": "stop",
    "unit": "solab-stt",
    "node": "SRV-4",
    "reason": "unit_not_deployed"
  }
}

Deploy / Undeploy

Install (write systemd unit + enable + optional start) or remove the runtime on the assignment's node.

DeployUndeploy

bash

curl -X POST "https://app.qevron.ai/api/cluster/assignments/42/deploy?start_now=1&enable_now=1" \
  -H "Authorization: Bearer $ADMIN_KEY"

bash

curl -X DELETE "https://app.qevron.ai/api/cluster/assignments/42/deploy" \
  -H "Authorization: Bearer $ADMIN_KEY"

The response mirrors the agent's /v1/services/<key>/install (or DELETE) result: {installed, enabled, started, unit, ...}. On failure, 502 + data.raw_error plus a structured translation (exit status 5 → "systemd unit not found", exit status 4 → "unit failed to load", etc.).

Logs — snapshot and live stream

Snapshot (last N lines)

bash

curl -G "https://app.qevron.ai/api/cluster/assignments/42/logs" \
  -H "Authorization: Bearer $ADMIN_KEY" \
  --data-urlencode tail=100

Live stream (SSE)

Two steps: mint a ticket, then open the EventSource. Tickets are single-use, 60-second TTL, Redis-backed. The long-lived JWT never enters the URL.

1. mint ticket2. open streambrowser

bash

TICKET=$(curl -sS -X POST \
  -H "Authorization: Bearer $ADMIN_KEY" \
  "https://app.qevron.ai/api/cluster/assignments/42/logs/ticket" \
  | jq -r '.data.ticket')

bash

curl -N "https://app.qevron.ai/api/cluster/assignments/42/logs/stream?tail=10&ticket=$TICKET"

javascript

// fetch ticket via standard axios (header auth)
const t = await api.post(`/cluster/assignments/${id}/logs/ticket`)
const es = new EventSource(`/api/cluster/assignments/${id}/logs/stream?ticket=${t.ticket}`)
es.onmessage = (ev) => console.log(ev.data)
es.addEventListener('end', (ev) => console.log('stream ended:', ev.data))

Response is text/event-stream — each line emitted as data: <line>\n\n. A : keepalive comment frame goes out every 25 seconds so reverse-proxy idle timeouts don't drop the connection. Client disconnect kills the server-side journalctl -f child.

Service status

Live systemd state (active, enabled, sub_state, active_enter_timestamp, main_pid).

bash

curl -H "Authorization: Bearer $ADMIN_KEY" \
  "https://app.qevron.ai/api/cluster/assignments/42/service-status"

json

{
  "data": {
    "unit": "vllm",
    "active": "active",
    "enabled": "enabled",
    "sub_state": "running",
    "main_pid": 12345,
    "active_enter_timestamp": "2026-05-27T14:30:00Z"
  },
  "success": true
}

Runtime — tier flag and VRAM suggestion

cp_cluster_runtimes records carry two optional fields:

Field	Behaviour
`requires_gpu`	When `true`, assignments cannot be placed on CPU-only nodes (Create/Update/Move return 422 with `reason: tier_mismatch`). UI hides CPU-only servers from selectors.
`default_vram_gb`	Auto-suggested reservation in the assignment form (the "Reserve VRAM" input pre-fills). `0` = no suggestion.

Seed defaults: vllm=24, blab-tts=4, blab-fast=2, blab-stable=2, solab-stt=4, spook-vision=3 (all requires_gpu=true). Toggle via the operator UI.

Agent push

When adding a new node or refreshing the agent binary:

bash

curl -X POST "https://app.qevron.ai/api/cluster/nodes/2/agent/install" \
  -H "Authorization: Bearer $ADMIN_KEY"

SSH into the node, push the binary, write the systemd unit, restart, probe /v1/health. Response carries stdout_tail with the install log.

Failure modes — friendly message examples

Earlier versions surfaced raw JSON like agent HTTP 502: {"error":"exit status 5"}. v2.10+ wraps failures in a structured envelope:

Scenario	HTTP	`message`
Stop on an undeployed unit	422	`solab-stt is not deployed on SRV-4. Click Deploy first.`
GPU-required runtime on a CPU-only node	422	`blab-tts requires a GPU; SRV-3 has none`
Channel.models drift on Move	422	`target channel "prod-chat" does not list model "verinova"; add the model to the channel first`
Agent unreachable	502	`agent on SRV-1 is unreachable`
Agent unit absent (systemctl exit 5)	502	`systemd unit not found on target server`

Every error carries structured detail in data (action, unit, node, reason, raw_error …) so UIs can render Cannot <action> <unit> on <node>: <message> instead of parsing the message string.

Local Models Catalogue — model inventory + base URL info
Errors — general API error envelope

Local Models Administration ​

Core concepts ​

Move — relocate an assignment ​

Request body ​

Pre-flight validation errors ​

Response — final_state enum ​

Bulk Move — relocate multiple assignments at once ​

Control — start / stop / restart ​

Deploy / Undeploy ​

Logs — snapshot and live stream ​

Snapshot (last N lines) ​

Live stream (SSE) ​

Service status ​

Runtime — tier flag and VRAM suggestion ​

Agent push ​

Failure modes — friendly message examples ​

Related pages ​

Local Models Administration

Core concepts

Move — relocate an assignment

Request body

Pre-flight validation errors

Response — `final_state` enum

Bulk Move — relocate multiple assignments at once

Control — start / stop / restart

Deploy / Undeploy

Logs — snapshot and live stream

Snapshot (last N lines)

Live stream (SSE)

Service status

Runtime — tier flag and VRAM suggestion

Agent push

Failure modes — friendly message examples

Related pages