Strata router

A Linux Rust daemon owns one foreground Strata server at a time. It exposes /v1/models, /v1/chat/completions, /health and read-only /metrics. Chat JSON and SSE pass through without collecting the response. It does not implement every OpenAI endpoint.

Build and verify

nix develop
cargo fmt --check
cargo check --locked
cargo clippy --all-targets --locked -- -D warnings
cargo test --locked
nix build
nix flake check

Cargo.lock pins Rust dependencies. flake.lock pins nixpkgs. nix run . -- daemon /absolute/config.json runs the package. Nix packages the router, not the GPU driver, Python Strata installation or weights. Tests run a lightweight Python subprocess with the same CLI and readiness contract. No downloads of model weights occur.

Configure and run

Copy config.example.json to a private location. Its 23 concrete entries distinguish family, fine-tune and quantization. The example enabled_models list contains all 23 IDs. All start uninstalled and incompatible. Install and prepare Strata separately, following the pinned upstream references in RESEARCH.md. Set absolute python, strata_dir, and per-model profile paths. Each profile is the JSON written by Strata setup for those exact weights. Set upstream_model to that profile's actual model_name, not necessarily the public router ID. Check memory and backend compatibility before setting compatible: true; set the per-model installed: true only for installed weights. The daemon cannot establish hardware compatibility from a model's name. Startup errors never trigger a different model.

Profiles must disable lazy loading and before_load commands. They must not contain an API key. The managed backend binds loopback; inherited STRATA_API_KEY is removed. The manager overrides Strata's internal idle timeout to zero and controls process shutdown itself. Do not use strata launch as a wrapper: it detaches a server. This daemon invokes python -m serve.server --engine strata --config PROFILE directly, supplies a private readiness file, verifies its child PID and exact model identity, and checks /health reports the same loaded model. Strata must support the CLI at the revision in RESEARCH.md. Reserve the backend port for this service.

The aliases object may contain only the family aliases qwen3.8-flash-next and qwen3.6-35b-a3b; there is no cross-family qwen alias. Each alias contains an ordered allowed_models array. Only concrete IDs may appear. The first item is preferred. A loaded allowed model wins; otherwise the daemon loads the first item. The router filters each alias through enabled_models, preserving order. It never skips an uninstalled or incompatible entry in that effective list. Each alias rejects entries from another family. Unknown IDs, empty arrays and alias references, including cycles, reject the whole configuration. Configuration is read at daemon startup.

enabled_models is the global list of concrete IDs that clients may discover and request. Omit it to enable all registered concrete IDs, or use [] to expose none. Unknown IDs, alias IDs and duplicate IDs are configuration errors. Models outside this list disappear from /v1/models and reject direct requests. Alias metadata contains only enabled IDs. An alias with no enabled entries disappears from the listing and rejects requests. Removing the first configured alias entry through this global list makes the next enabled entry preferred.

Enablement controls access and advertising, not installation. An enabled model may still report not_installed, incompatible, missing_profile or prepared_unverified. For older configs, per-model enabled remains accepted as a legacy spelling of installed; move access restrictions to the top-level enabled_models list.

install -d -m 700 /run/user/$(id -u)/strata-router
# Set control_socket to this directory's control.sock in your config.
strata-router daemon /absolute/config.json

Send a request to http://127.0.0.1:8088/v1/chat/completions, with model set to a concrete ID or alias. x-strata-concrete-model identifies the resolved public ID. Payload model names come from Strata itself. /v1/models distinguishes kind: alias from kind: concrete. Preparation metadata does not guarantee weights can load. Only successful readiness establishes runtime availability. Models excluded by enabled_models do not appear in this endpoint or alias metadata. A model's optional metadata object can supply meta, architecture and thinking fields. The router also reads those fields from router_metadata in its profile.

max_request_body_bytes sets the inference JSON body limit for both Content-Length and chunked requests. It defaults to 268435456 bytes, or 256 MiB, matching Strata's default body limit instead of Axum's implicit 2 MiB limit. Existing configs inherit this default. Values must be positive and at most 1 GiB. This byte limit is separate from the model's token context limit. Large accepted bodies consume memory while queued, so do not disable authentication on a public proxy.

GET /metrics reports the loaded model, in-flight request count, queued request count and its split by service tier, configured request-body limit and the last streamed request's TTFT and decode rate. A decode rate is null when the upstream stream omits completion token usage.

Per-request accounting

Every POST to /v1/chat/completions gets one completed record, including JSON/body-limit rejections, routing failures, queued abandonment and generation cancellation. Health, discovery, metrics, /requests, unsupported endpoints and method rejections do not generate inference records. The router currently supports chat completions only. x-strata-request-id appears on its chat responses, including errors. x-strata-concrete-model remains the concrete model header.

The memory ring is always enabled. File persistence is opt-in. These top-level config keys are accepted by the JSON config, including the config generated by modules/hosts/poseidon/strata-router.nix in dotfiles:

{
  "request_log_path": "/var/lib/strata-router/requests.jsonl",
  "request_log_max_bytes": 16777216,
  "request_log_rotate_files": 3,
  "request_log_capacity": 2048
}

request_log_path defaults to null, disabling files. A configured path must be absolute. request_log_max_bytes defaults to 16 MiB and accepts 4096 through 1073741824 bytes. request_log_rotate_files defaults to 3 and accepts 0 through 100 backups. request_log_capacity defaults to 2048 and accepts 1 through 100000 records. Config changes require a daemon restart.

For poseidon's service, set StateDirectory = "strata-router", StateDirectoryMode = "0700" and UMask = "0077" in serviceConfig. The service user must own and be able to write /var/lib/strata-router. If the unit uses ProtectSystem = "strict", permit that state directory through StateDirectory or ReadWritePaths. Keep the existing private RuntimeDirectory and control socket under /run/strata-router; persistent records belong under /var/lib, not /run. No deployment or dotfiles changes are part of this router change.

The active file contains append-only JSONL. Before a line would exceed the size limit, the writer moves the active file to .1, .1 to .2, and so on, replacing the oldest backup. Zero backups truncates the old active file at rotation. A line larger than the limit is dropped. Retention is at most max_bytes * (rotate_files + 1) bytes, with no time-based expiry. New files use mode 0600. Use a private parent directory and one writer per path. On restart the writer discards any incomplete final line left by an interrupted write, then appends to the existing active file; it does not replay files into the ring.

Serving never waits for disk writes, rotation or synchronization. A bounded 2048-line writer queue uses nonblocking enqueue. The writer calls sync_data after each line. Overflow drops file writes; filesystem failures leave serving and the memory ring usable. /metrics.request_log reports write_errors, write_drops, ring evicted, records_in_ring, config values, next_seq and instance_id. Here next_seq is the next completion sequence to allocate. A clean shutdown gives the writer up to two seconds to finish queued writes. A crash, forced kill, storage failure or sustained queue overflow can lose records. This is best-effort durable accounting, not a transaction required for inference success.

Reading recent records

GET /requests?after=123&limit=100 returns:

{
  "records": [],
  "next_seq": 123,
  "dropped": 0,
  "oldest_seq": null,
  "instance_id": "process-and-start-time-identifier"
}

records contains the schema below, in increasing completion seq. after is an exclusive unsigned integer cursor. limit defaults to 100 and clamps to 1 through 1000. next_seq is the last returned record's sequence, or the supplied cursor if no record matched. It is safe to pass it back as after. Start with after=0 to read all retained records. Omitting after selects only the newest limit records. An empty initial ring returns next_seq: 0. oldest_seq is the oldest retained sequence or null. dropped is the number of records after the supplied cursor that the ring has already evicted, independent of the page limit. Without a cursor, dropped is zero. All integer fields here are nonnegative; only oldest_seq is nullable.

Reads never drain the ring. Each reader stores its own cursor and instance_id. Sequences start at 1 for each daemon instance. After a router restart, instance_id changes and readers must reset after to 0. Records and cursors do not survive in memory across restart. A collector that restarts can resume its stored cursor if the instance is unchanged. A dropped count signals a gap; retained JSONL files may supply missing records through a separate file reader. request_id contains the instance identifier and arrival sequence, so it distinguishes requests across instances.

The endpoint has the same access policy as /metrics: the configured loopback listener, no HTTP authentication, and access for local users. With the example port, another host can run:

ssh -N -L 18088:127.0.0.1:8088 poseidon
curl 'http://127.0.0.1:18088/requests?after=0&limit=1000'

SSH supplies remote authentication. Use poseidon's configured router port if it differs from 8088. Do not publish the listener to the network.

Record schema, version 1

The JSONL line and each HTTP records item have exactly the same fields. Every field is present. Nullable fields use JSON null, never an omitted key or an invented zero. Counts and router durations are unsigned integers. Engine durations and rates are JSON numbers and can contain fractional milliseconds. All timestamps are RFC3339 UTC strings with millisecond precision, ending in Z.

Fields	Type and availability	Meaning
schema_version	integer, always 1	Schema version.
seq, arrival_seq	integer, always	Completion order and arrival order, respectively. Both start at 1 per instance. Model grouping and concurrency mean they can differ.
request_id	string, always	Unique correlation ID, also returned in x-strata-request-id.
endpoint	string, always	/v1/chat/completions. No query strings are recorded.
stream	boolean, always	Requested streaming mode. False if the JSON body could not be parsed.
model_requested	nullable string	Recognized configured model ID or alias. Unknown or missing model strings are null to avoid persisting arbitrary input.
model_resolved, upstream_model	nullable string	Resolved public ID and backend profile's model name. Null before resolution.
arrived_at, completed_at	timestamp string, always	Middleware entry after HTTP headers, before JSON body extraction, and response-body EOF or cancellation. Arrival is not TCP accept time. Completion does not prove the client received every byte.
lease_at	nullable timestamp	Admission before backend loading starts.
first_token_at, last_token_at	nullable timestamp	First and last observed generated SSE deltas, including reasoning and tool-call deltas. Heartbeats, role-only frames and usage-only frames do not count. Unobservable for nonstream JSON.
queue_depth_at_arrival, in_flight_at_arrival	integer, always	Concurrent snapshots of waiting tickets and admitted leases. In legacy mode queue depth derives from pending permits minus admitted leases. Admission includes loading and header waits. Snapshots can race with concurrent arrivals/completions.
queue_wait_ms	nullable integer ms	Arrival to admission, including request-body extraction and admission scheduling, excluding the subsequent model load. Null if no lease was acquired. A queued cancellation's total_ms shows how long it waited.
model_switch	boolean, always	Admission found a different loaded ID or no loaded model. Includes cold starts. False before admission.
model_load_ms	nullable integer ms	Backend stop/start/readiness work, including cleanup on failed starts. Null when no load ran or cancellation interrupted measurement. A same-ID reload after child death can have a duration with model_switch: false.
upstream_headers_ms	nullable integer ms	Arrival to upstream response headers.
ttft_ms, ttft_after_lease_ms	nullable integer ms	Arrival to first generated delta, and admission to that delta. Include model loading; the second excludes queue wait. Null if no generated delta was observed.
generation_ms	nullable integer ms	First to last observed generated delta. Zero is possible for a single delta.
total_ms	integer ms, always	Arrival to completion or cancellation.
prompt_tokens, completion_tokens, reasoning_tokens	nullable integer tokens	Backend usage counts. Reasoning is a subset when the backend defines it that way.
prompt_n, cache_n, predicted_n	nullable integer tokens	Engine fresh prompt work, cached/reused prompt tokens, and generated tokens. cache_n accepts usage.prompt_tokens_details.cached_tokens, with engine timings.cache_n taking precedence in the same frame.
prompt_ms, predicted_ms	nullable number ms	Engine prompt-processing and generation times, copied without inferring them from router latency.
prompt_per_second, predicted_per_second	nullable number tokens/s	Engine prefill and generation rates. Null when omitted, including engines with no measured work.
finish_reason	nullable string enum	stop, length, tool_calls, function_call, content_filter, or other. Unknown strings are replaced with other.
status, upstream_status	nullable integer HTTP code	Client-facing status and upstream status. Queued streaming requests can have status: 200 with a failed upstream status or outcome. status is null if cancelled before a response exists.
success	boolean, always	True only for completed responses with a 2xx client status and no detected failure.
reason	string enum, always	completed, client_disconnect, upstream_error, timeout, queue_timeout, routing_off, shutdown, model_load_error, model_not_prepared, invalid_request, or queue_full.
response_bytes	integer bytes, always	Body bytes observed while forwarding to the HTTP stack, including SSE comments and in-band errors. Excludes HTTP headers and transport framing. Not a delivery acknowledgement.
disconnect_phase	nullable string enum	queued, loading, or generating for client disconnect. loading includes header wait; generating means upstream headers had arrived. Null for other outcomes.

The parser handles fragmented single-line SSE JSON, final usage/timing frames and nonstream JSON. It keeps at most 256 KiB of one SSE line or one nonstream JSON response for inspection. Oversized accounting frames or JSON responses are forwarded unchanged but their accounting is unavailable. Nonstream responses have no observed token arrival time, so TTFT and generation duration stay null. This parser does not interpret multiline SSE data fields. Current Strata emits single-line frames.

At the pinned Strata revision in RESEARCH.md, source inspection verifies that openai_stream emits prompt_tokens, completion_tokens, cached-token usage, finish reason and the llama.cpp-style timings fields listed above. openai_collect preserves usage and timings in JSON. request_timings returns no timings if the engine has no prompt clock. Reasoning counts exist internally but this revision does not include them in chat usage, so reasoning_tokens stays null with that backend. Fixture tests verify parsing, including reasoning counts from other compatible backends. No live GPU inference was run to establish availability on poseidon.

The router does not own KV checkpoints and the chat response does not report per-request parking saves/restores, their milliseconds/bytes, or budget skips. These are not guessed from global server metrics. cache_n records reused prompt tokens, not a count of checkpoint restores. An upstream follow-up should attach numeric per-request save/restore/skip accounting to the final frame; scraping stderr would mix requests and risk logging content.

For CLIProxyAPI joins, match model_requested or model_resolved, the arrived_at through completed_at interval, token counts, status and latency. Confirm which instant its event timestamp describes and allow for proxy overhead. Identical overlapping requests can remain ambiguous. If the collector can preserve x-strata-request-id, prefer that exact join. CLIProxyAPI's existing usage event does not provide this ID. Compare total_ms with total_ms - queue_wait_ms and ttft_ms with ttft_after_lease_ms to include or exclude the queue. Use engine prompt_per_second for prefill and predicted_per_second for decode speed. Null means unknown, not zero. Model load and KV restore time cannot be separated by subtracting TTFT alone.

Records never contain prompts, generated text, tool names/arguments, headers, API keys, client addresses or error bodies. Only recognized model identifiers, fixed endpoint/reason strings, timestamps and numeric/boolean accounting enter the record. The observer inspects response JSON transiently to extract those allowed fields. Persistence and the HTTP endpoint use the same explicit typed record; neither serializes request/response objects.

Per-request accounting

Every POST to /v1/chat/completions gets one completed record, including JSON/body-limit rejections, routing failures, queued abandonment and generation cancellation. Health, discovery, metrics, /requests, unsupported endpoints and method rejections do not generate inference records. The router currently supports chat completions only. x-strata-request-id appears on its chat responses, including errors. x-strata-concrete-model remains the concrete model header.

The memory ring is always enabled. File persistence is opt-in. These top-level config keys are accepted by the JSON config, including the config generated by modules/hosts/poseidon/strata-router.nix in dotfiles:

{
  "request_log_path": "/var/lib/strata-router/requests.jsonl",
  "request_log_max_bytes": 16777216,
  "request_log_rotate_files": 3,
  "request_log_capacity": 2048
}

request_log_path defaults to null, disabling files. A configured path must be absolute. request_log_max_bytes defaults to 16 MiB and accepts 4096 through 1073741824 bytes. request_log_rotate_files defaults to 3 and accepts 0 through 100 backups. request_log_capacity defaults to 2048 and accepts 1 through 100000 records. Config changes require a daemon restart.

For poseidon's service, set StateDirectory = "strata-router", StateDirectoryMode = "0700" and UMask = "0077" in serviceConfig. The service user must own and be able to write /var/lib/strata-router. If the unit uses ProtectSystem = "strict", permit that state directory through StateDirectory or ReadWritePaths. Keep the existing private RuntimeDirectory and control socket under /run/strata-router; persistent records belong under /var/lib, not /run. No deployment or dotfiles changes are part of this router change.

The active file contains append-only JSONL. Before a line would exceed the size limit, the writer moves the active file to .1, .1 to .2, and so on, replacing the oldest backup. Zero backups truncates the old active file at rotation. A line larger than the limit is dropped. Retention is at most max_bytes * (rotate_files + 1) bytes, with no time-based expiry. New files use mode 0600. Use a private parent directory and one writer per path. On restart the writer discards any incomplete final line left by an interrupted write, then appends to the existing active file; it does not replay files into the ring.

Serving never waits for disk writes, rotation or synchronization. A bounded 2048-line writer queue uses nonblocking enqueue. The writer calls sync_data after each line. Overflow drops file writes; filesystem failures leave serving and the memory ring usable. /metrics.request_log reports write_errors, write_drops, ring evicted, records_in_ring, config values, next_seq and instance_id. Here next_seq is the next completion sequence to allocate. A clean shutdown gives the writer up to two seconds to finish queued writes. A crash, forced kill, storage failure or sustained queue overflow can lose records. This is best-effort durable accounting, not a transaction required for inference success.

Reading recent records

GET /requests?after=123&limit=100 returns:

{
  "records": [],
  "next_seq": 123,
  "dropped": 0,
  "oldest_seq": null,
  "instance_id": "process-and-start-time-identifier"
}

records contains the schema below, in increasing completion seq. after is an exclusive unsigned integer cursor. limit defaults to 100 and clamps to 1 through 1000. next_seq is the last returned record's sequence, or the supplied cursor if no record matched. It is safe to pass it back as after. Start with after=0 to read all retained records. Omitting after selects only the newest limit records. An empty initial ring returns next_seq: 0. oldest_seq is the oldest retained sequence or null. dropped is the number of records after the supplied cursor that the ring has already evicted, independent of the page limit. Without a cursor, dropped is zero. All integer fields here are nonnegative; only oldest_seq is nullable.

Reads never drain the ring. Each reader stores its own cursor and instance_id. Sequences start at 1 for each daemon instance. After a router restart, instance_id changes and readers must reset after to 0. Records and cursors do not survive in memory across restart. A collector that restarts can resume its stored cursor if the instance is unchanged. A dropped count signals a gap; retained JSONL files may supply missing records through a separate file reader. request_id contains the instance identifier and arrival sequence, so it distinguishes requests across instances.

The endpoint has the same access policy as /metrics: the configured loopback listener, no HTTP authentication, and access for local users. With the example port, another host can run:

ssh -N -L 18088:127.0.0.1:8088 poseidon
curl 'http://127.0.0.1:18088/requests?after=0&limit=1000'

SSH supplies remote authentication. Use poseidon's configured router port if it differs from 8088. Do not publish the listener to the network.

Record schema, version 1

The JSONL line and each HTTP records item have exactly the same fields. Every field is present. Nullable fields use JSON null, never an omitted key or an invented zero. Counts and router durations are unsigned integers. Engine durations and rates are JSON numbers and can contain fractional milliseconds. All timestamps are RFC3339 UTC strings with millisecond precision, ending in Z.

Fields	Type and availability	Meaning
schema_version	integer, always 1	Schema version.
seq, arrival_seq	integer, always	Completion order and arrival order, respectively. Both start at 1 per instance. Model grouping and concurrency mean they can differ.
request_id	string, always	Unique correlation ID, also returned in x-strata-request-id.
endpoint	string, always	/v1/chat/completions. No query strings are recorded.
stream	boolean, always	Requested streaming mode. False if the JSON body could not be parsed.
model_requested	nullable string	Recognized configured model ID or alias. Unknown or missing model strings are null to avoid persisting arbitrary input.
model_resolved, upstream_model	nullable string	Resolved public ID and backend profile's model name. Null before resolution.
arrived_at, completed_at	timestamp string, always	Middleware entry after HTTP headers, before JSON body extraction, and response-body EOF or cancellation. Arrival is not TCP accept time. Completion does not prove the client received every byte.
lease_at	nullable timestamp	Admission before backend loading starts.
first_token_at, last_token_at	nullable timestamp	First and last observed generated SSE deltas, including reasoning and tool-call deltas. Heartbeats, role-only frames and usage-only frames do not count. Unobservable for nonstream JSON.
queue_depth_at_arrival, in_flight_at_arrival	integer, always	Concurrent snapshots of waiting tickets and admitted leases. In legacy mode queue depth derives from pending permits minus admitted leases. Admission includes loading and header waits. Snapshots can race with concurrent arrivals/completions.
queue_wait_ms	nullable integer ms	Arrival to admission, including request-body extraction and admission scheduling, excluding the subsequent model load. Null if no lease was acquired. A queued cancellation's total_ms shows how long it waited.
model_switch	boolean, always	Admission found a different loaded ID or no loaded model. Includes cold starts. False before admission.
model_load_ms	nullable integer ms	Backend stop/start/readiness work, including cleanup on failed starts. Null when no load ran or cancellation interrupted measurement. A same-ID reload after child death can have a duration with model_switch: false.
upstream_headers_ms	nullable integer ms	Arrival to upstream response headers.
ttft_ms, ttft_after_lease_ms	nullable integer ms	Arrival to first generated delta, and admission to that delta. Include model loading; the second excludes queue wait. Null if no generated delta was observed.
generation_ms	nullable integer ms	First to last observed generated delta. Zero is possible for a single delta.
total_ms	integer ms, always	Arrival to completion or cancellation.
prompt_tokens, completion_tokens, reasoning_tokens	nullable integer tokens	Backend usage counts. Reasoning is a subset when the backend defines it that way.
prompt_n, cache_n, predicted_n	nullable integer tokens	Engine fresh prompt work, cached/reused prompt tokens, and generated tokens. cache_n accepts usage.prompt_tokens_details.cached_tokens, with engine timings.cache_n taking precedence in the same frame.
prompt_ms, predicted_ms	nullable number ms	Engine prompt-processing and generation times, copied without inferring them from router latency.
prompt_per_second, predicted_per_second	nullable number tokens/s	Engine prefill and generation rates. Null when omitted, including engines with no measured work.
finish_reason	nullable string enum	stop, length, tool_calls, function_call, content_filter, or other. Unknown strings are replaced with other.
status, upstream_status	nullable integer HTTP code	Client-facing status and upstream status. Queued streaming requests can have status: 200 with a failed upstream status or outcome. status is null if cancelled before a response exists.
success	boolean, always	True only for completed responses with a 2xx client status and no detected failure.
reason	string enum, always	completed, client_disconnect, upstream_error, timeout, queue_timeout, routing_off, shutdown, model_load_error, model_not_prepared, invalid_request, or queue_full.
response_bytes	integer bytes, always	Body bytes observed while forwarding to the HTTP stack, including SSE comments and in-band errors. Excludes HTTP headers and transport framing. Not a delivery acknowledgement.
disconnect_phase	nullable string enum	queued, loading, or generating for client disconnect. loading includes header wait; generating means upstream headers had arrived. Null for other outcomes.

The parser handles fragmented single-line SSE JSON, final usage/timing frames and nonstream JSON. It keeps at most 256 KiB of one SSE line or one nonstream JSON response for inspection. Oversized accounting frames or JSON responses are forwarded unchanged but their accounting is unavailable. Nonstream responses have no observed token arrival time, so TTFT and generation duration stay null. This parser does not interpret multiline SSE data fields. Current Strata emits single-line frames.

At the pinned Strata revision in RESEARCH.md, source inspection verifies that openai_stream emits prompt_tokens, completion_tokens, cached-token usage, finish reason and the llama.cpp-style timings fields listed above. openai_collect preserves usage and timings in JSON. request_timings returns no timings if the engine has no prompt clock. Reasoning counts exist internally but this revision does not include them in chat usage, so reasoning_tokens stays null with that backend. Fixture tests verify parsing, including reasoning counts from other compatible backends. No live GPU inference was run to establish availability on poseidon.

The router does not own KV checkpoints and the chat response does not report per-request parking saves/restores, their milliseconds/bytes, or budget skips. These are not guessed from global server metrics. cache_n records reused prompt tokens, not a count of checkpoint restores. An upstream follow-up should attach numeric per-request save/restore/skip accounting to the final frame; scraping stderr would mix requests and risk logging content.

For CLIProxyAPI joins, match model_requested or model_resolved, the arrived_at through completed_at interval, token counts, status and latency. Confirm which instant its event timestamp describes and allow for proxy overhead. Identical overlapping requests can remain ambiguous. If the collector can preserve x-strata-request-id, prefer that exact join. CLIProxyAPI's existing usage event does not provide this ID. Compare total_ms with total_ms - queue_wait_ms and ttft_ms with ttft_after_lease_ms to include or exclude the queue. Use engine prompt_per_second for prefill and predicted_per_second for decode speed. Null means unknown, not zero. Model load and KV restore time cannot be separated by subtracting TTFT alone.

Records never contain prompts, generated text, tool names/arguments, headers, API keys, client addresses or error bodies. Only recognized model identifiers, fixed endpoint/reason strings, timestamps and numeric/boolean accounting enter the record. The observer inspects response JSON transiently to extract those allowed fields. Persistence and the HTTP endpoint use the same explicit typed record; neither serializes request/response objects.

Control the existing daemon

strata-router status /run/user/1000/strata-router/control.sock
strata-router off [--now] /run/user/1000/strata-router/control.sock
strata-router on /run/user/1000/strata-router/control.sock [MODEL]

These commands connect to the existing Unix socket. They never launch or signal the daemon itself. status prints one JSON line with the daemon PID and, when idle, child PID, loaded ID and last startup error. During an operation it reports busy instead of waiting for a stream to finish.

off frees the model's memory and returns only when that is done. It disables routing first, so new and queued inference gets 503 (health, model listing and control stay usable). It then waits for work that holds the model lease to finish, unloads the model by stopping the backend child, and returns once the backend process group is gone. The daemon, its control socket and the systemd listener stay up. --now skips the wait and interrupts active responses.

$ strata-router off /run/strata-router/control.sock
[   0.0s] routing disabled: new and queued requests get 503
[   0.0s] 1 active request, waiting for it to finish (use --now to interrupt)
[   6.1s] unloading model qwen3.8-flash-next-gsq-rco-iq3_xxs
[   6.1s] stopping backend pid 48213
[   6.4s] backend pid 48213 exited, memory released
off: model unloaded, routing disabled

on enables routing and loads a model, then returns only when the backend is healthy, its model identity matches the profile and the router accepts requests. It loads MODEL (a concrete ID or alias) if given, else the model currently loaded, else the last model that was loaded, else the first entry of enabled_models (the first model in ID order if that list is omitted). That model must be installed and compatible; there is no fallback to another model. Loading is bounded by startup_ms. On failure the command exits non-zero with the startup error and routing returns to its previous state. An explicit on ignores the failure cooldown. Running on when routing is on and the model is loaded and healthy prints on: already on at once. A request that arrives during the load waits for it. The engine lock is held for the whole load, so idle unload cannot interfere. Idle unloading still applies afterwards: the idle clock restarts when on finishes, so the model unloads again after idle_ms without requests.

$ strata-router on /run/strata-router/control.sock
[   0.0s] routing enabled
[   0.0s] starting backend for model qwen3.8-flash-next-gsq-rco-iq3_xxs
[   0.0s] backend pid 48390 started, waiting for readiness (loading model qwen3.8-flash-next-gsq-rco-iq3_xxs)
[   2.0s] still waiting for readiness (2s of 120s allowed)
[   9.3s] backend reported ready, running health check
[   9.4s] health check passed, model identity verified
on: model qwen3.8-flash-next-gsq-rco-iq3_xxs loaded and healthy, routing enabled

Progress lines go to stderr. The daemon's final JSON event goes to stdout, with ok, message, and the same fields as status. The exit status is zero only for a successful final event. on and off have no client timeout and fail if the daemon closes the connection without a final event. Only one on or off runs at a time; a second waits for the first. Interrupting the client (Ctrl-C) cancels the wait or load in progress. A cancelled off leaves routing disabled; run off --now or on next.

Protocol: a client sends one line, either a bare word or a JSON object. A bare status returns one JSON line, as before. Bare on and off remain the original routing-flag-only commands with a single {"routing":...} reply. They do not load or unload. The CLI instead sends {"cmd":"off","now":false} or {"cmd":"on","model":null} and reads newline-delimited events: {"event":"progress","op","phase","message","elapsed_ms"} followed by exactly one {"event":"done","ok":true|false,...}. An older daemon answers these requests with {"error":"unknown control command"}, so restart the daemon after upgrading the binary.

The socket's directory must belong to the daemon user and have no group or other permissions. The socket is mode 0600. Do not share that directory. A second daemon cannot overwrite its socket. After an unclean crash, verify the service is dead before removing a stale socket. Configuration and Strata profiles are trusted executable configuration, writable only by the service operator. No shell evaluates user input. Model IDs accept ASCII letters, digits, underscores, dots and hyphens. HTTP binds loopback only. Local users can send inference requests; use a separately authenticated proxy if exposing inference beyond a trusted machine. No SSO changes are needed. Child stdout/stderr are discarded to avoid accidentally logging profile secrets. Startup diagnostics are intentionally bounded and generic.

Scheduling and timers

Wait-first scheduling

The example enables wait_when_busy: true. Busy requests wait without a queue deadline or an overload 429. max_pending limits dispatched upstream requests, not waiting clients. response_ms: 0 disables the total response deadline. A positive value retains a deadline after admission. queue_ms is ignored in this mode. Waiting connections and their request bodies still consume memory, so expose this loopback API only through a trusted proxy. This does not guarantee recovery from a backend crash, invalid request, client timeout, or exhausted host resources.

The router reserves an active model lease before sending upstream, then releases the engine mutex. A slow response-header wait cannot block admission of other requests for the same model. The lease lasts through EOF, error or client disconnect. Model changes wait until every active lease ends.

Requests for the loaded model run together. After its last request ends, model_grace_ms keeps it selected through tool-call gaps. A new same-model request resets that quiet period. Once quiet, the earliest waiting model group goes next. Other model groups wait without expiring; continuous current-model work can delay them. The router sees HTTP requests, not T3 thread completion, so the grace interval is an approximation, not a thread-completion guarantee. A zero grace interval disables this preference between requests. Idle unloading also waits for the grace interval and an empty queue.

Streaming clients receive HTTP 200 and SSE comments every five seconds while queued, loading, or generating. Upstream errors after those headers become event: error frames with error.status and no success [DONE]. Non-streaming requests retain upstream HTTP status. CLIProxyAPI v8.0.30 recognizes these SSE errors. Configure requests.streaming.keepalive-seconds and send-keepalive-before-ft: true in that proxy to keep downstream clients connected before generation starts. Disable cooling on the single local provider so a transient backend error does not disable its route for other clients.

tests/waiting.py covers overloaded admission, delayed headers, model groups, tool-gap grace, disconnects, errors and shutdown. Set CLIPROXYAPI_BIN to the v8.0.30 executable when running it to also check real Responses translation, pre-token heartbeats, in-band failure handling and immediate route reuse. The optional proxy test uses fixture credentials and no real models.

Service tiers (fast mode)

Requests choose a tier with OpenAI's service_tier field: priority, default (also auto, or omitted) or flex. Anthropic-style speed: "fast" also selects priority, and speed: "standard" leaves the tier unchanged. Other values get 400. The router removes both fields before forwarding to Strata. Upstream responses and queued streaming responses carry x-strata-service-tier; early router errors may omit it. /metrics reports queue_depth_by_tier, including startup and upstream header waits in legacy mode.

Tiers only order waiting requests. A waiting higher-tier request holds back every lower-tier request, including ones for the loaded model, so the engine drains toward it. Requests of the same tier keep their existing order and model-group rules. Priority also skips the model_grace_ms hold. No tier cancels or preempts a generation that has already been admitted: a priority request for another model waits for active responses to finish, then switches. Flex requests wait behind all default and priority work, so a steady stream of higher-tier requests can delay them indefinitely.

In legacy bounded mode, a waiting higher tier likewise holds back lower tiers, but max_pending, the 429 limit and queue_ms still apply to every tier.

Current cache matches

Within each service tier and model group, requests with a higher cached_tokens / input_tokens fraction run first. A request qualifies only when its input meets the backend's configured conversation_cache_min_tokens threshold. Equal fractions and inputs below that threshold keep arrival order. Cache reuse never promotes a request above its service tier. The existing preference for the loaded model and its grace interval still applies.

This requires the Strata cache lookup extension, including a rebuilt native engine. The router asks the loaded backend for current matches before admission when multiple requests compete in the highest waiting tier. It discards scores when requests enter or leave the queue and when generation ends, including cancellation. It never uses previous response usage or client-supplied cache scores.

cache_lookup_ms bounds each complete lookup, including all batches. It defaults to 1000 milliseconds; zero disables lookup, and the maximum is 10000. An unsupported, busy or unavailable backend, a timeout, or an invalid reply falls back to the existing tier and model-group rules, with arrival order within a group. Legacy queue deadlines still apply during lookup. The lookup does not hold the engine mutex, so status and stream cleanup can proceed. Already admitted generations finish normally.

The example sets cache_max_active: 1 so requests stay in the router queue until the current generation finishes. This matters because the native probe requires an idle engine. max_pending alone can allow a private backlog inside Strata, where the router cannot reorder it. cache_max_active defaults to zero for existing configurations, which preserves unrestricted dispatch up to max_pending. Set it to one for reliable lookup opportunities with a single generation slot. Larger values allow concurrent generation or batching, but lookup can fall back while Strata is busy. Zero disables this additional limit, as does cache_lookup_ms: 0. The limit applies in both scheduling modes and accepts values up to 1024.

Legacy bounded scheduling

Omitting wait_when_busy, or setting it to false, preserves the original contract. In this mode the daemon holds its engine lock while it selects or starts a model and sends the upstream request. It releases the lock once response headers arrive. Requests for the loaded model can then overlap; Strata decides whether to queue them or use configured batch slots. A request for another model waits until all active responses finish. max_pending bounds active and waiting requests. Excess admission gets 429. Queue expiry gets 504 without switching. Concrete requests either get that model or an error.

Legacy durations are positive milliseconds:

• queue_ms starts after admission and bounds waiting for the model lease.
• startup_ms starts after spawning and bounds readiness. Failure kills and reaps the process group, returns 503, and starts cooldown_ms. Requests during cooldown receive 503 without another launch. There are no automatic retry loops.
• response_ms starts before sending upstream and bounds headers and the full stream. Header expiry returns 504. Expiry or upstream failure after headers terminates the response with a transport error. Existing HTTP status and content type, including upstream errors, pass through.
• idle_ms starts at successful response completion or cancellation. It resets when the response worker releases its lease. While any response is active, idle unloading cannot run. A 50 ms maintenance tick unloads by stopping/reaping the process, leaving the daemon alive. The next request reloads. strata-router off uses the same stop path on demand.
• shutdown_ms bounds graceful child termination before SIGKILL. It also supplies the legacy request-drain timeout when drain_ms is omitted. A separate drain_ms: 0 waits for accepted requests and their streaming/cleanup workers without a deadline. Positive drain_ms bounds that wait, after which work can be cancelled. Startup observes forced shutdown cancellation on its next bounded readiness poll.

Streaming uses a two-chunk channel for backpressure, not an output-sized buffer. Client disconnect drops the upstream stream and frees the lease. A disconnect during startup can leave the child starting until startup timeout, but cannot transfer the lease to another request early. Upstream work is not retried. Strata's own request cancellation handles the closed upstream connection; switching kills its old process group before starting another profile.

Service deployment

Install both units in contrib/, adjust paths and create the dedicated account. The .socket unit owns the loopback listener with Accept=no and FlushPending=no. Set require_socket_activation: true in a managed production config and match the configured address to ListenStream. The daemon validates LISTEN_PID, exactly one descriptor, its listening state and address, then adopts fd 3. It marks the descriptor close-on-exec and removes activation variables from the backend environment. Missing activation is an error when required; malformed activation never falls back to binding another listener. Direct binding remains available for standalone development configs.

On SIGTERM, the old process stops accepting connections but keeps routing and cancellation state unchanged for accepted work. With drain_ms: 0, it waits for queued requests, header waits, streaming bodies and tracked cleanup workers to finish. Only then does it stop the backend and remove its private control socket. New connections wait in the systemd listener's finite TCP backlog until the replacement process accepts them. Leave the .socket unit running during ordinary service restarts. A socket-unit restart discards that protection.

Use KillMode=mixed, TimeoutStopSec=infinity and SendSIGKILL=yes. Systemd sends the initial stop signal only to the main process, not its backend. Once the main process exits, systemd kills any remaining cgroup descendants. The private RuntimeDirectory is removed after failure, allowing Restart=on-failure to recover without a stale control socket. Tests cover real systemd crash recovery as well as ordinary draining. Linux parent-death signaling kills the direct backend on router death; systemd remains responsible for descendants. A crash cannot preserve responses that were already being generated.

An indefinitely stalled request deliberately stalls a graceful restart. Inspect the private status command for draining: true; do not send inference or health probes to the retained HTTP listener to decide whether the old process is still draining. An operator can force recovery with systemctl kill --kill-whom=all --signal=SIGKILL strata-router.service, but that discards active work. Backlog exhaustion and client-side deadlines can still fail requests during a sufficiently long drain.

NixOS must set systemd.services.strata-router.stopIfChanged = false after converting to socket activation. NixOS's default stop/start policy also stops associated socket units, which would discard queued connections. The false setting requests a service-only restart after activation, preserving the unchanged socket. Changing the listener address or socket unit itself needs a separate maintenance plan.

The first conversion cannot inherit the old router's privately bound socket. Wait for active work to finish before stopping that old process, then start the socket and socket-activated service. That one-time cutover still has a short connection gap. Do not introduce the socket while an old process is still bound to the same port.

tests/restart.py checks a real process handoff with a retained descriptor, including accepted model queues and delayed response headers. The optional root-only python3 tests/systemd_restart.py /absolute/path/to/strata-router uses unique temporary units and the fake backend to test actual systemd restart and SIGKILL recovery. It does not touch the production service or use model weights.

CLIProxyAPI is optional in front, configured with this daemon as its OpenAI-compatible upstream. It does not own this daemon's Strata processes. See RESEARCH.md for the source evidence and responsibility boundary.

License

Strata router is licensed under the Mozilla Public License 2.0. Third-party source snapshots under research/ retain their upstream licenses, included in each snapshot directory. Rust dependencies retain their respective licenses.