LocalAI Workspace

This checkout is currently rooted at:

C:\Users\Samuel\WIN_Dev\Personal\AI\LocalAI

That path documents the reference machine; scripts derive the workspace root from their own location. For moving an existing workspace to another machine or user, use PORTABILITY.md. For recreating the installation from scratch, use SETUP_REPLICATION.md.

Mission

This workspace exists to continuously identify, run, and compare the most capable state-of-the-art local models that fit the available hardware. It is an experimentation environment, not a fixed installation centered on one model. As stronger models and runtimes appear, the workspace should make it quick to:

1. Bring up a candidate model with the correct runtime.
2. Tune model quantization, KV-cache precision, context length, batching, and GPU offload for this machine.
3. Measure practical VRAM fit, prefill speed, generation speed, stability, and usable context.
4. Compare the candidate with its predecessors under reproducible conditions.
5. Preserve successful configurations as named launch presets backed by retained results.

The target is maximum useful model capability while keeping inference practical and, where possible, keeping the model and active context entirely in dedicated VRAM. Largest context, smallest quantization, and highest throughput are measurements—not substitutes for model quality.

Evaluation vision

The workspace currently has operational benchmarks, not model-quality evaluations. Existing scripts establish whether a configuration loads, fits, remains on the GPU, and performs acceptably at the requested context. They do not establish that one model reasons better, follows instructions better, retrieves long-context information more reliably, or produces more useful answers than another.

A future evaluation layer may add repeatable, versioned comparisons for representative real work, reasoning, instruction following, coding, structured output, and long-context retrieval. Those evaluations should compare each candidate directly with the current preferred model and retain enough metadata to reproduce the result. Until that exists, claims that a model is “better” require deliberate side-by-side use; benchmark throughput or successful context allocation alone is insufficient.

Every model-specific preset is therefore provisional. Keep it while evidence says it is useful; replace it cleanly when a more capable model and configuration wins.

What this workspace does

The workspace combines:

• LlamaStash for model discovery, process management, its TUI, and the OpenAI-compatible proxy.
• PrismML's llama.cpp runtime for CUDA inference and custom Prism quantizations such as PQ2_0.
• PowerShell launch scripts that select arbitrary GGUF models, remember recent choices, create Desktop shortcuts, and benchmark practical context capacity.

The current baseline was installed with:

• LlamaStash v0.4.0.
• Prism llama.cpp build 10685, commit 7dffb158d.
• NVIDIA CUDA device CUDA0 on an RTX 5070 Ti with approximately 16 GB VRAM.
• Initial model Bonsai-2-27B-PQ2_0-CRACK.gguf.

These versions describe the installed baseline, not a requirement to freeze future repairs or upgrades. The models the workspace currently runs, how each fits in 16 GB, and how each is tuned are documented per model under docs\models\.

Set up on a new machine

The repository carries scripts and documentation. It does not carry the runtime: the binaries are 677 MB, the state is machine-specific, and the benchmark history is personal. Everything omitted lives under .private\, and one script rebuilds it.

Requirements: Windows with an NVIDIA GPU and a current driver, PowerShell 5.1 or later, and internet access. About 520 MB is downloaded and roughly 700 MB occupied. Setup also installs Bun, which the gateway runs on. Nothing here requires WSL.

git clone <repository-url> LocalAI
cd LocalAI
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\Setup-Private.ps1

In order, that script creates the .private tree, downloads and checksum-verifies the LlamaStash release, downloads the PrismML llama.cpp CUDA build together with its separately published CUDA runtime archive, runs both binaries to prove they actually start, and reports which model files are present.

Read its exit code, not its output: 0 ready, 2 runtime installed but model weights are missing, 1 setup failed.

Argument	When to use it
-CudaVariant 12.4	The driver is too old for the CUDA 13 build. Default is 13.3.
-PrivateRoot <path>	Keep the runtime on another drive, or provision a throwaway copy when testing.
-Force	Reinstall over an existing runtime instead of skipping it.
-PrismTag prism-b10685-7dffb158d	Pin the validated build rather than taking the latest release.
-LlamaStashTag v0.4.0	Pin the LlamaStash release.
-WithReferenceClone	Also clone the LlamaStash source for reading.

Model weights are deliberately not downloaded. The Hugging Face repository may require authentication and licence acceptance, and quietly pulling an unofficial mirror is the wrong default. When the script reports missing files, fetch them from the official prism-ml/Ternary-Bonsai-2-27B-gguf repository into %USERPROFILE%\.lmstudio\models\prism-ml\Ternary-Bonsai-2-27B-gguf\.

Three files are needed: the PQ2_0 weights, the mmproj-BF16 projector for image input, and the kv-bias calibration file. The bias file is not optional. The launch profile quantizes the KV cache to q4_0, which is only calibrated when that bias is supplied, and llama.cpp will happily run without it and silently produce worse long-context output.

Then install the shortcuts and start the model:

.\scripts\Install-Desktop-Shortcuts.ps1
.\scripts\Start-Bonsai2-MaxContext.ps1        # desktop profile, opens the TUI
.\scripts\Start-Bonsai2-Headless.ps1          # API server on port 41100, no TUI

When setup fails

Symptom	Cause and fix
No asset matching '...' in release	The release naming changed, or the CUDA variant does not exist for that tag. The error prints every available asset name; pick one and pass -PrismTag/-CudaVariant.
llama-server.exe did not run	The CUDA runtime DLLs are missing or mismatched. The runtime and cudart archives must come from the same release and CUDA version. Rerun with -Force.
Checksum mismatch	A truncated or tampered download. Delete %TEMP%\localai-setup and rerun.
Required file not found: ...llamastash.exe from any other script	.private is absent or pointed elsewhere. Run this setup script, or set LOCALAI_PRIVATE_ROOT.
Setup provisions the real .private despite an override	An environment variable exported in WSL does not reach a Windows process. Pass -PrivateRoot instead.
GitHub API returns 403	Unauthenticated rate limit. Wait, or run from a network with fewer callers.

Verify an installation without reinstalling it:

. .\scripts\Common.ps1                        # throws if the private runtime is missing
& $LlamaServer --list-devices                 # must list CUDA0

Portability

The workspace can move to another Windows user or directory without rewriting its core scripts. Common.ps1 resolves the root from $PSScriptRoot, model locations use $HOME, and Desktop shortcuts are generated for the current checkout.

Machine-local state must not be copied blindly:

• Regenerate Desktop shortcuts and Pi/OMP installed configuration.
• Delete ephemeral daemon files and reset persisted model selections containing absolute paths.
• Install runtime binaries compatible with the destination GPU and architecture.
• Verify or replace the CUDA0 device assumptions, then rerun context-capacity benchmarks on changed hardware.
• Place the GGUFs and model-specific cache biases under the destination user's model directory, or update the tuned launcher's source paths.
• Replace the reference Tailscale MagicDNS hostname and diagnostic IP in the launcher and remote-client templates.
• Let the tuned launcher recreate the stable bonsai-2.gguf hard link locally; do not copy that generated link as though it were workspace data.

The client-facing selector remains office-pc/bonsai-2. office-pc is a logical provider ID and may remain unchanged even when the Windows computer or Tailscale hostname changes. The complete file inventory, migration procedure, hard-link constraints, security checklist, and verification commands are in docs\PORTABILITY.md.

Source of truth

The files under scripts\ are authoritative. Desktop shortcuts are generated pointers, not editable copies.

Desktop\Local AI\
  1 - Open LlamaStash TUI.lnk
  2 - Select and Start a Model.lnk
  3 - Benchmark Context Capacity.lnk
  4 - Enable Tailnet Model API.lnk
  9 - Stop All Local AI.lnk
  Presets\
    Bonsai 2 27B - 262K Max Context.lnk
    Bonsai 2 27B CRACK - 262K Max Context.lnk
    (one shortcut per roster row: Bonsai, Qwen3.8, Ridge, Fusion, EXL3, Vision)

Regenerate those shortcuts with:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\Install-Desktop-Shortcuts.ps1

Never repair behavior by editing a .lnk directly. Change the relevant script, then regenerate the shortcuts.

AGENTS.md at the repository root is the agent-facing summary: the anchor-edit discipline and the invariants that must not be broken. Pi loads it into every session automatically, so keep it short and pointed at these docs rather than restating them.

Interfaces: how humans and agents reach the model

Humans use the Desktop shortcuts listed above. Numbered ones are model-agnostic; Presets\ holds benchmark-backed, model-specific launchers. Nothing else is needed for ordinary use.

Programs use an HTTP endpoint. All of them speak the OpenAI-compatible API, so any client that accepts a base URL works.

Every client points at the gateway on http://127.0.0.1:41200/v1. That includes OMP on Windows and in WSL, Pi on either side, and remote machines over the tailnet. This is not a preference, it is what makes concurrent callers work: the model has one inference slot, and a client that reaches llama-server directly interleaves with everyone else and evicts their prefix cache. One base URL for everything, and the queue is universal.

The configuration files that carry it:

Client	File	Points at
OMP, Windows	%USERPROFILE%\.omp\agent\models.yml, an NTFS link to the workspace file	http://127.0.0.1:41200/v1
OMP, WSL	~/.omp/agent/models.yml, a copy refreshed by setup-wsl.sh	http://127.0.0.1:41200/v1
Pi, Windows	%USERPROFILE%\.pi\agent\models.json, an NTFS link to infrastructure\pi\models.json	http://127.0.0.1:41200/v1
Pi, WSL	~/.pi/agent/models.json, a copy refreshed by infrastructure\pi\setup-wsl.sh	http://127.0.0.1:41200/v1
Remote, over tailnet	infrastructure\omp\models.tailnet.yml, rendered per client	https://<machine>.<tailnet>.ts.net/v1

The ports behind the gateway, for debugging only:

Port	Serves	Started by
41200	The gateway. Every client uses this.	infrastructure\gateway\local_ai_gateway.mjs
41100	llama-server directly. Speak to it only to test the model itself.	scripts\Start-Bonsai2-Headless.ps1
11435	The LlamaStash proxy, used by the desktop TUI profile.	scripts\Start-Bonsai2-MaxContext.ps1

Do not point a client at 11435. A request that reaches the LlamaStash proxy for a model it is not already running makes LlamaStash try to auto-start it, and its admission estimate for these Prism quantizations is badly wrong. The refusal reads needs 68.6 GiB but only 30.1 GiB is free and returns 503, which looks like a broken model rather than a misrouted request. This is exactly how the tailnet route used to fail.

Tailscale is optional. Only the last row above needs it, and the scripts that print or install a tailnet URL discover the name at runtime rather than storing it. <machine>.<tailnet>.ts.net is a placeholder: the real MagicDNS name identifies the tailnet, so it is deliberately not recorded here. Read it with tailscale status, or on the serving machine with (tailscale status --json | ConvertFrom-Json).Self.DNSName.

The model ID is bonsai-2; the Pi and OMP selector is office-pc/bonsai-2. Both stay stable across machines and across which Bonsai variant supplies the bytes.

The original Qwen3.8-27B is exposed alongside it as office-pc/qwen3.8-27b (all-GPU preset A) and office-pc/qwen3.8-27b-maxquality (preset B), from both the host and WSL agent configs. The gateway is a single upstream on port 41100, so the selector does not choose the preset: whichever one the desktop launcher started is the one answering. See docs/models/qwen3.8-27b.md.

curl http://127.0.0.1:41200/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"bonsai-2","messages":[{"role":"user","content":"hello"}]}'

Four things a caller has to know:

• The server has one inference slot, and the gateway queues for it. Callers do not run concurrently; two conversations take turns, and each turn costs a cold prefix when the cache is evicted. Measured on a 6.6k-token context: 3.7s cold against 0.08s warm, so the penalty is large. That is handled for you: point at the gateway and let it serialize. Two concurrent streams through it stay strictly ordered, and a second conversation waits at most --handoff seconds (default 5) for the current one to come back before it takes its turn.
• Reasoning is on and defaults to xhigh effort. Answers arrive with the thinking in reasoning_content and the answer in content. A short max_tokens can be consumed entirely by reasoning, returning an empty content: that is a budget that was too small, not a failure. medium is the shorter override; low is unsupported by this model.
• Image input works on the official variant, which loads a projector. The -Crack variant is text-only.
• Sampling defaults are set at launch from the model card. Clients should not re-specify temperature and top-p unless they mean to override calibrated values.

Running the gateway

# Normally you do not run this by hand: the model launchers start it if it is not already up.
# This is for running it on its own.
bun .\infrastructure\gateway\local_ai_gateway.mjs

Bun runs the gateway and is a Windows program here, alongside llama-server. It is deliberately not a WSL one: requiring WSL would put a whole second operating system between a desktop shortcut and the model. scripts\Setup-Private.ps1 installs it, or do it by hand with winget install Oven-sh.Bun. The gateway is a single file using Bun.serve, with no package.json, no dependency install, and no build step. Details and the design argument are in infrastructure\gateway\README.md.

The launchers keep it up, and it outlives them. Every model launcher calls Start-LocalAIGateway first, which starts it hidden and does not wait for it, so closing the shortcut leaves the endpoint running. Starting it when it is already up is a no-op. It runs before the model without complaint, answering a clear 502 until llama-server is listening.

Clients on WSL reach it too, through the loopback interface that WSL2 shares with Windows. The gateway itself is one process on one machine; WSL is only ever a client.

Admission belongs at the server for that reason: clients arrive from both sides of the machine, and none of them has to cooperate for the slot to stay protected.

Directory map

LocalAI\                     tracked in version control
  config\                    LlamaStash configuration and the KV calibration note
  docs\                      Workspace documentation
  infrastructure\eval\       Evaluation source, pinned PPL corpus, ablation configs, vendored BetterBench and EXL3 setup
  infrastructure\gateway\    Single-slot gateway, run with Bun
  infrastructure\omp\        Backup-friendly OMP config and Windows/WSL setup scripts
  infrastructure\pi\         Backup-friendly Pi agent config: model providers and the editor plugin
  scripts\                   Authoritative launch, benchmark, and setup scripts

LocalAI\.private\            never committed; rebuilt by scripts\Setup-Private.ps1
  cache\                     LlamaStash logs and disposable cache data
  repos\llamastash\          Optional reference clone of the upstream LlamaStash repository
  results\context\           Permanent JSON and CSV context benchmark results
  state\                     LlamaStash runtime state and remembered UI selections
  tools\llamastash\          Installed LlamaStash executable
  tools\prism-llama\         Installed Prism llama.cpp binaries and CUDA DLLs

.private is authoritative for machine-local runtime, tools and scratch; downloaded model weights use the separate store below. Common.ps1 derives runtime paths from .private. Override it with LOCALAI_PRIVATE_ROOT to keep the runtime on another drive (this does not relocate model weights).

The evaluation harness is versioned under infrastructure/eval/, not under .private. All 71 previously tracked .private/eval source files have been relocated; .private/ is blanket-ignored and must have no entries in git ls-files. Runtime results, archives and installed tools remain local. This removes the directory from the next commit's tree, not from historical commits. Run python infrastructure/eval/test_layout.py to check the source/runtime boundary and byte-exact corpus.

Important state files:

• .private\state\selection.json: last selected GGUF, launch context, and benchmark maximum. This is intentionally persistent and may be inspected or reset when troubleshooting selection behavior.
• .private\state\state.json: LlamaStash-managed state. Prefer LlamaStash commands over manual edits.
• .private\state\runtime.json and .private\state\daemon.pid: ephemeral daemon bookkeeping. Do not treat stale presence as proof that the daemon is alive.

OMP's local-model configuration is authoritative at infrastructure\omp\models.yml. Windows OMP uses a symbolic link or NTFS hard link to that file; WSL receives a copy through infrastructure\omp\setup-wsl.sh. See infrastructure\omp\README.md before changing OMP integration.

Model storage on this machine

Downloaded weights live physically at F:\AI\LocalAI\models\<publisher>\<repo>, on the user-selected Gen5 drive. C:\Users\Samuel\.lmstudio\models is an NTFS directory junction to that store, not another copy. Existing LM Studio settings, profiles, stable hard-link aliases and Windows/WSL clients can keep their old paths; new downloads should use the F: store explicitly. Keep the publisher/repository layout and do not replace the junction with a separate directory. The F: volume must be mounted for those legacy paths to work.

The verified migration inventory and procedure are recorded in docs/agents/archive/plans/2026-10-05-model-store-migration.md. Moving storage does not alter model bytes, qualify a dual-GPU preset, or establish faster inference; its expected benefit is loading/download I/O. Existing benchmark results and runtime tools remain in their previous locations.

Script architecture

scripts\Common.ps1

All entry-point scripts dot-source this file. It owns the shared contract:

• Resolves the workspace from $PSScriptRoot, and the private runtime from .private or LOCALAI_PRIVATE_ROOT.
• Sets all LLAMASTASH_* environment variables.
• Pins LLAMASTASH_LLAMA_SERVER to .private\tools\prism-llama\llama-server.exe.
• Configures UTF-8 console output.
• Defines default launch arguments.
• Throws when a required binary is absent, naming Setup-Private.ps1 as the fix.
• Loads and saves .private\state\selection.json.
• Implements the GGUF file picker and editable token-window selector.
• Stops existing launches of a selected model.

Do not duplicate this logic in entry-point scripts. Extend Common.ps1 when behavior is genuinely shared.

scripts\Start-Model.ps1

Interactive path:

1. Select a GGUF using a native Windows file dialog.
2. Select or type a context size.
3. Stop another launch of the same model.
4. Start it through LlamaStash.
5. Open the TUI after readiness.

Automation path:

.\scripts\Start-Model.ps1 `
  -ModelPath "C:\path\to\model.gguf" `
  -Context 32768 `
  -NoTui

Keep the command-line path functional. It is the reliable way for an LLM or CI-style process to verify behavior without interacting with native dialogs.

scripts\Test-ContextCapacity.ps1

The benchmark validates two different things for each candidate context:

1. The backend can allocate and load that context.
2. A single request can actually prefill almost the entire context.

It tokenizes the generated prompt through the backend rather than estimating tokens from characters. Results are written to results\context\ as JSON and CSV.

Interactive use offers 64K, 128K, 256K, 512K, and 1M, while allowing a custom value. Noninteractive examples:

# Full prefill at selected candidates
.\scripts\Test-ContextCapacity.ps1 `
  -ModelPath "C:\path\to\model.gguf" `
  -CandidateContexts 65536,131072,262144

# Fast allocation-only smoke test
.\scripts\Test-ContextCapacity.ps1 `
  -ModelPath "C:\path\to\model.gguf" `
  -CandidateContexts 8192 `
  -SkipPrefill

A successful allocation above a model's native context proves runtime capacity only. It does not prove that the model remains accurate at that context.

Model preset launchers

Model-specific launchers and their tuned profiles are documented under docs\models\; the root README does not repeat them.

• Bonsai 2 27B: Start-Bonsai2-MaxContext.ps1 (desktop/TUI), Start-Bonsai2-Headless.ps1 (API), with the shared profile in Bonsai2-Profile.ps1.
• Qwen3.8-27B: Start-Qwen38-Headless.ps1 -Variant Performance|Balanced|Max, with the shared profile in Qwen38-Profile.ps1.

Each *-Profile.ps1 is the one definition of that model's quality-affecting arguments. A launcher may add serving concerns (port, metrics, cache-reuse); it must not hold a second opinion about the model. The generic Start-Model.ps1 and the Test-ContextCapacity.ps1 benchmark above are model-agnostic.

scripts\Test-ModelEval.ps1

A small, fast, deterministic battery for comparing presets: instruction following, arithmetic, JSON, Python code shape, a one-step logic puzzle, and needle-in-a-haystack retrieval at the given context sizes. It records pass/fail, latency, token counts, and decode speed as JSON and CSV under .private\results\evals\.

.\scripts\Test-ModelEval.ps1 -Label "A-allgpu"
.\scripts\Test-ModelEval.ps1 -Label "B-maxquality" -NeedleContexts 4096

It is a smoke battery, not a leaderboard: both Qwen presets pass 7/7, so it separates speed and long-context robustness far more than it separates reasoning quality. Claims that one model is "better" still require deliberate side-by-side use.

scripts\context-length-test.ts

Bun script that measures large-context behavior against a live endpoint (default: the local gateway). For each context size (4096 baseline plus 50k/100k/120k by default, capped at the model's maxContextLength) it runs needle-in-a-haystack retrieval at 10/50/90% depth plus one generation request, measuring:

• TTFT and effective prefill throughput, with the queue/startup floor subtracted using the first request's TTFT.
• Client-side decode throughput.
• Needle accuracy — the row passes when the secret code appears in the visible answer or in the model's reasoning, so a truncated visible answer is not misread as a retrieval failure.

Every request gets a distinct seed (prefix caching cannot assist) and requests are strictly sequential because the gateway serializes the single inference slot. The assistant itself runs on the same model, so the run competes for the slot between agent turns; the floor is visible in the baseline row. Results go to .private\results\context\<label>-ctxlen-<timestamp>\ as results.json, summary.csv, and report.md.

bun scripts\context-length-test.ts --label qwen38-27b
bun scripts\context-length-test.ts --quick --label smoke   # 3 small requests

scripts\Set-DefaultModel.ps1

Switches the default model and thinking level for pi and omp on both the Windows host and the WSL Ubuntu distro in one deterministic pass: reads all four live values, prints a before/after plan, writes every target, then re-reads and verifies. Targets are pi's ~/.pi/agent/settings.json and omp's ~/.omp/agent/config.yml (home-relative on both platforms). Files are patched in place — line surgery for the omp YAML, JSON round-trip for pi — to preserve key order, comments, unrelated keys, and the omp models.yml hard link (which the script never touches).

The WSL half runs as one python3 process inside WSL with the patch code passed base64-encoded: a multi-line quoted here-string does not survive the PowerShell-to-WSL argument path, and path-like arguments get rewritten by the terminal they pass through. Values travel as plain argv tokens, so nothing in the patch needs quoting.

.\scripts\Set-DefaultModel.ps1            # dropdown pickers (model, thinking)
.\scripts\Set-DefaultModel.ps1 -Model office-pc/qwen3.8-27b -Thinking xhigh
.\scripts\Set-DefaultModel.ps1 ... -DryRun  # plan only, no writes

Falls back to a numbered console menu when the WinForms UI cannot initialize.

scripts\Setup-Private.ps1

Rebuilds .private on a new machine: directories, the LlamaStash release, the PrismML llama.cpp CUDA build plus its separate CUDA runtime archive, and a verification pass that runs both binaries. Never dot-sources Common.ps1, which would throw on exactly the missing binaries this script installs. See Set up on a new machine.

Other entry points

• Open-TUI.ps1: opens LlamaStash with the workspace environment.
• Stop-All.ps1: stops a running battery first (delegating to Restart-ModelEvals.ps1 -Force -StopOnly), then every managed model, then any llama-server running from this workspace's own binary, then the LlamaStash daemon.
• Install-Desktop-Shortcuts.ps1: recreates the five general Desktop shortcuts and the tuned preset shortcuts.
• Test-ModelEval.ps1: runs the basic eval battery against whatever model is serving on the default base URL; see above.
• Test-StopControls.ps1: regression test for the stop/start controls -- stale daemons, orphaned eval runners, log-handle false alarms, the controller lock, and Stop-All's battery delegation; refuses to run while a real battery is live.

Runtime invariants

Preserve these unless there is evidence for changing them:

• --device CUDA0 keeps inference on the NVIDIA GPU. The AMD integrated GPU is not part of this CUDA runtime.
• --parallel 1 gives one request the entire configured context instead of dividing it among multiple slots.
• --flash-attn is enabled.
• --n-gpu-layers 99 requests full practical GPU offload for supported models.
• The generic launcher retains --reasoning off; named model presets may enable reasoning with their own defaults.
• --force bypasses LlamaStash's incorrect admission estimate for unknown Prism quantizations.
• The generic launcher's selected model path is dynamic. Named presets are intentionally model-specific and backed by retained benchmark results.
• Scripts set process-local environment variables. Do not add machine-wide environment changes without a concrete need.
• One GPU holds one model. Every launcher calls Stop-WorkspaceModelServer before starting, so the previous server is stopped whether or not LlamaStash started it. Do not add a launcher that skips this.

Model-specific invariants - calibrated KV caches, image-input requirements, and sampler defaults - live with the model under docs\models\.

Known quirks

Model-specific traps - Prism quantization support, KV-cache/flash-attention coverage, MTP behavior, projector interactions, and loop guards - are documented with the model under docs\models\ rather than here.

The TUI VRAM gauge can include the AMD iGPU

On this machine, LlamaStash aggregates the NVIDIA GPU with the Ryzen integrated AMD GPU's shared-memory pool. The TUI can therefore show roughly 26 GB rather than the RTX 5070 Ti's real approximately 16 GB VRAM.

The actual Prism CUDA server exposes only CUDA0, and the scripts explicitly select it. Treat the aggregate TUI number as a telemetry display issue, not usable CUDA memory.

Context size and quality are separate

A benchmark pass means the selected runtime can allocate and prefill the tested number of tokens. It does not override the GGUF's native context metadata or validate long-context answer quality.

Stopping a model means stopping the process, not asking its manager

LlamaStash can only stop launches it made. A llama-server started directly, by the headless launcher or by a supervisor in WSL, is absent from its catalog, so llamastash stop --all reports success while the GPU stays fully allocated. This looked exactly like a broken shortcut.

Stop-WorkspaceModelServer, in Common.ps1, is the one answer to that, and every launcher calls it before starting a model, not only Stop-All.ps1. One GPU cannot hold two copies of the weights. Without the call, running the desktop preset while a headless server is up loads a second model: it either fails to allocate or spills into shared memory and crawls. Matching is on the executable path, so other llama.cpp installations are left alone.

Releasing the GPU is not instant either: a 15 GiB resident model took over 20 seconds to disappear from the process list. A short wait reports a failure that did not happen, so it polls for up to 90 seconds. Confirm with VRAM, not with the absence of an error.

Native command stderr is a terminating error in PowerShell

Both llamastash.exe --version and llama-server.exe --version write their banner to stderr. Under $ErrorActionPreference = "Stop", redirecting with 2>&1 turns that into a terminating NativeCommandError, so a perfectly working binary fails a verification step. Judge native commands by $LASTEXITCODE, and relax the preference around the call.

WSL environment variables do not reach Windows processes

Exporting LOCALAI_PRIVATE_ROOT in WSL and then launching powershell.exe does not pass it through; that requires WSLENV. A test written this way silently provisions the real .private instead of the throwaway one. Pass -PrivateRoot explicitly.

A port can be unbindable while nothing is listening on it

Bun reports a refused port as "already in use", which sends you hunting for a process that does not exist. This bit the gateway: 41099 became unbindable for every Windows process while netstat and ss both showed nothing and netsh listed no excluded range. The cause was a WSL mirrored-networking reservation: once a WSL process has bound a port, Windows can no longer bind it until WSL's networking restarts, and any port that WSL process used is affected, not just the one in use.

Start-LocalAIGateway therefore reports this distinctly, testing whether any process can bind the port at all before blaming a stray gateway. The gateway moved to 41200 rather than requiring a wsl --shutdown, which would have killed whatever was running inside WSL.

The Windows OMP config is a hard link, and editors break it

%USERPROFILE%\.omp\agent\models.yml is an NTFS hard link to infrastructure\omp\models.yml, so both names share one file. An editor or a script that replaces the file instead of writing through it, including sed -i, gives the workspace path a new file and leaves the link pointing at the old one. The two then disagree silently: the workspace shows the new baseUrl while OMP keeps using the old one. Rerun setup-windows.ps1 to relink after editing that file.

Troubleshooting order for future LLMs

Use evidence in this order; do not immediately reinstall everything.

1. Read the authoritative scripts. Start with Common.ps1, then the failing entry point.

2. Check the private runtime exists. Most "nothing works" reports are an absent or misplaced .private. Dot-sourcing Common.ps1 throws and names the fix.

3. Inspect remembered selection. Check .private\state\selection.json for a moved/deleted GGUF or surprising context.

4. Inspect logs. Read the newest entries under .private\cache\logs\.

5. Check managed status using the workspace environment. Dot-source Common.ps1 before invoking LlamaStash directly:

   . .\scripts\Common.ps1
   & $LlamaStash status --json
   

6. Separate manager failure from runtime failure. Launch .private\tools\prism-llama\llama-server.exe directly with the model when diagnosing custom tensor, CUDA, or allocation errors.

7. Use noninteractive script parameters. Native dialogs are for users; explicit -ModelPath, -Context, and -CandidateContexts are for reproducible automation.

8. Run the smallest meaningful smoke check. An 8K allocation-only benchmark verifies wiring quickly. Use a full prefill only when validating context behavior.

9. Stop everything after diagnosis. Run Stop-All.ps1, then confirm with VRAM that the GPU was released: nvidia-smi --query-gpu=memory.used --format=csv,noheader. A freed card reads on the order of 1 GB, not 15.

Safe change workflow

When extending or repairing this workspace:

1. Preserve existing benchmark results unless explicitly asked to delete them.

2. Make shared behavior changes in Common.ps1 rather than creating a second convention.

3. Keep the general Desktop shortcuts model-agnostic; put only benchmark-backed, model-specific launchers under Desktop\Local AI\Presets.

4. Preserve noninteractive parameters for verification.

5. Parse every PowerShell file after editing:

   $failed = $false
   Get-ChildItem .\scripts\*.ps1 | ForEach-Object {
       $tokens = $null
       $errors = $null
       [void][System.Management.Automation.Language.Parser]::ParseFile(
           $_.FullName,
           [ref]$tokens,
           [ref]$errors
       )
       if ($errors.Count) {
           $failed = $true
           $errors
       }
   }
   if ($failed) { exit 1 }
   

6. Smoke the changed path, not merely the script parser.

7. Delete temporary smoke-result JSON/CSV files, but retain real user benchmark results.

8. Regenerate and inspect shortcuts after changing their installer or entry-point names.

9. Remove downloaded archives after verified extraction.

10. Never claim the workspace is stopped without checking processes and listening ports.

Useful commands

Run from the LocalAI root in Windows PowerShell:

# Provision or repair the private runtime
.\scripts\Setup-Private.ps1

# Open the interactive model launcher
.\scripts\Start-Model.ps1

# Open the interactive context benchmark
.\scripts\Test-ContextCapacity.ps1

# Start the benchmark-selected 262K Bonsai 2 profile
.\scripts\Start-Bonsai2-MaxContext.ps1

# Start the original Qwen3.8-27B: Performance (A), Balanced (B), or Max (C)
.\scripts\Start-Qwen38-Headless.ps1 -Variant Performance
.\scripts\Start-Qwen38-Headless.ps1 -Variant Balanced
.\scripts\Start-Qwen38-Headless.ps1 -Variant Max

# Compare presets with the basic eval battery
.\scripts\Test-ModelEval.ps1 -Label "A-allgpu"

# Start the same profile headless for API clients, on port 41100
.\scripts\Start-Bonsai2-Headless.ps1

# Open the TUI
.\scripts\Open-TUI.ps1

# Stop all managed services, including a directly launched llama-server
.\scripts\Stop-All.ps1

# Reinstall Desktop shortcuts
.\scripts\Install-Desktop-Shortcuts.ps1

# Ask Prism which compute devices it sees
.\.private\tools\prism-llama\llama-server.exe --list-devices

# Confirm the GPU is actually free
nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader

Run the gateway from either Windows PowerShell or WSL; one instance covers both:

bun infrastructure/gateway/local_ai_gateway.mjs

Documentation files

• README.md: setup, interfaces, architecture, invariants, troubleshooting, and maintenance guidance.
• docs/models/README.md: per-model index and the shared 16 GB placement tradeoff.
• docs/models/bonsai-2-27b.md: Bonsai 2 27B profile, Prism runtime requirements, and calibration invariants.
• docs/models/qwen3.8-27b.md: Qwen3.8-27B presets, tuning knobs, KV/flash-attention coverage, and measurements.
• infrastructure/gateway/README.md: the single-slot gateway, why it exists, and how to run it.
• PORTABILITY.md: migration inventory, machine-specific values, security guidance, and destination verification.
• SETUP_REPLICATION.md: the original from-scratch build prompt, kept as the record of how the workspace was specified.
• docs/REFERENCES.md: the running collection of external blogs, releases, and threads, each assessed against local measurements and tagged adopted/verify/watch/rejected.
• docs/TODO.md: the parking lot for deferred cross-cutting investigations, each with the trigger that un-parks it.