LocalAI Workspace

This checkout is currently rooted at:

C:\Users\Samuel\WIN_Dev\Personal\AI\LocalAI

That path documents the reference machine; scripts derive the workspace root from their own location. For moving an existing workspace to another machine or user, use PORTABILITY.md. For recreating the installation from scratch, use SETUP_REPLICATION.md.

Mission

This workspace exists to continuously identify, run, and compare the most capable state-of-the-art local models that fit the available hardware. It is an experimentation environment, not a fixed installation centered on one model. As stronger models and runtimes appear, the workspace should make it quick to:

1. Bring up a candidate model with the correct runtime.
2. Tune model quantization, KV-cache precision, context length, batching, and GPU offload for this machine.
3. Measure practical VRAM fit, prefill speed, generation speed, stability, and usable context.
4. Compare the candidate with its predecessors under reproducible conditions.
5. Preserve successful configurations as named launch presets backed by retained results.

The target is maximum useful model capability while keeping inference practical and, where possible, keeping the model and active context entirely in dedicated VRAM. Largest context, smallest quantization, and highest throughput are measurements—not substitutes for model quality.

Evaluation vision

The workspace currently has operational benchmarks, not model-quality evaluations. Existing scripts establish whether a configuration loads, fits, remains on the GPU, and performs acceptably at the requested context. They do not establish that one model reasons better, follows instructions better, retrieves long-context information more reliably, or produces more useful answers than another.

A future evaluation layer may add repeatable, versioned comparisons for representative real work, reasoning, instruction following, coding, structured output, and long-context retrieval. Those evaluations should compare each candidate directly with the current preferred model and retain enough metadata to reproduce the result. Until that exists, claims that a model is “better” require deliberate side-by-side use; benchmark throughput or successful context allocation alone is insufficient.

Every model-specific preset is therefore provisional. Keep it while evidence says it is useful; replace it cleanly when a more capable model and configuration wins.

What this workspace does

The workspace combines:

• LlamaStash for model discovery, process management, its TUI, and the OpenAI-compatible proxy.
• PrismML's llama.cpp runtime for CUDA inference and custom Prism quantizations such as PQ2_0.
• PowerShell launch scripts that select arbitrary GGUF models, remember recent choices, create Desktop shortcuts, and benchmark practical context capacity.

The current baseline was installed with:

• LlamaStash v0.4.0.
• Prism llama.cpp build 10685, commit 7dffb158d.
• NVIDIA CUDA device CUDA0 on an RTX 5070 Ti with approximately 16 GB VRAM.
• Initial model Bonsai-2-27B-PQ2_0-CRACK.gguf.

These versions describe the installed baseline, not a requirement to freeze future repairs or upgrades.

Set up on a new machine

The repository carries scripts and documentation. It does not carry the runtime: the binaries are 677 MB, the state is machine-specific, and the benchmark history is personal. Everything omitted lives under .private\, and one script rebuilds it.

Requirements: Windows with an NVIDIA GPU and a current driver, PowerShell 5.1 or later, and internet access. About 520 MB is downloaded and roughly 700 MB occupied. Setup also installs Bun, which the gateway runs on. Nothing here requires WSL.

git clone <repository-url> LocalAI
cd LocalAI
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\Setup-Private.ps1

In order, that script creates the .private tree, downloads and checksum-verifies the LlamaStash release, downloads the PrismML llama.cpp CUDA build together with its separately published CUDA runtime archive, runs both binaries to prove they actually start, and reports which model files are present.

Read its exit code, not its output: 0 ready, 2 runtime installed but model weights are missing, 1 setup failed.

Argument	When to use it
-CudaVariant 12.4	The driver is too old for the CUDA 13 build. Default is 13.3.
-PrivateRoot <path>	Keep the runtime on another drive, or provision a throwaway copy when testing.
-Force	Reinstall over an existing runtime instead of skipping it.
-PrismTag prism-b10685-7dffb158d	Pin the validated build rather than taking the latest release.
-LlamaStashTag v0.4.0	Pin the LlamaStash release.
-WithReferenceClone	Also clone the LlamaStash source for reading.

Model weights are deliberately not downloaded. The Hugging Face repository may require authentication and licence acceptance, and quietly pulling an unofficial mirror is the wrong default. When the script reports missing files, fetch them from the official prism-ml/Ternary-Bonsai-2-27B-gguf repository into %USERPROFILE%\.lmstudio\models\prism-ml\Ternary-Bonsai-2-27B-gguf\.

Three files are needed: the PQ2_0 weights, the mmproj-BF16 projector for image input, and the kv-bias calibration file. The bias file is not optional. The launch profile quantizes the KV cache to q4_0, which is only calibrated when that bias is supplied, and llama.cpp will happily run without it and silently produce worse long-context output.

Then install the shortcuts and start the model:

.\scripts\Install-Desktop-Shortcuts.ps1
.\scripts\Start-Bonsai2-MaxContext.ps1        # desktop profile, opens the TUI
.\scripts\Start-Bonsai2-Headless.ps1          # API server on port 41100, no TUI

When setup fails

Symptom	Cause and fix
No asset matching '...' in release	The release naming changed, or the CUDA variant does not exist for that tag. The error prints every available asset name; pick one and pass -PrismTag/-CudaVariant.
llama-server.exe did not run	The CUDA runtime DLLs are missing or mismatched. The runtime and cudart archives must come from the same release and CUDA version. Rerun with -Force.
Checksum mismatch	A truncated or tampered download. Delete %TEMP%\localai-setup and rerun.
Required file not found: ...llamastash.exe from any other script	.private is absent or pointed elsewhere. Run this setup script, or set LOCALAI_PRIVATE_ROOT.
Setup provisions the real .private despite an override	An environment variable exported in WSL does not reach a Windows process. Pass -PrivateRoot instead.
GitHub API returns 403	Unauthenticated rate limit. Wait, or run from a network with fewer callers.

Verify an installation without reinstalling it:

. .\scripts\Common.ps1                        # throws if the private runtime is missing
& $LlamaServer --list-devices                 # must list CUDA0

Portability

The workspace can move to another Windows user or directory without rewriting its core scripts. Common.ps1 resolves the root from $PSScriptRoot, model locations use $HOME, and Desktop shortcuts are generated for the current checkout.

Machine-local state must not be copied blindly:

• Regenerate Desktop shortcuts and Pi/OMP installed configuration.
• Delete ephemeral daemon files and reset persisted model selections containing absolute paths.
• Install runtime binaries compatible with the destination GPU and architecture.
• Verify or replace the CUDA0 device assumptions, then rerun context-capacity benchmarks on changed hardware.
• Place the GGUFs and model-specific cache biases under the destination user's model directory, or update the tuned launcher's source paths.
• Replace the reference Tailscale MagicDNS hostname and diagnostic IP in the launcher and remote-client templates.
• Let the tuned launcher recreate the stable bonsai-2.gguf hard link locally; do not copy that generated link as though it were workspace data.

The client-facing selector remains office-pc/bonsai-2. office-pc is a logical provider ID and may remain unchanged even when the Windows computer or Tailscale hostname changes. The complete file inventory, migration procedure, hard-link constraints, security checklist, and verification commands are in docs\PORTABILITY.md.

Source of truth

The files under scripts\ are authoritative. Desktop shortcuts are generated pointers, not editable copies.

Desktop\Local AI\
  1 - Open LlamaStash TUI.lnk
  2 - Select and Start a Model.lnk
  3 - Benchmark Context Capacity.lnk
  4 - Enable Tailnet Model API.lnk
  9 - Stop All Local AI.lnk
  Presets\
    Bonsai 2 27B - 262K Max Context.lnk
    Bonsai 2 27B CRACK - 262K Max Context.lnk

Regenerate those shortcuts with:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\Install-Desktop-Shortcuts.ps1

Never repair behavior by editing a .lnk directly. Change the relevant script, then regenerate the shortcuts.

Interfaces: how humans and agents reach the model

Humans use the Desktop shortcuts listed above. Numbered ones are model-agnostic; Presets\ holds benchmark-backed, model-specific launchers. Nothing else is needed for ordinary use.

Programs use an HTTP endpoint. All of them speak the OpenAI-compatible API, so any client that accepts a base URL works.

Every client points at the gateway on http://127.0.0.1:41200/v1. That includes OMP on Windows and in WSL, Pi on either side, and remote machines over the tailnet. This is not a preference, it is what makes concurrent callers work: the model has one inference slot, and a client that reaches llama-server directly interleaves with everyone else and evicts their prefix cache. One base URL for everything, and the queue is universal.

The configuration files that carry it:

Client	File	Points at
OMP, Windows	%USERPROFILE%\.omp\agent\models.yml, an NTFS link to the workspace file	http://127.0.0.1:41200/v1
OMP, WSL	~/.omp/agent/models.yml, a copy refreshed by setup-wsl.sh	http://127.0.0.1:41200/v1
Pi, either side	~/.pi/agent/models.json	http://127.0.0.1:41200/v1
Remote, over tailnet	infrastructure\omp\models.tailnet.yml, rendered per client	https://<machine>.<tailnet>.ts.net/v1

The ports behind the gateway, for debugging only:

Port	Serves	Started by
41200	The gateway. Every client uses this.	infrastructure\gateway\local_ai_gateway.mjs
41100	llama-server directly. Speak to it only to test the model itself.	scripts\Start-Bonsai2-Headless.ps1
11435	The LlamaStash proxy, used by the desktop TUI profile.	scripts\Start-Bonsai2-MaxContext.ps1

Do not point a client at 11435. A request that reaches the LlamaStash proxy for a model it is not already running makes LlamaStash try to auto-start it, and its admission estimate for these Prism quantizations is badly wrong. The refusal reads needs 68.6 GiB but only 30.1 GiB is free and returns 503, which looks like a broken model rather than a misrouted request. This is exactly how the tailnet route used to fail.

Tailscale is optional. Only the last row above needs it, and the scripts that print or install a tailnet URL discover the name at runtime rather than storing it. <machine>.<tailnet>.ts.net is a placeholder: the real MagicDNS name identifies the tailnet, so it is deliberately not recorded here. Read it with tailscale status, or on the serving machine with (tailscale status --json | ConvertFrom-Json).Self.DNSName.

The model ID is bonsai-2; the Pi and OMP selector is office-pc/bonsai-2. Both stay stable across machines and across which Bonsai variant supplies the bytes.

curl http://127.0.0.1:41200/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"bonsai-2","messages":[{"role":"user","content":"hello"}]}'

Four things a caller has to know:

• The server has one inference slot, and the gateway queues for it. Callers do not run concurrently; two conversations take turns, and each turn costs a cold prefix when the cache is evicted. Measured on a 6.6k-token context: 3.7s cold against 0.08s warm, so the penalty is large. That is handled for you: point at the gateway and let it serialize. Two concurrent streams through it stay strictly ordered, and a second conversation waits at most --handoff seconds (default 5) for the current one to come back before it takes its turn.
• Reasoning is on and defaults to xhigh effort. Answers arrive with the thinking in reasoning_content and the answer in content. A short max_tokens can be consumed entirely by reasoning, returning an empty content: that is a budget that was too small, not a failure. medium is the shorter override; low is unsupported by this model.
• Image input works on the official variant, which loads a projector. The -Crack variant is text-only.
• Sampling defaults are set at launch from the model card. Clients should not re-specify temperature and top-p unless they mean to override calibrated values.

Running the gateway

# Normally you do not run this by hand: the model launchers start it if it is not already up.
# This is for running it on its own.
bun .\infrastructure\gateway\local_ai_gateway.mjs

Bun runs the gateway and is a Windows program here, alongside llama-server. It is deliberately not a WSL one: requiring WSL would put a whole second operating system between a desktop shortcut and the model. scripts\Setup-Private.ps1 installs it, or do it by hand with winget install Oven-sh.Bun. The gateway is a single file using Bun.serve, with no package.json, no dependency install, and no build step. Details and the design argument are in infrastructure\gateway\README.md.

The launchers keep it up, and it outlives them. Every model launcher calls Start-LocalAIGateway first, which starts it hidden and does not wait for it, so closing the shortcut leaves the endpoint running. Starting it when it is already up is a no-op. It runs before the model without complaint, answering a clear 502 until llama-server is listening.

Clients on WSL reach it too, through the loopback interface that WSL2 shares with Windows. The gateway itself is one process on one machine; WSL is only ever a client.

Admission belongs at the server for that reason: clients arrive from both sides of the machine, and none of them has to cooperate for the slot to stay protected.

Directory map

LocalAI\                     tracked in version control
  config\                    LlamaStash configuration and the KV calibration note
  docs\                      Workspace documentation
  infrastructure\gateway\    Single-slot gateway, run with Bun
  infrastructure\omp\        Backup-friendly OMP config and Windows/WSL setup scripts
  scripts\                   Authoritative launch, benchmark, and setup scripts

LocalAI\.private\            never committed; rebuilt by scripts\Setup-Private.ps1
  cache\                     LlamaStash logs and disposable cache data
  repos\llamastash\          Optional reference clone of the upstream LlamaStash repository
  results\context\           Permanent JSON and CSV context benchmark results
  state\                     LlamaStash runtime state and remembered UI selections
  tools\llamastash\          Installed LlamaStash executable
  tools\prism-llama\         Installed Prism llama.cpp binaries and CUDA DLLs

.private is the one authoritative location for everything machine-local, large, or personal. Common.ps1 derives every such path from it, so nothing else needs to know the layout. Override it with the LOCALAI_PRIVATE_ROOT environment variable to keep the runtime on another drive.

Important state files:

• .private\state\selection.json: last selected GGUF, launch context, and benchmark maximum. This is intentionally persistent and may be inspected or reset when troubleshooting selection behavior.
• .private\state\state.json: LlamaStash-managed state. Prefer LlamaStash commands over manual edits.
• .private\state\runtime.json and .private\state\daemon.pid: ephemeral daemon bookkeeping. Do not treat stale presence as proof that the daemon is alive.

OMP's local-model configuration is authoritative at infrastructure\omp\models.yml. Windows OMP uses a symbolic link or NTFS hard link to that file; WSL receives a copy through infrastructure\omp\setup-wsl.sh. See infrastructure\omp\README.md before changing OMP integration.

Script architecture

scripts\Common.ps1

All entry-point scripts dot-source this file. It owns the shared contract:

• Resolves the workspace from $PSScriptRoot, and the private runtime from .private or LOCALAI_PRIVATE_ROOT.
• Sets all LLAMASTASH_* environment variables.
• Pins LLAMASTASH_LLAMA_SERVER to .private\tools\prism-llama\llama-server.exe.
• Configures UTF-8 console output.
• Defines default launch arguments.
• Throws when a required binary is absent, naming Setup-Private.ps1 as the fix.
• Loads and saves .private\state\selection.json.
• Implements the GGUF file picker and editable token-window selector.
• Stops existing launches of a selected model.

Do not duplicate this logic in entry-point scripts. Extend Common.ps1 when behavior is genuinely shared.

scripts\Start-Model.ps1

Interactive path:

1. Select a GGUF using a native Windows file dialog.
2. Select or type a context size.
3. Stop another launch of the same model.
4. Start it through LlamaStash.
5. Open the TUI after readiness.

Automation path:

.\scripts\Start-Model.ps1 `
  -ModelPath "C:\path\to\model.gguf" `
  -Context 32768 `
  -NoTui

Keep the command-line path functional. It is the reliable way for an LLM or CI-style process to verify behavior without interacting with native dialogs.

scripts\Test-ContextCapacity.ps1

The benchmark validates two different things for each candidate context:

1. The backend can allocate and load that context.
2. A single request can actually prefill almost the entire context.

It tokenizes the generated prompt through the backend rather than estimating tokens from characters. Results are written to results\context\ as JSON and CSV.

Interactive use offers 64K, 128K, 256K, 512K, and 1M, while allowing a custom value. Noninteractive examples:

# Full prefill at selected candidates
.\scripts\Test-ContextCapacity.ps1 `
  -ModelPath "C:\path\to\model.gguf" `
  -CandidateContexts 65536,131072,262144

# Fast allocation-only smoke test
.\scripts\Test-ContextCapacity.ps1 `
  -ModelPath "C:\path\to\model.gguf" `
  -CandidateContexts 8192 `
  -SkipPrefill

A successful allocation above a model's native context proves runtime capacity only. It does not prove that the model remains accurate at that context.

scripts\Start-Bonsai2-MaxContext.ps1

Starts the benchmark-selected Bonsai 2 profile at its full native 262,144-token context:

• PQ2_0 packing
• calibrated Q4_0 K/V cache
• K-cache rotation disabled to match the calibration bias
• 2,048 logical / 512 physical batch
• one inference slot
• flash attention and full CUDA offload
• reasoning enabled, with the model-card thinking sampler: temperature 1.0, top-p 0.95, top-k 20, min-p 0, presence penalty 0, and repetition penalty 1.0
• Pi and OMP default thinking level xhigh, matching the model default; medium remains the shorter/faster override
• official PrismML profile loads Ternary-Bonsai-2-27B-mmproj-BF16.gguf for image input

The launcher defaults to the official PrismML model and explicitly passes its multimodal projector to llama.cpp. Pass -Crack to select Bonsai-2-27B-PQ2_0-CRACK.gguf; every shared runtime parameter remains identical, but image input is unavailable because the CRACK distribution has no compatible projector. Because the two model GGUF files have different SHA-256 hashes, each variant uses a bias calibrated from that exact model rather than reusing the official model's bias.

Before restarting LlamaStash, the launcher refreshes the hard link %USERPROFILE%\.lmstudio\models\local\Bonsai-2-Stable-GGUF\bonsai-2.gguf to the selected source GGUF. LlamaStash therefore publishes the stable model ID bonsai-2, and Pi and OMP use the invariant selector office-pc/bonsai-2 across model swaps without duplicating the model file.

The calibrated cache reduced the measured 262K allocation estimate to 12,644 MiB and completed a 261,888-token prefill without shared-memory spill. The corresponding generated shortcut is under Desktop\Local AI\Presets.

scripts\Bonsai2-Profile.ps1

The one definition of the calibrated Bonsai 2 profile: model and bias file locations, context, KV-cache types, batch sizes, and the model card's sampler. Both Bonsai launchers consume it.

This file exists because the two launchers previously carried their own copies and drifted. The headless launch used a q4_0 KV cache without the mean-centering bias and without LLAMA_ATTN_ROT_DISABLE, so it was quantizing the cache with no calibration. Nothing errored; only long-context quality would have suffered. Add quality-affecting arguments here, never to a launcher.

scripts\Start-Bonsai2-Headless.ps1

The API-facing launcher. Runs llama-server in the foreground with no daemon and no TUI, so a process supervisor owns it directly and a crash surfaces as an exit code. Defaults to port 41100. Adds only serving concerns on top of the shared profile: listen address, metrics, and --cache-reuse.

scripts\Setup-Private.ps1

Rebuilds .private on a new machine: directories, the LlamaStash release, the PrismML llama.cpp CUDA build plus its separate CUDA runtime archive, and a verification pass that runs both binaries. Never dot-sources Common.ps1, which would throw on exactly the missing binaries this script installs. See Set up on a new machine.

Other entry points

• Open-TUI.ps1: opens LlamaStash with the workspace environment.
• Stop-All.ps1: stops every managed model, then any llama-server running from this workspace's own binary, then the LlamaStash daemon.
• Install-Desktop-Shortcuts.ps1: recreates the five general Desktop shortcuts and both tuned preset shortcuts.

Runtime invariants

Preserve these unless there is evidence for changing them:

• --device CUDA0 keeps inference on the NVIDIA GPU. The AMD integrated GPU is not part of this CUDA runtime.
• --parallel 1 gives one request the entire configured context instead of dividing it among multiple slots.
• --flash-attn is enabled.
• --n-gpu-layers 99 requests full practical GPU offload for supported models.
• The generic launcher retains --reasoning off. The Bonsai 2 preset intentionally uses --reasoning on; Pi and OMP default this model to xhigh, matching the model's native reasoning default.
• --force bypasses LlamaStash's incorrect admission estimate for unknown Prism quantizations.
• The generic launcher's selected model path is dynamic. The separately named Bonsai 2 preset is intentionally model-specific and backed by retained benchmark results.
• Scripts set process-local environment variables. Do not add machine-wide environment changes without a concrete need.
• The calibrated profile requires all three of --cache-type-k q4_0, --kv-mean-center <bias>, and LLAMA_ATTN_ROT_DISABLE=1. They are one setting in three parts: the bias was measured with K-cache rotation disabled, and a quantized cache without its bias is uncalibrated. Never carry one without the others.
• Quality-affecting arguments live in Bonsai2-Profile.ps1. A launcher may add a port or metrics; it may not hold a second opinion about the model.
• Image input is required. The official profile always passes --mmproj. This is settled, not a performance trade: it is not to be removed to recover --cache-reuse or anything else. The -Crack variant is text-only solely because no compatible projector exists for it.
• One GPU holds one model. Every launcher calls Stop-WorkspaceModelServer before starting, so the previous server is stopped whether or not LlamaStash started it. Do not add a launcher that skips this.

Known quirks

Prism quantizations require the Prism runtime

The initial Bonsai model uses the custom PQ2_0 quantization. Stock llama.cpp and stock LM Studio runtimes reject its custom tensor type. Test this class of model with tools\prism-llama\llama-server.exe, not a stock binary.

Known initial model checksum:

Bonsai-2-27B-PQ2_0-CRACK.gguf
SHA-256: 5b24ea3eebc3e0bccd05fb474eb88b10c57699d71a5db2f29485e3789a70d55d
Native context: 262144 tokens

LlamaStash may misread custom quantization size

LlamaStash v0.4.0 identifies this Prism quantization as unknown and can report a false model weight/admission estimate around 51–52 GiB. That is why these scripts use --force.

Do not remove --force solely because the admission estimate looks alarming. First compare it with a direct Prism runtime launch and actual GPU allocation.

The TUI VRAM gauge can include the AMD iGPU

On this machine, LlamaStash aggregates the NVIDIA GPU with the Ryzen integrated AMD GPU's shared-memory pool. The TUI can therefore show roughly 26 GB rather than the RTX 5070 Ti's real approximately 16 GB VRAM.

The actual Prism CUDA server exposes only CUDA0, and the scripts explicitly select it. Treat the aggregate TUI number as a telemetry display issue, not usable CUDA memory.

Context size and quality are separate

A benchmark pass means the selected runtime can allocate and prefill the tested number of tokens. It does not override the GGUF's native context metadata or validate long-context answer quality.

Stopping a model means stopping the process, not asking its manager

LlamaStash can only stop launches it made. A llama-server started directly, by the headless launcher or by a supervisor in WSL, is absent from its catalog, so llamastash stop --all reports success while the GPU stays fully allocated. This looked exactly like a broken shortcut.

Stop-WorkspaceModelServer, in Common.ps1, is the one answer to that, and every launcher calls it before starting a model, not only Stop-All.ps1. One GPU cannot hold two copies of the weights. Without the call, running the desktop preset while a headless server is up loads a second model: it either fails to allocate or spills into shared memory and crawls. Matching is on the executable path, so other llama.cpp installations are left alone.

Releasing the GPU is not instant either: a 15 GiB resident model took over 20 seconds to disappear from the process list. A short wait reports a failure that did not happen, so it polls for up to 90 seconds. Confirm with VRAM, not with the absence of an error.

A projector makes --cache-reuse inert

Passing --mmproj makes llama-server log cache_reuse is not supported by multimodal, it will be disabled. Slot-level prefix matching is unaffected, so an agentic run still gets its warm cache; only the --cache-reuse chunk-shifting path does not apply. The flag is harmless and stays.

Image input is a requirement of this workspace. The projector is not a tunable. Do not propose removing --mmproj to recover chunk reuse, and do not treat this as a trade to be revisited.

Native command stderr is a terminating error in PowerShell

Both llamastash.exe --version and llama-server.exe --version write their banner to stderr. Under $ErrorActionPreference = "Stop", redirecting with 2>&1 turns that into a terminating NativeCommandError, so a perfectly working binary fails a verification step. Judge native commands by $LASTEXITCODE, and relax the preference around the call.

WSL environment variables do not reach Windows processes

Exporting LOCALAI_PRIVATE_ROOT in WSL and then launching powershell.exe does not pass it through; that requires WSLENV. A test written this way silently provisions the real .private instead of the throwaway one. Pass -PrivateRoot explicitly.

A port can be unbindable while nothing is listening on it

Bun reports a refused port as "already in use", which sends you hunting for a process that does not exist. This bit the gateway: 41099 became unbindable for every Windows process while netstat and ss both showed nothing and netsh listed no excluded range. The cause was a WSL mirrored-networking reservation: once a WSL process has bound a port, Windows can no longer bind it until WSL's networking restarts, and any port that WSL process used is affected, not just the one in use.

Start-LocalAIGateway therefore reports this distinctly, testing whether any process can bind the port at all before blaming a stray gateway. The gateway moved to 41200 rather than requiring a wsl --shutdown, which would have killed whatever was running inside WSL.

The Windows OMP config is a hard link, and editors break it

%USERPROFILE%\.omp\agent\models.yml is an NTFS hard link to infrastructure\omp\models.yml, so both names share one file. An editor or a script that replaces the file instead of writing through it, including sed -i, gives the workspace path a new file and leaves the link pointing at the old one. The two then disagree silently: the workspace shows the new baseUrl while OMP keeps using the old one. Rerun setup-windows.ps1 to relink after editing that file.

Troubleshooting order for future LLMs

Use evidence in this order; do not immediately reinstall everything.

1. Read the authoritative scripts. Start with Common.ps1, then the failing entry point.

2. Check the private runtime exists. Most "nothing works" reports are an absent or misplaced .private. Dot-sourcing Common.ps1 throws and names the fix.

3. Inspect remembered selection. Check .private\state\selection.json for a moved/deleted GGUF or surprising context.

4. Inspect logs. Read the newest entries under .private\cache\logs\.

5. Check managed status using the workspace environment. Dot-source Common.ps1 before invoking LlamaStash directly:

   . .\scripts\Common.ps1
   & $LlamaStash status --json
   

6. Separate manager failure from runtime failure. Launch .private\tools\prism-llama\llama-server.exe directly with the model when diagnosing custom tensor, CUDA, or allocation errors.

7. Use noninteractive script parameters. Native dialogs are for users; explicit -ModelPath, -Context, and -CandidateContexts are for reproducible automation.

8. Run the smallest meaningful smoke check. An 8K allocation-only benchmark verifies wiring quickly. Use a full prefill only when validating context behavior.

9. Stop everything after diagnosis. Run Stop-All.ps1, then confirm with VRAM that the GPU was released: nvidia-smi --query-gpu=memory.used --format=csv,noheader. A freed card reads on the order of 1 GB, not 15.

Safe change workflow

When extending or repairing this workspace:

1. Preserve existing benchmark results unless explicitly asked to delete them.

2. Make shared behavior changes in Common.ps1 rather than creating a second convention.

3. Keep the general Desktop shortcuts model-agnostic; put only benchmark-backed, model-specific launchers under Desktop\Local AI\Presets.

4. Preserve noninteractive parameters for verification.

5. Parse every PowerShell file after editing:

   $failed = $false
   Get-ChildItem .\scripts\*.ps1 | ForEach-Object {
       $tokens = $null
       $errors = $null
       [void][System.Management.Automation.Language.Parser]::ParseFile(
           $_.FullName,
           [ref]$tokens,
           [ref]$errors
       )
       if ($errors.Count) {
           $failed = $true
           $errors
       }
   }
   if ($failed) { exit 1 }
   

6. Smoke the changed path, not merely the script parser.

7. Delete temporary smoke-result JSON/CSV files, but retain real user benchmark results.

8. Regenerate and inspect shortcuts after changing their installer or entry-point names.

9. Remove downloaded archives after verified extraction.

10. Never claim the workspace is stopped without checking processes and listening ports.

Useful commands

Run from the LocalAI root in Windows PowerShell:

# Provision or repair the private runtime
.\scripts\Setup-Private.ps1

# Open the interactive model launcher
.\scripts\Start-Model.ps1

# Open the interactive context benchmark
.\scripts\Test-ContextCapacity.ps1

# Start the benchmark-selected 262K Bonsai 2 profile
.\scripts\Start-Bonsai2-MaxContext.ps1

# Start the same profile headless for API clients, on port 41100
.\scripts\Start-Bonsai2-Headless.ps1

# Open the TUI
.\scripts\Open-TUI.ps1

# Stop all managed services, including a directly launched llama-server
.\scripts\Stop-All.ps1

# Reinstall Desktop shortcuts
.\scripts\Install-Desktop-Shortcuts.ps1

# Ask Prism which compute devices it sees
.\.private\tools\prism-llama\llama-server.exe --list-devices

# Confirm the GPU is actually free
nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader

Run the gateway from either Windows PowerShell or WSL; one instance covers both:

bun infrastructure/gateway/local_ai_gateway.mjs

Documentation files

• README.md: setup, interfaces, architecture, invariants, troubleshooting, and maintenance guidance.
• infrastructure/gateway/README.md: the single-slot gateway, why it exists, and how to run it.
• PORTABILITY.md: migration inventory, machine-specific values, security guidance, and destination verification.
• SETUP_REPLICATION.md: the original from-scratch build prompt, kept as the record of how the workspace was specified.