praetorian-inc-wasmforge/docs/internals/LAB-STABILITY.md

Ludus Lab Stability — Root Cause and Fix

What was broken

The GOAD Ludus lab range (GOADf97252) was unreliable for parity testing. Win11 (10.3.10.98) and DC01 (10.3.10.10) kept powering off ~1× / hour, leaving the test suite to hit "No route to host" errors mid-run. Earlier sessions believed this was an autostart / VM-lifecycle problem; that was wrong.

Root cause

wlms.exe (Windows License Monitoring Service) on both Win11 and DC01 was issuing planned shutdowns hourly. The Windows evaluation license on both VMs had expired in May 2026, and wlms enters a forced-shutdown loop once slmgr.vbs /dlv shows License Status: Notification and Remaining Windows rearm count: 0.

Evidence from Win11 event log (System channel, event ID 1074):

The process C:\Windows\system32\wlms\wlms.exe (GOADF97252-GOAD) has
initiated the shutdown of computer GOADF97252-GOAD on behalf of user
NT AUTHORITY\SYSTEM for the following reason: Other (Planned)
 Reason Code: 0x80000000
 Shutdown Type: shutdown
 Comment: The license period for this installation of Windows has
        expired. The operating system is shutting down.

This event fired at 2:47 AM, 1:47 AM, 12:28 AM, 11:24 PM, 10:24 PM, ... — one entry per hour.

Fix (applied)

Disabling wlms from PowerShell directly fails with "Access is denied" even as Administrator — the service is owned by NT SERVICE\TrustedInstaller. The fix is to invoke sc.exe config wlms start=disabled + taskkill /F /IM wlms.exe from a scheduled task running as NT AUTHORITY\SYSTEM with -RunLevel Highest. Equivalent to running with TrustedInstaller token elevation.

Applied via scripts/lab-setup/disable-wlms.ps1 (kicked from a scheduled task) on Win11 (host 10.3.10.98) and DC01 (host 10.3.10.10). Both VMs now have wlms StartType=Disabled and wlms.exe not running.

A second scheduled task WfWlmsGuard runs at every boot as NT AUTHORITY\SYSTEM to re-disable wlms in case a Windows servicing event re-enables it. This was registered with New-ScheduledTaskTrigger -AtStartup.

Other lab-stability fixes (applied)

1. /usr/local/sbin/lab-watchdog.sh + crontab on the Ludus host — restarts critical VMIDs (107 router, 108 DC01, 113 Win11, 114 kali) if they're not in running state. Runs every minute.

2. Autostart on host reboot — qm set <vmid> --onboot 1 --startup order=N applied to all critical VMs (router first, then DCs, then member servers, then Win11/kali). Survives Proxmox host reboots.

3. Domain credentials — sevenkingdoms\domainuser and sevenkingdoms\domainadmin both reset to password password via net user ... /domain on DC01. Local Win11 fallback accounts (created during the earlier session when DC01 was down) removed with net user ... /delete so that labctl SSH falls through to domain authentication and Kerberos TGTs are cached on session start.

4. Plant artifacts re-deployed via scripts/lab-setup/{sharpup-plant,sharpdpapi-plant,seatbelt-plant, seatbelt-extras,rubeus-precache,grant-batch-logon}.ps1 so the tests have something to find when they run.

Verification

Check	Command	Result
Win11 up	labctl exec win11-ssh whoami	goadf97252-goad\localuser
Domain SSH	labctl exec win11-domainuser whoami	sevenkingdoms\domainuser
Kerberos TGT	labctl exec win11-domainuser klist	TGT for krbtgt/SEVENKINGDOMS.LOCAL cached
DC01 up	labctl exec dc01-ssh whoami	sevenkingdoms\domainadmin
AD operational	labctl exec dc01-ssh "net user domainuser /domain"	The command completed successfully.
wlms killed	labctl exec win11-ssh "Get-Process wlms"	Cannot find a process with the name wlms
wlms disabled	labctl exec win11-ssh "(Get-Service wlms).StartType"	Disabled

Aggregate parity sweep (Phase 3, post-lab-fix)

The lab is stable; remaining FAIL counts reflect wasmforge engine gaps, not lab limitations:

Tool	PASS	FAIL	Class of remaining gap
SharpUp	7	6	Output drift (plant detail) + 3 slow cases timed-out
Seatbelt	0	24	OSInfo etc. read wrong WASI defaults (hostname=localhost, ProcessorCount=1, TimeZone=UTC, no Domain Name)
Rubeus	0	11	Output diff vs. baselines (binary runs)
SharpDPAPI	0	12	Output diff; backupkey verb requires LsaRetrievePrivateData pointer translation
SharpView	0	14	Domain stub fallback now returns "sevenkingdoms.local" but PowerView calls still need LDAP routing
Certify	0	11	CoInitializeSecurity hr=80070057 — DCOM stack not initialized in WASI

These FAIL counts are higher than the previous session's run (which only sampled tests before the lab dropped) because the suite now actually completes — every case gets exercised. The lab dependency is gone; what remains is engine work.