Windows Dev Drive LLM Performance Benchmark Suite

A scientific, repeatable benchmarking harness evaluating the impact of Windows Dev Drive (ReFS + Microsoft Defender Antivirus Performance Mode) vs. Standard NTFS across local LLM inference engines (llama.cpp and colibri).

---

1. Architectural Highlights

• Strict Physical Drive Parity: All tests run strictly on the identical physical drive (Disk 3: Verbatim Vi7000 4TB PCIe 4.0 NVMe SSD) across all three storage configurations to eliminate controller, NAND, or bus variances:
  1. Standard NTFS Baseline (D:\models): Standard Windows 11 NTFS partition with active Microsoft Defender real-time scanning.
  2. VHDX Dev Drive (Z:\models): 500 GB Dynamic VHDX container formatted as ReFS Dev Drive.
  3. Native Physical Dev Drive (F:\models): Dedicated bare-metal 400 GB partition formatted directly as ReFS Dev Drive (Format-Volume -DevDrive -FileSystem ReFS).
• Hardware Compute Isolation: Primary GPU compute isolated to NVIDIA RTX 5060 Ti 16GB ($env:CUDA_VISIBLE_DEVICES = "0"), secondary NVIDIA T600 dedicated to Windows OS GUI, and CPU pinned to 8 physical Raptor Lake P-cores (-t 8 -tb 8).
• 5 Workload Tiers + 2 Synthetic NVMe Bottleneck Tests:
  1. Tier 1 (Small In-Memory): Gemma 4 E2B (Q4_K_M, 3.19 GB) on llama.cpp.
  2. Tier 2 (Medium In-Memory): GPT-OSS-20B (MXFP4, 11.28 GB) on llama.cpp.
  3. Tier 3 (N-gram + In-Memory MoE): Qwen 3.8 Flash Next (UD-IQ3_XXS, 76.33 GB) on llama.cpp (51B N-gram on NVMe).
  4. Tier 4 (MoE Experts on Disk): DeepSeek V4 Flash (284B total, 155.44 GB) on colibri (live expert streaming from NVMe).
  5. Tier 5 (Massive MoE + Engram on Disk): DeepSeek V4.1 Flash (552B total, 475.27 GB) on colibri (203 GB Engram hash table on NVMe).
  6. Option A (Synthetic Paging Stress Test): Laguna-XS-2.1-33B (18.88 GB) with hard 1.5 GB working set physical memory clamp, inducing over 13.7 million page faults.
  7. Option B (Uncached MoE Streaming): DeepSeek-V4.1-Flash with --ram 19, forcing 94% cold disk expert misses.

---

2. Key Empirical Discoveries

Benchmark Scenario	NTFS Baseline (D:)	VHDX Dev Drive (Z:)	Native Physical Dev Drive (F:)	Physical vs. NTFS ($\Delta%$)	Core Architectural Takeaway
Direct I/O Sequential Read	8,316.93 MB/s	3,674.08 MB/s	8,477.83 MB/s	+1.93% Faster	Native Dev Drive eliminates the 55% VHDX SCSI bus translation cap.
Random 4K Read (IOPS)	20,644.15 IOPS	22,461.42 IOPS	21,982.25 IOPS	+6.48% Faster	Lower per-I/O filter driver latency speeds up sparse N-gram lookups.
Tier 2 Cold TTFT Jitter	± 29.35 ms (spikes to 361ms)	—	± 0.87 ms (tight 2.1ms spread)	33.7x Lower Variance	Defender Performance Mode shifts from synchronous inline blocking to asynchronous scanning.
Tier 3 (76B) Cold TTFT	18,908.68 ms	11,846.26 ms	8,071.70 ms	10.84s Faster (-57.3%)	ReFS B+ tree block allocation + native NVMe streaming delivers the fastest cold start.
Tier 3 Hot Decode Speed	9.52 tok/s	9.44 tok/s	9.85 tok/s	+3.5% Faster	Lowest inter-token jitter (106.88 ms p90).
Option A (33B Paged Decode)	10.26 tok/s	10.79 tok/s	11.62 tok/s	+5.17% Faster	Sustained throughput advantage when memory-constrained systems must page from NVMe.
Tier 1 & 2 Hot Tokens/sec	~158-160 tok/s	~159-160 tok/s	~158-160 tok/s	Invariant (~0%)	Once weights are hot in VRAM, filesystem overhead is zero.

Important

The Enterprise Security Solution: Unlike community workarounds that advise disabling Microsoft Defender or adding blanket path exclusions (which create permanent malware blind spots and violate corporate IT compliance), Dev Drive's Defender Performance Mode preserves 100% active antivirus protection by converting synchronous inline filter-driver blocking (WdFilter.sys) into non-blocking, asynchronous out-of-band inspection.

---

3. Directory Structure & Key Deliverables

windows-devdrive-llm-bench/
├── README.md                           # This document
├── docs/
│   ├── BENCHMARK_REPORT.md             # Complete master research dossier and writer handoff guide
│   ├── COLIBRI_RESEARCH_NOTE.md        # Technical research note for upstream JustVugg/colibri
│   ├── colibri_experiment_manifest.json# Formatted and validated Colibri experiment manifest
│   ├── SYSTEM_SPECIFICATIONS.md        # Complete hardware profile and bus bandwidth matrix
│   ├── TESTING_METHODOLOGY.md          # Cold/hot protocols, TTFT formulas, jitter metrics
│   └── RAW_STORAGE_BENCHMARKS.md       # Empirical Direct I/O storage benchmarks across D:, Z:, and F:
├── harness/
│   ├── admin/                          # Elevated scripts (Run as Administrator)
│   │   ├── Setup-DevDrive-Admin.ps1    # Provisions ReFS Dev Drive VHDX on Drive D:
│   │   ├── Purge-StandbyCache-Admin.ps1# Clears Windows standby list & cache between cold runs
│   │   ├── Configure-DefenderExclusion-Admin.ps1 # Toggles NTFS folder exclusions
│   │   └── Expand-DevDrive-Admin.ps1   # Expands VHDX partition
│   ├── Run-BenchmarkMatrix.ps1         # Automated non-elevated orchestrator (Tiers 1-5)
│   ├── Run-NVMe-OptionA-Laguna.ps1     # Working set memory clamp runner (Laguna-XS 33B)
│   ├── Run-NVMe-OptionB-Colibri.ps1    # Uncached MoE streaming runner (DeepSeek-V4.1-Flash)
│   ├── Run-StorageBenchmarks.ps1       # Storage benchmark runner for Disk 3
│   ├── Measure-ServerEndpoint.py       # High-precision SSE streaming benchmark client
│   ├── Stage-Models.ps1                # Model staging utility
│   └── prompts/                        # Standardized Agentic, Coding, and Creative prompt suites
├── data/
│   ├── raw/                            # 87 raw JSON streaming telemetry records
│   └── processed/                      # Aggregated CSVs and summary tables
│       ├── summary_metrics.csv
│       └── summary_table.md
└── analysis/
    ├── compute_statistics.py           # Calculates medians, p90/p99 jitter, and percentage deltas
    └── generate_charts.py              # Visualizations generator

---

4. Documentation & Hand-off Links

• 📖 Comprehensive Benchmark Report & Writer Dossier: docs/BENCHMARK_REPORT.md
• 🧪 Upstream Colibri Research Note: docs/COLIBRI_RESEARCH_NOTE.md
• 📊 Processed Summary Table: data/processed/summary_table.md