Repository navigation
Benchmarks: REA on Linux (Node 22, 24 and 26 startup, plus five local models) #1998
Replies: 2 comments
|
A side benchmark: REA's CLI feeding free local models. A script ran three REA calls on a small test program of my own. Then each model read the same output and answered the same three questions. More on the setup is in my note on #1958. The models
They run on llama.cpp (MIT) behind llama-swap (MIT), one at a time, on a Radeon RX 6700 XT with 12 GB. REA calls (Ghidra 12.1.4)
One run each on a quiet box, all exit 0, so the gap is within noise. In 6.4.0, How well each model read it3 asks per model, 3 questions each. The secret word is not in the file, so "partly" is the best possible answer.
Speed, memory and power
The first ask reads the whole prompt, about 2,800 tokens. The next two reuse the cache. Load times may be low, because the files were likely still cached. The first KAT-Coder run, on a busy box
The load average counts programs running or waiting for a CPU, and this box has 24 threads. My code editor's file search had run away during the busy run. Same model, same prompt, so the busy CPU was the cause. Not run yet
Edit: on 12 October I reran on a quiet box and added four more models. The busy-box numbers are in their own table. 🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊 |
|
TL;DR: I reran the startup timer on rea-agents 6.5.0 and 6.4.0. In the same session, 6.5.0 reaches the tool list 0.05 to 0.10 s sooner than 6.4.0. It also lists 207 tools, not 170. Seconds from process start, median of 5 runs. "Tool list" is the reply to the tool list request. "Init" is the first reply. Both packages ran on the same box, one after the other, in two passes. The second pass ran in reverse order.
6.5.0 against 6.4.0, tool list, first pass. The reverse pass gave 0.07, 0.07 and 0.10 s.
Saved and Faster by come from the exact times, so they can differ by a hair from the rounded cells. Earlier numbers, tool list median in seconds:
6.4.0 ran 0.34 to 0.50 s faster today than in the first post. I do not know why. So compare only rows from the same session. 6.5.0 is 890 commits past 13e03e9, so the main column is not a like-for-like step. The load average counts programs running or waiting for a CPU, over 1 minute. This box has 24 threads. Another test job ran at the same time. The load was 1.38 to 1.94 when each row started. Still not run
Raw numbersSeconds. Each cell is the median, with the fastest and slowest run in brackets.
Init, 6.5.0 against 6.4.0, first pass: 0.22 s sooner on Node 22, 0.19 s on Node 24, 0.18 s on Node 26. The reverse pass gave 0.22, 0.21 and 0.21 s. The gap between the init reply and the tool list reply grew. It was 0.19 to 0.25 s on 6.4.0 and 0.31 to 0.40 s on 6.5.0. I did not look into why. I threw away one earlier attempt. A stray loop of my own ran on the box during it, so its load was wrong. It is not in these numbers. CommandsInstall each target in its own folder, then run the timer from the first post. One change: I added a temp folder (TMPDIR) to the server's environment. A small runner script waited for a 1-minute load of 2.0 or less, ran the timer once per row, and wrote one CSV line per row. Test setup
🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
In plain words: when an AI app starts REA, it waits until REA sends its list of tools. Only then can the AI use REA. This page times that wait on Linux, for each REA release and each Node version. Smaller numbers are better. Newer Node versions start REA faster. 6.5.0 update (14 October): it is a little faster again, by about 5%, even though it now lists 207 tools. The numbers are in the comments.
Speed at a glance. Seconds until REA's tools are ready for the AI. Lower is better. Both versions were timed on the same PC on 14 October.
Newest Node plus newest REA is the fastest pair.
Release to release. 6.4.0 was a bit slower than 6.3.0, by about 10 to 14%. It added 31 tools, from 139 to 170. 6.5.0 is a bit faster than 6.4.0, by about 4 to 6%. That is with 37 more tools, now 207. Each pair was timed on the same day, so the two jumps are fair. Different days can shift all the numbers a little.
How fast REA's server starts on Linux, on Node 22, 24 and 26. I update it each release. A local model side test is in the comments.
It started as a comment on #1435 and moved here on 11 October, so it has its own page. Last updated for 6.4.0 and main at 13e03e9.
Startup: time to the tool list
Seconds, from process start. "Tool list" is the median of 5 runs, with the fastest and slowest run in brackets. "Init" is the median time to the first reply.
Main (13e03e9) against 6.4.0, time to the tool list (median):
Saved and Faster by come from the exact times, so they can differ by a hair from the rounded cells.
First real job (Node 24)
Main saves 0.18 s at the tool list. The browser capture then takes 0.21 s longer, so the saving does not survive the first capture. The other three jobs keep it.
Seconds from process start to the job's reply, median of 5 runs. The job alone is in brackets (job reply minus tool list reply).
Browser capture, 6.4.0 against main:
The Change column comes from the exact times.
What changed between releases
Newest first. Time to the tool list, median.
Not run
Startup timing (the first table)
First real job (Node 24 only)
The four commits main has that 6.4.0 lacks
#1691 says it should make MCP startup faster. Its own numbers:
client.connectwent from about 490 ms to about 400 ms. My numbers fit that claim, but I did not time it alone.More on the first real job
Other test jobs ran on this box while I timed. The load average was 3.0 to 7.4 in the main pass and 0.8 to 4.8 in the check pass. The load average counts programs running or waiting for a CPU, over 1 minute. The box has 24 threads. The tool list times here are lower than in the first table (6.4.0 was 1.75 s there, 1.38 s here). The earlier timer gives 1.36 s for 6.4.0 and 1.17 s for main on this box now, at a load of 0.3 to 0.5. So the two timers agree, and the box ran faster that evening. I do not know why, and I did not test it. Main saves 0.18 s here, not 0.28 s. Every row has 5 timed runs, all ok except the failed 6.3.0 scan.
How to rerun
Install the target, then run the timer. The timer starts the server and sends the three requests. It does 1 warm-up and 5 timed runs. It prints one line of numbers in ms: init median, min and max, then tool list median, min and max. After those come the tool count and the Node version.
For main, build a plain copy of the commit first. I used Node 24 for the build.
My wrapper scripts check the load before each row and add one row to a CSV file. They start only if the load is 2.0 or less. The load average counts programs that are running or waiting for a CPU, averaged over 1 minute. This box has 24 threads.
The timer, mcp-time.mjs:
Inputs and rerun
Each run starts the server, sends the first two requests, then makes the job's call. The server is closed after each run. Each run gets a new empty home and temp folder. Each row has 1 warm-up run and 5 timed runs. The Ghidra row took about 8 s per run, so 5 was fine.
python3 -m http.serveron 127.0.0.1 serves it. The call iscapture_browser_scenario. It opens the page and clicks the button. The browser is headless Chromium from Fedora. The license is BSD-3-Clause, among others. The version is 154.0.8037.57. No sandbox flag was needed.analyze_javascript_applicationon a folder with one file. The file isapp/main.jsfrom Stretchly 1.22.1 (BSD-2-Clause). I setREA_MCP_MAX_RESPONSE_BYTES=268435456on all targets.open_binary, theninspect_artifact, on Coffee 2.27 (Apache-2.0). The tool needs an open app, so the time runs through the second reply.open_binary, thenbinary_overview, on an ELF built from my own 3-line C file withcc -O0 -g0. Ghidra 12.1.4 (Apache-2.0) ran on Eclipse Temurin JDK 21.0.12.1+1. Temurin isGPL-2.0-only WITH Classpath-exception-2.0.Inputs, checked with
sha256sum -cbefore each run:To rerun all 12 rows (about 5 minutes):
Test setup
sha512-1l3L22PYqlzW.sha512-Vj1zWfSlna5b.🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊
All reactions