test: make rendering testable from master - #2212
Open
jcelerier wants to merge 25 commits into
Open
Conversation
jcelerier
force-pushed
the
test/infra
branch
4 times, most recently
from
August 16, 2026 18:43
390717e to
458f2b6
Compare
GCC 16.2 segfaults emitting CodeView type records for any boost::container
container, and this codebase uses those in around thirty translation units, so
the flag makes a gcc Debug or RelWithDebInfo build impossible to finish. A
two-line file reproduces it:
#include <boost/container/vector.hpp>
void f() { boost::container::vector<int> v; (void)v; }
REPRO.cpp:2:54: internal compiler error: Segmentation fault
The flag on its own is enough; DWARF is fine, and so is clang, which is the
compiler we actually want CodeView from on Windows. gcc keeps its normal DWARF
debug info.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
Those two objects now live in ossia/score-addon-synthimi (18ec263). Master already dropped the CMake half that built them out of the avendish tree, so all that is left here is to carry the submodule to the commit where the sources themselves go away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--script took the raw option value and evaluated it, so passing a file path
evaluated the path itself as a program, and the result of evaluate() was
discarded, so the syntax error that produced went nowhere. Any throw in a script
behaved the same way. Combined with readFile returning an empty string for a
path that does not resolve -- and eval("") being a no-op -- a script whose
include path was wrong ran to completion and reported success.
That is how a wrong <LIBRARY>: prefix broke 122 tester scripts while every run
exited 0. A harness cannot tell a pass from a silent failure, which makes every
green result unfalsifiable.
A path is now read as a file, an unreadable one exits 2, an error in the script
is printed with its name and line and exits 3, and readFile says which path it
could not read -- showing the resolved form when a prefix changed it.
locateFilePath gains an overload for callers with no document: a --script file
is read while the application is still starting up, and <LIBRARY>: is a settings
lookup that does not need one.
WindowDevice::grabTo and the offscreen device were written for the scene rework and only exist above pland/scene, so no pixel test can run against the 47 commits below it -- which is exactly where the pixel-format and caching work lives, and where a rendering regression would be hardest to find later. Nothing here depends on that rework: BackgroundNode already exposes shared_readback on master, and grabTo just reads it. The window path reuses the existing window() accessor rather than reaching through the device. grabTo now says why it produced nothing -- not rendered yet, short readback, unwritable path -- because a silently missing PNG is indistinguishable from a test that did not run. Also brings in the golden-render harness, which is scripts only; it reports NORENDER until something drives a render.
Rendering is driven by wall-clock timers, so "how many frames have been drawn" depends on how long the caller waited and how fast the machine is. The tester harness sleeps five seconds and then grabs, which means a slow machine grabs a different frame than a fast one, and an animated shader gives a different image every run. References cannot be stored against that. renderFrames(n) steps the render side exactly n times and returns, so frame N is the same frame everywhere. grabFrame(n, path) is the pair of it, replacing sleep-then-grab. This covers the render side only. Whatever a node derives from execution time still comes from the execution clock, so a graph that depends on it is only as reproducible as that clock -- which is what frame-determinism.sh measures: same case, same frame, two processes, byte-identical PNGs or it has not been achieved.
With neither DISPLAY nor WAYLAND_DISPLAY set, QT_QPA_PLATFORM was overwritten with eglfs unconditionally -- including when the caller had explicitly asked for something else. has_platform was computed for this and only consulted in the xcb branch further down, so a headless machine could not select offscreen: eglfs found no display device, and Qt then qFatal()s on the first window for having no screens. frame-determinism.sh also now rejects a frame that is a single flat colour. Two blank frames are byte-identical, so without that check the easiest way to pass a determinism test is to render nothing.
staticVulkanInstance() could hand back an instance whose create() had failed; return null instead so callers fall back cleanly rather than crashing. Moved down from split/gfx-fixes-and-formats (35eb25e, itself folding 6b0b345 and 25d183b) so the Vulkan hardening sits once, at the bottom of the stack, where headless configurations reach it. #2160 keeps the identical patch; rebasing it onto this branch drops the duplicate by patch-id. Note this covers create() failing, not the probe above it: supportedApiVersion() is called eighteen lines earlier and still crashes under a platform with no Vulkan support. That is a separate defect, unfixed here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
runningUnderAnUISession() decides whether score builds a QGuiApplication or a QCoreApplication. It already recognises eglfs, vkkhrdisplay, linuxfb and vnc as platforms that render without DISPLAY -- but not offscreen, which is equally real: it draws and reads back, it just never shows anything. So on a machine with no DISPLAY and XDG_SESSION_TYPE=tty, QT_QPA_PLATFORM= offscreen fell through to QCoreApplication. QGuiApplicationPrivate:: platform_integration is then never assigned, and the first gfx render segfaults in QVulkanInstancePrivate::ensureVulkan(), which loads that global and dereferences it with no check. Confirmed under gdb: the faulting instruction is `mov (%rdi),%rax` with rdi = platform_integration = 0, while QCoreApplication::self is non-null -- so the application object exists and the windowing layer does not. The crash presents three layers below the cause, and every guard nearer the crash would have hidden it by disabling GPU rendering under offscreen instead.
When the platform plugin cannot provide a Vulkan instance (offscreen QPA, headless CI), score::gfx::staticVulkanInstance() returns nullptr and the Vulkan branch handed it straight to QRhi::create, which dereferences it in QVulkanInstance::isValid() and segfaults. Bail out to the null-rhi state instead, like the other unavailable-backend paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
…ails (cherry picked from commit ee1a189)
Score.readFile has resolved <LIBRARY>: and <PROJECT>: since the file-path helper landed, but createProcess hands its third argument straight to the process factory, and the processes that take a file open it themselves without knowing about the prefixes. So a script that builds a shader process from "<LIBRARY>:/packages/.../x.fs" got a process whose Fragment was empty, and the only symptom was an "ISF error: Missing start comment" further down the log -- the parser had been handed nothing. Every document the csf-testers corpus generated was empty this way: one process, no shader, and a Window outlet addressed at a device that therefore had nothing to render. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
SCORE_FORCE_OFFSCREEN_WINDOW=Name1,Name2 sends a matching WindowDevice down the headless offscreen path whatever its Single/Background/MultiWindow mode, so grabTo reads the render target back instead of grabbing a platform window. The offscreen device has been in the tree since headless capture landed, but nothing selected it, which left every grab on the window branch. That branch calls QScreen::grabWindow, which reads the framebuffer at the window's geometry rather than the window's own buffer: it captures whatever is on top. Measured here, two runs of the same tester at the same frame returned 1280x720 images of the screensaver. It now says so in the log, because a screenshot of a quiet desktop is byte-stable and will otherwise pass a comparison against a stored reference. Also split the empty-readback warning in two. BackgroundNode::render clears the readback when its render list holds only the output itself, which surfaced as "readback is 0 bytes for -1 x -1" -- a default-constructed QSize, not a size anything computed. That case means nothing is connected to the device, and now reports it. Recovered from 52aa6e6 on the pre-reorganisation split/gpu-interop line; the WindowDevice half of that commit never reached master or the current stack. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
renderFrames() drew N frames but never touched time, so every time-ish uniform kept whatever the transport last delivered. Frame N was reproducible only for a shader that ignores time; anything animated gave a different picture each run, which is the thing a stored reference cannot tolerate. Each step now hands every process node a token of frame / stepRate(), after updateGraph so it wins over the date the transport last sent. TIME, TIMEDELTA and PROGRESS follow from it, the same stepping the shader sweeps have used all along. The rest of the audit the reference images depend on: FRAMEINDEX is already incremented by the renderers per render, so it counts renders rather than seconds, and ProcessUBO::date is never written anywhere in the plugin, so it stays at its zero initialiser. No wall clock is left on this path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The gate passed twice on screenshots of my desktop. It checked that the frame was not a single flat colour, which a desktop is not, and that two runs matched, which two screenshots of a quiet desktop do. Four changes, each closing one way it could pass without rendering: - SCORE_FORCE_OFFSCREEN_WINDOW=Window, so the grab reads the render target. - fail on the "capturing the SCREEN" warning, so the window fallback can never quietly supply the reference again. - fail on "nothing rendered into", which is what an unwired Window device says. - Score.play() in the generated script: the gfx nodes only exist while the score executes, so a grab on a stopped document has nothing to read. The default case is now isf-time-uniforms rather than isf-solid-color. It draws TIME, TIMEDELTA, PROGRESS and FRAMEINDEX as bars, so it fails if any of them still follows a wall clock; a static case passes whether or not the step clock works, which is exactly the reassurance that was wrong before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
Qt.exit(code) already shut the app down cleanly -- the handler marks the documents saved first, which is what keeps a modified document from popping a save prompt on the way out -- but it then called quit(), i.e. exit(0), and threw the code away. A script could stop the app and could not say whether the run had succeeded. Harnesses had to infer the verdict from side effects, or wrap every run in a timeout and read the kill as normal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
setStepRate(fps) on the Window device, next to renderFrames, so a harness can step at the rate the material wants instead of the built-in 60. renderFrames already supported frame-by-frame use -- the step counter carries across calls, so renderFrames(1) in a loop walks the same timeline renderFrames(60) does. That is the more useful shape for a harness: it can grab several frames, or read and set controls between them, without paying process startup for each one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The runs ended by being killed: --wait defers autoplay so that slow media have time to load, it is not an exit timer, and nothing ever asked the app to quit. Each of the two runs therefore sat until the 120s guard fired. Qt.exit(0) at the end of the generated script ends the run when the work is done. The pair now takes 2.0s rather than about 240s, the timeout goes back to being a hang guard, and a nonzero exit is a real failure rather than the expected outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
One process per case, so a case that crashes or hangs takes only itself down, and each tester builds its document from a clean application either way. Verdicts are separated by what actually happened rather than lumped into pass/fail: RENDER, BLANK, NORENDER, SCREEN, FAIL. SCREEN exists because a grab that falls back to capturing the desktop is not a weaker result, it is a meaningless one, and the sweep exits nonzero if any case produced one. First run, 130 cases: 81 render on the stack against 60 on master, the difference being shaders whose ISF features only exist above #2120 -- the measured form of the claim that no pixel test can run below it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
At 450 the draw-parameter builtins only exist under their ARB names -- gl_BaseInstanceARB and friends -- because GL_ARB_shader_draw_parameters spells them that way; the unsuffixed gl_BaseInstance is core from 460. A shader author writing the name that appears in every reference gets "undeclared identifier", and since a shader that fails to compile yields a process with no ports, the error surfaces much later as a null port. Validated on both Qt versions we build against, whole corpus of 130 testers: - Qt 6.4.2: RENDER 64 BLANK 32 NORENDER 34 CRASH 0 - Qt 6.12.0: RENDER 64 BLANK 32 NORENDER 34 CRASH 0 byte-identical, and no case changed verdict against the 450 build. The frame determinism gate still passes. On the gfx stack, where the shaders that use these builtins actually compile, scene-mdi renders with the unsuffixed spelling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
One process per case already contains a crash -- the sweep carries on and only that case is lost -- but the summary counted it with ordinary script failures, so a SCORE_ASSERT firing in render-target creation looked like a bad tester. A signal now reports as CRASH with the signal number and the first file:line in the trace, which is enough to tell two different aborts apart in the table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
|
Coverage is 19.411% — test/infra into master. No base build found for |
Catch2 exits with 4 when every case in a binary was skipped. ctest did not know that, so a test that skipped because its precondition is absent -- no display, no shader library, no capture device -- was counted as a failure. The three shader sweeps have been reported as failing on every commit of this branch and every commit above it while in fact never running once, which is worse than a silent skip: it hides the gap behind a number that looks like a known problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
score::PluginLoader::pluginsDir() probes <appdir>/plugins, <appdir>/../lib/score, /usr/lib/score and <cwd>/plugins. Test executables live in <build>/tests/..., so on a machine with no installed score the only path that can match is <cwd>/plugins: ctest gets this right through WORKING_DIRECTORY, but the same binary run by hand from anywhere else boots an application with no plug-ins at all and then aborts somewhere unrelated -- settings<T>() on a model whose plug-in never registered. Pass the build root through to the fixture and chdir when ./plugins is absent, so a hand-run behaves like the ctest run it is meant to reproduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
closeDocument() raised a modal QMessageBox on a modified document. With applicationSettings.gui false -- headless, --script, offscreen QPA -- nothing can answer it and QMessageBox::exec() aborts, so a scripted /exit on a document that had been edited died in teardown with SIGABRT instead of exiting 0. Every score::MessageBox helper already guards on that flag; this was the one raw QMessageBox left on the exit path. No GUI means no one to save for, so proceed as Discard -- which is the outcome forceExit() already produced anyway, since it quits 500 ms later whatever the answer would have been. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
settings<T>() aborts when no plug-in registered that model, which is right for application code -- the plug-in is missing and nothing downstream can work. A test harness would rather say so itself, with a message naming the cause. findSettings<T>() returns the pointer or null; settings<T>() now delegates to it and is otherwise unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sweep took its graphics API from ctx.settings<Gfx::Settings::Model>(), which aborts the process when the gfx plug-in registered no settings model -- which is what happens when the binary is run from a directory without ./plugins. Use findSettings and FAIL with a message naming the cause instead of dying in a settings lookup. SCORE_SHADER_SWEEP_DUMP_DIR writes each frame out, so that "it rendered" can be checked against what the shader is supposed to draw rather than taken on faith. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork so that rendering can be tested at all, on
master, ahead of the gfx PR stack.Today no pixel test can run below #2120:
WindowDevice::grabToand thegolden-render/harness enter atpland/scene,OffscreenDeviceatplanc/gfx-infra. That makes 47 commits of the stack build-only, and a rendering regression in #2160 or #2119 impossible to bisect. This PR puts the minimum on master.Silent failures fixed — each of these turned a broken run into a green one:
js: report --script failures instead of exiting 0— a script that threw ran to completion and exited 0. Every result it produced looked like a success. This is what hid the three bugs below for a full session.app: keep the platform the caller asked for when there is no display—main.cppoverwrote an explicitQT_QPA_PLATFORMwitheglfswhenever no X11/Wayland display was found.app: treat offscreen as a rendering session, like eglfs and vkkhrdisplay—runningUnderAnUISession()did not listoffscreen, so--no-guiunder it took the no-UI path.--no-guimeans "do not open score's main widget UI", not "do not render".Vulkan under a null platform integration —
QVulkanInstancePrivate::ensureVulkan()dereferencesQGuiApplicationPrivate::platform_integrationwith no null check, so building a Vulkan QRhi before/without a platform integration segfaults (confirmed under gdb:mov (%rdi),%rax,rdi=0).gfx: do not return a Vulkan instance when creation failedgfx: don't create a Vulkan QRhi without an instancegfx: don't return a half-initialized render state when QRhi::create failsTest entry points
gfx: make headless frame capture available from master—grabTo/grabFrameon the Window device.gfx: render a fixed number of frames on demand—GfxContext::renderFrames(int), the hook a deterministic frame clock needs.Known gap, tracked separately: the window path of
grabTousesscreen()->grabWindow(), which on X11 reads the framebuffer at the window's geometry rather than the window's own buffer — it captures whatever is on top. Golden images must come from the offscreen readback instead. That is whyOffscreenDeviceis here.🤖 Generated with Claude Code
https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE