ci: make inference test matrix test-owned

[fl0rianr]

Ideally we only test what is required for the code change to safe resources.

Current state:

We have a mid sized test suite and many additional tests maintainers have to activate.
It happend regular that such an activation is missed and the PR fails inside the merge queue.

Target:

Even smaller test suite for some PRs, not no less test for some other.

Proposed Approach:

• each test suite describes which part is tested e.g. llama.cpp and for what e.g. CPU in a way machines can read it
• we should differentiate direct tested and dependency
• those meta-data become source of Truth for the test coverage (not current workflow YAML which does not adapt when a test changes etc.)
• PR ci: make inference test matrix test-owned #3022 does this for Inference-Backends with direct and consumes
• form changes, CI can automatically determine which test are affected
• we also see automatically if a packend or component has no test at all
• CI shows clearly why a test was chosen or skipped
• safe fall back if no detection was possible or fallback to current state
• We can start with backends only and expand over router, MCP, installer, server, GUI etc.

@jeremyfowers I thought we would agree on this, maybe an misunderstanding.

[jeremyfowers]

It happend regular that such an activation is missed and the PR fails inside the merge queue.

How common is this today, especially now that reviewers are more familiar with the test activation system? I want to ensure we are solving a real problem before we invest energy in validating a change that makes the CI system significantly more complex.

[fl0rianr]

Don't really know, happened to me maybe 3 times and saw it with other PRs so review also 5 - 6 times maybe.

There is a real benefit beyond that knowing what is tested etc. Also that it can be maintained dezentral with the test changes instead of central like now.

[jeremyfowers]

Let's see how much people upvote this RFC. If it gets a lot of votes let's spend the effort on reviewing and merging the change.

[jeremyfowers]

Here's one more data point: #3539

although this was the distro tests, not the inference tests.