Repository navigation
Remove tests that rely on undefined testing APIs #172
Description
Activity
FWIW:
gsnedders@gsnedders-marsha web-platform-tests % rg -F resources/chromium --files-with-matches | sort battery-status/resources/battery-status-helpers.js bluetooth/README.md bluetooth/resources/bluetooth-test.js compute-pressure/resources/pressure-helpers.js contacts/resources/helpers.js content-index/resources.js generic-sensor/README.md html/semantics/links/downloading-resources/header-origin-no-referrer-when-downgrade.html html/semantics/links/downloading-resources/header-origin-no-referrer.html html/semantics/links/downloading-resources/header-origin-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-origin-origin.html html/semantics/links/downloading-resources/header-origin-same-origin.html html/semantics/links/downloading-resources/header-origin-strict-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-origin-strict-origin.html html/semantics/links/downloading-resources/header-origin-unsafe-url.html html/semantics/links/downloading-resources/header-origin.html html/semantics/links/downloading-resources/header-referrer-no-referrer-when-downgrade.html html/semantics/links/downloading-resources/header-referrer-no-referrer.html html/semantics/links/downloading-resources/header-referrer-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-referrer-origin.html html/semantics/links/downloading-resources/header-referrer-same-origin.html html/semantics/links/downloading-resources/header-referrer-strict-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-referrer-strict-origin.html html/semantics/links/downloading-resources/header-referrer-unsafe-url.html html/semantics/links/downloading-resources/header-referrer.html idle-detection/resources/idle-detection-helper.js lint.ignore managed/resources/managed-configuration-helper.js mediacapture-image/resources/imagecapture-helpers.js orientation-event/README.md orientation-event/resources/orientation-event-helpers.js serial/README.md serial/resources/automation.js shape-detection/README.md shape-detection/resources/shapedetection-helpers.js subapps/resources/subapps-helpers.js web-nfc/README.md web-nfc/resources/nfc-helpers.js webhid/resources/automation.js webusb/README.md webusb/resources/usb-helpers.js webxr/resources/webxr_util.jsI think at least some of those are things where the spec does define the testing API.
But I agree that tests which depend on undefined browser internal APIs aren't a good fit for wpt, and vendors should find a way to run those outside of the shared suite (note that doesn't necessarily mean they can't be written to use the wpt harness and infrastructure; we have gecko-specific tests that may use gecko's internal
SpecialPowersAPI as a separate set of web-platform-tests in mozilla-central, and these are not synced with upstream).I think at least some of those are things where the spec does define the testing API.
Yes, that list of files certainly is superset of what is implicated in this issue (e.g., WebUSB and WebXR do define testing APIs, and are thus fine for WPT), but they are probably what needs to be audited.
Sorry for the delay in responding here, I just saw this ping. I had reached out to the test owners and they agreed to move those tests to wpt_internal in the near term (2024) and create test APIs in the longer term (2025+).
So the list has somewhat changed; some tests have lost their dependency on
resources/chromiumbut some have gained one:% rg -F resources/chromium --files-with-matches | sort battery-status/resources/battery-status-helpers.js bluetooth/README.md bluetooth/resources/bluetooth-test.js contacts/resources/helpers.js content-index/resources.js css/css-view-transitions/navigation/chromium-paint-holding-timeout.html html/semantics/links/downloading-resources/header-origin-no-referrer-when-downgrade.html html/semantics/links/downloading-resources/header-origin-no-referrer.html html/semantics/links/downloading-resources/header-origin-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-origin-origin.html html/semantics/links/downloading-resources/header-origin-same-origin.html html/semantics/links/downloading-resources/header-origin-strict-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-origin-strict-origin.html html/semantics/links/downloading-resources/header-origin-unsafe-url.html html/semantics/links/downloading-resources/header-origin.html html/semantics/links/downloading-resources/header-referrer-no-referrer-when-downgrade.html html/semantics/links/downloading-resources/header-referrer-no-referrer.html html/semantics/links/downloading-resources/header-referrer-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-referrer-origin.html html/semantics/links/downloading-resources/header-referrer-same-origin.html html/semantics/links/downloading-resources/header-referrer-strict-origin-when-cross-origin.html html/semantics/links/downloading-resources/header-referrer-strict-origin.html html/semantics/links/downloading-resources/header-referrer-unsafe-url.html html/semantics/links/downloading-resources/header-referrer.html idle-detection/resources/idle-detection-helper.js lint.ignore managed/resources/managed-configuration-helper.js mediacapture-image/resources/imagecapture-helpers.js shape-detection/README.md shape-detection/resources/shapedetection-helpers.js subapps/resources/subapps-helpers.js web-nfc/README.md web-nfc/resources/nfc-helpers.js webusb/README.md webusb/resources/usb-helpers.js webxr/resources/webxr_util.jsLooking at these and current engine implementations (even off by default, if I've done my research correct):
Directory Chromium Firefox WebKit battery-status ✅ ❌ ❌ bluetooth ✅ ❌ ❌ contacts ✅ ❌ ❌ content-index ✅ ❌ ❌ css/css-view-transitions ✅ ✅ ✅ html/semantics/links/downloading-resources ✅ ❌ ✅ idle-detection ✅ ❌ ❌ managed ✅ ❌ ❌ mediacapture-image ✅ ❌ ✅ shape-detection ✅ ❌ ✅ subapps ❌ ❌ ❌ web-nfc ❌ ❌ ❌ webusb ✅ ❌ ❌ webxr ✅ ✅ ✅ I think it's fair to say those with cross-browser support are the ones that are most important to resolve, but as noted before, starting implementation on something just to find WPT tests that you cannot meaningfully pass is very frustrating. I really would like to see progress here.
The fact the
mediacapture-imagetests, per web-platform-tests/wpt#51350, have a comment saying "These tests rely on the User Agent providing an implementation of platform image capture backends. In Chromium-based browsers this implementation is provided by a polyfill" leads to questions how much those tests are even testing in Chromium.I really want to push back strongly on any of this getting worse, which makes the new
css/css-view-transitionstests incredibly frustrating.Reacted by Luke WarlowMost of this list falls under "Chromium device APIs" (battery-status, bluetooth, idle-detection, shape-detection, web-nfc, webusb) On our side we're trying to get on top of this by bringing up new testing APIs for device simulation.
Web Bluetooth now has an automated testing API and all tests have been migrated to the new API. I think we can remove the remaining references to
resources/chromiumand get it off the list.We're bringing up a testing API for Gamepad which is not on this list, but already has cross-browser support and suffers from poor WPT coverage as there's no cross-browser way to simulate device behavior. w3c/gamepad#224
WPT contains a variety of tests that rely on MojoJS to run in Chromium-based browsers.
Chromium's own WPT docs say:
However, we have many cases where things are "polyfilled" via MojoJS where the spec doesn't define the testing API, and nor has there been any attempt within WPT to document what the testing API the tests expect is. (Furthermore, https://groups.google.com/a/chromium.org/g/blink-dev/search?q=mojojs makes it look like basically nobody has ever followed those steps, as nobody has done the "reach out" step.)
The tests are, thus, practically Chromium-specific, short of someone reverse-engineering what the testing API is.
For example, the Shape Detection tests that were added to WPT in web-platform-tests/wpt@dfc0080 rely on /shape-detection/resources/shapedetection-helpers.js to load testing APIs, except nowhere in https://wicg.github.io/shape-detection-api/ (or any other document) are
FaceDetectionTest,BarcodeDetectionTest, orTextDetectionTestactually defined.Notwithstanding questions of whether any such testing API would ship in the browser, this means wpt.fyi will forever show Safari as failing many of the shape detection tests, even if the feature flag were enabled by default, unless we invest in reverse-engineering Chromium's testing API. Part of the very goal of standards and WPT is to avoid vendors having to invest in reverse-engineering one-another.
In extreme, /serial/README.md even notes:
WPT cannot be a dumping ground for inherently vendor-specific tests, written with little-to-no regard for other vendors using them, as this undermines the very premise of it being a shared test suite. It is unreasonable and unhelpful for Chromium engineers to export tests which other vendors must reverse-engineer the testing API to gain any benefit from. Chromium can maintain these tests in their own, non-exported tree, and do so until such point as they define what the testing APIs actually are.
As such, we should remove all tests that rely on undefined testing APIs from WPT.