test,tools: update wpt weekly #50567
Open
+197
−17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To keep the Web API behavior synchronized with the latest Web specs and WPT tests, add a GitHub Action to automatically update WPT fixtures with WPT weekly epoch. The status files are updated with the test results of the main branch as well.
|
Review requested:
|
nodejs-github-bot
added
build
panva
requested changes
Nov 6, 2023
There was a problem hiding this comment.
I am -1 on automated WPT fixture updates. They change the status json expectations without notice to the subsystem maintainers, making tests skipped and lowering our chance of finding regressions.
This could get bundled with automated issue creation for newly skipped tests but I think that would create too much noise resulting in unresolved and ignored bot issues anyway.
What about, instead of this, have the daily WPT job do something similar to what's done in nodejs/reliability and create / update a single issue with a list of unexpected passes & unexpected failures, that's effectively a link to this query
|
+1 to @panva's suggestion |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To keep the Web API behavior synchronized with the latest Web specs and
WPT tests, add a GitHub Action to automatically update WPT fixtures with
WPT weekly epoch. The status files are updated with the test results of
the main branch as well.
Example of automatic PR: legendecas#3