From 637bc1e5d69c98d2e152f7d2ab6050b9c309347e Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 15:06:37 +0100 Subject: [PATCH 1/8] Nuke the blacklist files to see what succeeds! --- .ci/worker-blacklist | 6 +++--- sytest-blacklist | 28 ++++++++++++++-------------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/.ci/worker-blacklist b/.ci/worker-blacklist index 5975cb98cfda..589ac2fe132d 100644 --- a/.ci/worker-blacklist +++ b/.ci/worker-blacklist @@ -1,10 +1,10 @@ # This file serves as a blacklist for SyTest tests that we expect will fail in # Synapse when run under worker mode. For more details, see sytest-blacklist. -Can re-join room if re-invited +# Can re-join room if re-invited # new failures as of https://github.com/matrix-org/sytest/pull/732 -Device list doesn't change if remote server is down +# Device list doesn't change if remote server is down # https://buildkite.com/matrix-dot-org/synapse/builds/6134#6f67bf47-e234-474d-80e8-c6e1868b15c5 -Server correctly handles incoming m.device_list_update +# Server correctly handles incoming m.device_list_update diff --git a/sytest-blacklist b/sytest-blacklist index de9986357b9a..cb25e4d171d8 100644 --- a/sytest-blacklist +++ b/sytest-blacklist @@ -10,34 +10,34 @@ # explanation for why the test has been excluded. # Blacklisted due to https://github.com/matrix-org/synapse/issues/1679 -Remote room members also see posted message events +# Remote room members also see posted message events # Blacklisted due to https://github.com/matrix-org/synapse/issues/2065 -Guest users can accept invites to private rooms over federation +# Guest users can accept invites to private rooms over federation # Blacklisted due to https://github.com/vector-im/riot-web/issues/7211 -The only membership state included in a gapped incremental sync is for senders in the timeline +# The only membership state included in a gapped incremental sync is for senders in the timeline # Blacklisted due to https://github.com/matrix-org/synapse/issues/1658 -Newly created users see their own presence in /initialSync (SYT-34) +# Newly created users see their own presence in /initialSync (SYT-34) # Blacklisted due to https://github.com/matrix-org/synapse/issues/1396 -Should reject keys claiming to belong to a different user +# Should reject keys claiming to belong to a different user # Blacklisted due to https://github.com/matrix-org/synapse/issues/1531 -Enabling an unknown default rule fails with 404 +# Enabling an unknown default rule fails with 404 # Blacklisted due to https://github.com/matrix-org/synapse/issues/1663 -New federated private chats get full presence information (SYN-115) +# New federated private chats get full presence information (SYN-115) # Blacklisted due to https://github.com/matrix-org/matrix-doc/pull/2314 removing # this requirement from the spec -Inbound federation of state requires event_id as a mandatory paramater +# Inbound federation of state requires event_id as a mandatory paramater # Blacklisted until MSC2753 is implemented -Local users can peek into world_readable rooms by room ID -We can't peek into rooms with shared history_visibility -We can't peek into rooms with invited history_visibility -We can't peek into rooms with joined history_visibility -Local users can peek by room alias -Peeked rooms only turn up in the sync for the device who peeked them +# Local users can peek into world_readable rooms by room ID +# We can't peek into rooms with shared history_visibility +# We can't peek into rooms with invited history_visibility +# We can't peek into rooms with joined history_visibility +# Local users can peek by room alias +# Peeked rooms only turn up in the sync for the device who peeked them From d196f9f156118a5b04dc2397246810b2c0317cd6 Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 15:12:38 +0100 Subject: [PATCH 2/8] Changelog --- changelog.d/10788.misc | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/10788.misc diff --git a/changelog.d/10788.misc b/changelog.d/10788.misc new file mode 100644 index 000000000000..a688b54b11ba --- /dev/null +++ b/changelog.d/10788.misc @@ -0,0 +1 @@ +Remove blacklisted tests whose underlying issues have been fixed. \ No newline at end of file From 2b3fba5c9e77410da23295c3671185967efcc224 Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 15:45:01 +0100 Subject: [PATCH 3/8] I think these things all need blacklisting --- sytest-blacklist | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/sytest-blacklist b/sytest-blacklist index cb25e4d171d8..7a513d426b96 100644 --- a/sytest-blacklist +++ b/sytest-blacklist @@ -10,34 +10,34 @@ # explanation for why the test has been excluded. # Blacklisted due to https://github.com/matrix-org/synapse/issues/1679 - # Remote room members also see posted message events + # Blacklisted due to https://github.com/matrix-org/synapse/issues/2065 +Guest users can accept invites to private rooms over federation -# Guest users can accept invites to private rooms over federation # Blacklisted due to https://github.com/vector-im/riot-web/issues/7211 +The only membership state included in a gapped incremental sync is for senders in the timeline -# The only membership state included in a gapped incremental sync is for senders in the timeline # Blacklisted due to https://github.com/matrix-org/synapse/issues/1658 +Newly created users see their own presence in /initialSync (SYT-34) -# Newly created users see their own presence in /initialSync (SYT-34) # Blacklisted due to https://github.com/matrix-org/synapse/issues/1396 +Should reject keys claiming to belong to a different user -# Should reject keys claiming to belong to a different user # Blacklisted due to https://github.com/matrix-org/synapse/issues/1531 - # Enabling an unknown default rule fails with 404 -# Blacklisted due to https://github.com/matrix-org/synapse/issues/1663 +# Blacklisted due to https://github.com/matrix-org/synapse/issues/1663 # New federated private chats get full presence information (SYN-115) + # Blacklisted due to https://github.com/matrix-org/matrix-doc/pull/2314 removing # this requirement from the spec +Inbound federation of state requires event_id as a mandatory paramater -# Inbound federation of state requires event_id as a mandatory paramater # Blacklisted until MSC2753 is implemented -# Local users can peek into world_readable rooms by room ID -# We can't peek into rooms with shared history_visibility -# We can't peek into rooms with invited history_visibility -# We can't peek into rooms with joined history_visibility -# Local users can peek by room alias -# Peeked rooms only turn up in the sync for the device who peeked them +Local users can peek into world_readable rooms by room ID +We can't peek into rooms with shared history_visibility +We can't peek into rooms with invited history_visibility +We can't peek into rooms with joined history_visibility +Local users can peek by room alias +Peeked rooms only turn up in the sync for the device who peeked them From 922ece92990f359fbc616b9348b8baf6931db20c Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 16:42:40 +0100 Subject: [PATCH 4/8] Commit to unblacklisting test for fixed issue --- sytest-blacklist | 3 --- 1 file changed, 3 deletions(-) diff --git a/sytest-blacklist b/sytest-blacklist index 7a513d426b96..8b71ba49b31d 100644 --- a/sytest-blacklist +++ b/sytest-blacklist @@ -24,9 +24,6 @@ Newly created users see their own presence in /initialSync (SYT-34) # Blacklisted due to https://github.com/matrix-org/synapse/issues/1396 Should reject keys claiming to belong to a different user -# Blacklisted due to https://github.com/matrix-org/synapse/issues/1531 -# Enabling an unknown default rule fails with 404 - # Blacklisted due to https://github.com/matrix-org/synapse/issues/1663 # New federated private chats get full presence information (SYN-115) From a62393e2d4577080663214233d46901e9a6e76c9 Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 17:13:04 +0100 Subject: [PATCH 5/8] Re-blacklist something that's known to be flakey. --- .ci/worker-blacklist | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.ci/worker-blacklist b/.ci/worker-blacklist index 589ac2fe132d..6d99b387fddf 100644 --- a/.ci/worker-blacklist +++ b/.ci/worker-blacklist @@ -4,7 +4,8 @@ # Can re-join room if re-invited # new failures as of https://github.com/matrix-org/sytest/pull/732 -# Device list doesn't change if remote server is down +# Flakey: doesn't consistently fail. +Device list doesn't change if remote server is down # https://buildkite.com/matrix-dot-org/synapse/builds/6134#6f67bf47-e234-474d-80e8-c6e1868b15c5 # Server correctly handles incoming m.device_list_update From c048d641f128b5f6c1843039bf7266c296ac5d32 Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 17:38:30 +0100 Subject: [PATCH 6/8] Revert "Re-blacklist something that's known to be flakey." Blacklist only known, consistent failures. Deflake where necessary. This reverts commit a62393e2d4577080663214233d46901e9a6e76c9. --- .ci/worker-blacklist | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.ci/worker-blacklist b/.ci/worker-blacklist index 6d99b387fddf..589ac2fe132d 100644 --- a/.ci/worker-blacklist +++ b/.ci/worker-blacklist @@ -4,8 +4,7 @@ # Can re-join room if re-invited # new failures as of https://github.com/matrix-org/sytest/pull/732 -# Flakey: doesn't consistently fail. -Device list doesn't change if remote server is down +# Device list doesn't change if remote server is down # https://buildkite.com/matrix-dot-org/synapse/builds/6134#6f67bf47-e234-474d-80e8-c6e1868b15c5 # Server correctly handles incoming m.device_list_update From bb5a60dbde06e9a5333b567c8d4af97f334c9409 Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 17:41:52 +0100 Subject: [PATCH 7/8] Unblacklist tests that aren't consistently failing This commit removes five additional tests. They were all commented out of the blacklist in the successful run below. So they're either fixed or flakey. Either way, we don't want them in the blacklist. https://github.com/matrix-org/synapse/runs/3557620105 --- .ci/worker-blacklist | 8 -------- sytest-blacklist | 8 +------- 2 files changed, 1 insertion(+), 15 deletions(-) diff --git a/.ci/worker-blacklist b/.ci/worker-blacklist index 589ac2fe132d..cb8eae5d2a34 100644 --- a/.ci/worker-blacklist +++ b/.ci/worker-blacklist @@ -1,10 +1,2 @@ # This file serves as a blacklist for SyTest tests that we expect will fail in # Synapse when run under worker mode. For more details, see sytest-blacklist. - -# Can re-join room if re-invited - -# new failures as of https://github.com/matrix-org/sytest/pull/732 -# Device list doesn't change if remote server is down - -# https://buildkite.com/matrix-dot-org/synapse/builds/6134#6f67bf47-e234-474d-80e8-c6e1868b15c5 -# Server correctly handles incoming m.device_list_update diff --git a/sytest-blacklist b/sytest-blacklist index 8b71ba49b31d..65bf1774e353 100644 --- a/sytest-blacklist +++ b/sytest-blacklist @@ -1,5 +1,5 @@ # This file serves as a blacklist for SyTest tests that we expect will fail in -# Synapse. +# Synapse. This doesn't include flakey tests---better to deflake them instead. # # Each line of this file is scanned by sytest during a run and if the line # exactly matches the name of a test, it will be marked as "expected fail", @@ -9,9 +9,6 @@ # Test names are encouraged to have a bug accompanied with them, serving as an # explanation for why the test has been excluded. -# Blacklisted due to https://github.com/matrix-org/synapse/issues/1679 -# Remote room members also see posted message events - # Blacklisted due to https://github.com/matrix-org/synapse/issues/2065 Guest users can accept invites to private rooms over federation @@ -24,9 +21,6 @@ Newly created users see their own presence in /initialSync (SYT-34) # Blacklisted due to https://github.com/matrix-org/synapse/issues/1396 Should reject keys claiming to belong to a different user -# Blacklisted due to https://github.com/matrix-org/synapse/issues/1663 -# New federated private chats get full presence information (SYN-115) - # Blacklisted due to https://github.com/matrix-org/matrix-doc/pull/2314 removing # this requirement from the spec Inbound federation of state requires event_id as a mandatory paramater From 17fade835d502d311592381ad766c9fed11b5ef6 Mon Sep 17 00:00:00 2001 From: David Robertson Date: Thu, 9 Sep 2021 17:46:01 +0100 Subject: [PATCH 8/8] Update changelog --- changelog.d/10788.misc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/10788.misc b/changelog.d/10788.misc index a688b54b11ba..568a85ac52da 100644 --- a/changelog.d/10788.misc +++ b/changelog.d/10788.misc @@ -1 +1 @@ -Remove blacklisted tests whose underlying issues have been fixed. \ No newline at end of file +Remove fixed and flakey tests from the sytest-blacklist.