Conformance display profiles on gatewayapi docs #2874

xtineskim · 2024-03-14T19:04:20Z

What type of PR is this?
/kind documentation

What this PR does / why we need it:
Related to #2550. tldr, we lack a way to quickly show different implementations of the Gateway API and their features. After trying various plugins for mkdocs, the most convenient was a markdown table that could be sorted based on columns. Not the most ideal experience, but it's a start for showing conformance profile reports on the gateway api site
Which issue(s) this PR fixes:
#2550 (although this will need more iteration in the future.It's a start)

Fixes #

Does this PR introduce a user-facing change?:

NONE

k8s-ci-robot · 2024-03-14T19:04:29Z

Hi @xtineskim. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

shaneutt · 2024-03-14T19:55:23Z

/cc @robscott @youngnick @mlavacca @sunjayBhatia

robscott · 2024-03-14T20:48:57Z

Thanks @xtineskim!

/ok-to-test

mlavacca

Thanks for this PR, @xtineskim!

hack/mkdocs-copy-geps.py

site-src/implementation-table.md

candita · 2024-04-15T23:11:48Z

hack/mkdocs-copy-geps.py

+  http_reports.drop_duplicates(subset='organization', inplace=True, keep='last')
+
+  table = pandas.DataFrame(columns=http_reports['organization'])
+  table.insert(loc=0, column='Features', value=httproute_extended_conformance_features_list)


It might be better use of the space if we come up with list of shortened feature names. Just removing the HttpRoute prefix will help. Maybe there could be a link from the shortened feature name to its reference in the API or a similar option.

For example, HTTPRouteBackendRequestHeaderModification could be called Header Modification and point to github.com/kubernetes-sigs/gateway-api/blob/e81022784e13b041a91f73725e9f3e393fccdbfc/conformance/utils/suite/features.go#L101

This also helps if readers aren't familiar with the terms we use for our features.

But we can leave this for the next version, as we wouldn't want to hold the first iteration.

+1 on this being very nice to have but also not a blocker. If you run out of time to fit this into this PR, do you mind filing a follow up issue so we don't lose track of it?

candita · 2024-04-15T23:18:47Z

conformance/reports/v1.0.0/cilium/v1.15.0-pre.3-report.yaml

@@ -38,7 +38,7 @@ profiles:
        - HTTPRouteRequestTimeout
        - HTTPRouteResponseHeaderModification
        - HTTPRouteSchemeRedirect
-      name: HTTP
+    name: HTTP


If an implementation adds a broken-format report, will it mess up the display completely, or just for that implementation?

good question - it will be missing from the display. For example, currently the generate_table filters for the name "HTTP", and if that column is missing (the format results in the column being in an incorrect location in the dataframe), then it will be dropped. I guess it's somewhat of a check

hack/mkdocs-copy-geps.py

mlavacca

I like @robscott's proposal to group reports by macro-categories (Gateway and Mesh) and have implementations on the y-axis. I see the additional problems we need to tackle here:

for each Gateway API Version, an implementation can submit multiple reports for different implementation versions
for each Gateway API version and implementation version, there can be multiple reports related to the modes
an organization can have multiple projects (at Kong, for example, we have the Gateway Operator we'll add conformance for soon)

For this reason, I'd like to iterate on Rob's proposal and add the following details:

Gateway Profile

HTTPRoute

Org	Project	Version	Mode	HostRewrite	MethodMatching	MethodMatching	PathRedirect
Kong	kubernetes-ingress-controller	v3.1.2	default	✅	❌	❌	❌
Kong	project-2	v1.2.3	with-the-lot	✅	✅	✅	✅
EnvoyProxy	envoy-gateway	0.6.0	default	✅	✅	✅	✅

Mesh Profile

HTTPRoute

...

I understand this decreases the readability of the table, but I didn't find a better way to include all the needed information.

xtineskim · 2024-04-16T17:05:00Z

@mlavacca thanks for your input! I think with @candita's comment about having abbreviated names (maybe not in this pr), I think it would be better. I'll make the changes to switch the axis

Per your comments listed:

for each Gateway API Version, an implementation can submit multiple reports for different implementation versions

Currently I am just grabbing the latest report uploaded (nginx, contour both have 2 yaml reports, but i sort the versions and keep the latest one). I'm hoping that as new conformance reports are generated, all the profiles will be included in the latest report

2.for each Gateway API version and implementation version, there can be multiple reports related to the modes

Ah good catch with the mode, I forgot that use case (reference for others). I haven't seen any project report use that field either (yet). To address this, I can add a filter for mode for the loaded dataframes to look for the mode type, but also add in the default mode for current projects. Thanks for the reminder on this!

3.an organization can have multiple projects

I'm assuming though that this will have it's own folder, and a different project name? And if the project name is different, that shouldn't be a problem 👍

xtineskim · 2024-04-16T19:08:20Z

@robscott thank you for the suggestions! 🤝 slowly addressing the above comments, and i like the proposal of having the Gateway Profile and the Mesh Profile.
Just a few q's:

The Mesh Profiles information are pulled from the name of MESH in the reports (for example kuma's. Not sure if naming of MESH will have to be changed in the future if other routes will be added under the MESH conformance? I guess just assume keeping it for HTTPRoute for now?
Should I still have the Kuma listed under the Gateway Profile as well for HTTPRoute (pulling from this part of the report).

robscott · 2024-04-18T22:34:07Z

The Mesh Profiles information are pulled from the name of MESH in the reports (for example kuma's. Not sure if naming of MESH will have to be changed in the future if other routes will be added under the MESH conformance? I guess just assume keeping it for HTTPRoute for now?

This is actually one of my concerns with the current names of the conformance profiles. I think we may want to just settle on "mesh" and "gateway" conformance profiles and then have per-route details covered by the supported features. Interested in what @shaneutt and @mlavacca think about that though.

Should I still have the Kuma listed under the Gateway Profile as well for HTTPRoute (pulling from this part of the report).

Yes, I think both Kuma and Istio support the Mesh and Gateway concepts in Gateway API, so they'd be covered under both sections.

michaelbeaumont · 2024-04-19T11:38:20Z

Should I still have the Kuma listed under the Gateway Profile as well for HTTPRoute (pulling from this part of the report).

Yes, I think both Kuma and Istio support the Mesh and Gateway concepts in Gateway API, so they'd be covered under both sections.

Yep, Kuma does indeed support HTTPRoute for both Mesh and Gateway!

mikemorris · 2024-04-23T22:17:57Z

Confirming that Istio likewise supports HTTPRoute for both gateway and mesh use cases. This visualization is looking great and should be quite helpful in guiding users looking to adopt Gateway API!

site-src/implementation-table.md

robscott

Thanks for all the work on this @xtineskim! This is looking like it's really close, just a few more nits but otherwise LGTM.

robscott · 2024-04-29T21:40:11Z

site-src/implementation-table.md

+| kumahq         | :x:                                         | :white_check_mark:            | :white_check_mark:        | :white_check_mark:                    | :white_check_mark:      | :white_check_mark:        | :white_check_mark:      | :white_check_mark:     | :white_check_mark:     | :white_check_mark:       | :x:                               | :x:                       | :x:                       | :x:                      |
+| nginxinc       | :x:                                         | :white_check_mark:            | :white_check_mark:        | :x:                                   | :white_check_mark:      | :white_check_mark:        | :x:                     | :white_check_mark:     | :white_check_mark:     | :x:                      | :x:                               | :x:                       | :x:                       | :x:                      |
+| projectcontour | :x:                                         | :white_check_mark:            | :white_check_mark:        | :white_check_mark:                    | :white_check_mark:      | :white_check_mark:        | :white_check_mark:      | :white_check_mark:     | :white_check_mark:     | :white_check_mark:       | :white_check_mark:                | :white_check_mark:        | :white_check_mark:        | :x:                      |
+| solo.io        | :x:                                         | :white_check_mark:            | :white_check_mark:        | :white_check_mark:                    | :white_check_mark:      | :white_check_mark:        | :white_check_mark:      | :x:                    | :x:                    | :x:                      | :x:                               | :x:                       | :x:                       | :x:                      |


@xtineskim I think we might need to show both "Organization" and "Project" here for now. Otherwise we end up with incomplete information. For example, in this case, most people would want to know the product name ("Gloo Gateway"), while in the case of Kong above, the "Kubernetes Ingress Controller" product name is really not sufficient. Until we have a better solution in conformance reports, I think we'll just have to settle for reporting both.

@mlavacca Do you think it's worth adding a new field (or merging fields) in conformance reports to make this easier to handle? Maybe we should just have a single product name field that represents everything?

in the reports we already have organization and project, which if merged provide a unique tuple. Acutally, this is very related to my comment above: #2874 (review). I agree we should address both org+project and implementation version.

gotcha, i'll add them in 👍

robscott · 2024-04-29T21:44:19Z

site-src/implementation-table.md

@@ -0,0 +1,25 @@
+The following tables are populated from the conformance reports uploaded by project implementations. They are separated into the extended features that each project supports listed in their reports.
+
+## Gateway Profile


I'm not sure where/how to fit this in, but we'll likely want to always support the current and previous version of Gateway API here. Maybe we need something like:

## v1.1 ### Gateway #### HTTPRoute ... ## v1.0

Admittedly a report for v1.1 hasn't merged yet, but that should happen relatively soon with #3025.

I think this could also be a followup item though if we can't get it in this PR.

yeah i think i'd like to noodle on it a bit. I don't like the way i have the version handled, but maybe a way to have older pages potentially? 🤔

site-src/implementations.md

robscott · 2024-04-29T21:46:47Z

hack/mkdocs-copy-geps.py

@@ -15,10 +15,109 @@
 import shutil
 import logging
 from mkdocs import plugins
+import yaml


Nit: I think this file should probably be renamed to represent the broader scope of the script.

im thinking of putting this conformance report generation into it's own make-implementation-table.py and have it as it's own hook. Thoughts??

SGTM, thanks!

mkdocs.yml

robscott · 2024-04-29T21:50:12Z

hack/mkdocs-copy-geps.py

+  http_reports.drop_duplicates(subset='organization', inplace=True, keep='last')
+
+  table = pandas.DataFrame(columns=http_reports['organization'])
+  table.insert(loc=0, column='Features', value=httproute_extended_conformance_features_list)


+1 on this being very nice to have but also not a blocker. If you run out of time to fit this into this PR, do you mind filing a follow up issue so we don't lose track of it?

site-src/implementation-table.md

robscott

Thanks @xtineskim, a couple tiny nits but otherwise LGTM.

robscott · 2024-05-02T17:55:02Z

site-src/implementation-table.md

+
+### HTTPRoute
+
+| Organization   | project                       | version       | HTTPRouteMethodMatching   | HTTPRouteQueryParamMatching   | HTTPRouteResponseHeaderModification   | HTTPRouteBackendTimeout   | HTTPRouteHostRewrite   | HTTPRouteParentRefPort   | HTTPRoutePathRedirect   | HTTPRoutePathRewrite   | HTTPRoutePortRedirect   | HTTPRouteRequestMirror   | HTTPRouteRequestMultipleMirrors   | HTTPRouteRequestTimeout   | HTTPRouteSchemeRedirect   |


Nit: Would be nice if Project and Version were capitalized

site-src/implementation-table.md

robscott · 2024-05-02T17:56:32Z

hack/mkdocs-copy-geps.py

@@ -15,10 +15,109 @@
 import shutil
 import logging
 from mkdocs import plugins
+import yaml


SGTM, thanks!

robscott · 2024-05-02T20:15:18Z

Thanks @xtineskim, this is a great addition!

/lgtm
/approve

k8s-ci-robot · 2024-05-02T20:15:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: robscott, xtineskim

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [robscott]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Conformance display profiles on gatewayapi docs

6a649b0

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/documentation Categorizes issue or PR as related to documentation. labels Mar 14, 2024

k8s-ci-robot requested review from keithmattix and michaelbeaumont March 14, 2024 19:04

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 14, 2024

k8s-ci-robot requested review from mlavacca, robscott, sunjayBhatia and youngnick March 14, 2024 19:55

Update requirements.txt for pandas and tabulate

8fc1a6e

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 14, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 14, 2024

xtineskim added 2 commits March 15, 2024 20:53

Merge branch 'main' into conformance-profile-display

8e7bc97

Merge branch 'main' into conformance-profile-display

0dcacaf

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 19, 2024

mlavacca reviewed Mar 29, 2024

View reviewed changes

hack/mkdocs-copy-geps.py Outdated Show resolved Hide resolved

site-src/implementation-table.md Outdated Show resolved Hide resolved

xtineskim marked this pull request as draft April 2, 2024 18:05

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 2, 2024

xtineskim added 2 commits April 3, 2024 16:18

Typo, change select column to organization

de7cbcb

Resolved conflicts/typo, removed multiple versions of same project

b0e0dac

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 4, 2024

Merge branch 'main' into conformance-profile-display

06c7408

candita reviewed Apr 15, 2024

View reviewed changes

hack/mkdocs-copy-geps.py Outdated Show resolved Hide resolved

mlavacca reviewed Apr 16, 2024

View reviewed changes

robscott mentioned this pull request Apr 18, 2024

Adding changelog for v1.1.0-rc1 #2967

Merged

Switched axis, display the mesh and gateway profiles

11adfa6

howardjohn reviewed Apr 24, 2024

View reviewed changes

site-src/implementation-table.md Outdated Show resolved Hide resolved

Updated axis, small fixes

f0d3510

robscott mentioned this pull request Apr 29, 2024

[Conformance] Allow for reporting results before fully achieving conformance #3036

Open

robscott reviewed Apr 29, 2024

View reviewed changes

youngnick reviewed Apr 30, 2024

View reviewed changes

site-src/implementation-table.md Outdated Show resolved Hide resolved

Addressed comments

cdc5ca8

robscott reviewed May 2, 2024

View reviewed changes

xtineskim added 2 commits May 2, 2024 14:32

nits, created standalone py for conformance reports

5b61d50

remove conformance report code from copy-geps.py

0bba97b

k8s-ci-robot assigned robscott May 2, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 2, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 2, 2024

k8s-ci-robot merged commit cd175ec into kubernetes-sigs:main May 2, 2024
8 checks passed

xtineskim deleted the conformance-profile-display branch May 3, 2024 12:30

xtineskim mentioned this pull request May 3, 2024

Follow up tasks for conformance reporting #3050

Closed

shaneutt mentioned this pull request Jun 11, 2024

Display for conformance profiles results #2550

Closed

xtineskim mentioned this pull request Jun 27, 2024

REQUEST: New membership for xtineskim kubernetes/org#5038

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conformance display profiles on gatewayapi docs #2874

Conformance display profiles on gatewayapi docs #2874

xtineskim commented Mar 14, 2024

k8s-ci-robot commented Mar 14, 2024

shaneutt commented Mar 14, 2024

robscott commented Mar 14, 2024

mlavacca left a comment

candita Apr 15, 2024 •

edited

Loading

candita Apr 15, 2024

candita Apr 15, 2024

robscott Apr 29, 2024

candita Apr 15, 2024 •

edited

Loading

xtineskim Apr 16, 2024

mlavacca left a comment

xtineskim commented Apr 16, 2024

xtineskim commented Apr 16, 2024

robscott commented Apr 18, 2024

michaelbeaumont commented Apr 19, 2024

mikemorris commented Apr 23, 2024 •

edited

Loading

robscott left a comment

robscott Apr 29, 2024

mlavacca Apr 30, 2024

xtineskim Apr 30, 2024

robscott Apr 29, 2024

youngnick Apr 30, 2024

xtineskim May 1, 2024

robscott Apr 29, 2024

xtineskim May 1, 2024

robscott May 2, 2024

robscott Apr 29, 2024

robscott left a comment

robscott May 2, 2024

robscott May 2, 2024

robscott commented May 2, 2024

k8s-ci-robot commented May 2, 2024

		@@ -0,0 +1,25 @@
		The following tables are populated from the conformance reports uploaded by project implementations. They are separated into the extended features that each project supports listed in their reports.

		## Gateway Profile


		### HTTPRoute

		\| Organization \| project \| version \| HTTPRouteMethodMatching \| HTTPRouteQueryParamMatching \| HTTPRouteResponseHeaderModification \| HTTPRouteBackendTimeout \| HTTPRouteHostRewrite \| HTTPRouteParentRefPort \| HTTPRoutePathRedirect \| HTTPRoutePathRewrite \| HTTPRoutePortRedirect \| HTTPRouteRequestMirror \| HTTPRouteRequestMultipleMirrors \| HTTPRouteRequestTimeout \| HTTPRouteSchemeRedirect \|

Conformance display profiles on gatewayapi docs #2874

Conformance display profiles on gatewayapi docs #2874

Conversation

xtineskim commented Mar 14, 2024

k8s-ci-robot commented Mar 14, 2024

shaneutt commented Mar 14, 2024

robscott commented Mar 14, 2024

mlavacca left a comment

Choose a reason for hiding this comment

candita Apr 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

candita Apr 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlavacca left a comment

Choose a reason for hiding this comment

Gateway Profile

HTTPRoute

Mesh Profile

HTTPRoute

xtineskim commented Apr 16, 2024

xtineskim commented Apr 16, 2024

robscott commented Apr 18, 2024

michaelbeaumont commented Apr 19, 2024

mikemorris commented Apr 23, 2024 • edited Loading

robscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robscott commented May 2, 2024

k8s-ci-robot commented May 2, 2024

candita Apr 15, 2024 •

edited

Loading

candita Apr 15, 2024 •

edited

Loading

mikemorris commented Apr 23, 2024 •

edited

Loading