Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for IPv6 #2190

Merged
merged 28 commits into from
Jul 17, 2024
Merged

Add support for IPv6 #2190

merged 28 commits into from
Jul 17, 2024

Conversation

salonichf5
Copy link
Contributor

@salonichf5 salonichf5 commented Jul 2, 2024

Proposed changes

Problem: User wants to NGINX Gateway Fabric to support IPv6 and IPv4

Solution: Add a new field ipFamily to NginxProxy API to specify the IP family to use with server and update listen directives in the nginx.conf.

Testing:

  • Updates unit tests as needed
  • Manual Testing with cafe-examples:
  1. Only IPv6

values.yaml file

config
   ipFamily: ipv6

NGINX Config

server {
    listen [::]:80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen [::]:80;

    server_name cafe.example.com;
.
.
.
upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 512k;

    server [fd00:10:244:1::e]:8080;
}

upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 512k;

    server [fd00:10:244:1::d]:8080;
}

Response from Coffee and Tea services

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/coffee
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /coffee HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 20:49:51 GMT
< Content-Type: text/plain
< Content-Length: 166
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 20:49:50 GMT
< Cache-Control: no-cache
<
Server address: fd00:10:244:1::d:8080
Server name: coffee-6b8b6d6486-gqg68
Date: 07/Jul/2024:20:49:51 +0000
URI: /coffee
Request ID: 9723f8cc35b422b0a5e385aee4d28ded
curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/tea
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /tea HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 20:50:41 GMT
< Content-Type: text/plain
< Content-Length: 159
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 20:50:40 GMT
< Cache-Control: no-cache
<
Server address: fd00:10:244:1::e:8080
Server name: tea-9d8868bb4-s2sh2
Date: 07/Jul/2024:20:50:41 +0000
URI: /tea
Request ID: fddfb7555f4089c9d4cdce386464bc74
  1. Only IPv4

values.yaml file

config
   ipFamily: ipv4

NGINX Config

server {
    listen 80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen 80;
.
.
.

upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 512k;

    server 10.244.0.6:8080;
}

upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 512k;

    server 10.244.0.7:8080;
}

Response from Coffee and Tea services

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/tea
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /tea HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:08:20 GMT
< Content-Type: text/plain
< Content-Length: 154
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:08:19 GMT
< Cache-Control: no-cache
<
Server address: 10.244.0.6:8080
Server name: tea-596697966f-bvgln
Date: 07/Jul/2024:21:08:20 +0000
URI: /tea
Request ID: adfa0f1b4ea2d066f2a1b250325eae0c
* Connection #0 to host cafe.example.com left intact
curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/coffee
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /coffee HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:08:48 GMT
< Content-Type: text/plain
< Content-Length: 160
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:08:47 GMT
< Cache-Control: no-cache
<
Server address: 10.244.0.7:8080
Server name: coffee-56b44d4c55-885kb
Date: 07/Jul/2024:21:08:48 +0000
URI: /coffee
Request ID: be548614b053439199dfaa449f36b607
* Connection #0 to host cafe.example.com left intact
  1. IPv6 and IPv4

values.yaml file (by default it is set to ipFamily: dual)

config
   {}

NGF Pod has both IPs configured when installed with dual IP Family.

kubectl describe pod -n nginx-gateway ngf-nginx-gateway-fabric-7446d69b4-6hsjh
Name:             ngf-nginx-gateway-fabric-7446d69b4-6hsjh
Namespace:        nginx-gateway
Status:           Running
IP:               10.244.1.2
IPs:
  IP:           10.244.1.2
  IP:           fd00:10:244:1::2

Coffee Service created with IPv4

kubectl describe svc coffee
Name:              coffee
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.202.148
IPs:               10.96.202.148

Tea Service created with IPv6 stack

kubectl describe svc tea
Name:              tea
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv6
IP:                fd00:10:96::2e2e

NGINX Config

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen 80;
    listen [::]:80;
.
.
.
upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 512k;

    server 10.244.1.3:8080;
}

upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 512k;

    server [fd00:10:244:1::4]:8080;
}

Responses from Tea and Coffee Applications

curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/coffee
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /coffee HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:24:30 GMT
< Content-Type: text/plain
< Content-Length: 160
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:24:29 GMT
< Cache-Control: no-cache
<
Server address: 10.244.1.3:8080
Server name: coffee-6b8b6d6486-hvxp5
Date: 07/Jul/2024:21:24:30 +0000
URI: /coffee
Request ID: 1833d0a1fd620e91946ea184c7a358e5
* Connection #0 to host cafe.example.com left intact
curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP  http://cafe.example.com:$GW_PORT/tea
* Added cafe.example.com:8080:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to cafe.example.com (127.0.0.1) port 8080
> GET /tea HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx/1.27.0
< Date: Sun, 07 Jul 2024 21:24:43 GMT
< Content-Type: text/plain
< Content-Length: 159
< Connection: keep-alive
< Expires: Sun, 07 Jul 2024 21:24:42 GMT
< Cache-Control: no-cache
<
Server address: fd00:10:244:1::4:8080
Server name: tea-9d8868bb4-82gxc
Date: 07/Jul/2024:21:24:43 +0000
URI: /tea
Request ID: 3ac40253dd7c64e4e92ad65e7c87a322
  1. If the cluster supports only IPv6 and a service is created with ipFamily: IPv4, following error shows:
gateway.gateway.networking.k8s.io/gateway created
The Service "coffee" is invalid: spec.ipFamilies[0]: Invalid value: "IPv4": not configured on this cluster
  1. If NGF is configured with IPv4 and a Service is configured with IPv6, the HTTPRoute shows an error
  Parents:
    Conditions:
      Last Transition Time:  2024-07-12T04:09:11Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2024-07-12T04:09:11Z
      Message:               service configured with IPv6 family but NginxProxy is configured with IPv4
      Observed Generation:   1
      Reason:                InvalidIPFamily
      Status:                False
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller
  1. If NGF is configured with IPv6 and a Service is configured with IPv4, the HTTPRoute shows an error
Status:
  Parents:
    Conditions:
      Last Transition Time:  2024-07-12T04:12:03Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2024-07-12T04:12:03Z
      Message:               service configured with IPv4 family but NginxProxy is configured with IPv6
      Observed Generation:   1
      Reason:                InvalidIPFamily
      Status:                False
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.

Closes #732

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked that all unit tests pass after adding my changes
  • I have updated necessary documentation //TODO
  • I have rebased my branch onto main
  • I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Adds support for IPv6

@salonichf5 salonichf5 requested review from a team as code owners July 2, 2024 22:10
@github-actions github-actions bot added documentation Improvements or additions to documentation enhancement New feature or request helm-chart Relates to helm chart labels Jul 2, 2024
@salonichf5 salonichf5 marked this pull request as draft July 2, 2024 23:00
@salonichf5 salonichf5 changed the title Do not review: Add support for ipv6 Do not review: Add support for IPv6 Jul 2, 2024
Copy link

codecov bot commented Jul 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.76%. Comparing base (45190d3) to head (e802e82).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2190      +/-   ##
==========================================
+ Coverage   87.62%   87.76%   +0.14%     
==========================================
  Files          96       96              
  Lines        6715     6793      +78     
  Branches       50       50              
==========================================
+ Hits         5884     5962      +78     
  Misses        774      774              
  Partials       57       57              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@salonichf5 salonichf5 force-pushed the feat/ipv6 branch 2 times, most recently from 5ef4cd3 to d2d10fe Compare July 3, 2024 00:43
@salonichf5 salonichf5 changed the title Do not review: Add support for IPv6 Add support for IPv6 Jul 3, 2024
@salonichf5 salonichf5 self-assigned this Jul 3, 2024
@pleshakov
Copy link
Contributor

Let's make sure we have the cafe example work for three cases when the cluster is:

(1)
IPv6

Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.

I tried this case myself and NGINX wasn't configured with upstream servers (they were empty).

(See this area of the code Also

if endpointSlice.AddressType != discoveryV1.AddressTypeIPv4 {
return true
}
)

Let's make sure it doesn't bind to IPv4 sockets or use IPv4 addresses as upstream servers

(2)
IPv4
Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

If the host machine doesn't support IPv6, NGINX will fail to reload with

2024/07/05 22:46:53 [emerg] 454#454: socket() [::]:80 failed (97: Address family not supported by protocol)

if NGINX binds to IPv6 socket:

listen [::]:80;

Because of that , let's make sure it doesn't bind to IPv6 sockets or use IPv4 addresses as upstream servers

(3)
Dual

  • Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.
  • Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

In this case, make sure one of the services - tea or coffee - has IPv4 and the other IPv6

You can configure IP familiy for the kind cluster -- https://kind.sigs.k8s.io/docs/user/configuration/#networking

See also https://kubernetes.io/docs/tasks/network/validate-dual-stack/#validate-services for controlling whether service shall be ipv4 or ipv6.

@salonichf5
Copy link
Contributor Author

Let's make sure we have the cafe example work for three cases when the cluster is:

(1) IPv6

Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.

I tried this case myself and NGINX wasn't configured with upstream servers (they were empty).

(See this area of the code Also

if endpointSlice.AddressType != discoveryV1.AddressTypeIPv4 {
return true
}

)
Let's make sure it doesn't bind to IPv4 sockets or use IPv4 addresses as upstream servers

(2) IPv4 Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

If the host machine doesn't support IPv6, NGINX will fail to reload with

2024/07/05 22:46:53 [emerg] 454#454: socket() [::]:80 failed (97: Address family not supported by protocol)

if NGINX binds to IPv6 socket:

listen [::]:80;

Because of that , let's make sure it doesn't bind to IPv6 sockets or use IPv4 addresses as upstream servers

(3) Dual

  • Ensure you can establish IPv6 connection to NGINX, and you see responses from coffee and tea services.
  • Ensure you can establish IPv4 connection to NGINX, and you see responses from coffee and tea services.

In this case, make sure one of the services - tea or coffee - has IPv4 and the other IPv6

You can configure IP familiy for the kind cluster -- https://kind.sigs.k8s.io/docs/user/configuration/#networking

See also https://kubernetes.io/docs/tasks/network/validate-dual-stack/#validate-services for controlling whether service shall be ipv4 or ipv6.

Have updated description to demonstrate these use cases. Let me know if something doesn't look right

@salonichf5 salonichf5 marked this pull request as ready for review July 8, 2024 17:00
apis/v1alpha1/nginxproxy_types.go Outdated Show resolved Hide resolved
apis/v1alpha1/nginxproxy_types.go Outdated Show resolved Hide resolved
internal/mode/static/nginx/config/http/config.go Outdated Show resolved Hide resolved
internal/mode/static/nginx/config/servers.go Outdated Show resolved Hide resolved
internal/mode/static/nginx/config/servers_template.go Outdated Show resolved Hide resolved
internal/mode/static/state/resolver/resolver.go Outdated Show resolved Hide resolved
internal/mode/static/state/resolver/resolver_test.go Outdated Show resolved Hide resolved
docs/developer/quickstart.md Outdated Show resolved Hide resolved
salonichf5 and others added 27 commits July 17, 2024 14:51
Co-authored-by: Michael Pleshakov <pleshakov@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
Co-authored-by: Kate Osborn <50597707+kate-osborn@users.noreply.github.com>
@salonichf5 salonichf5 merged commit 4813408 into nginxinc:main Jul 17, 2024
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request helm-chart Relates to helm chart release-notes
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

IPv4 and IPv6 Listeners
5 participants