Multi-protocol IPv4 BGP overrides IPv6 nexthop on reflection

[bbs2web]

Description

We believe that the below patch, to force 6vPE prefixes to use an IPv4 gateway encoded as IPv6 (eg ::ffff:51.49.31.54), to break multi-protocol BGP sessions where there is no MPLS forwarding path, but instead another type of layer 2 tunnelling fabric.

Merged last month:
#15453

We're kindly asking that a work around or option exist to disable this behaviour. FRR acting as a route reflector correctly learns the route, sets appropriate next-hop gateways for exchanged prefixes and can route packets towards the destination. Route reflector clients however receive prefixes where the next hop has been set as the route reflector's router ID, encoded as IPv6.

Route reflector clients have access to the provider edge gateways and should, as the route reflector itself does, recursively resolve the prefix. Changing an attribute of a reflected prefix whilst processing it as wanted locally is IMHO unexpected.

Version

FRRouting 9.1 (uklon01-rr04) on Linux(6.6.19-amd64-vyos).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--localstatedir=/var/run/frr' '--sbindir=/usr/lib/frr' '--sysconfdir=/etc/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--enable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

How to reproduce

FRR configuration file for RR1, which learns route for Google (in the example provided):

Setting either set ipv6 next-hop prefer-global and/or attribute-unchanged next-hop in the import/export route maps and router settings makes zero difference:

route-map rr_client-in permit 10
 match ipv6 address prefix-list all
 on-match goto 20
 set ipv6 next-hop prefer-global
route-map rr_client-in permit 20

route-map rr_client-out permit 10
 match ipv6 address prefix-list all
 on-match goto 20
 set ipv6 next-hop prefer-global
route-map rr_client-out permit 20

router bgp 55514
 bgp router-id 51.49.22.46
 bgp log-neighbor-changes
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp cluster-id 51.49.22.6
 bgp bestpath as-path multipath-relax
 no bgp network import-check
 neighbor rr_client peer-group
 neighbor rr_client remote-as 55514
 neighbor rr_client update-source lo
 neighbor 51.49.22.32 peer-group rr_client
 neighbor 51.49.22.33 peer-group rr_client
 !
 address-family ipv4 unicast
  redistribute connected route-map connected-out
  redistribute static route-map static-out
  neighbor rr_client activate
  neighbor rr_client addpath-tx-all-paths
  neighbor rr_client route-reflector-client
  neighbor rr_client soft-reconfiguration inbound
  neighbor rr_client route-map rr_client-in in
  neighbor rr_client route-map rr_client-out out
  neighbor rr_client attribute-unchanged next-hop
  maximum-paths ibgp 2
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected route-map connected-out
  redistribute static route-map static-out
  neighbor rr_client activate
  neighbor rr_client addpath-tx-all-paths
  neighbor rr_client route-reflector-client
  neighbor rr_client soft-reconfiguration inbound
  neighbor rr_client route-map rr_client-in in
  neighbor rr_client route-map rr_client-out out
  neighbor rr_client attribute-unchanged next-hop
  maximum-paths ibgp 2
 exit-address-family
exit
!

Network:

zanjnb01-rr04# show interface brief
Interface       Status  VRF             Addresses
---------       ------  ---             ---------
br0             up      default         + 2001:db8:1:0:51:49:21:55/64    # IPv6 bridge
eth0            up      default
eth0.11         up      default         192.168.12.146/30    # Link to CR01
eth0.13         up      default         192.168.12.154/30    # Link to CR02
eth0.14         up      default         192.168.12.158/30    # Link to partner RR
eth0.15         up      default           # IPv6 over MPLS for non-l2vpn capable routers, connected to br0
lo              up      default         51.49.21.55/32
pim6reg         up      default

Expected behavior

Other route reflector clients should receive the iBGP route with the next-hop set as it shows on the FRR instance (the route reflector) that ingested the route.

On the first route reflector, the route is ingested with the IPv6 next-hop set as it was received and everything works:

uklon01-rr04# show ipv6 route 2001:4860:4860::8888
Routing entry for 2001:4860::/32
  Known via "bgp", distance 200, metric 0, best
  Last update 00:33:22 ago
  * 2001:db8:1:0:51:49:22:32, via br0, weight 1
  * 2001:db8:1:0:51:49:22:33, via br0, weight 1

uklon01-rr04# show bgp ipv6 2001:4860:4860::8888
BGP routing table entry for 2001:4860::/32, version 7735842
Paths: (3 available, best #3, table default)
  15169, (Received from a RR-client)
    2001:db8:1:0:51:49:22:1 from 51.49.22.1 (51.49.22.1)
      Origin IGP, metric 0, localpref 130, valid, internal, rpki validation-state: valid
      Community: 55514:1003
      AddPath ID: RX 0, TX-All 228655 TX-Best-Per-AS 0 TX-Best-Selected 0
      Advertised to: 51.49.22.45 51.49.21.54 51.49.21.55
      Last update: Sun Mar 24 09:57:01 2024
  15169, (Received from a RR-client)
    2001:db8:1:0:51:49:22:33 from 51.49.22.33 (51.49.22.33)
      Origin IGP, metric 0, localpref 200, valid, internal, multipath, rpki validation-state: valid
      Community: 55514:2000
      AddPath ID: RX 0, TX-All 85810 TX-Best-Per-AS 0 TX-Best-Selected 0
      Advertised to: 51.49.22.45 51.49.21.54 51.49.21.55
      Last update: Sat Mar 23 19:31:34 2024
  15169, (Received from a RR-client)
    2001:db8:1:0:51:49:22:32 from 51.49.22.32 (51.49.22.32)
      Origin IGP, metric 0, localpref 200, valid, internal, multipath, best (Router ID), rpki validation-state: valid
      Community: 55514:2000
      AddPath ID: RX 0, TX-All 298162 TX-Best-Per-AS 0 TX-Best-Selected 0
      Advertised to: 51.49.31.48 51.49.31.49 51.49.31.52 51.49.31.50 51.49.31.51 51.49.31.55 51.49.32.45 51.49.32.46
      Last update: Sun Mar 24 09:59:10 2024
uklon01-rr04# ping 2001:4860:4860::8888
PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes
64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=120 time=1.50 ms
^C
--- 2001:4860:4860::8888 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.502/1.502/1.502/0.000 ms

Actual behavior

FRR is assuming that I want 6vPE and 'helpfully' advertises ::ffff:3331:162e (IPv4 51.49.22.46 encoded as IPv6). This breaks other devices.

I'm kindly asking that attribute-unchanged next-hop could be supported for corner cases where multi-protocol BGP is of high value but transport is not wanted via MPLS.

Other router is is peering IPv4 and IPv6 over an IPv4 session still continues to receive routes with the route reflector's router ID set as the IPv6 encoded next hop:

[admin@uklon01-pe03] > ipv6 route print detail where dst-address=2001:4860::/32
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, o - ospf, b - bgp, U - unreachable
 0  Db  dst-address=2001:4860::/32 gateway=::ffff:51.49.22.45 gateway-status=::ffff:51.49.22.45 unreachable distance=200 scope=40 target-scope=30
        bgp-as-path="15169" bgp-local-pref=200 bgp-med=0 bgp-origin=igp bgp-communities=55514:2000 received-from=uklon01-rr03

 1  Db  dst-address=2001:4860::/32 gateway=::ffff:51.49.22.46 gateway-status=::ffff:51.49.22.46 unreachable distance=200 scope=40 target-scope=30
        bgp-as-path="15169" bgp-local-pref=200 bgp-med=0 bgp-origin=igp bgp-communities=55514:2000 received-from=uklon01-rr04

ie: Routes are inactive, as it can't resolve the gateways of ::ffff:51.49.22.45 or ::ffff:51.49.22.46 (the route reflector IPs. I would expect the prefix to be received and processed like it is on the route reflector itself.

IPv4 routes are reflected as expected:

[admin@uklon01-pe03] > /ip route print detail where dst-address=8.8.8.0/24
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 0 ADb  dst-address=8.8.8.0/24 pref-src=51.49.22.34 gateway=51.49.22.32 gateway-status=51.49.22.32 recursive via 192.168.28.13 vlan6 distance=200 scope=40
        target-scope=30 bgp-as-path="15169" bgp-local-pref=200 bgp-med=0 bgp-origin=igp bgp-communities=55514:2000
        bgp-ext-communities="000x4300:0x0 0x0 0x0 0x0 0x0 0x0" received-from=uklon01-rr03

 1  Db  dst-address=8.8.8.0/24 pref-src=51.49.22.34 gateway=51.49.22.32 gateway-status=51.49.22.32 recursive via 192.168.28.13 vlan6 distance=200 scope=40
        target-scope=30 bgp-as-path="15169" bgp-local-pref=200 bgp-med=0 bgp-origin=igp bgp-communities=55514:2000
        bgp-ext-communities="000x4300:0x0 0x0 0x0 0x0 0x0 0x0" received-from=uklon01-rr04

Additional context

Has been working perfectly on FRR 8.5.1, due for an upgrade and sad to see this broken in FRR 9.1.

Checklist

• I have searched the open issues for this bug.
• I have not included sensitive information in this report.

[ton31337]

@bbs2web could you check this patch #15611?

[ton31337]

If we don't find a quick solution, IMO we should revert it ASAP.

[bbs2web]

Many thanks, apologies for not being able to validate the provided patch but I have spent a couple of days unpuzzling how to now recompile FRR with patches and rebuild the VyOS nightly ISO.

Hoping to be able to contribute better in future.

[louis-6wind]

Fixed in
#15683 (10.0)
#15684 (9.1)