ip4_select_id harm performance dramatically

[hulinfan]

uint32_t ip4_select_id(struct ipv4_hdr *iph)
{
    uint32_t hash, id;
    rte_atomic32_t *p_id;

    hash = rte_jhash_3words(iph->dst_addr, iph->src_addr,
            iph->next_proto_id, ip4_id_hashrnd);

    p_id = ip4_idents + hash % IP4_IDENTS_SZ;
    id = htons(rte_atomic32_read(p_id));
    rte_atomic32_add(p_id, 1);

    return id;
}

dpvs 的这个函数对性能影响很大。我在测试 ipip 隧道模式的性能，16核，100万 udp 连接，注释 ip4_select_id，采用固定的 id， 性能达到 1800万pps；打开注释，性能只有 800 多万pps，下降了一半。

不难看到，ip4_select_id 中使用了全局的原子变量 ip4_idents，原子操作在多核情况下大大降低了转发性能。下面是 perf 热点图：

问题：
这个函数设计的目的是什么？为何不用每核一份自增的 packet_id ？

[ywc689]

ip4_select_id 意在实现 RFC #791中关于 packet id 选取的规范。

  Identification

    The choice of the Identifier for a datagram is based on the need to
    provide a way to uniquely identify the fragments of a particular
    datagram.  The protocol module assembling fragments judges fragments
    to belong to the same datagram if they have the same source,
    destination, protocol, and Identifier.  Thus, the sender must choose
    the Identifier to be unique for this source, destination pair and
    protocol for the time the datagram (or any fragment of it) could be
    alive in the internet.

    It seems then that a sending protocol module needs to keep a table
    of Identifiers, one entry for each destination it has communicated
    with in the last maximum packet lifetime for the internet.

    However, since the Identifier field allows 65,536 different values,
    some host may be able to simply use unique identifiers independent
    of destination.

    It is appropriate for some higher level protocols to choose the
    identifier. For example, TCP protocol modules may retransmit an
    identical TCP segment, and the probability for correct reception
    would be enhanced if the retransmission carried the same identifier
    as the original transmission since fragments of either datagram
    could be used to construct a correct TCP segment.

这个函数的性能问题之前测试没有发现过，我们会看看能否复现这个问题。由于 DPVS 核心代码不支持分片，且当前的 packet id选取也不完全符合 rfc 规范，现在使用 per-lcore 的 packet id 也不会有太大问题。

[hulinfan]

感谢回答。我们使用 dpdk-pktgen 发包，测试条件是配置 1个 service 1个 rs：

# ./tools/ipvsadm/ipvsadm -ln
IP Virtual Server version 0.0.0 (size=0)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  192.168.3.239:5555 rr
  -> 192.168.2.6:5555             Tunnel  1      0          1048576

可以尝试一下。

[hulinfan]

还有个问题请教一下，DPVS 代码应该是支持 IP 分片的吧？我看已经实现了 ipv4_fragment，该函数在 ipv4_output_fin 处调用：

static int ipv4_output_fin(struct rte_mbuf *mbuf)
{
    struct route_entry *rt = mbuf->userdata;

    if (mbuf->pkt_len > rt->mtu)
        return ipv4_fragment(mbuf, rt->mtu, ipv4_output_fin2);

    return ipv4_output_fin2(mbuf);
}