Unable to provide multiple parameters with tgt_cmd_extra_args

[pcuzner]

Using a single override works fine, but when I attempt to use something like

tgt_cmd_extra_args = "--cpumask=0xFF --msg-mempool-size=524288"

The gateway fails to start since it passes the same parameters to the start of nvmf

 INFO:control.server:Starting /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock --cpumask=0xFF --msg-mempool-size=524288
 INFO:control.server:Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0
 INFO: Setting log level to WARN
 INFO:JSONRPCClient(/var/tmp/spdk.sock):Setting log level to WARN
 [2023-09-13 04:58:10.051690] Starting SPDK v23.01.1 / DPDK 22.11.1 initialization...
 [2023-09-13 04:58:10.051765] [ DPDK EAL parameters: nvmf --no-shconf -c 0xFF --msg-mempool-size=524288 --no-pci --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x2000000>
 EAL: invalid characters in coremask: [ 0xFF --msg-mempool-size=524288]
 EAL: invalid coremask syntax

[caroav]

@gbregman please have a look

[gbregman]

The problem here is the usage of quotes in the conf file. As the string used was:

tgt_cmd_extra_args = "--cpumask=0xFF --msg-mempool-size=524288"

The two arguments were passed as a single argument to SPDK. The SPDK command line parser tries to parse the "--cpumask" argument and get:

ceph-nvmeof-nvmeof-1 | EAL: invalid characters in coremask: [ 0xFF --msg-mempool-size=524288] ceph-nvmeof-nvmeof-1 | EAL: invalid coremask syntax ceph-nvmeof-nvmeof-1 | EAL: FATAL: Invalid 'command line' arguments. ceph-nvmeof-nvmeof-1 | EAL: Invalid 'command line' arguments.

When I remove the quotes and used this line in the conf file:

tgt_cmd_extra_args = --cpumask=0xFF --msg-mempool-size=524288

I don't get this error. I still get an error but it's about the actual core mask being invalid:

ceph-nvmeof-nvmeof-1 | [2023-09-13 13:18:25.365223] [ DPDK EAL parameters: nvmf --no-shconf -c 0xFF --no-pci --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid7 ] ceph-nvmeof-nvmeof-1 | EAL: lcore 4 unavailable ceph-nvmeof-nvmeof-1 | EAL: lcore 5 unavailable ceph-nvmeof-nvmeof-1 | EAL: lcore 6 unavailable ceph-nvmeof-nvmeof-1 | EAL: lcore 7 unavailable ceph-nvmeof-nvmeof-1 | EAL: invalid coremask, please check specified cores are part of 0-3 ceph-nvmeof-nvmeof-1 | EAL: FATAL: Invalid 'command line' arguments. ceph-nvmeof-nvmeof-1 | EAL: Invalid 'command line' arguments. ceph-nvmeof-nvmeof-1 | [2023-09-13 13:18:25.373935] init.c: 602:spdk_env_init: *ERROR*: Failed to initialize DPDK

When I changed the mask value to 0x03 using the line:

tgt_cmd_extra_args = --cpumask=0x03 --msg-mempool-size=524288

In the conf file, SPDK went up with no problem.

[rkachach]

@pcuzner @gbregman I remember having to introduce some tricky change to generate the double quotes correctly when generating the daemon config file in cephadm, I understand this not needed anymore right?

[pcuzner]

Thanks @gbregman. I've confirmed that without the quotes I can use a cpumask upto the number of cores on a socket.