RocksDB fails to build on kfreebsd-amd64 due to 'malloc_np.h: No such file'

[ottok]

On systems with the kfreebsd kernel (not Linux) and arch amd64 RocksDB fails to compile with the error message:

In file included from /<<PKGBUILDDIR>>/storage/rocksdb/rocksdb/db/builder.cc:31:
--
/<<PKGBUILDDIR>>/storage/rocksdb/rocksdb/table/format.h:15:10: fatal error: malloc_np.h: No such file or directory
#include <malloc_np.h>
^~~~~~~~~~~~~

Full log can be seen here where MariaDB 10.3 is build on Debian official builders:
Full log at https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.3&arch=kfreebsd-amd64&ver=1%3A10.3.13-2&stamp=1554671911&raw=0

This was originally reported at:

• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=920994
• https://jira.mariadb.org/browse/MDEV-19251

The fix should be done as upstream as possible, that is in this Facebook/RocksDB repository.

[siying]

This include was explicitly placed in place by an open-source contributor: #1428 . Feel free to send a pull request to do appropriate adjustment to make it work.

[ottok]

There is no such file as malloc_np.h in the Debian archives: https://packages.debian.org/search?searchon=contents&keywords=malloc_np.h&mode=exactfilename&suite=unstable&arch=any

There is no real commit message in d1b1835 explaining the stuff commit messages usually cover.

Could you @wjwithagen please elaborate a bit on what that change was, and where the malloc_np.h should come from?

Thanks!

[siying]

I am not sure @wjwithagen will respond. I Googled the file and it shows up here: https://github.com/freebsd/freebsd/blob/master/include/malloc_np.h

Maybe the right fix is to improve the test here: https://github.com/facebook/rocksdb/blob/master/build_tools/build_detect_platform#L394-L406 to run exactly the same code as in the source files and set ROCKSDB_MALLOC_USABLE_SIZE properly.

[wjwithagen]

Sorry, it took a while, but it got pushed down in my mailbox and just now I got triggered again...

On FreeBSD is the _np part of filenames used for those parts that are Non-Posix.
And malloc_np.h lives at:
[~] wjw@router.digiware.nl> locate malloc_np.h
/usr/include/malloc_np.h

And contains those jemalloc parts that are not required by Posix, like:

size_t  malloc_usable_size(const void *ptr);

void    malloc_stats_print(void (*write_cb)(void *, const char *),
    void *cbopaque, const char *opts);
int     mallctl(const char *name, void *oldp, size_t *oldlenp, void *newp,
    size_t newlen);
int     mallctlnametomib(const char *name, size_t *mibp, size_t *miblenp);
int     mallctlbymib(const size_t *mib, size_t miblen, void *oldp,
    size_t *oldlenp, void *newp, size_t newlen);
......
size_t  __malloc_usable_size(const void *ptr);

And if I remeber correctly was malloc_stats_print the reason to add malloc_np.h.

So I guess that kfreebsd triggers the OS_FREEBSD but has the debian files.
Sollution would be to rewrite the test such that it takes into account that there is no malloc_np.h on this mix of platforms.
I'm willing to test the regular FreeBSD side of things, in case of changes/PR

[siying]

If just because of malloc_stats_print(), we don't necessarily need to support to print out malloc stats in FreeBSD. It's just nice to have. But if it is for malloc_usable_size(), then it's a better idea to try the best to be able to find and call this function.

[wjwithagen]

@siying
I submitted this patch because RocksDB is used in Ceph.
And it is used there for reporting and more, and I did not feel like suppressing that code for FreeBSD.

Also by itself my submitted patch is not wrong, it just did not take into account dat kFreeBSD announces itself as OS_FREEBSD but does not have the FreeBSD files....
So the fix should be to add to the test something that identifies kFreeBSD uniquely.

[siying]

@wjwithagen you can resolve this with @ottok in whatever way you agree with. I'll merge the PR you agree on.

[wjwithagen]

@siying @ottok
Sure, no problem.
I think I would require a list of preprocessor defines at the point of the inclusion.
Perhaps you can pastebin it somewhere.
-E Run the preprocessor stage.
Will generate the code after preprocession on stdout.
Then we can check if there is a define that is significant for kFreeBSD, and not for FreeBSD

[ottok]

Hello! If somebody watching this makes an attempt to make a patch, I an happy to include it in the next upload of MariaDB to Debian experimental.

[wjwithagen]

@ottok
Willing to make a patch, but I do need to know what preprocessor defines are available.
on kFreeBSD
Run something like:
clang -dM -E - < /dev/null
and pastebin the result.
But I'm not sure how we moved from RocksDB to MariaDB??

[ottok]

the librocksdb in Debian does actually not even try to build for kFreeBSD (https://buildd.debian.org/status/package.php?p=rocksdb). MariaDB steps into the picture as it is the place where RocksDB is tried to build for all Debian platforms (https://buildd.debian.org/status/package.php?p=mariadb-10.3&suite=sid).

The kFreeBSD testing images directory is empty at the moment (https://d-i.debian.org/daily-images/kfreebsd-amd64/) so I am not sure how to spin up a virtual machine with kFreeBSD at the moment etc. Maybe it is so marginal that it does not make sense to invest more time in?

[wjwithagen]

@ottok
I'm fine with that.
When it arisses again, feel free to contact.

[ottok]

Mitigated in Debian via https://salsa.debian.org/mariadb-team/mariadb-10.3/commit/fc8eb8f634f60ddef64ece979d58e09cb5318d62

[ottok]

The original report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=920994 was closed via https://salsa.debian.org/mariadb-team/mariadb-10.4/-/blob/master/debian/patches/rocksdb-kfreebsd.patch by @jrtc27

Please review that patch and apply it upstream so we don't have to carry a patched version of RocksDB downstream.