Open dynamic library and call to it using dlopen() from Chapel?

[bradcray]

This issue asks whether it is currently possible to create a Chapel program that would open a dynamic library using dlopen() and make calls to it.

If so, I'd be interested in an example program demonstrating the capability; if not, I'd like to understand what the limiting factors to doing so are today, and what would be required to resolve them (either as a heroic programmer or through changes to the language or implementation).

[e-kayrakli]

On a toy (and by this time should be very stale, if I can find it) branch I was able to to dynamic function loading in the runtime. If something is doable in the runtime, it should be relatively easy to do in Chapel. So, I am hopeful that it should be doable without things getting too ugly.

Doing this (in the runtime, that is) is still of interest to me to support a better cpu-as-device mode, which can allow us to find a function in the same binary using its name.

[mppf]

Here is a complete example.

I am aware of the following issues in this area:

1. dlsym will generally return a function pointer. We can represent that with c_fn_ptr, but we don't have a way (currently) in the Chapel type system to turn this into a callable pointer. So we currently have to do the call from C.
2. dlopen of a library written in C is fine, but loading a library written in Chapel into a running Chapel program will run into problems because we don't yet have a way to update the virtual functions table or ftable (for on etc). Instead, what you will get today is that the library has its own tables and runtime loaded. One specific way that might cause problems is that we'll end up with redundant qthreads worker threads.
3. There are problems with generic functions. We can't dynamically load a Chapel library containing a generic function that isn't instantiated yet (that would require doing some compilation at load-time). Even if all of the generic functions are already instantiated, since we currently name the instantiations things like foo5, we won't know when we could possibly reuse one of these.
4. dlopen is not portable to Windows. If we were to make wrappers for dlopen in the OS module, we could make space for having a Windows implementation.
   Here is a complete example:

test.sh

#!/bin/sh
#

echo building shared library clib.so
gcc -shared -fPIC -o clib.so clib.c
echo building C code to use dlopen
gcc c-dlopen.c -o c-dlopen -ldl
echo running C code using dlopen
LD_LIBRARY_PATH=. ./c-dlopen

echo building Chapel code using dlopen
chpl chapel-dlopen.chpl
echo running Chapel code using dlopen
LD_LIBRARY_PATH=. ./chapel-dlopen

clib.h

void clibfn(void);

clib.c

#include "clib.h"

#include <stdio.h>

void clibfn(void) {
  printf("in clibfn\n");
}

c-dlopen.c

#include <dlfcn.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>

int main() {
  void* lib = NULL;
  void (*fn)(void) = NULL;

  lib = dlopen("clib.so", RTLD_LAZY);
  if (!lib) {
    fprintf(stderr, "dlopen failed: %s\n", dlerror());
    exit(-1);
  }

  fn = dlsym(lib, "clibfn");
  if (!fn) {
    fprintf(stderr, "dlsym failed: %s\n", dlerror());
    exit(-1);
  }

  fn();

  dlclose(lib);
  return 0;
}

chapel-dlopen.chpl

extern {
  #include <dlfcn.h>
  #include <stddef.h>
  #include <stdio.h>
  #include <stdlib.h>

  static int callit(void) {
    void* lib = NULL;
    void (*fn)(void) = NULL;

    lib = dlopen("clib.so", RTLD_LAZY);
    if (!lib) {
      fprintf(stderr, "dlopen failed: %s\n", dlerror());
      exit(-1);
    }

    fn = dlsym(lib, "clibfn");
    if (!fn) {
      fprintf(stderr, "dlsym failed: %s\n", dlerror());
      exit(-1);
    }

    fn();

    dlclose(lib);
    return 0;
  }
}

proc main() {
  callit();
}

expected behavior:

$ ./test.sh 
building shared library clib.so
building C code to use dlopen
running C code using dlopen
in clibfn
building Chapel code using dlopen
running Chapel code using dlopen
in clibfn

[bradcray]

I was curious whether the dlopen/dlclose calls could be pushed outside of the C code easily, and it appears that's the case:

extern {
  #include <dlfcn.h>
  #include <stdio.h>
  #include <stdlib.h>

  static void callit(void* lib) {
    void (*fn)(void) = NULL;

    fn = dlsym(lib, "clibfn");
    if (!fn) {
      fprintf(stderr, "dlsym failed: %s\n", dlerror());
      exit(-1);
    }

    fn();
  }
}

proc main() {
  const lib = dlopen("clib.so", RTLD_LAZY);
  if (!lib) then halt("dlopen failed: " + dlerror():string);

  callit(lib);

  dlclose(lib);
}

I also wrote a variant on the above that accepts an integer, prints it, and returns a variant of it, and that worked fine as well.

[jabraham17]

I think you can go a step further and move the dlsym call into Chapel as well. Then the pointer returned can be passed to callit, defined as

extern {
  typedef void (*void_func_t)(void);
  void callit(void* f);
  void callit(void* f) { ((void_func_t)f)(); }
}

So all that is missing the ability to represent c function pointers in Chapel (beyond just c_fn_ptr, which has no type info) and to call c function pointers

[bradcray]

Great point—I considered doing that, but wasn't sure I liked the idea of casting the Chapel void* pointer to a C function pointer. But on reflection, I was probably being too guarded given that that's what the C code was effectively doing anyway, just implicitly. Thanks!

[bradcray]

Here's a Chapel code that uses Jade's proposed cleanup to call a routine taking and returning a (C) integer:

extern {
  #include <dlfcn.h>

  static int callit(void *fnptr) {
    typedef int (*my_func_t)(int);

    return ((my_func_t)fnptr)(42);
  }
}

proc main() {
  const lib = dlopen("clib.so", RTLD_LAZY);
  if !lib then halt("dlopen failed: " + string.createBorrowingBuffer(dlerror()));

  const fn = dlsym(lib, "clibfn2");
  if !fn then halt("dlsym failed: " + string.createBorrowingBuffer(dlerror()));

  writeln(callit(fn));

  dlclose(lib);
}

where I added this routine to Michael's clib.c/h:

int clibfn2(int x) {
  printf("in clibfn(%d)\n", x);
  return x + 3;
}

---

I was also able to call into a simple Chapel routine using the following:

chpllib.chpl:

export proc chpllibfn(x: int): int {
  extern proc printf(x...);  // note that we may also be able to call into writeln() if we called into `chpl__init_chpllib()`…
  printf("x is: %lld\n", x);
  return x+3;
}

mytest.chpl:

extern {
  #include <dlfcn.h>
  #include <stdint.h>

  static int callit(void *fnptr) {
    typedef int64_t (*my_func_t)(int64_t);

    return ((my_func_t)fnptr)(42);
  }
}

proc main() {
  const lib = dlopen("libchpllib.so", RTLD_LAZY);
  if !lib then halt("dlopen failed: " + string.createBorrowingBuffer(dlerror())\
);

  const fn = dlsym(lib, "chpllibfn");
  if !fn then halt("dlsym failed: " + string.createBorrowingBuffer(dlerror()));

  writeln(callit(fn));

  dlclose(lib);
}

Using these commands:

$ chpl --library --dynamic chpllib.chpl
$ ln -s lib/libchpllib.so .  // alternatively, I could add `lib/` to my dynamic library path
$ chpl mytest.chpl
$ ./mytest

[edit: Note that I've only tried all of this in single-locale settings so far; I expect that, for multi-locale settings, we'd need to make the dl*() calls on each locale that wanted to make calls]

[dlongnecke-cray]

I've opened: #26099

This allows you to write:

// Just includes the header containing 'dlopen', nothing more.
extern {
  #include <dlfcn.h>
}

proc main() {
  // Open the library.
  const lib = dlopen("SomeCLibrary.so", RTLD_LAZY);
  if !lib then halt("dlopen failed: " + string.createBorrowingBuffer(dlerror()));

  // Get a 'c_ptr(void)' to the function.
  const vp1 = dlsym(lib, "clibfn1");
  if !vp1 then halt("dlsym failed: " + string.createBorrowingBuffer(dlerror()));

  // Cast to the appropriate function type and call.
  const f1 = vp1 : (proc(): void);
  f1();

  // Get a 'c_ptr(void)' to the function.
  const vp2 = dlsym(lib, "clibfn2");
  if !vp2 then halt("dlsym failed: " + string.createBorrowingBuffer(dlerror()));

  // Cast to the appropriate function type and call.
  const f2 = vp2 : (proc(_: int): int);
  var x = f2(42);
  writeln(x);

  // Close the library.
  dlclose(lib);
}

You have to turn on fcfsUsePointerImplementation which is a param flag.

An important caveat here is that these function pointers are usable only on the locale that loaded the library - the moment you try to call them on another locale the whole program explodes.

This is a problem that the entire pointer implementation of FCFs is facing at the moment - due to ASLR a pointer can live at a different address on each locale. For symbols that are known at compile-time the fix will be to add them to the global procedure table of the program.

That can't really happen for symbols produced by dlopen because, well, they're loaded at runtime...so on top of ASLR the other locales won't even have the library loaded.

There are a couple of approaches we could investigate as ways to prevent the program from exploding:

• We could introduce a notion of locality or a sort of widePtr type that wraps function pointers (basically restricting them to only be called on the locale they were created) then have our dlsym return those
• We could figure out a way to restrict our dlopen wrapper to only be used in a local block (and ostensibly keep the returned pointers being copied out of it - seems hard to do)
• We could smack on a big red warning onto our dlopen and dlsym wrappers and just not care
• We could have our dlopen wrapper return a sort of "context". The context maintains state such as whether or not the .so is loaded on each locale (and loads it if not). The context is locale sensitive and returns a callable object that is safe to use somehow...

[mppf]

An important caveat here is that these function pointers are usable only on the locale that loaded the library - the moment you try to call them on another locale the whole program explodes.

This is a problem that the entire pointer implementation of FCFs is facing at the moment - due to ASLR a pointer can live at a different address on each locale. For symbols that are known at compile-time the fix will be to add them to the global procedure table of the program.

I think we need to introduce a more generalized table mapping integers to function pointers on all locales. This is similar to the existing ftable / dispatch table, but the main difference is that it's not entirely known before execution. It can be modified at runtime. Our future separate compilation / dynamic loading efforts could even use this same mechanism. Note that we already have something like this in runtime/include/chpl-privatization.h (which helps Chapel code _newPrivatizedClass to implement privatization; the summary is that there is an atomic on Locale 0 that counts up to assign the integer IDs & the ID to pointer mapping is saved in per-locale arrays).

Presumably, at the same time, we would need to have our dlopen / dlsym wrapper functionality that loads a symbol on all locales and updates the dispatch table in a coordinated way. Or, maybe, it would proactively assign an integer ID in a way that is globally unique -- and then it could load the actual data on-demand. (I think the proactive mode is better to start with though, because dlopen/dlsym can fail in various ways, and it might be hard to get that to be something one can respond to if there is lazy loading. Long-term, that might mean that, if you want nice error handling, you need to ask for proactive loading).