call talib within numba

[untoreh]

Following numba docs, this example with ADX appears to be working.

the cython version exposes only the relevant inputs, and does some checks so it's not 1:1.

I don't think the cython version can be called directly because it wraps the C code.

Null elements appear at the end because before calling the indicator function the lookback period should be computed (by another ta_lib function, which this example is lacking) and the out pointer passed to the indicator function should be shifted accordingly

sooo..it's not just an import away :(

from numba import jit
import numba as nb
import numpy as np
from ctypes import *
import ctypes
import talib.abstract as ta

ta_lib_path = "/usr/local/lib/libta_lib.so"

talib = CDLL(ta_lib_path)

np.random.seed(42)
arr = np.random.uniform(0, 1, 100)

beg = np.full(1, 0).ctypes.data_as(c_void_p)
nbe = np.full(1, 0).ctypes.data_as(c_void_p)

high = arr.copy().ctypes.data_as(c_void_p)
low = high
close = high
np_high = arr.copy()
np_low = np_high
np_close = np_high

np_out = np.full(arr.shape[0], np.nan)
out = np_out.ctypes.data_as(c_void_p)
window = 20
start, end = 0, 99
c_window = c_int64(window)
c_start, c_end = c_int64(start), c_int64(end)

adx_fn = talib.TA_ADX
adx_fn.argtypes = [
    c_int64,
    c_int64,
    c_void_p,
    c_void_p,
    c_void_p,
    c_int64,
    c_void_p,
    c_void_p,
    c_void_p,
]
adx_fn.restype = c_int64

adx_fn_type = ctypes.CFUNCTYPE(adx_fn.restype, *adx_fn.argtypes)(adx_fn)

@nb.extending.intrinsic
def address_as_void_pointer(typingctx, src):
    """ returns a void pointer from a given memory address """
    from numba.core import types, cgutils

    sig = types.voidptr(src)

    def codegen(cgctx, builder, sig, args):
        return builder.inttoptr(args[0], cgutils.voidptr_t)

    return sig, codegen

@nb.njit
def ADX_jit(start, end, high_addr, low_addr, close_addr, window, beg, nbe, out_addr):
    high = address_as_void_pointer(high_addr)
    low = address_as_void_pointer(low_addr)
    close = address_as_void_pointer(close_addr)
    out = address_as_void_pointer(out_addr)
    return adx_fn_type(start, end, high, low, close, window, beg, nbe, out)

def ADX_cython(x):
    for _ in range(x):
        np_out[:] = ta.ADX(arr, arr, arr, timeperiod=window)
        
def ADX_ctypes(x):
    for _ in range(x):
        adx_fn_type(c_start, c_end, high, low, close, c_window, beg, nbe, out)

@nb.njit
def ADX_nb(x, start, end,np_high, np_low, np_close, window, np_out):
    for _ in range(x):
        ADX_jit(
            start,
            end,
            np_high.ctypes.data,
            np_low.ctypes.data,
            np_close.ctypes.data,
            window,
            np.full(1, 0).ctypes.data,
            np.full(1, 0).ctypes.data,
            np_out.ctypes.data,
        )
def ADX_numba(x):
    ADX_nb(x, start, end, np_high, np_low, np_close, window, np_out)

# x = 1
# print(np_out)
# ADX_cython(x)
# print(np_out)
# np_out[:] = np.nan
# ADX_jit(start, end, np_high.ctypes.data, np_low.ctypes.data, np_close.ctypes.data, window, np.full(1, 0).ctypes.data, np.full(1, 0).ctypes.data, np_out.ctypes.data,)
# print(np_out)

x = 100000
%timeit ADX_numba(x)
%timeit ADX_ctypes(x)
%timeit ADX_cython(x)

267 ms ± 15.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
419 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
1.15 s ± 24.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[polakowo]

Kudos for testing this out!

Being able to run talib from within njited code would fit perfectly to the idea of vectorbt, but as you see it seems to require just too much of effort. Although the current implementation in vectorbt uses an ugly pythonic loop to run talib on each column and some other tricks, it can match or sometimes overtake a comparable indicator written in pure numba, so the performance is already pretty solid. Where it could truly shine is building complex things like indicators on top of other indicators, so that's definitely a direction I'd love to move in. I have opened an issue in TA-Lib and there are tips to make it compatible, right now I'm a bit too busy with other features though.

[smjure]

@untoreh great work. Have you maybe managed to get none-null elements/outputs eventually? Thank you.

[quantfreedom]

@untoreh ... were you ever able to figure out how to get talib to work with numba?

[xyffar]

@polakowo have you ever figured out how to build indicators on top of other indicators?