Problems with multiprocessing

[ironbar]

There is a problem with multiprocessing explained here. When calling a function on the main thread and after that launching another on multiprocessing the program hangs.

Apparently compiling using intel TBB solves the problem. "-D BUILD_TBB=ON -D WITH_TBB=ON"

Code that reproduces the error:

import cv2
import multiprocessing
import glob
import numpy

def job(path):

    image = cv2.imread(path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    return path

if __name__ == "__main__":

    main_image = cv2.imread("./image.png")
    main_image = cv2.cvtColor( main_image, cv2.COLOR_BGR2GRAY)

    paths = glob.glob("./data/*")
    pool = multiprocessing.Pool()
    result = pool.map(job, paths)

    print 'Finished'

    for value in result:
        print value

I have tried with all versions of opencv from conda-forge from 3.0 and it happens the same.

[jakirkham]

The morale of the story is multiprocessing typically uses POSIX fork on platforms that support it (i.e. Unixy). However fork often is problematic for a variety of reasons. Please see this blog for a good overview of these problems. With Qt, this is a known issue as demonstrated by this SO question. CPython developers provided some alternatives in Python 3 with contexts allowing users to choose spawn or forkserver instead. As noted in the thread, using spawn works. If you are using OpenCV 3 and Python 3, this would be something to consider. However if you are not, would encourage you to start thinking about how to migrate. Support for OpenCV 2 in conda-forge has been dwindling and Python 2 is already on the ropes.

[ironbar]

Thanks jakirkham for your great answer.

Meanwhile I have created a fork of opencv that works with intel tbb.
https://anaconda.org/ironbar/opencv
https://github.com/ironbar/opencv-feedstock

[sdvillal]

Compiling against conda-forge TBB package is possible (example). Just add TBB to requirements in meta.yaml and cmake it like this:

 -DCMAKE_CXX_FLAGS=-isystem\ ${PREFIX}/include
 -DWITH_TBB=ON
 -DBUILD_TBB=OFF
 -DTBBROOT=${PREFIX}

Sorry, not much time for PRs these days.