Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

configure / make fail to find MKL on Ubuntu 20.04 #4262

Closed
4gra opened this issue Sep 8, 2020 · 11 comments · Fixed by #4350
Closed

configure / make fail to find MKL on Ubuntu 20.04 #4262

4gra opened this issue Sep 8, 2020 · 11 comments · Fixed by #4350
Assignees
@kkm000
Labels
enhancement waiting-for-feedback Reporter's feedback has been requested

Comments

@4gra
Copy link

4gra commented Sep 8, 2020

Steps to reproduce:

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
[...]
$ sudo apt install intel-mkl-full gfortran libsox-dev [...etc]
$ git clone https://github.com/kaldi-asr/kaldi.git
$ ls -l /usr/lib/x86_64-linux-gnu/libmkl_*
$ ls -l /usr/include/mkl/
  (i.e. they exist)
$ cd tools
$ extras/check_dependencies.sh
./extras/check_dependencies.sh: Intel MKL is not installed. Run extras/install_mkl.sh to install it.
 [...]

I realised the approval of check_dependencies wasn't required to actually make it work, so I just went ahead and made the tools, which worked fine. But then MKL hit a snag again on building kaldi proper:

$ cd ../src
$ ./configure --shared
    (doesn't work)
$ ./configure --shared --mkl-libdir=/usr/lib/x86_64-linux-gnu
   (also doesn't work)
$ CXXFLAGS="-I/usr/include/mkl/" ./configure --shared --mkl-libdir=/usr/lib/x86_64-linux-gnu
MKL configured libs: -L/usr/lib/x86_64-linux-gnu -Wl,-rpath=/usr/lib/x86_64-linux-gnu -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -ldl -lpthread -lm
MKL include directory: -I/usr/lib/x86_64-linux-gnu/../../include
Using Intel MKL as the linear algebra library.
mkl-test.cc:21:10: fatal error: mkl.h: No such file or directory
   21 | #include <mkl.h>
      |          ^~~~~~~
compilation terminated.
***configure failed: Cannot validate the MKL switches ***

so, I patched ./configure:

diff --git a/src/configure b/src/configure
index e0ac02115..da1253e26 100755
--- a/src/configure
+++ b/src/configure
@@ -266,6 +266,8 @@ configure_mkl_includes() {
   local mklroot=$1 mkllibdir=$2
   if [[ -d $mklroot/include ]]; then
     echo "-I$1/include"
+  elif [[ -d /usr/include/mkl ]]; then
+    echo "-I/usr/include/mkl"
   elif [[ -d $mkllibdir/../../include ]]; then
     echo "-I$mkllibdir/../../include"
   else

Now configure succeeds:

$ CXXFLAGS="-I/usr/include/mkl" ./configure --shared --mkl-libdir=/usr/lib/x86_64-linux-gnu
Configuring KALDI to use MKL.
 [...]
MKL configured libs: -L/usr/lib/x86_64-linux-gnu -Wl,-rpath=/usr/lib/x86_64-linux-gnu -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -ldl -lpthread -lm
MKL include directory: -I/usr/include/mkl
Using Intel MKL as the linear algebra library.
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
 [...]

but make then failed as it passed in a mandatory reference to an include directory "$(MKLROOT)/include". So, I must confess at this point I lost patience and patched kaldi.mk rather than figure out how it's made:

--- src/kaldi.mk        2020-09-08 20:50:00.000000001 +0100
+++ src/kaldi.mk        2020-09-08 20:51:00.903004252 +0100
@@ -49,17 +49,12 @@
 ifndef OPENFSTLIBS
 $(error OPENFSTLIBS not defined.)
 endif
-ifndef MKLROOT
-$(error MKLROOT not defined.)
-endif
-
-MKLLIB ?= $(MKLROOT)/lib/intel64
 
 CXXFLAGS = -std=c++11 -I.. -isystem $(OPENFSTINC) -O1 $(EXTRA_CXXFLAGS) \
            -Wall -Wno-sign-compare -Wno-unused-local-typedefs \
            -Wno-deprecated-declarations -Winit-self \
            -DKALDI_DOUBLEPRECISION=$(DOUBLE_PRECISION) \
-           -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_MKL -I$(MKLROOT)/include \
+           -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_MKL -I/usr/include/mkl \
            -m64 -msse -msse2 -pthread \
            -g

having done this, make finally succeeded and kaldi appears to have been built completely successfully.

So at the least, the auto-detection is broken, and at worst the build process simply can't cope with the include/library structure imposed by an upstream Ubuntu install of intel-mkl.
@4gra 4gra added the bug label Sep 8, 2020
@danpovey
Copy link
Contributor

danpovey commented Sep 9, 2020 via email

@danpovey
Copy link
Contributor

danpovey commented Oct 5, 2020

@kkm000 any time to work on this?

@kkm000
Copy link
Contributor

kkm000 commented Nov 10, 2020 via email

@kkm000
Copy link
Contributor

kkm000 commented Nov 10, 2020 •
edited
Loading

@4gra, is it critical to use the repackaged MKL that comes with Ubuntu? It is much better to remove the intel-mkl-full Ubuntu package, and use our script extra/install_mkl.sh from the Intel's repository.

@kkm000 kkm000 added waiting-for-feedback Reporter's feedback has been requested discussion and removed bug labels Nov 10, 2020
@4gra
Copy link
Author

4gra commented Nov 11, 2020 •
edited
Loading

It's not crucial, but I administer a large system which deploys these packages across many machines and it would make my life significantly easier if it gets along with as many upstream packages as possible. If not I'd have to distribute the MKL twice on those systems, which is obviously not great.

@kkm000
Copy link
Contributor

kkm000 commented Nov 11, 2020 via email

@kkm000 kkm000 self-assigned this Nov 11, 2020
@kkm000 kkm000 removed the waiting-for-feedback Reporter's feedback has been requested label Nov 11, 2020
@4gra
Copy link
Author

4gra commented Nov 12, 2020

@kkm000 (slightly off-topic but I somewhat agree. The deluge of updates for their own sake can be counterproductive and we hold many packages, MKL included, for a stable environment. We intend to stick to 20.04 LTS for some time, I hope its rate of churn will reduce over time.)

@kkm000
Copy link
Contributor

kkm000 commented Nov 15, 2020 via email

kkm000 added a commit to kkm000/kaldi that referenced this issue Nov 23, 2020
@kkm000
[configure] Recognize MKL installed from Ubuntu package
e89f1bc 
Ubuntu 20.04+ packages an unknown 20.x MKL version. The installation
in /opt/intel is still preferred, but if none were found or supplied
by the user, the package is probes as well, the last in order of
probing.

Closes kaldi-asr#4262
@kkm000 kkm000 mentioned this issue Nov 23, 2020
Merged
@kkm000 kkm000 added enhancement in progress Issue has been taken and is being worked on and removed discussion labels Nov 24, 2020
kkm000 added a commit that referenced this issue Nov 24, 2020
@kkm000
[configure] Recognize MKL installed from Ubuntu package [#4262] (#4350)
58c4323 
Ubuntu 20.04+ packages an unknown 20.x MKL version. The installation
in /opt/intel is still preferred, but if none were found or supplied
by the user, the package is probed as well, the last in order of
probing.

Closes #4262
@4gra
Copy link
Author

4gra commented Nov 25, 2020

Thanks for your careful consideration.

@kkm000
Copy link
Contributor

kkm000 commented Nov 25, 2020 via email

@4gra
Copy link
Author

4gra commented Nov 25, 2020

I'll be doing a slew of software updates in the next week or so, so I'll get back to you.

@kkm000 kkm000 added waiting-for-feedback Reporter's feedback has been requested and removed in progress Issue has been taken and is being worked on labels Nov 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kkm000 kkm000

Labels
enhancement waiting-for-feedback Reporter's feedback has been requested
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[configure] Recognize MKL installed from Ubuntu package (#4262) kkm000/kaldi
3 participants
@4gra @danpovey @kkm000