~johanvandegriff/tensorflow-custom-build

ref: 6e63b2ba122b9ab04e03a60b63bc38a4e26b692f tensorflow-custom-build/README.md -rw-r--r-- 4.9 KiB
6e63b2baJohan Vandegriff added build for python 3.8 9 months ago

#The Issue

I was getting Illegal instruction (core dumped) when running tensorflow 2.4.1 on my computer. I found this github issue which said that newer versions of tensorflow require CPUs with the AVX instruction, which mine apparently does not have. The solution suggested in that thread was to build from source, which solved it for me, but took several hours to complete. So I decided to publish the build for anyone who has the same issue and trusts me to not modify the build. If you want better security, build it yourself using the process in this guide or the rest of this document.

#My Build

I built tensorflow 2.4.1 for python 3.6 and 3.8:

#For python 3.6:

tensorflow-2.4.1-cp36-cp36m-linux_x86_64.whl

And install it with

pip3 install tensorflow-2.4.1-cp36-cp36m-linux_x86_64.whl

#For python 3.8:

tensorflow-2.4.1-cp38-cp38-linux_x86_64.whl

And install it with

pip3 install tensorflow-2.4.1-cp38-cp38-linux_x86_64.whl

This was built with the flags -march=native -mcx16 for 64-bit linux on the following CPU:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       40 bits physical, 48 bits virtual
CPU(s):              16
On-line CPU(s) list: 0-15
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               26
Model name:          Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
Stepping:            5
CPU MHz:             1646.208
CPU max MHz:         2267.0000
CPU min MHz:         1600.0000
BogoMIPS:            4533.42
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            8192K
NUMA node0 CPU(s):   0-3,8-11
NUMA node1 CPU(s):   4-7,12-15
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm pti tpr_shadow vnmi flexpriority ept vpid dtherm

#Build Process

The following commands are what I ran based on this document. I also built this inside a docker container for my own convenience, but that is not needed. Note that you have to paste in the flags that are outputted from the first step into the ./configure when it asks you.

#find compiler flags to use later in ./configure
grep flags -m1 /proc/cpuinfo | cut -d ":" -f 2 | tr '[:upper:]' '[:lower:]' | { read FLAGS; OPT="-march=native"; for flag in $FLAGS; do case "$flag" in "sse4_1" | "sse4_2" | "ssse3" | "fma" | "cx16" | "popcnt" | "avx" | "avx2") OPT+=" -m$flag";; esac; done; MODOPT=${OPT//_/\.}; echo "$MODOPT"; }
#-march=native -mcx16

#apt dependencies
apt update
apt install -y python3-dev python3-pip #python 3.6

#pip dependencies
pip3 install -U --user pip numpy wheel
pip3 install -U --user keras_preprocessing --no-deps

#for python 3.8:
apt install -y python-3.8
python3.8 -m pip install -U --user pip numpy wheel
python3.8 -m pip install -U --user keras_preprocessing --no-deps

#install bazel
wget https://github.com/bazelbuild/bazel/releases/download/3.7.2/bazel-3.7.2-linux-x86_64
chmod +x bazel-3.7.2-linux-x86_64
cp bazel-3.7.2-linux-x86_64 /bin/bazel

## SKIP THIS SECTION ##
#this version of bazel was too new:
#wget https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-amd64
#chmod +x bazelisk-linux-amd64
#./bazelisk-linux-amd64
#cp /root/.cache/bazelisk/downloads/bazelbuild/bazel-4.0.0-linux-x86_64/bin/bazel /bin/bazel
## END OF SECTION TO SKIP ##

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow

git checkout r2.4
./configure
#for python 3.8, specify "/usr/bin/python3.8" for the python location
#IMPORTANT!!! enter the compiler flags when prompted "Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: "

#build the actual thing -- takes a while!
bazel build //tensorflow/tools/pip_package:build_pip_package

#build the pip package, takes only a few minutes
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

This will drop a file in /tmp/tensorflow_pkg, for me this was called tensorflow-2.4.1-cp36-cp36m-linux_x86_64.whl but it might be different for you, depending on which version you selected with git checkout r2.4. Save this file somewhere so you don't have to rebuild to get it again. You can install the file with pip, and tensorflow will work on your CPU without AVX support.