I’m having a problem with highs.linux-intel64.20221228. It works okay if by hand I run “highs -v” or “highs -?” but when the Python-based app I’m working with tries to invoke it via Pyomo (BTW that works with other solvers) it errors out opaquely but this shows up in dmesg:
[714305.886477] traps: highsamp[13107] trap invalid opcode ip:8158c2 sp:7fff8678a9a0 error:0 in highs[400000+5bb000]
It’s been suggested to me that this sort of thing is due to a mismatch between compile target and CPU; in this case it’s an Intel Xeon E5420 (x86_64) and to save anyone from having to look up its flags, here they are:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm pti dtherm
If It is the case that this HiGHS binary was compiled for new hardware than I’ve got, is it possible to obtain it built with e.g. mtune=generic or is there a source tree such that I can use cmake/make on it myself?
Thanks!
See “Building from source” here: Developer documentation — ampl::mp 20211021 documentation
You run
cmake .. -DBUILD=highsmp -DHIGHS_INCLUDE_DIRS=(your dir) -DHIGHS_LIBS=…../libhighs.a
OK, thank you - I’m trying this but running into a problem at the make stage.
I followed the process you linked me to:
git clone https://github.com/ampl/mp.git
cd mp
git checkout develop # For newest code (but possibly work-in-progress)
git submodule init
git submodule update
mkdir build # Create a directory to hold the build output
cd build
And then my interpretation of the cmake invocation you posted was
cmake .. -DBUILD=highsmp -DHIGHS_INCLUDE_DIRS=/usr/local/include/highs -DHIGHS_LIBS=/usr/local/lib64/libhighs.a
Everything up to here ran without incident. I’d already built HiGHS on this machine so those include/lib paths check out. Then I assumed the next stop is to invoke make. However, here is how that ends:
.
.
.
[ 90%] Linking CXX executable ../bin/nl-reader-test
[ 90%] Built target nl-reader-test
[ 91%] Linking CXX static library ../lib/libamplhighsmp-static.a
[ 91%] Built target amplhighsmp-static
[ 92%] Building C object solvers/CMakeFiles/highsmp-lib.dir/highsmp/highsmp-lib.c.o
[ 93%] Building CXX object solvers/CMakeFiles/highsmp-lib.dir/__/src/solver.cc.o
[ 95%] Building CXX object solvers/CMakeFiles/highsmp.dir/highsmp/main.cc.o
[ 96%] Linking CXX shared library ../bin/libhighsmp-lib.so
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /usr/local/lib64/libhighs.a(HighsTaskExecutor.cpp.o): relocation R_X86_64_TPOFF32 against `__tls_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status
make[2]: *** [solvers/CMakeFiles/highsmp-lib.dir/build.make:117: bin/libhighsmp-lib.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:1208: solvers/CMakeFiles/highsmp-lib.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 97%] Linking CXX executable ../bin/highsmp
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /usr/local/lib64/libhighs.a(HMPSIO.cpp.o): in function `zstr::istreambuf::~istreambuf()':
HMPSIO.cpp:(.text._ZN4zstr10istreambufD2Ev[_ZN4zstr10istreambufD5Ev]+0x26): undefined reference to `deflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZN4zstr10istreambufD2Ev[_ZN4zstr10istreambufD5Ev]+0x71): undefined reference to `inflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /usr/local/lib64/libhighs.a(HMPSIO.cpp.o): in function `zstr::istreambuf::~istreambuf()':
HMPSIO.cpp:(.text._ZN4zstr10istreambufD0Ev[_ZN4zstr10istreambufD5Ev]+0x29): undefined reference to `deflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZN4zstr10istreambufD0Ev[_ZN4zstr10istreambufD5Ev]+0x79): undefined reference to `inflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /usr/local/lib64/libhighs.a(HMPSIO.cpp.o): in function `zstr::istreambuf::underflow()':
HMPSIO.cpp:(.text._ZN4zstr10istreambuf9underflowEv[_ZN4zstr10istreambuf9underflowEv]+0xdf): undefined reference to `inflate'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZN4zstr10istreambuf9underflowEv[_ZN4zstr10istreambuf9underflowEv]+0x240): undefined reference to `inflateInit2_'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZN4zstr10istreambuf9underflowEv[_ZN4zstr10istreambuf9underflowEv]+0x267): undefined reference to `deflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZN4zstr10istreambuf9underflowEv[_ZN4zstr10istreambuf9underflowEv]+0x29a): undefined reference to `inflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZN4zstr10istreambuf9underflowEv[_ZN4zstr10istreambuf9underflowEv]+0x2b1): undefined reference to `deflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZN4zstr10istreambuf9underflowEv[_ZN4zstr10istreambuf9underflowEv]+0x2c1): undefined reference to `inflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /usr/local/lib64/libhighs.a(HMPSIO.cpp.o): in function `std::unique_ptr<zstr::detail::z_stream_wrapper, std::default_delete<zstr::detail::z_stream_wrapper> >::~unique_ptr()':
HMPSIO.cpp:(.text._ZNSt10unique_ptrIN4zstr6detail16z_stream_wrapperESt14default_deleteIS2_EED2Ev[_ZNSt10unique_ptrIN4zstr6detail16z_stream_wrapperESt14default_deleteIS2_EED5Ev]+0x13): undefined reference to `deflateEnd'
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: HMPSIO.cpp:(.text._ZNSt10unique_ptrIN4zstr6detail16z_stream_wrapperESt14default_deleteIS2_EED2Ev[_ZNSt10unique_ptrIN4zstr6detail16z_stream_wrapperESt14default_deleteIS2_EED5Ev]+0x21): undefined reference to `inflateEnd'
collect2: error: ld returned 1 exit status
make[2]: *** [solvers/CMakeFiles/highsmp.dir/build.make:101: bin/highsmp] Error 1
make[1]: *** [CMakeFiles/Makefile2:1235: solvers/CMakeFiles/highsmp.dir/all] Error 2
[ 98%] Linking CXX executable ../bin/converter-mip-test
[ 98%] Built target converter-mip-test
In file included from /home/jeff/mp/test/converter-flat-test.cpp:4:
/home/jeff/mp/test/converter-flat-test.h: In instantiation of ‘{anonymous}::InterfaceTesterWithBackendAcceptingConstraints<Constraint>::InterfaceTesterWithBackendAcceptingConstraints() [with Constraint = mp::CustomFunctionalConstraint<std::vector<int>, std::array<int, 0>, mp::NumericFunctionalConstraintTraits, mp::MaxConstraintId>]’:
/home/jeff/mp/test/converter-flat-test.cpp:11:1: required from here
/home/jeff/mp/test/converter-flat-test.h:91:16: warning: member ‘{anonymous}::InterfaceTesterWithBackendAcceptingConstraints<mp::CustomFunctionalConstraint<std::vector<int>, std::array<int, 0>, mp::NumericFunctionalConstraintTraits, mp::MaxConstraintId> >::env_’ is used uninitialized [-Wuninitialized]
91 | interface_(env_) { }
| ^~~~
/home/jeff/mp/test/converter-flat-test.h: In instantiation of ‘{anonymous}::InterfaceTesterWithBackendAcceptingConstraints<Constraint>::InterfaceTesterWithBackendAcceptingConstraints() [with Constraint = mp::AlgebraicConstraint<mp::QuadAndLinTerms, mp::AlgConRange>]’:
/home/jeff/mp/test/converter-flat-test.cpp:25:1: required from here
/home/jeff/mp/test/converter-flat-test.h:91:16: warning: member ‘{anonymous}::InterfaceTesterWithBackendAcceptingConstraints<mp::AlgebraicConstraint<mp::QuadAndLinTerms, mp::AlgConRange> >::env_’ is used uninitialized [-Wuninitialized]
[100%] Linking CXX executable ../bin/converter-flat-test
[100%] Built target converter-flat-test
make: *** [Makefile:146: all] Error 2
Can you tell what went wrong here?
Actually I link libhighs.so (Highs builds SHARED=on by default). On up-to-date Kubuntu 22.04, with current Highs master. If with .so it still fails, provide your system config and Highs setup.
OK, I’m starting to get somewhere. I redid my build of HiGHS from source today because I wanted to include -mtune=native in the compiler options and I’ve done that. However, this time …/build/lib64/libhighs.so did not get created but it had previously. If I tried building mp using the previous build’s …/build/lib64/libhighs.so for -DHIGHS_LIBS= instead of /usr/local/lib64/libhighs.a, the make finished normally. Even though my modeling app errored out when invoking highs/mp, at least i had a binary to execute. Can you tell me what controls whether or not …/build/lib64/libhighs.so gets created versus just libhighs.a?
It would be the CMake option -DSHARED=on. It’s actually on by default, see CMake. You might need to wipe out the build folder and rerun configuration.
What do you mean by “rerun configuration?” As described on the process is this:
git clone https://github.com/ERGO-Code/HiGHS.git
cd HiGHS
mkdir build
cd build
cmake -DFAST_BUILD=ON ..
cmake --build . (NOTE: I've been adding "-j 9 -v" here to speed things up and verify -mtune=native)
followed by (as root)
cmake --install .
Are you saying to change “cmake -DFAST_BUILD=ON …” to “cmake -DFAST_BUILD=ON -DSHARED=on …” and if that addition is the default, why would anything about the HiGHS build operation behave any differently than what I’m describing?
“Rerun CMake config” means delete CMakeCache.txt in the build directory, and all build files. To make it easiest, best to clean the build directory. The reason to do this from time to time is that some settings can get messed up. In particular, your previous settings implied SHARED=off.
Moreover for me FAST_BUILD=on did not work, something failed during compilation or linking.
Suggest:
cd build
rm -rf *
cmake .. ## SHARED=on by default
cmake --build . -- -j 1000
sudo make install
I have a partial answer to myself. if in the HiGHS build directory I do “cmake …” instead of “cmake -DFAST_BUILD=ON …” then I’ll be left with …/lib64/libhighs.so but then ctest fails in several places like this:
The following tests FAILED:
3 - unit_tests_all (ILLEGAL)
5 - osi_unit_tests_all (SEGFAULT)
111 - lseu--presolve=off (ILLEGAL)
112 - lseu--presolve=on (ILLEGAL)
113 - lseu--random_seed=1 (ILLEGAL)
114 - lseu--random_seed=2 (ILLEGAL)
115 - lseu--random_seed=3 (ILLEGAL)
116 - egout--presolve=off (ILLEGAL)
117 - egout--presolve=on (ILLEGAL)
.
.
.
And it continues on like that all the way down to #150. That is without having done anything to add compiler flags. I do wind up with …/lib64/libhighs.so etc. instead of just …/lib64/libhighs.a when I leave off -DFAST_BUILD=ON but how do I know …/lib64/libhighs.so isn’t defective somehow going forward? I think I need to take that up with the HiGHS folks.
If it produces wrong code without FAST_BUILD, this is weird: ILLEGAL means wrong CPU instruction? I’d say check your compiler.
Okay, I have a really important update to this; I think my problem seems to be solved if not every problem but here’s what’s happened so far.
First, HiGHS, as provided in the form of its developers’ git repository, is set up to be built with the compiler (gcc c++ or LLVM clang++) option -mpopcnt enabled if the CPU is x86-64 even if the target cpu doesn’t have the feature. So if you go into the cloned repo and comment out the line that does that in CMakeLists.txt like so:
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86\_64|i686)")
#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mpopcnt")
then …/build/highs will pass ctest. I’ve bee doing this on Gentoo Linux but I think this problem is OS-orthogonal on x86-64.
Having gotten past that, in the HiGHS build directory I ran
cmake --install .
as root so that HiGHS stuff would get put in the intended places. Then I moved on to the mp build for HiGHS and referencing the previous posts, what finally worked was
cmake .. -DBUILD=highsmp -DHIGHS_INCLUDE_DIRS=/usr/local/include/highs -DHIGHS_LIBS=/home/jeff/HiGHS/build/li
b64/libhighs.so
The result didn’t pass all of ctest either:
2 - util-test (Failed)
7 - expr-test (Subprocess aborted)
8 - expr-visitor-test (Failed)
10 - nl-reader-test (Failed)
14 - solver-test (SEGFAULT)
20 - converter-flat-test (Failed)
21 - converter-mip-test (Failed)
But I soldiered on regardless, running the modeling system with highsmp as the solver and it worked on a drastically-cut-down version of the dataset. I’m running it on the full dataset now.
-mpopcnt seems to need SSE 4.2.
The unit tests are not maintained. We use end2end testing, see Testing — ampl::mp 20211021 documentation.