We are running a set of experiments using AMPL with different solvers: Gurobi, IPOPT, and Knitro. Our goal is to benchmark their performance on a common problem. However, we are running into difficulties when trying to compare computational times in a consistent and meaningful way.
It seems that the times reported back to AMPL (via solve_result or solve_message) differ not only in format but also in what they actually represent. For instance:
Gurobi exposes “Work” units, which are deterministic and repeatable, but we don’t see a way to access this from AMPL.
IPOPT and Knitro report CPU times, but they vary from run to run, even with the same input and settings.
solve_result_num sometimes gives inconsistent or implementation-specific values depending on the solver.
Does anyone have advice or best practices on how to compare solver times in AMPL?
Is there a reliable way to extract solver CPU time or wall time for different solvers in a uniform way from AMPL?
Any help or suggestions would be greatly appreciated.
There are two solver time values available from AMPL, in the following built-in parameters:
solve_time reports the CPU time
solve_elapsed_time reports the wall-clock time
These do not include any time spent in the AMPL process, but only time spent in the (separate) solver process. You can show them with a display or print statement after the solver has finished.
When evaluating solvers that may run multiple threads in parallel — for example, Gurobi and Knitro — solve_time does not reflect the effectiveness of the parallelization, because it is the sum of CPU times on all the individual threads. Thus you should rely on solve_elapsed_time, which will of course be most reliable if you can use a computer that does not have any other time-consuming processes running at the same time.
The Gurobi work units are only useful for comparing different Gurobi options. Gurobi’s Performance page has links to some interesting discussions of how to make performance comparisons, and some parts of those discussions are even useful for comparing other solvers.