06-08-2024, 06:30 AM
Hi TIMES modelling community,
After the conversation at the Bonn meeting around open-source tools + solvers I was curious what the time and memory penalty of moving away from GAMS / CPLEX would be for the very small TIMES-NZ model. I am with those who argued there would be big benefits if we could migrate to more open and modern tools - if we can get them to work for us.
I had a bit of a play - I'm coming at this without much experience of optimization problems at this scale, and I know the performance of different algorithms can depend on details of the problem, so I'm not sure how this would generalize to other / larger models.
In any case the results, although a little confusing, seem to corroborate what was said in the conversation. I did a simple experiment for TIMES-NZ (kea scenario) and got the following results (not attempting to be scientific, just ballpark):
===================================
Running time for TIMES-NZ scenario kea-v2_1_2
VEDA (GAMS/CPLEX solver) running kea-v2_1_2 : 76 seconds * 7 threads
Julia (HiGHS solver) running kea-v2_1_2 : 242 seconds * 6 threads
Python (HiGHS solver) running kea-v2_1_2 : 405 seconds
Open-source solver relative slowdown : 3 to 5 times longer (approx)
===================================
Memory usage for TIMES-NZ scenario kea-v2_1_2
On-disk kea-v2_1_2, saved as an mps file : 85 MB
VEDA (GAMS/CPLEX solver) running kea-v2_1_2 : 8 MB # used by GAMS process. How can this be? Am I missing some of the memory usage?
Julia (HiGHS solver) running kea-v2_1_2 : 2426 MB # total memory used by julia process. Not clear how much of this is due to the model. The size of the model object in memory is 159 MB according to Julia.
Python (HiGHS solver) running kea-v2_1_2 : 400 MB # total memory used by python process.
Open-source memory usage increase : 50 to 300 times more (approx)
===================================
I'm a bit confused by how much less memory is being used by GAMS / CPLEX while solving the problem.
First thought was perhaps the CPLEX solver does a more effective presolve? But from the logging it looks like Highs is just as successful. Presolved Problem Specifications:
GAMS/CPLEX:
Rows: 166,606
Columns: 139,519
Nonzeros: 1,192,044
HiGHS:
Rows: 166,922
Columns: 140,319
Nonzeros: 1,196,177
A few things I tried made little difference to the results:
- Pulled out the Matrix representation of the LP and create a new JuMP model, attempting to suppress variable naming (result: same size and solve time)
- Implemented the model as a Convex.jl optimization problem and tried solving it with SCS (Bad idea. Way too slow!)
Will
PS - in case anyone else wants to try profiling their TIMES model in other languages, here's a crude way: the following steps are one way to ensure a "writelp" line gets added at the bottom of the cplex.opt file when you run TIMES using VEDA. The "writelp" line tells cplex to dump the LP to disk.
In my case the line that gets added is "writelp TIMES_kea-v2_1_2.lp", because my scenario is called kea-v2_1_2.
1) Go to the latest TIMES model file directory (GAMS_SrcTIMES.vA.B.C) present in your VEDA directory
2) Open the file solve.mod using a text editor like VSCode.
3) Find a line (e.g. for TIMES v4.8.1, it's line 33) that looks like:
$ IF %METHOD% == P $SETLOCAL METHOD LP
4) Immediately below it, insert the following six lines:
DISPLAY "Configure cpex to export an MPS file for use with other solvers.";
$onecho >> cplex.opt
* Keep the line empty above this one - it puts a newline where it's needed!
writelp %MODEL_NAME%_%RUN_NAME%.lp
$offecho
5) Run your TIMES scenarios using VEDA as usual.
6) After running TIMES, in the relevant scenario directory e.g. GAMS_WrkTIMES\kea-v2_1_2, you should find a new ".lp" file that contains the full model problem. In my case it's called TIMES_kea-v2_1_2.lp
After the conversation at the Bonn meeting around open-source tools + solvers I was curious what the time and memory penalty of moving away from GAMS / CPLEX would be for the very small TIMES-NZ model. I am with those who argued there would be big benefits if we could migrate to more open and modern tools - if we can get them to work for us.
I had a bit of a play - I'm coming at this without much experience of optimization problems at this scale, and I know the performance of different algorithms can depend on details of the problem, so I'm not sure how this would generalize to other / larger models.
In any case the results, although a little confusing, seem to corroborate what was said in the conversation. I did a simple experiment for TIMES-NZ (kea scenario) and got the following results (not attempting to be scientific, just ballpark):
===================================
Running time for TIMES-NZ scenario kea-v2_1_2
VEDA (GAMS/CPLEX solver) running kea-v2_1_2 : 76 seconds * 7 threads
Julia (HiGHS solver) running kea-v2_1_2 : 242 seconds * 6 threads
Python (HiGHS solver) running kea-v2_1_2 : 405 seconds
Open-source solver relative slowdown : 3 to 5 times longer (approx)
===================================
Memory usage for TIMES-NZ scenario kea-v2_1_2
On-disk kea-v2_1_2, saved as an mps file : 85 MB
VEDA (GAMS/CPLEX solver) running kea-v2_1_2 : 8 MB # used by GAMS process. How can this be? Am I missing some of the memory usage?
Julia (HiGHS solver) running kea-v2_1_2 : 2426 MB # total memory used by julia process. Not clear how much of this is due to the model. The size of the model object in memory is 159 MB according to Julia.
Python (HiGHS solver) running kea-v2_1_2 : 400 MB # total memory used by python process.
Open-source memory usage increase : 50 to 300 times more (approx)
===================================
I'm a bit confused by how much less memory is being used by GAMS / CPLEX while solving the problem.
First thought was perhaps the CPLEX solver does a more effective presolve? But from the logging it looks like Highs is just as successful. Presolved Problem Specifications:
GAMS/CPLEX:
Rows: 166,606
Columns: 139,519
Nonzeros: 1,192,044
HiGHS:
Rows: 166,922
Columns: 140,319
Nonzeros: 1,196,177
A few things I tried made little difference to the results:
- Pulled out the Matrix representation of the LP and create a new JuMP model, attempting to suppress variable naming (result: same size and solve time)
- Implemented the model as a Convex.jl optimization problem and tried solving it with SCS (Bad idea. Way too slow!)
Will
PS - in case anyone else wants to try profiling their TIMES model in other languages, here's a crude way: the following steps are one way to ensure a "writelp" line gets added at the bottom of the cplex.opt file when you run TIMES using VEDA. The "writelp" line tells cplex to dump the LP to disk.
In my case the line that gets added is "writelp TIMES_kea-v2_1_2.lp", because my scenario is called kea-v2_1_2.
1) Go to the latest TIMES model file directory (GAMS_SrcTIMES.vA.B.C) present in your VEDA directory
2) Open the file solve.mod using a text editor like VSCode.
3) Find a line (e.g. for TIMES v4.8.1, it's line 33) that looks like:
$ IF %METHOD% == P $SETLOCAL METHOD LP
4) Immediately below it, insert the following six lines:
DISPLAY "Configure cpex to export an MPS file for use with other solvers.";
$onecho >> cplex.opt
* Keep the line empty above this one - it puts a newline where it's needed!
writelp %MODEL_NAME%_%RUN_NAME%.lp
$offecho
5) Run your TIMES scenarios using VEDA as usual.
6) After running TIMES, in the relevant scenario directory e.g. GAMS_WrkTIMES\kea-v2_1_2, you should find a new ".lp" file that contains the full model problem. In my case it's called TIMES_kea-v2_1_2.lp