07-08-2024, 10:36 AM
Hi Antti-L, thanks for your response!
> Thanks for reporting your findings form the experiments. However, am I correct that the experiments were basically only about open source solver performance?
Yes! I should have paused to explain my thinking. As you say, all I have done here is to intercept the LP that is produced by TIMES / GAMS, loaded it into other languages, and checked how an open-source solver performs when called from these open languages. The results are indicative of overall performance only if the model generation part, currently done by GAMS, is a relatively small part of the total computation. (Dr Parzen pointed out at the ETSAP meeting that most of the computational work typically happens in the solver.)
In the case of TIMES-NZ (GAMS), model-generation is about the first 6s and roughly the next 70s is spent with the solver. So I should have reported the CPLEX time as 70s instead of 76. And hypothetically, if we did re-implement TIMES in Julia, the slowdown will be greater than reported here if model generation in Julia turns out to be slower than in GAMS. The assumption here is that since it's a small fraction (under 10 percent) of the time, the impact is less important - the goal here being a quick / rough estimate. (And translating TIMES looks harder!) For what it's worth, re-generating the model from its sparse matrix representation in Julia (attached script) took 2 seconds. Probably not that informative, but maybe slightly reassuring.
> I have understood those expected benefits to be referring to migrating the whole TIMES code to Julia (or some other open source modeling environment). Should that not be accomplished, GAMS would be needed anyway, and many open source solvers like HiGHS are already available under GAMS as a free solver options (no solver cost).
Yes, the cheap experiment I performed here doesn't address the much more challenging task of porting TIMES to Julia, and doesn't yield any of the benefits. But I was curious what the order of magnitude of performance change would be. Based on the evidence to hand, I suspect TIMES-NZ would take not much more than about 3x longer if we used Julia to do the model generation and HiGHs to do the solve. I can't speak for other modelling groups, especially those running much bigger models, but for TIMES-NZ this performance penalty would be outweighed by the benefits.
> ...I am a bit confused about this, because that would seem to suggest that you have already converted the full TIMES code into Julia and Python, but that would sound too good to be true! So, do you mean that the role of Julia and Python is here limited to reading the model from the file IMES_kea-v2_1_2.lp (written out by GAMS/Cplex) into a Julia/Python model object and then simply solving that with HiGHS? If so, do you in fact see some notable benefit of using Julia/Python for just that purpose (only the solver interface)?
Sorry, no, I haven't done the conversion, and like you, I can't see any benefit of what I've done here, except as a way of getting a feel for what the performance might be (subject to the assumptions discussed) if we did move the whole thing across to Julia/Python.
> I suspect that you may be missing the memory use of the Cplex solver process (which is not "gams.exe" but I guess would be gmsgennx.exe).
Ah - thanks! With the gmsgennx.exe process it looks like memory usage by GAMS / CPLEX may be in the same ballpark (gmsgennx.exe was using 645 MB for most of the solve) as what is seen with HiGHS called from the open-source languages.
> I think editing the solve.mod file is in general not a very good idea, but of course could be done for a quick experiment. A better option would be to modify the options file by vtrun.cmd (with user commands that can be added there automatically), if the writelp file should have the casename, or create a new version of the cplex.opt having the writelp option (if a generic writelp file name is sufficient when creating that file). Note also that GAMS convert can also convert the model into a Cplex lp format file.
Thanks! (Still learning)
> Anyway, in my view the main issue would be about migrating the TIMES code, while the open source solvers are in fact available to users regardless of that, unless I am missing something? Could you perhaps consider submitting a proposal to the related ETSAP call about a migration evaluation?
I'd be happy to submit this information to the migration evaluation (sorry, I don't see a thread for this), but as you point out all that I have done here is to get a rough estimate (in the case of TIMES-NZ) of the performance penalty of going open source. If others did similar exercises, we could build up a picture of how the community would be affected by a migration to Julia, possibly this has already been done. If so I'd be interested in the results! Sharing my (hacky) approach here was aimed to make that easier if others want to do the same experiment for their models.
Hope that makes my original post a bit clearer!
> Thanks for reporting your findings form the experiments. However, am I correct that the experiments were basically only about open source solver performance?
Yes! I should have paused to explain my thinking. As you say, all I have done here is to intercept the LP that is produced by TIMES / GAMS, loaded it into other languages, and checked how an open-source solver performs when called from these open languages. The results are indicative of overall performance only if the model generation part, currently done by GAMS, is a relatively small part of the total computation. (Dr Parzen pointed out at the ETSAP meeting that most of the computational work typically happens in the solver.)
In the case of TIMES-NZ (GAMS), model-generation is about the first 6s and roughly the next 70s is spent with the solver. So I should have reported the CPLEX time as 70s instead of 76. And hypothetically, if we did re-implement TIMES in Julia, the slowdown will be greater than reported here if model generation in Julia turns out to be slower than in GAMS. The assumption here is that since it's a small fraction (under 10 percent) of the time, the impact is less important - the goal here being a quick / rough estimate. (And translating TIMES looks harder!) For what it's worth, re-generating the model from its sparse matrix representation in Julia (attached script) took 2 seconds. Probably not that informative, but maybe slightly reassuring.
> I have understood those expected benefits to be referring to migrating the whole TIMES code to Julia (or some other open source modeling environment). Should that not be accomplished, GAMS would be needed anyway, and many open source solvers like HiGHS are already available under GAMS as a free solver options (no solver cost).
Yes, the cheap experiment I performed here doesn't address the much more challenging task of porting TIMES to Julia, and doesn't yield any of the benefits. But I was curious what the order of magnitude of performance change would be. Based on the evidence to hand, I suspect TIMES-NZ would take not much more than about 3x longer if we used Julia to do the model generation and HiGHs to do the solve. I can't speak for other modelling groups, especially those running much bigger models, but for TIMES-NZ this performance penalty would be outweighed by the benefits.
> ...I am a bit confused about this, because that would seem to suggest that you have already converted the full TIMES code into Julia and Python, but that would sound too good to be true! So, do you mean that the role of Julia and Python is here limited to reading the model from the file IMES_kea-v2_1_2.lp (written out by GAMS/Cplex) into a Julia/Python model object and then simply solving that with HiGHS? If so, do you in fact see some notable benefit of using Julia/Python for just that purpose (only the solver interface)?
Sorry, no, I haven't done the conversion, and like you, I can't see any benefit of what I've done here, except as a way of getting a feel for what the performance might be (subject to the assumptions discussed) if we did move the whole thing across to Julia/Python.
> I suspect that you may be missing the memory use of the Cplex solver process (which is not "gams.exe" but I guess would be gmsgennx.exe).
Ah - thanks! With the gmsgennx.exe process it looks like memory usage by GAMS / CPLEX may be in the same ballpark (gmsgennx.exe was using 645 MB for most of the solve) as what is seen with HiGHS called from the open-source languages.
> I think editing the solve.mod file is in general not a very good idea, but of course could be done for a quick experiment. A better option would be to modify the options file by vtrun.cmd (with user commands that can be added there automatically), if the writelp file should have the casename, or create a new version of the cplex.opt having the writelp option (if a generic writelp file name is sufficient when creating that file). Note also that GAMS convert can also convert the model into a Cplex lp format file.
Thanks! (Still learning)
> Anyway, in my view the main issue would be about migrating the TIMES code, while the open source solvers are in fact available to users regardless of that, unless I am missing something? Could you perhaps consider submitting a proposal to the related ETSAP call about a migration evaluation?
I'd be happy to submit this information to the migration evaluation (sorry, I don't see a thread for this), but as you point out all that I have done here is to get a rough estimate (in the case of TIMES-NZ) of the performance penalty of going open source. If others did similar exercises, we could build up a picture of how the community would be affected by a migration to Julia, possibly this has already been done. If so I'd be interested in the results! Sharing my (hacky) approach here was aimed to make that easier if others want to do the same experiment for their models.
Hope that makes my original post a bit clearer!