@Olex : Yes, it was already much better, on my computer the whole cycle took now about 35 secs:
Define sets: 0.071533 seconds (90.67 k allocations: 6.188 MiB, 35.27% gc time, 97.53% compilation time)
Load demo data (sets and parameters): 4.993969 seconds (14.83 M allocations: 1.004 GiB, 8.33% gc time, 92.74% compilation time)
Compute additional parameters: 0.161982 seconds (370.70 k allocations: 25.610 MiB, 98.97% compilation time)
Compute additional sets: 1.584292 seconds (3.08 M allocations: 212.797 MiB, 2.28% gc time, 99.04% compilation time)
Create Variables: 1.606490 seconds (9.44 M allocations: 739.344 MiB, 28.36% gc time, 55.39% compilation time)
Objective Function: 0.240101 seconds (190.58 k allocations: 13.097 MiB, 26.72% gc time, 98.79% compilation time)
Specify constraints: 23.129035 seconds (477.64 M allocations: 22.244 GiB, 6.23% gc time, 22.82% compilation time)
Running HiGHS 1.7.2 (git hash: 5ce7a2753): Copyright © 2024 HiGHS under MIT licence terms
Presolve : Reductions: rows 228(-376); columns 245(-1393846); elements 826(-1094)
Solving the presolved LP
Model status : Optimal
Objective value : 3.8706151237e+04
HiGHS run time : 0.38
Solve model: 2.802878 seconds (8.42 M allocations: 725.451 MiB, 10.65% gc time, 3.70% compilation time: 56% of which was recompilation)
However, there were still lots of unnecessary variables (1.4 million vars in total, while there should be only about 650 variables. In my 2018 experiment, I used the following kind of tighter specs:
@variable(m,PrcAct[r in REGION,t in MILEYR,p in [p for p in PROCESS if RTP_VARA[r,t,p]],v in RTP_VNT[r,t,p],s in PRC_TS[r,p]] >=0)
@variable(m,PrcFlo[r in REGION,t in MILEYR,p in [p for p in PROCESS if RP_FLO[r,p] && RTP_VARA[r,t,p]],v in RTP_VNT[r,t,p],c in RPC[r,p],s in RPC_TS[r,p,c]] >=0)
@variable(m,IreFlo[r in REGION,t in MILEYR,p in [p for p in PROCESS if RP_IRE[r,p] && RTP_VARA[r,t,p]],v in RTP_VNT[r,t,p],c in RPC[r,p],s in PRC_TS[r,p],ie in IMPEXP; RPC_IRE[r,p,c,ie]] >=0)
Of course, the various control attributes used there had to be created before that, but they could be used in many constraints as well. But I agree that Julia works unexpectedly poorly with respect to variables declaration (compared to e.g. GAMS), if one needs to use complex conditionals to avoid lots of spurious entries.
On my computer, the Utopia model is generated by GAMS in just 0.05 secs, and solved in 0.02 secs by Cplex, and so the performance gap is still high. Looks like there is a lot of room for improvement in the constraint formulation (e.g. the memory usage suggests so, 22.244 GiB, if I understand it correctly). I think one should get it at least under 1 second...
Define sets: 0.071533 seconds (90.67 k allocations: 6.188 MiB, 35.27% gc time, 97.53% compilation time)
Load demo data (sets and parameters): 4.993969 seconds (14.83 M allocations: 1.004 GiB, 8.33% gc time, 92.74% compilation time)
Compute additional parameters: 0.161982 seconds (370.70 k allocations: 25.610 MiB, 98.97% compilation time)
Compute additional sets: 1.584292 seconds (3.08 M allocations: 212.797 MiB, 2.28% gc time, 99.04% compilation time)
Create Variables: 1.606490 seconds (9.44 M allocations: 739.344 MiB, 28.36% gc time, 55.39% compilation time)
Objective Function: 0.240101 seconds (190.58 k allocations: 13.097 MiB, 26.72% gc time, 98.79% compilation time)
Specify constraints: 23.129035 seconds (477.64 M allocations: 22.244 GiB, 6.23% gc time, 22.82% compilation time)
Running HiGHS 1.7.2 (git hash: 5ce7a2753): Copyright © 2024 HiGHS under MIT licence terms
Presolve : Reductions: rows 228(-376); columns 245(-1393846); elements 826(-1094)
Solving the presolved LP
Model status : Optimal
Objective value : 3.8706151237e+04
HiGHS run time : 0.38
Solve model: 2.802878 seconds (8.42 M allocations: 725.451 MiB, 10.65% gc time, 3.70% compilation time: 56% of which was recompilation)
However, there were still lots of unnecessary variables (1.4 million vars in total, while there should be only about 650 variables. In my 2018 experiment, I used the following kind of tighter specs:
@variable(m,PrcAct[r in REGION,t in MILEYR,p in [p for p in PROCESS if RTP_VARA[r,t,p]],v in RTP_VNT[r,t,p],s in PRC_TS[r,p]] >=0)
@variable(m,PrcFlo[r in REGION,t in MILEYR,p in [p for p in PROCESS if RP_FLO[r,p] && RTP_VARA[r,t,p]],v in RTP_VNT[r,t,p],c in RPC[r,p],s in RPC_TS[r,p,c]] >=0)
@variable(m,IreFlo[r in REGION,t in MILEYR,p in [p for p in PROCESS if RP_IRE[r,p] && RTP_VARA[r,t,p]],v in RTP_VNT[r,t,p],c in RPC[r,p],s in PRC_TS[r,p],ie in IMPEXP; RPC_IRE[r,p,c,ie]] >=0)
Of course, the various control attributes used there had to be created before that, but they could be used in many constraints as well. But I agree that Julia works unexpectedly poorly with respect to variables declaration (compared to e.g. GAMS), if one needs to use complex conditionals to avoid lots of spurious entries.
On my computer, the Utopia model is generated by GAMS in just 0.05 secs, and solved in 0.02 secs by Cplex, and so the performance gap is still high. Looks like there is a lot of room for improvement in the constraint formulation (e.g. the memory usage suggests so, 22.244 GiB, if I understand it correctly). I think one should get it at least under 1 second...