feat(xpress): add direct io

[CharlieFModo]

Closes # (if applicable).

Changes proposed in this Pull Request

• Adds xpress direct solver IO
• To ensure xpress direct IO is faster than lp-polars this PR removes pandas from the hot-path of matrix operations required to get problem coefficients to write to xpress
  • This seems to have large performance gains throughout the package beyond what I have tested for this PR
  • I would like some guidance on whether these changes are ok in the broader context of the package. I've only tested them on the problems that I am interested in solving and found this gives a c. 10x improvement from the lp-polars IO

Checklist

• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Unit tests for new features were added (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the MIT license.

[coroa]

Hi @CharlieFModo,

thanks, i can see quite some performance impacts (ie. up to 30% improvement for a typical highs direct io model i am sometimes using), with your changes here.

So, i'd like very much if we were able to include it, but we will have to be very careful about the memory of these changes.

1. We had memory spikes in earlier versions, which is why we introduced the whole iterate_slices of the constraints objects.
2. You are putting quite a lot of data into these variable_data and constraint_data cached properties .

I don't understand the impact of these changes to well yet, but the speed-up gain is very promising.

[coroa]

We probably will need to restructure the package a bit as more and more alternative io paths are crowding into io.py. Maybe some of it should go into solvers or we start an io directory with specific sub modules.

[CharlieFModo]

Thanks @coroa - what would you like the scope of this PR to be to include the changes? I'd prefer to avoid refactoring the package if possible and just focus on adding the xpress IO and these performance changes I needed to make to make it a large enough improvement from lp-polars.

I'm hopeful these changes should be pretty respectful of memory because the parts of code that I've refactored work with the raw ndarray data which I guess should already be in memory as part of the model. It avoid loading them into pandas dataframes, which are really awful for consuming memory. Not 100% though because I haven't profiled it but I can have a look at adding iterate_slices style logic if you think that would be better. Can you suggest a benchmark that would indicate if this is relevant? One of the pypsa models?