OMX

[bstabler]

Have you considered the work done under the OMX project?

[PeterKucirek]

Oh indeed! I tested using OMX as my matrix-caching format however found the implementation to be much too slow for what I needed. However, ideally I would like to include an OMX-to-DataFrame reader in this project, when I have some time.

[bstabler]

Are you reading and writing to matrix rows/cols/cells from/to the disk during your model run? If so, then, yes, OMX is too slow. It is designed more for entire matrix I/O at the start or end of a model run.

[PeterKucirek]

I do each matrix in one shot, but in general, yes I'm doing within a model run.

The other thing to note is that this package converts these formats to a DataFrame which I believe is a different output format from the current OMX library (which IIRC returns a numpy array).

[bstabler]

so just reading or writing the entire matrix from OMX was too slow? Typically you just need to read, say, skims, at the start of the demand model, so if it takes 5 or 10 minutes, then that is acceptable since the demand model takes hours. Are you doing something different? I'm curious what your use case is and why OMX doesn't work. Thanks.

Yes OMX returns a numpy array not a pandas DataFrame.