Resolve other_indexes

[olejandro]

This is a follow-up on #225

Currently other_indexes is used to fill in varios dimensions, e.g. io or ie, through a mapping. A one-to-one relation is assumed, however multiple dimensions may be present (i.e. separated with ~). We can resolve other_indexes into multiple columns to handle this (i.e. in addition to cg). However, the majority of the cells in those columns will be empty.

[olejandro]

@siddharth-krishna is there an efficient way of handling this? E.g. should we use None or pd.NA or anything else to optimise memory usage?

[siddharth-krishna]

Sorry for the delay, Olex. Yes, I think we should use pd.NA for now as it will allow us to use other inbuilt pandas functions easily. If we run into memory issues (perhaps we should start monitoring this on our biggest model?) then we can think about using sparse data frames: https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html

If you'd like me to look into implementing this, please could you point me to the relevant bits of the code? And should we resolve other_indexes into existing columns, or create new columns when splitting on ~? Thanks

[olejandro]

Thanks @siddharth-krishna! Actually, removing unused columns and dropping tables that are no longer used has helped with the memory issues.
I'll start working in resolving other_indexes into multiple columns soon (at the moment we create cg from it, but for some attributes only). Let's see if we run into any memory issues then.