Memory efficiency of Cashvalue_ME_EX4

[MatthewCaseres]

Code like this seems to be having impacts on memory efficiency

    cl = np.array(list(claims(t, kind) for t in range(max_proj_len()))).transpose()

    return cl @ disc_factors()[:max_proj_len()]

I imagine that the logic described in this blog post (https://modelx.io/blog/2022/03/26/running-model-while-saving-memory/) will allow for some slight modifications to be made so that the memory efficiency is much improved.

[MatthewCaseres]

Did some more reading and I see that the blog post linked above reports that -

The maximum additional consumption is less then 9 GB, 30% of the original run, and the run time is actually shorter than the original run.

The implementation of the model appears to be O(P*T) where P=policies and T=timesteps. It might be possible to achieve savings of over 100x on memory by making the memory complexity O(P) by changing the way the calculation is done?

I am trying to make some publication for the benchmarks repository in a journal like Journal of Open Source Software. It will be discussing the memory and time complexity of actuarial modeling and we are focusing on the basic term and cashvalue_ME_EX4 models. A coauthor reports that lifelib is consuming more memory than Julia, and I believe it is related to this issue.

[fumitoh]

The vectorized models(those with an '_M' suffix), are designed to load extensive data onto memory. This approach allows us to pass large datasets to the native level, where homogeneous primitive operations can be applied simultaneously via vectorized or matrix operations. This is an area where numpy excels, though at the cost of significant memory consumption. So, this high memory usage is an expected outcome rather than an unforeseen issue.
Of course, after you've completed the necessary operations, you can free up the memory, retaining only the final results you ultimately require.

[MatthewCaseres]

I will do some experimenting and see if there is opportunity for improved memory consumption, and if it impacts performance

[MatthewCaseres]

Experiments indicate that matrix multiplication is not as performant as a loop, also an implementation of BasicTerm_M that is optimized for memory consumption:

https://github.com/actuarialopensource/benchmarks/blob/memory_optimizations/computation_graph/experiments.ipynb

[fumitoh]

Interesting. Perhaps this part is handling a significant amount of processing: result = np.array([pols_if(t) for t in range(policy_timesteps)]).transpose().

[MatthewCaseres]

Yes I think some of the processing is not necessary. The list comprehension, conversion from List[np.ndarray] to np.ndarray and transpose can all basically be avoided.

The for loop is not concerning because multiplications are still done between NumPy arrays, we don't loop over the policy dimension. The time dimension is not big, so looping over it in Python is not so much of a problem.

Not familiar enough with Modelx internals to know how to optimize from O(P*T) to O(P) memory consumption, but it seems possible

[fumitoh]

So, according to your tests, pv_loop significantly outperforms pv_matmul. I'll incorporate this finding into lifelib. Also, disc_factors doesn't necessarily have to be a matrix; they could be t-indexed vectors instead.

With respect to memory usage, each pv_* formula generates a large numpy array matrix. These matrices are not used and referenced after the formula execution, so they become eligible for memory deallocation.

Python has a garbage collector that identifies and frees unreferenced objects, but this process isn't instantaneous. So, memory consumption can remain high until the garbage collection is activated.

Fortunately, the gc module allows for manual control over the garbage collection process. As a matter of fact, timeit uses gc here to temporarily disable garbage collection to prevent it from interfering with time measurement.