[Graphbolt][Performance] Reduce the memory usage of preprocess_ondisk_dataset

[czkkkkkk]

🚀 Feature

Motivation

Currently, preprocess_ondisk_dataset consumes much more memory than the topology of a graph itself during the preprocessing. When loading a graph with 2B nodes and 8B edges, it cannot be finished in a machine with 380 GB memory. After a rough profiling, I found that the peak memory usage is reached when converting a DGL graph to a fused sampling graph.

dgl/python/dgl/graphbolt/impl/ondisk_dataset.py

Line 212 in 4ee0a8b

fused_csc_sampling_graph = from_dglgraph(

There could be two factors contributing to the peak memory usage.

1. The input DGL graph is passed to the function, which consumes about 160 GB memory.
2. from_dglgraph creates a temporary homogeneous graph and also its CSC format.

Alternatives

Pitch

Additional context

[Rhett-Ying]

@Skeleton003 could you look into it and try with the new implementation: #6986

[github-actions]

This issue has been automatically marked as stale due to lack of activity. It will be closed if no further activity occurs. Thank you