Use too much memory when use repodata cache.

[way-zer]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

# clean install with 0.41.4
pixi global install gdu #first download, no cache
pixi global install rclone #get OOM, VmRSS 857 MiB
# pixi global update # get same OOM

rm .cache/rattler/cache/repodata -rf
pixi global install rclone #OK, no cache

Issue description

Maybe repodata cache problem. It seems too large, and maybe fully load into memory.

 --- /root/.cache/rattler/cache/repodata ---
                         /..
  308.1 MiB ███████▏  ▏3018e552.json
  118.3 MiB ██▋       ▏36521c4e.json
    4.0 KiB           ▏36521c4e.info.json
    4.0 KiB           ▏3018e552.info.json
        0 B           ▏36521c4e.lock
        0 B           ▏3018e552.lock

Expected behavior

As a package manager, it should memory<100MB or same as no cache.

[ruben-arts]

Are you using the https://prefix.dev/conda-forge as your default channel already?

You can set that with:

pixi config set --global default-channels '["https://prefix.dev/conda-forge"]'
pixi config set --global repodata-config.disable-sharded false 

This implements a much faster and less memory intensive repodata fetching technology.

Edit: it should have been the disable-sharded config

[way-zer]

Are you using the https://prefix.dev/conda-forge as your default channel already?

It's anaconda source, and the first time(without cache) is OK.

You can set that with:

pixi config set --global default-channels '["https://prefix.dev/conda-forge"]'
pixi config set --global repodata-config."https://prefix.dev".disable-zstd true 

This implements a much faster and less memory intensive repodata fetching technology.

➜  ~ pixi config set --global repodata-config."https://prefix.dev".disable-zstd true
Error:
  × Unknown key: repodata-config.https://prefix.dev.disable-zstd
  │ Supported keys:

EDIT:
default-channels seems not affect pixi global update

 --- /root/.cache/rattler/cache/repodata/3018e552.info.json ---
{
  "url": "https://conda.anaconda.org/conda-forge/linux-64/repodata.json.zst",

[ruben-arts]

That is a surprising result... @baszalmstra I'm I wrong in thinking that the sharded repodata should improve the RAM usage?

[way-zer]

It seem not use sharded repodata. Try with fast.prefix.dev, only use 57MB (prefix.dev used 720MB)

➜  third /bin/time -v pixi global install -c https://fast.prefix.dev/conda-forge rclone
Global environments as specified in '/root/.pixi/manifests/pixi-global.toml'
└── rclone: 1.69.1 (installed)
    └─ exposes: rclone
        Command being timed: "pixi global install -c https://fast.prefix.dev/conda-forge rclone"
        User time (seconds): 2.33
        System time (seconds): 0.71
        Percent of CPU this job got: 28%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.80
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 57740
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 21455
        Voluntary context switches: 28139
        Involuntary context switches: 95
        Swaps: 0
        File system inputs: 0
        File system outputs: 179912
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
➜  third /bin/time -v pixi global install -c https://prefix.dev/conda-forge rclone 
Global environments as specified in '/root/.pixi/manifests/pixi-global.toml'
└── rclone: 1.69.1 (installed)
    └─ exposes: rclone
        Command being timed: "pixi global install -c https://prefix.dev/conda-forge rclone"
        User time (seconds): 6.65
        System time (seconds): 4.50
        Percent of CPU this job got: 52%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.12
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 720492
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 50182
        Voluntary context switches: 173441
        Involuntary context switches: 230
        Swaps: 0
        File system inputs: 0
        File system outputs: 1240080
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[wolfv]

Did you configure pixi to use sharded repodata?

[way-zer]

Did you configure pixi to use sharded repodata?

No, all default installation.

[ruben-arts]

Could you retry with setting pixi config set --global repodata-config.disable-sharded false first?

[way-zer]

Could you retry with setting pixi config set --global repodata-config.disable-sharded false first?

It works. Is it default values problem?

[ruben-arts]

Because it is a feature that could create a lot of issues if it doesn't work absolutely perfect we've made it not the default. But we're seeing a great amount of users have already switched so I think it would be a good time to make it the default strategy now.