Proper way to use GCToolkit with big files

[bric3]

Hi everyone, thanks for the work done at GCtoolkit.

So I have a setup with six rolling files of 20 MiB each

.rwxr--r--   17M brice.dutheil  4 Jun  2020  fixed-gc-logs-2020-05-29_18-13-46.log
.rwxr--r--   21M brice.dutheil  4 Jun  2020  fixed-gc-logs-2020-05-29_18-13-46.log.0
.rwxr--r--   21M brice.dutheil  4 Jun  2020  fixed-gc-logs-2020-05-29_18-13-46.log.1
.rwxr--r--   21M brice.dutheil  4 Jun  2020  fixed-gc-logs-2020-05-29_18-13-46.log.2
.rwxr--r--   21M brice.dutheil  4 Jun  2020  fixed-gc-logs-2020-05-29_18-13-46.log.3
.rwxr--r--   21M brice.dutheil  4 Jun  2020  fixed-gc-logs-2020-05-29_18-13-46.log.4

Parsing a single file works well with a heap of 512m, however parsing all six log files with a RotatingGCLogFile results in an OOM. Raising Xmx to 1024m did the trick.
And for the record the OOM can happen even without any aggregation being registered. So most allocation pressure seem to comes from the parser usually around Pattern.Matcher (new instance, .group(n), etc.)

I'm not familiar with gctoolkit architecture, but by any chance do you have any number to share, eg. is there a relation between the size of the data source and the heap size. With the all the various GC logs files I have in my possession 1024m seem to be OK, but maybe you encountered other situations.

• gclogs.zip
• gctoolkit-recordings.zip

[karianna]

@kcpeppe may have a rough calculation. We are certainly working on reducing the footprint here (double capture groups for regexes and the like)

[kcpeppe]

We know the performance of the parsers is not great but it’s always been good enough that other tasks have taken priority. I’m in the middle of a refactoring which will break the unwanted api to vertx dependency after which the performance of the parsers will be addressed.

…

On Fri, Sep 16, 2022 at 02:58 Martijn Verburg ***@***.***> wrote: @kcpeppe <https://github.com/kcpeppe> may have a rough calculation. We are certainly working on reducing the footprint here (double capture groups for regexes and the like) — Reply to this email directly, view it on GitHub <#233 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE2NJ2RXLTR7F7URDAQJY3V6RAFTANCNFSM6AAAAAAQNPOIII> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bric3]

Yes, I wasn't complaining, indeed it's good enough for me as well, I just wanted to confirm/discuss what's the best environment to run the GCToolkit.

Thanks a lot for reaching back !