Performance

[dnhkng]

Hi,

Is there any benchmarks on LLMs using bitsandbytes on Jetsons?

I´m very curious to know if this approach works, and how it compares to something like CPU based llama.cpp

[g588928812]

Yes. I've used "ai-benchmark"
https://ai-benchmark.com/ranking_deeplearning.html
https://pypi.org/project/ai-benchmark/

Here's the scores I got recently ("AI-Score", last column):
My MacBook Pro: 1086
Apple M1 Pro: 2391 (from a YT video)
Apple M1 Max: 2436 (from a YT video)
NVIDIA Jetson Xavier developer kit 32GB: 2860
RTX3090 on a Desktop PC: 56431
Google Colab (NVIDIA A100-SXM4-40GB) 57728

--> mediocre performance of the Jetson Xavier ;-)

[dnhkng]

I just got the Orin NX 16GB Module, I'll post the results. It has double the vida cores, but the NX models are not clocked as fast as the larger systems.