Integration in exllama v2

[xonfour]

Nice approach. I'm wondering if and how this could be implemented in exllama v2 and just started another discussion at turboderp-org/exllamav2#500

Let's see. ;)

[xonfour]

It looks like it was implemented for hidden states just a week ago. So before I start my experiments: Any recommendations on how to add the noise here?

[EGjoni]

There's already an experimental exllamav2 implementation I made for an oobabooga extension.

https://github.com/EGjoni/BrainHackingChip/tree/DRUGS

It's rough around the edges and I don't know what happened to the brainhackingchips author, but turboderp's hidden state hook implementation seem's to be getting at the same general idea, and I believe the hooks approach definitely has potential for inference time orthogonalization which he mentioned in the thread you linked to (as well as any number of other neat hacks).

Hopefully the code in that repo (along with the porting guide) helps with your implementation, but feel free to drop questions here. I'll keep an eye out on this thread and the one you linked to.

[xonfour]

I guess I need some help with the math ;-)
Please see the discussion at the exllamav2 repo.

This has a lot of potential for dynamic model tuning.