embedding: support cybertron local embeddings

[corani]

Using https://github.com/nlpodyssey/cybertron you can generate embeddings locally in pure Go code. Not all models are supported, but I've successfully tested it with all-MiniLM-L6-v2, bge-small-en-v1.5 and e5-small-v2 (also base and large variants, but they get exponentially slower)

As this is CPU-based the performance isn't stellar, I'm getting about 20-30 embeddings per second on my laptop. Nevertheless, there are no dependencies on C-code or on CUDA, so it's very easy to get it to work.

Would it make sense to add an adapter for this in langchaingo?

Some sample code:

m, err := tasks.Load[textencoding.Interface](&tasks.Config{
    ModelsDir: "models",
    ModelName: "sentence-transformers/all-MiniLM-L6-v2",
})
if err != nil {
    panic(err)
}

result, err := m.Encode(ctx, text, int(bert.MeanPooling))
if err != nil {
    panic(err)
}

embedding := result.Vector.Normalize2().Data().F32()

Cybertron will automatically fetch the model from Huggingface and cache it locally in the ModelsDir. The last statement will give you an []float32 of the embedding.

[corani]

I had some idle time, so I've created a PR for it: #531

[tmc]

Very cool! Are you open to writing an example and perhaps a docs site guide?

[corani]

I have an example for the examples folder, but as the examples require a tagged version of langchaingo I can't submit it together with #531.

Let me know whether I should add it with a replace directive in go.mod or whether I should create a follow-up once #531 is merged and we have a new version.