Unity Multimodal Interface

Ummi for short, pronounced yummy.

Requirements

This project uses .NET packages

$ dotnet add package Microsoft.ML
$ dotnet add package Microsoft.ML.OnnxRuntime
$ dotnet add package Microsoft.ML.OnnxTransformer

If you init a new Unity project, simply use NuGetForUnity to install these dependencies.

Further instructions

• Download a sentence transformer model from huggingface (only all-MiniLM-L6-v2 has been tested for now).
  • You will need to download the entire ONNX folder, and place it inside the StreamingAssets folder of your project (Assets/StreamingAssets/<onnx-folder>).
  • Either update the path specified to the UmmiSTC Monobehaviour, or make sure the folder is named all-MiniLM-L6-v2 and contains model.onnx and vocab.txt files.
• Download a Whisper model and place it inside StreamingAssets/Whisper.
  • Again, make sure the path provided to the WhisperManager Monobehaviour matches your model path.

Example scene

Check this repository for an example scene.

Acknowledgements

Other dependencies used are

• Whisper.unity (MIT Licensed),
• BertTokenizers (MIT Licensed).

However, these were cloned inside the repo and therefore, there is no need to import them.

We also thank Sentence Transformers: Multilingual Sentence, Paragraph, and Image Embeddings using BERT & Co. for their work which inspired this project.

License

This project is licensed under the MIT License.