From 4ea73cdd7142071b1e244a4a897e3769990372a3 Mon Sep 17 00:00:00 2001 From: Li Bo Date: Fri, 8 Mar 2024 16:04:41 +0800 Subject: [PATCH] Update README.md --- README.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 8aede3289..b2067bce9 100644 --- a/README.md +++ b/README.md @@ -250,7 +250,23 @@ Please refer to our [documentation](docs/README.md). lmms_eval is a fork of [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness). We recommend you to read through the [docs of lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs) for relevant information. Below are the changes we made to the original API: - - Build context now only pass in idx and process image and doc during the model responding phase. This is due to the fact that dataset now contains lots of images and we can't store them in the doc like the original lm-eval-harness other wise the cpu memory would explode. - Instance.args (lmms_eval/api/instance.py) now contains a list of images to be inputted to lmms. - lm-eval-harness supports all HF language models as single model class. Currently this is not possible of lmms because the input/output format of lmms in HF are not yet unified. Thererfore, we have to create a new class for each lmms model. This is not ideal and we will try to unify them in the future. + +We also thank: +- [Jingkang Yang](https://jingkang50.github.io/), [Dong Guo](https://www.linkedin.com/in/dongguoset/) and [Sheng Shen](https://sincerass.github.io/) for early discussion and testing. + +## Citations + +```shell +@misc{lmms_eval2024, + title={LMMs-Eval: Accelerating the Development of Large-scale Multi-modality Models}, + url={https://github.com/EvolvingLMMs-Lab/lmms-eval}, + author={Bo Li*, Peiyuan Zhang*, Kaicheng Zhang*, Fanyi Pu*, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Yue Xiang, Chunyuan Li and Ziwei Liu}, + publisher = {Zenodo}, + version = {v0.1.0}, + month={March}, + year={2024} +} +```