We introduce an advanced Russian general language understanding evaluation benchmark.
Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models andan overall leaderboard of transformer modelsfor the Russian language.
You can see our documentation at diagnostics description
- LiDiRus link
- RCB link
- PARus link
- MuSeRC link
- TERRa link
- RUSSE link
- RWSD link
- DaNetQA link
- RuCoS link
TBA Submitted at EMNLP