Second only to humans: SberDevices language models best in the world at Russian text comprehension

SberDevices' ruRoberta-large finetune text model received the best text comprehension score, coming second only to humans in terms of accuracy according to Russian SuperGLUE, the main Russian-language benchmark for evaluating large text models. Four more models designed by SberDevices made the top six: ruT5-large-finetune, ruBert-large finetune, ruT5-base-finetune, ruBert-base finetune.

Having successfully trained the ruBERT language model, Sber began to develop its more advanced version - ruRoBERTa. In terms of architecture, the advanced version is the same BERT trained on a large text corpus, but for the task of masked token recovery, featuring a large batch size and the BBPE tokenizer from the ruGPT-3 neural network. Training the model on the Christofari supercomputer took three weeks. The final dataset (250 GB of text) was similar to the one used for ruGPT-3, but with English and a section of the 'dirty' Common Crawl removed.

The Russian SuperGLUE (General Language Understanding Evaluation) leaderboard is the first neural network ranking for the Russian language. The ranking is based on how well a neural network completes tasks on logic, common sense, goal-setting, and meaning. It is an open project that is used by all data researchers working with Russian-language neural networks.

The overall language comprehension evaluation begins with the diagnostic dataset, a set of tests reflecting different linguistic phenomena. It shows the language's linguistic phenomena and how well the ruRoberta-large finetune model understands certain particularities. The model's high score (LiDiRus) means that it is not only memorizing tasks or guessing the result, but also learning the particularities and multitude of phenomena of the Russian language.

Each model is evaluated through a variety of tasks, including DaNetQA, a set of questions on common sense and knowledge with yes/no answers, RCB (Russian Commitment Bank), a classification of causal linkages between a text and a hypothesis, and PARus (Plausible Alternatives for Russian), which assesses goal-setting, choice of alternative options based on common sense, and more.

Sber's leading experts have spent several years perfecting Russian-language neural networks. For objective evaluation purposes, we developed the Russian SuperGLUE leaderboard, which is the first of its kind and clearly shows the progress being made. Our ultimate goal is the creation of reliable intelligent systems that will solve a variety of Russian-language tasks and will serve as the predecessors of robust, homegrown artificial intelligence.

David Rafalovsky

Executive Vice President, Sberbank; CTO, Head of Technology, Sber