Researchers from Yandex published two new articles on EMNLP on LLM assessment EMNLP 2026 is one of the two main NLP conferences along with ACL, and by the way, it has grown considerably in scope (this year the organizers received 17,669 applications compared to 8,000 last year). The topic of benchmarks is particularly relevant at this year's EMNLP. The old ones are saturated at a tremendous speed, and the industry is in dire need of fresh LLM diagnostic methods. One of the trends is gaming metrics. Games provide what static systems lack, namely testing strategic thinking, maintaining context, and multi-step thinking. MTGBench is only a bilingual gaming benchmark, consisting of seven multiplayer games, each in English and Russian, including Wordle, "Word Chain", "Dantaki", "20 Questions", "Field of Miracles", "20 Signs" and "Nickname". In addition to the content, the benchmark is interesting because all the moves of the model can be answered objectively. In four games, the result is checked by the algorithm; In the other three, the LLM judge strictly monitors compliance with the rules. The judge himself was independently verified separately. Leaderboard for 16 top tested models. Strong thinking models catch up or bypass the human baseline that the authors put together due to efficient play and getting the answer in fewer moves. The second article deals with evaluating machine translation. Its quality is improving, but mistakes in conveying meaning and unnatural phrasing still occur and are not always apparent on first reading. Finding them requires careful work by skilled experts, and it is difficult to expand such marking. The authors investigated whether language models can estimate expert-level translation, and how such an estimator should be designed. For the purpose of the study, we collected 600 English-Russian translations with three independent experts marking each of them and sequentially tested the effect of the instructions, methods of preparation and selection of examples, selection of models and integration of their reports.

Researchers from Yandex published two new articles on EMNLP on LLM assessment EMNLP 2026 is one of the two main NLP conferences along with ACL, and by the way, it has grown considerably in scope (this year the organizers received 17,669 applications compared to 8,000 last year). The topic of benchmarks is particularly relevant at this year's EMNLP. The old ones are saturated at a tremendous speed, and the industry is in dire need of fresh LLM diagnostic methods. One of the trends is gaming metrics. Games provide what static systems lack, namely testing strategic thinking, maintaining context, and multi-step thinking. MTGBench is only a bilingual gaming benchmark, consisting of seven multiplayer games, each in English and Russian, including Wordle, "Word Chain", "Dantaki", "20 Questions", "Field of Miracles", "20 Signs" and "Nickname". In addition to the content, the benchmark is interesting because all the moves of the model can be answered objectively. In four games, the result is checked by the algorithm; In the other three, the LLM judge strictly monitors compliance with the rules. The judge himself was independently verified separately. Leaderboard for 16 top tested models. Strong thinking models catch up or bypass the human baseline that the authors put together due to efficient play and getting the answer in fewer moves. The second article deals with evaluating machine translation. Its quality is improving, but mistakes in conveying meaning and unnatural phrasing still occur and are not always apparent on first reading. Finding them requires careful work by skilled experts, and it is difficult to expand such marking. The authors investigated whether language models can estimate expert-level translation, and how such an estimator should be designed. For the purpose of the study, we collected 600 English-Russian translations with three independent experts marking each of them and sequentially tested the effect of the instructions, methods of preparation and selection of examples, selection of models and integration of their reports.