Language model benchmark

A language model benchmark is a standardized test designed to evaluate the performance of language models on various natural language processing tasks. These tests are intended for comparing different models' capabilities in areas such as language understanding, generation, and reasoning. As language models began to power AI agents, AI agent benchmarks evolved out of language model benchmarks.

Benchmarks generally consist of a dataset and corresponding evaluation metrics. The dataset provides text samples and annotations, while the metrics measure a model's performance on tasks like answering questions, text classification, and machine translation. These benchmarks are developed and maintained by academic institutions, research organizations, and industry players to track progress in the field. In addition to accuracy, the metrics can include throughput, energy efficiency, bias, trust, and sustainability.

Overview

Types

Benchmarks may be described by the following adjectives, not mutually exclusive:

Classical: These tasks are studied in natural language processing, even before the advent of deep learning. Examples include the Penn Treebank for testing syntactic and semantic parsing, as well as bilingual translation benchmarked by BLEU scores.
Question answering: These tasks have a text question and a text answer, often multiple-choice. They can be open-book or closed-book. Open-book QA resembles reading comprehension questions, with relevant passages included as annotation in the question, in which the answer appears. Closed-book QA includes no relevant passages. Closed-book QA is also called open-domain question-answering. Before the era of large language models, open-book QA was more common, and understood as testing information retrieval methods. Closed-book QA became common since GPT-2 as a method to measure knowledge stored within model parameters.
Omnibus: An omnibus benchmark combines many benchmarks, often previously published. It is intended as an all-in-one benchmarking solution.
Index: An index is not a benchmark, but a summary of many benchmarks. It is intended to be interpreted as an overall score of models's "general performance".
Reasoning: These tasks are usually in the question-answering format, but are intended to be more difficult than standard question answering.
Multimodal: These tasks require processing not only text, but also other modalities, such as images and sound. Examples include OCR and transcription.
Agency: These tasks are for a language-model–based software agent that operates a computer for a user, such as editing images, browsing the web, etc.
Adversarial: A benchmark is "adversarial" if the items in the benchmark are picked specifically so that certain models do badly on them. Adversarial benchmarks are often constructed after state of the art (SOTA) models have saturated (achieved 100% performance) a benchmark, to renew the benchmark. A benchmark is "adversarial" only at a certain moment in time, since what is adversarial may cease to be adversarial as newer SOTA models appear.
Public/Private: A benchmark might be partly or entirely private, meaning that some or all of the questions are not publicly available. The idea is that if a question is publicly available, then it might be used for training, which would be "training on the test set" and invalidate the result of the benchmark. Usually, only the guardians of the benchmark have access to the private subsets, and to score a model on such a benchmark, one must send the model weights, or provide API access, to the guardians.

The boundary between a benchmark and a dataset is not sharp. Generally, a dataset contains three "splits": training, test, and validation. Both the test and validation splits are essentially benchmarks. In general, a benchmark is distinguished from a test/validation dataset in that a benchmark is typically intended to be used to measure the performance of many different models that are not trained specifically for doing well on the benchmark, while a test/validation set is intended to be used to measure the performance of models trained specifically on the corresponding training set. In other words, a benchmark may be thought of as a test/validation set without a corresponding training set.

Conversely, certain benchmarks may be used as a training set, such as the English Gigaword or the One Billion Word Benchmark, which in modern language is just the negative log-likelihood loss on a pretraining set with 1 billion words. Indeed, the distinction between benchmark and dataset in language models became sharper after the rise of the pretraining paradigm, whereby a model is first trained on massive, unlabeled datasets to learn general language patterns, syntax, and knowledge (pretraining), and the base model is then adapted to specific, downstream tasks using smaller, labeled datasets (fine-tuning).

Lifecycle

Generally, the life cycle of a benchmark consists of the following steps:

Inception: A benchmark is published. It can be simply given as a demonstration of the power of a new model (implicitly) that others then picked up as a benchmark, or as a benchmark that others are encouraged to use (explicitly).
Growth: More papers and models use the benchmark, and the performance on the benchmark grows.
Maturity, degeneration or deprecation: A benchmark may be saturated, after which researchers move on to other benchmarks. Progress on the benchmark may also be neglected as the field moves to focus on other benchmarks.
Upgrade: A benchmark may be upgraded to remove bad items or replace them with good items. A saturated benchmark may be upgraded to make it no longer saturated. Upgrading may be versioned or continuous.

Construction

Like datasets, benchmarks are typically constructed by several methods, individually or in combination:

Web scraping: Ready-made question-answer pairs may be scraped online, such as from websites that teach mathematics and programming.
Conversion: Items may be constructed programmatically from scraped web content, such as by blanking out named entities from sentences, and asking the model to fill in the blank. This was used for making the CNN/Daily Mail Reading Comprehension Task.
Crowd sourcing: Items may be constructed by paying people to write them, such as on Amazon Mechanical Turk. This was used for making the MCTest.
Items may be constructed anew specifically for the benchmark. This is expensive and may result in less realistic items, but decreases contamination risk.

Some benchmarks contain provenence for their items.

Evaluation

Generally, benchmarks are fully automated. This limits the questions that can be asked. For example, with mathematical questions, "proving a claim" would be difficult to automatically check, while "calculate an answer with a unique integer answer" or "write a proof for the statement that passes a proof checker" would be automatically checkable. With programming tasks, the answer can generally be checked programmatically using a test fixture, by running unit tests, with an upper limit on runtime.

The benchmark scores are of the following kinds:

For multiple choice or cloze questions, common scores are accuracy (frequency of correct answer), precision, recall, F1 score, etc.
pass@n: The model is given $n$ attempts to solve each problem. If any attempt is correct, the model earns a point. The pass@n score is the model's average score over all problems.
k@n: The model makes $n$ attempts to solve each problem, but only $k$ attempts out of them are selected for submission. If any submission is correct, the model earns a point. The k@n score is the model's average score over all problems.
cons@n: The model is given $n$ attempts to solve each problem. If the most common answer is correct, the model earns a point. The cons@n score is the model's average score over all problems. Here "cons" stands for "consensus" or "majority voting".

The pass@n score can be estimated more accurately by making $N>n$ attempts, and use the unbiased estimator $1-{\frac {\binom {N-c}{n}}{\binom {N}{n}}}$ , where $c$ is the number of correct attempts.

For less well-formed tasks in language processing, where the output can be any sentence, there are the following commonly used scores including BLEU ROUGE, METEOR, NIST, word error rate, LEPOR, CIDEr, and SPICE. With the rise of large language models, open-ended language tasks can be scored by LLM-as-a-judge.

With the advent of reasoning language models, an AI model may spend a variable amount of cost on the same item to improve performance. Consequently, benchmark scores for reasoning language models are typically interpreted on a cost-performance curve, and models on the Pareto frontier are considered Pareto-optimal.

Issues

error: Some benchmark answers may be wrong.
ambiguity: Some benchmark questions may be ambiguously worded.
subjective: Some benchmark questions may not have an objective answer at all. This problem generally prevents creative writing benchmarks. Similarly, this prevents benchmarking writing proofs in natural language, though benchmarking proofs in a formal language is possible.
open-ended: Some benchmark questions may not have a single answer of a fixed size. This problem generally prevents programming benchmarks from using more natural tasks such as "write a program for X", and instead uses tasks such as "write a function that implements specification X".
inter-annotator agreement: Some benchmark questions may be not fully objective, such that even people would not agree with 100% on what the answer should be. This is common in natural language processing tasks, such as syntactic annotation.
indiscrimination: Some benchmark items may fail to distinguish capability between models. That is, models score at the item may be uncorrelated with actual model capability. This may happen if the item is too easy, too difficult, or essentially random.
shortcut: Some benchmark questions may be easily solved by an "unintended" shortcut. For example, in the SNLI benchmark, having a negative word like "not" in the second sentence is a strong signal for the "Contradiction" category, regardless of what the sentences actually say.
contamination/leakage: Some benchmark questions may have answers already present in the training set. Also called "training on the test set" or "memorizing the answers". Some benchmarks (such as Big-Bench) may use a "canary string", so that documents containing the canary string can be voluntarily removed from the training set.
saturation: As time goes on, many models reach the highest performance level practically possible, and so the benchmark can no longer differentiate these models. For example, GLUE had been saturated, necessitating SuperGLUE.
Goodhart's law: If new models are designed or selected to score highly on a benchmark, the benchmark may cease to be a good indicator for model quality.
cherry picking: New model publications may only point to benchmark scores on which the new model performed well, avoiding benchmark scores that it did badly on.
noise: Benchmark scores may have high noise, making interpretation difficult. This is particularly a problem when a benchmark is too costly to run multiple times.

List of benchmarks

General language modeling

Essentially any dataset can be used as a benchmark for statistical language modeling, with the perplexity (or near-equivalently, negative log-likelihood and bits per character, as in the original Shannon's test of the entropy of the English language) being used as the benchmark score. For example, the original GPT-2 announcement included those of the model on WikiText-2, enwik8, text8, and WikiText-103 (all being standard language datasets made from the English Wikipedia).

However, there have been datasets more commonly used, or specifically designed, for use as a benchmark.

One Billion Word Benchmark: The negative log likelihood loss on a dataset of 1 billion words.
Paloma (Perplexity Analysis for Language Model Assessment): A collection of English and code texts, divided into 546 domains. Used to measure the perplexity of a model on specific domains.
Penn Treebank: The error or negative log likelihood loss for part-of-speech tags on a dataset of text.

General language understanding

See for a review of over 100 such benchmarks.

CNN/Daily Mail Reading Comprehension Task: Articles from CNN (380K training, 3.9K development, 3.2K test) and Daily Mail (879K training, 64.8K development, 53.2K test) were scraped. The bullet point summaries accompanying the news articles were used. One entity in a bullet point was replaced with a placeholder, creating a cloze-style question. The goal is to identify the masked entity from the article.
CoLA (Corpus of Linguistic Acceptability): 10,657 English sentences from published linguistics literature that were manually labeled either as grammatical or ungrammatical.
HellaSwag (Harder Endings, Longer contexts, and Low-shot Activities for SWAG): A harder version of SWAG. Contains 10K items.
LAMBADA: 10,000 narrative passages from books, each with a missing last word that humans can guess if given the full passage but not from the last sentence alone.
MultiNLI (Multi-Genre Natural Language Inference): Similar to SNLI, with 433K English sentence pairs from ten distinct genres of written and spoken English.
RACE (ReAding Comprehension Examinations): 100,000 reading comprehension problems in 28,000 passages, collected from the English exams for middle and high school Chinese students in the age range between 12 and 18.
SNLI (Stanford Natural Language Inference: 570K human-written English sentence pairs manually labeled for balanced classification with 3 labels "entailment", "contradiction", and "neutral".
SWAG (Situations With Adversarial Generations): 113K descriptions of activities or events, each with 4 candidate endings; the model must choose the most plausible ending. Adversarial against a few shallow language models (MLP, bag of words, one-layer CNN, etc).
WSC (Winograd schema challenge): 273 sentences with ambiguous pronouns. The task is to determine what the pronoun refers to.
WinoGrande: A larger version of WSC with 44,000 items. Designed to be adversarial to 2019 SOTA, since the original had been saturated. This dataset consists of fill-in-the-blank style sentences, as opposed to the pronoun format of previous datasets.
WMT 2014 (Workshop on Statistical Machine Translation): a collection of 4 machine translation benchmarks at the Ninth Workshop on Statistical Machine Translation. The Attention Is All You Need paper used it as a benchmark.

General language generation

CharXiv: 9292 descriptive questions (examining basic chart elements) and 2323 reasoning questions (synthesizing information across complex visual elements) about 2323 charts from scientific papers.
IFEval (Instruction-Following Eval): 541 instructions to be followed, each containing at least one verifiable constraint, such as "mention the keyword of AI at least 3 times".
LMArena (formerly Chatbot Arena): Human users vote between two outputs from two language models. An Elo rating for each language model is computed based on these human votes. The same organization released Arena Hard in 2024, containing 500 user queries with a right answer (math problems, programming problems, etc) and 250 creative writing queries sourced from Chatbot Arena.
MT-Bench (multi-turn benchmark): An automated version of Chatbot Arena where LLMs replace humans in generating votes.
MultiChallenge: 273 instances. Each instance is a multi-turn (up to 10 turns) conversation history between two parties, ending with a final user turn containing a requirement/question. Designed to test for instruction-following, context allocation, and in-context reasoning at the same time. Scored by LLM as judge with instance-level rubrics.
NaturalInstructions: 61 distinct tasks with human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema.
Super-NaturalInstructions: 1,616 diverse NLP tasks and their expert-written instructions, and 5M task instances.

Open-book question-answering

ARC (AI2 Reasoning Challenge): Multiple choice questions, with a Challenge Set (2590 questions) and an Easy Set (5197 questions). Designed specifically to be adversarial against models that had saturated SNLI and SQuAD.
ChartQA: 32,719 questions about 20,882 charts crawled from four diverse online sources (Statista, Pew Research Center, Our World In Data, OECD). Of these, 9,608 were human-written (in ChartQA-H), and 23,111 were machine-generated (in ChartQA-M). The answers are either verbatim texts from the chart or integers calculated based on the chart's data.
CoQA (Conversational QA): 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains.
DocVQA: multimodal, 50,000 questions on 12,767 document images, sectioned from 6,071 distinct documents. The documents were sourced from 5 industries (tobacco, food, drug, fossil fuel, chemical) of the UCSF Industry Documents Library, mostly from the 1940-2010 period. Documents with structured elements like tables, forms, lists, and figures were prioritized. The answers are verbatim extracts from the document text.
DROP (Discrete Reasoning Over the content of Paragraphs): 96,567 questions along with Wikipedia passages, especially from narratives rich in numerical information (like sports summaries and history), often involving multi-step numerical reasoning over several text spans. Adversarial against 2019 SOTA.
GRS-QA: Graph Reasoning-Structured Question Answering Dataset. A dataset designed to evaluate question answering models on graph-based reasoning tasks.
HotpotQA: 113K multi-hop questions that require reading multiple Wikipedia-based passages to answer. They were produced by showing crowd workers multiple supporting context documents and asking them to produce questions that requiring reasoning about all of the documents.
MCTest (Machine Comprehension Test): 500 fictional stories, each with 4 multiple-choice questions (with at least 2 requiring multi-sentence understanding), designed to be understandable by a 7-year-old. The vocabulary was limited to approximately 8,000 words probably known by a 7-year-old. The stories were written by workers on Amazon Mechanical Turk.
Natural Questions: 323045 items. Each containing a question that had been searched on Google, a Wikipedia page relevant for answering the question, a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or "null" if no long/short answer is present.
OpenBookQA: 5960 multiple choice questions, each coming with an elementary level science fact (the "open book"). There are 1329 such facts in total.
SearchQA: 140,461 question-answer pairs from the J! Archive, with each pair augmented with (on average 50) snippets and urls obtained by searching the question on Google.
SQuAD (Stanford Question Answering Dataset): 100,000+ questions posed by crowd workers on 500+ Wikipedia articles. The task is, given a passage from Wikipedia and a question, find a span of text in the text that answers the question.

SQuAD 2.0: 50,000 unanswerable questions that look similar to SQuAD questions. Every such unanswerable question must be answered with an empty string. Written by crowd workers.

StrategyQA: 2,780 questions annotated with relevant passages from Wikipedia, such that the question require multi-hop reasoning over the passages to answer. For example, "Did Aristotle use a laptop?" is annotated with passages from the Wikipedia pages for "laptop" and "Aristotle".
WebQuestions: 6,642 question-answer pairs designed to be answerable with knowledge present in the 2013 version of Freebase.

Closed-book question-answering

C-Eval (Chinese Eval): 13948 multiple choice questions about in 52 subjects at 4 levels of difficulty. In Chinese.
MedQA: 61097 questions from professional medical board exams, in English, Simplified Chinese, Traditional Chinese.
OpenEQA (Open Embodied QA): over 1600 questions accompanying about videos, scans of real-world environments, and simulations.
PIQA (Physical Interaction QA): 17951 two-choice questions. Each question gives a goal (like separating egg yolk from egg white with a water bottle), and 2 choices for accomplishing it.
RealWorldQA: 765 multimodal multiple-choice questions. Each containing an image and a question. Designed to test spatial understanding. Images are drawn from various real-world scenarios, including those captured from vehicles.
ScienceQA: 21208 multiple choice questions in natural science, social science, and linguistics, with difficulty level from grade 1 to grade 12, sourced from elementary and high school science curricula. Some questions require reading a diagram. Most questions are annotated with lecture textual lectures and explanations.
SimpleQA: 4,326 short questions that are answerable with knowledge as of 2023. Each answer is graded as either "correct", "incorrect", or "not attempted". Adversarial against GPT-4 specifically.
TruthfulQA: 817 questions in health, law, finance and politics with common misconceptions. Adversarial against GPT-3 and T5.

Omnibus

Some benchmarks are "omnibus", meaning they are made by combining several previous benchmarks.

Big-Bench (Beyond the Imitation Game): A benchmark collection of 204 tasks. A particular subset of 23 tasks is called BBH (Big-Bench Hard). An adversarial variant of BBH is called BBEH (Big-Bench Extra Hard), made by replacing each of the 23 tasks from BBH with a similar but adversarial variant.
GLUE (General Language Understanding Evaluation): collection of 9 benchmarks designed for testing general language understanding. The tasks are in the format of sentence- or sentence-pair. There are over 1M items.

SuperGLUE: An update to GLUE. Designed to be still challenging to the SOTA models of the time (2019) since the original had been saturated. Includes 8 additional tasks (e.g. logical reasoning, commonsense inference, coreference resolution).

HELM (Holistic Evaluation of Language Models): A continuously updated benchmark framework of several benchmarks, maintained by the Stanford Center for Research on Foundation Models.
MMLU (Measuring Massive Multitask Language Understanding): 16,000 multiple-choice questions spanning 57 academic subjects including mathematics, philosophy, law, and medicine. Upgraded to MMLU-Pro which increases the number of choices from 4 to 10, eliminated the trivial and noisy questions from MMLU, and added harder problems.

CMMLU (Chinese MMLU): 1,528 multiple-choice questions across 67 subjects, 16 of which are "China-specific", like Classical Chinese. Some data collected from non-publicly available materials, mock exam questions, and questions from quiz shows to avoid contamination. More than 80% of the data was crawled from PDFs after OCR.
MMMLU (Multilingual MMLU): The test set of MMLU, translated into 14 languages by professional human translators.

Relatedly, certain organizations dedicated to AI benchmarking publish their own indexes. Indices are numerical scores constructed from model performance on many benchmarks, over many configurations (varied computational budget, varied tool access, pass@n, and so on). Different benchmarks may be included, at different weights. The precise construction of an index is usually continuously updated over time, released in versions, according the organization's own methodology.

Epoch Capabilities Index, run by Epoch AI. It is based on a statistical model using Item Response Theory. Each model $m$ has a single capability index $C_{m}$ , and each benchmark $b$ has a sharpness $\alpha _{b}$ and a difficulty level $D_{b}$ , and the predicted performance is $\mathrm {score} (m,b)=\mathrm {sigmoid} (\alpha _{b}(C_{m}-D_{b}))$ . The parameters $C_{m},\alpha _{b},D_{b}$ are fitted to model performance on various benchmarks.
Artificial Analysis Intelligence Index, run by Artificial Analysis. Aggregation of a suite of benchmarks on reasoning, knowledge, maths and programming.

Multimodal

Some benchmarks specifically test for multimodal ability, usually between text, image, video, and audio.

MMMU (Massive Multi-discipline Multimodal Understanding): A vision-language version of MMLU. 11550 questions collected from college exams, quizzes, and textbooks, covering 30 subjects. The questions require image-understanding to solve. Includes multiple-choice questions and open-ended QA (which are scored by regex extraction). Human expert baseline is 89%.

MMMU-Pro: 1730 multiple-choice multimodal questions in the same format as MMMU, designed to be adversarial against text-only models. Some problems in MMMU turned out to be answerable without looking at the images, necessitating MMMU-Pro. Each question has 10 choices, and presented in both text-image format, and screenshot/photo format.

MMT-Bench: benchmark designed to assess LVLMs performance on massive multimodal tasks that involve expert knowledge, visual recognition, localization, reasoning, and planning. The test bench includes 31,325 multi-choice questions from visual multimodal scenarios (like driving and navigation) that cover 32 core meta-tasks and 162 subtasks.
OmniDocBench: 981 annotated PDF pages (expanded to 1,355 in v1.5) sampled from over 200,000 documents, spanning 9 document types (academic papers, textbooks, slides, newspapers, handwritten notes, etc) in English, Simplified Chinese, and mixed. Evaluates document parsing across end-to-end parsing, layout detection, table recognition, formula recognition, and text OCR.
Vibe-Eval: 269 visual understanding prompts, with standard responses written by experts. Of these, 100 were "hard" meaning they could not be solved by an LLM (Reka Core) at the time of publication. Automatic scoring by LLMs.
VideoMMMU: Like MMMU, but with videos. Contains 300 college-level lecture videos in 30 subjects in 6 disciplines (Art, Business, Science, Medicine, Humanities, and Engineering), with 900 questions.

Context length

Some benchmarks were designed specifically to test for processing continuous text that is very long.

FACTS Grounding: 1,719 items divided into a public set (860) and a private held-out (859) set. Each contains a document, a system instruction requiring the LLM to exclusively reference the provided document, and a user request that requires understanding of the document. Answers are scored by frontier LLMs.
InfiniteBench: 3946 items in 12 tasks from 5 domains (retrieval, code, math, novels, and dialogue) with context lengths exceeding 100K tokens.
L-Eval: 2,000+ human-labeled query-response pairs over 508 long documents in 20 tasks, including diverse task types, domains, and input length (3K—200K tokens).
LOFT (Long-Context Frontiers): 6 long-context task categories (text retrieval, visual retrieval, audio retrieval, retrieval-augmented generation, SQL-like dataset query, many-shot in-context learning) in 35 datasets and 4 modalities. Up to 1 million tokens.
LongBench: 4,750 tasks on 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). Updated with LongBench v2 that contained 503 more tasks, that require a context length ranging from 8K to 2M words, with the majority under 128K.
Long Range Arena: 6 synthetic tasks that required 1K to 16K tokens of context length to solve.
Michelangelo: 3 tasks generated programmatically, and can be arbitrarily long. They are Multi-Round Co-reference Resolution (MRCR, track identities and references in adversarial conversation histories up to 1M tokens), Latent List, and I Don't Know (IDK).
MTOB (Machine Translation from One Book): translate sentences between English and Kalamang after reading a grammar book of Kalamang (~570 pages), a bilingual word list (2,531 entries, with Part-of-Speech tags) and a small parallel corpus of sentence pairs (~400 train sentences, 100 test sentences, filtered to exclude examples from the book), both published on Dictionaria.
Needle in a haystack tests (NIH): This is not a specific benchmark, but a method for benchmarking context lengths. In this method, a long context window is filled with text, such as Paul Graham's essays, and a random statement is inserted. The task is to answer a question about the inserted statement.
NoLiMa: Long-Context Evaluation Beyond Literal Matching. The benchmark assesses long-context models beyond simple keyword matching. Specifically, the words in the question have minimal or no direct lexical overlap with the words in the "needle" sentence. The "haystacks" are 10 open-licensed books.
RULER: 13 tasks in 4 categories (retrieval, multi-hop, aggregation, question answering). Each task is specified by a program which can generate arbitrarily long instances of each task on demand.
ZeroSCROLLS: 4,378 items in 6 tasks. Includes 6 tasks from SCROLLS and introduces 4 new datasets. Named "zero" because it was designed for zero-shot learning during the early days of pretraining paradigm, back when zero-shot capability was uncommon.

Reasoning

Reasoning benchmarks benchmark for the ability of language models to perform tasks that, when performed by a human, is considered to involve "logical thought", "formal reasoning", "intelligence", etc. They are generally similar to academic examinations and intelligence tests taken by human subjects.

As AI capability increases, reasoning benchmarks has gradually glided towards agency benchmarks, since easier items have been saturated, and harder items are constructed. These harder test items are intended to test for capability that are intrinsically agentic. These are capability for acting coherently in an action-observation-response loop, for using tools, for building tools that are then used to perform tasks, and so on.

Programming

Aider Polyglot: 225 of the hardest coding exercises from Exercism, in languages of C++, Go, Java, JavaScript, Python and Rust.
APPS: 10,000 problems from Codewars, AtCoder, Kattis, and Codeforces.
BigCodeBench: 1140 tasks that requires multiple function calls. The benchmark involves 139 libraries and 7 domains. A subset BigCodeBench-Hard involves just a 148-task subset of the full benchmark.
BFCL (Berkeley Function-Calling Leaderboard): The task is to write API calls according to a specification. Released in 3 versions, with 1760, 2251, and 1000 items respectively. Some calls are evaluated by parsing into an AST and comparing against the reference answer, while others are evaluated by calling and comparing the response against the reference response. Includes Python, Java, JavaScript, SQL, and REST API.
CodeElo: 387 contest problems from Codeforces during 2024, annotated with metadata such as contest divisions, problem difficulty ratings, and problem algorithm tags. Benchmarking is run by directly submitting to Codeforces, resulting in an Elo rating. Limited to 8 submissions per problem.
DS-1000: 1000 data science problems obtained by reformulating 451 unique StackOverflow problems, requiring the use of 7 Python libraries, such as NumPy and Pandas. The responses are scored by running test cases and comparing outputs, and checking for the presence/absence of specific APIs or keywords.
DSBench: 466 data analysis tasks and 74 data modeling tasks sourced from Kaggle and ModelOff competitions, spanning exploratory analysis, multi‑table joins, and predictive modeling with large CSVs and multimodal prompts.
HumanEval: 164 problems where the solution is always a python function, often just a few lines long.
MBPP (Mostly Basic Programming Problems): 974 short Python functions designed to be solved by entry-level programmers. Each comes with a text description and unit tests. They were written by an internal pool of crowdworkers who have basic knowledge of Python.

Mathematics

As of 2026, frontier models have saturated all mathematics benchmarks whose solutions are known, thus frontier progress is measured by solving open problems. Nevertheless, some of the following benchmarks remain useful for benchmarking capability in non-frontier models.

Alg514: 514 algebra word problems and associated equation systems gathered from Algebra.com.
AQuA-RAT (Algebra Question Answering with Rationales): Also known as just "AQuA". 100,000 algebraic word problems with 5 choices per problem, and an annotation for the correct choice with natural language rationales. 34,202 "seed problems" were collected from many sources, such as GMAT and GRE, which were then expanded to the full dataset with Amazon Turk.
FrontierMath: Several hundred questions from areas of modern math that are difficult for professional mathematicians to solve. Many questions have integer answers, so that answers can be verified automatically. Held-out to prevent contamination. Divided into Tiers. Tiers 1-3 contained 300 problems and finished construction in 2024-11. Tier 4 contained 50 problems and finished construction in 2025-06. In 2026-01, Open Problems was added to the benchmark, consisting of unsolved problems.
GSM1K: 1205 items with the same format and difficulty as GSM8K. More securely contained to avoid the data contamination concerns with the previous GSM8K.
GSM8K (Grade School Math): 8.5K linguistically diverse elementary school math word problems that require 2 to 8 basic arithmetic operations to solve. Contains errors that had been corrected with GSM8K-Platinum.
IMO-Bench: Benchmark based on the IMO. Consisting of 3 parts: IMO-AnswerBench (400) for getting the right answer, IMO-ProofBench (60) for writing a proof, IMO-GradingBench (1,000) for grading a proof according to a given rubric.
MATH: 12,500 competition-level math problems divided into difficulty levels 1 to 5 (as the Art of Problem Solving), with AIME problems being level 5. There are 1,324 level 5 items. An adversarial version is MATH-P, obtained by modifying a few characters in the original questions.
Math23K: 23,164 elementary school Chinese mathematical word problems, collected from various online educational websites.
MathArena: Instead of a purpose-built benchmark, the MathArena benchmark simply takes the latest math competitions (AIME and HMMT) as soon as possible and uses those to benchmark LLMs, to prevent contamination.
MathBench: 3709 questions in English and Chinese, divided into 5 difficulty levels (basic arithmetic, primary school, middle school, high school, college). Divided into 2,209 questions of MathBench-T (theoretical) and 1,500 questions of MathBench-A (applied).
MathEval: An omnibus benchmark that contains 20 other benchmarks, such as GSM8K, MATH, and the math subsection of MMLU. Over 20,000 math problems. Difficulty ranges from elementary school to high school competition.
MathQA: 37,200 word problems in English. Each problem came from AQuA-RAT, and annotated with an "operation program" which exactly specifies the mathematical operations required to solve the problem, written in a domain-specific language with 58 operators. Has a variant, MathQA-Python, consisting of 23,914 problems, produced by taking the solutions to a subset of the MathQA dataset, and rewriting into Python.
miniF2F (mini formal-to-formal): 488 Olympiad-level mathematics problems from AIME, AMC, and IMO, stated in formal languages (Metamath, Lean, Isabelle (partially) and HOL Light (partially)). The task is to formally prove the formal statement, which can be verified automatically.
Omni-MATH: 4428 competition-level math problems with human annotation.
ProofNet: 371 theorems in undergraduate-level mathematics, each consisting of a formal statement in Lean, a natural language statement, and a natural language proof. There are two tasks: given an informal (formal) statement, produce a corresponding formal (informal) statement; given an informal theorem statement, its informal proof, and its formal statement, produce a formal proof. Originally was in Lean 3, but the original authors deprecated it in favor of the Lean 4 version.
PutnamBench: 1709 formalized versions of Putnam competition questions during 1962 - 2023. The task is to compute the numerical answer (if there is a numerical answer) and to provide a formal proof. The formalizations are in Lean 4, Isabelle, and Rocq (then: Coq).
Riemann-Bench: 25 mathematics problems, solicited from researchers who were asked to find difficult problems they have solved but not yet published. Test subject is given access to Python and web search. Each problem is estimated by the researcher to have taken weeks of effort to solve.
TheoremQA: 800 questions that test for the use of 350 theorems from math, physics, electric engineering, computer science, and finance.
U-MATH: 1100 math problems sourced from real-world university curricula, balanced across six subjects with 20% of problems including visual elements.

Academic

AGIEval: questions from 20 official, public, and high-standard admission and qualification exams, such as SAT, Gaokao, law school admission tests, math competitions, lawyer qualification tests, and national civil service exams.
GPQA (Google-Proof Q&A): 448 multiple-choice questions written by domain experts in biology, physics, and chemistry, designed to be PhD-level. OpenAI found that human experts achieve an average score of 69.7% on the Diamond subset. It is composed of 3 sets: "Extended" with 546 problems, containing all problems solicited from writers; "Main" with 448 problems, which were an expert-validated subset from "Extended"; "Diamond" with 198 problems, which were the hardest problems from "Main". In the dataset, there is also an anonymized list of 60 experts who validated the dataset, and their qualifications. The inter-expert agreement on the Extended set is only 74%. The construction of the dataset cost ~$120K. Each question cost an average of 2 expert-hours. Each expert was paid $100/hr.

SuperGPQA: 26,529 multiple-choice questions collected by domain experts in 285 graduate-level disciplines. The questions were collected by individuals with or pursuing a PhD and then refined and inspected with the help of large language models.

Humanity's Last Exam (HLE): 2,500 multimodal questions across over a hundred academic subjects, with a held-out private dataset left unreleased to prevent contamination. 10% of questions requires both image and text comprehension and the rest are fully text-based. 80% of questions are scored by exact string matching, and the rest are multiple-choice.
- On September 22, 2026, HLE-Diamond was released. It contained 1,000 items filtered and cleaned from the original HLE question collection, removing errors and incorporating community feedback.
LiveBench: A series of benchmarks released monthly, including high school math competition questions, competitive coding questions, logic puzzles, and other tasks.
MathVista: 6,141 questions involving quantitative reasoning that requires reading a picture to solve.
OlympicArena: 11,163 problems from 62 distinct Olympic competitions.
PhilosophyBench: A Stanford University benchmark that evaluates whether AI can develop original philosophical ideas with clarity, depth and sophistication.
OlympiadBench: 8,476 math and physics problems in English and Chinese, sourced from International Olympiads, Chinese Olympiads, and Gaokao.
SimpleBench: A multiple-choice text benchmark with over 200 questions covering spatio-temporal reasoning, social intelligence, and linguistic adversarial robustness (or trick questions). It is designed to test "everyday human reasoning".

Values and Opinions

ETHICS: Over 130,000 scenarios testing knowledge of basic concepts in normative ethics: justice, deontology, virtue ethics, utilitarianism, and commonsense moral intuitions.
OpinionQA: 1,498 multiple-choice questions on topics such as politics, science, and religion from the Pew Research Center's American Trends Panel surveys.
GlobalOpinionQA: 2,556 multiple-choice questions taken from cross-national surveys (the Pew Research Center's Global Attitudes survey and the World Values Survey).
ValueBench: Items from 44 established psychometic inventories, covering 453 value dimensions.

References

↑ Liang, Percy; Bommasani, Rishi; Lee, Tony; Tsipras, Dimitris; Soylu, Dilara; Yasunaga, Michihiro; Zhang, Yian; Narayanan, Deepak; Wu, Yuhuai (2023-10-01). "Holistic Evaluation of Language Models". Annals of the New York Academy of Sciences. 1525 (1): 140–146. arXiv:2211.09110. Bibcode:2023NYASA1525..140B. doi:10.1111/nyas.15007. PMID 37230490.
↑ Chang, Yupeng; Wang, Xu; Wang, Jindong; Wu, Yuan; Yang, Linyi; Zhu, Kaijie; Chen, Hao; Yi, Xiaoyuan; Wang, Cunxiang; Wang, Yidong; Ye, Wei; Zhang, Yue; Chang, Yi; Yu, Philip S.; Yang, Qiang (2024-06-30). "A Survey on Evaluation of Large Language Models". ACM Transactions on Intelligent Systems and Technology. 15 (3): 1–45. doi:10.1145/3641289. ISSN 2157-6904.
↑ Chen, Danqi; Yih, Wen-tau (July 2020). Savary, Agata; Zhang, Yue (eds.). "Open-Domain Question Answering". Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts. Online: Association for Computational Linguistics: 34–37. doi:10.18653/v1/2020.acl-tutorials.8.
↑ Weng, Lilian (2020-10-29). "How to Build an Open-Domain Question Answering System?". lilianweng.github.io. Retrieved 2025-03-05.
1 2 Radford, Alec; Wu, Jeffrey; Child, Rewon; Luan, David; Amodei, Dario; Sutskever, Ilya (February 14, 2019). "Language Models are Unsupervised Multitask Learners" (PDF). OpenAI.
↑ "English Gigaword Fifth Edition". Linguistic Data Consortium. June 17, 2011. Retrieved 2025-05-17.
1 2 Chelba, Ciprian; Mikolov, Tomas; Schuster, Mike; Ge, Qi; Brants, Thorsten; Koehn, Phillipp; Robinson, Tony (2013). "One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling". arXiv:1312.3005 [cs.CL].
1 2 Dehghani, Mostafa; Tay, Yi; Gritsenko, Alexey A.; Zhao, Zhe; Houlsby, Neil; Diaz, Fernando; Metzler, Donald; Vinyals, Oriol (2021-07-14). "The Benchmark Lottery". arXiv:2107.07002 [cs.LG].
↑ DeepSeek-AI; Guo, Daya; Yang, Dejian; Zhang, Haowei; Song, Junxiao; Zhang, Ruoyu; Xu, Runxin; Zhu, Qihao; Ma, Shirong (2025-01-22). "DeepSeek-R1 incentivizes reasoning in LLMS through reinforcement learning". Nature. 645 (8081): 633–638. arXiv:2501.12948. Bibcode:2025Natur.645..633G. doi:10.1038/s41586-025-09422-z. PMC 12443585. PMID 40962978.
1 2 Chen, Mark; Tworek, Jerry; Jun, Heewoo; Yuan, Qiming; Pinto, Henrique Ponde de Oliveira; Kaplan, Jared; Edwards, Harri; Burda, Yuri; Joseph, Nicholas (2021-07-14). "Evaluating Large Language Models Trained on Code". arXiv:2107.03374 [cs.LG].
↑ Vedantam, Ramakrishna; Lawrence Zitnick, C.; Parikh, Devi (2015). "CIDEr: Consensus-based image description evaluation". 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4566–4575. arXiv:1411.5726. doi:10.1109/CVPR.2015.7299087. ISBN 978-1-4673-6964-0.
↑ Anderson, Peter; Fernando, Basura; Johnson, Mark; Gould, Stephen (2016). "SPICE: Semantic Propositional Image Caption Evaluation". In Leibe, Bastian; Matas, Jiri; Sebe, Nicu; Welling, Max (eds.). Computer Vision – ECCV 2016. Lecture Notes in Computer Science. Vol. 9909. Cham: Springer International Publishing. pp. 382–398. arXiv:1607.08822. doi:10.1007/978-3-319-46454-1_24. ISBN 978-3-319-46454-1.
↑ Northcutt, Curtis G.; Athalye, Anish; Mueller, Jonas (2021-11-07). "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks". arXiv:2103.14749 [stat.ML].
↑ Richie, Russell; Grover, Sachin; Tsui, Fuchiang (Rich) (May 2022). Demner-Fushman, Dina; Cohen, Kevin Bretonnel; Ananiadou, Sophia; Tsujii, Junichi (eds.). "Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations". Proceedings of the 21st Workshop on Biomedical Language Processing. Dublin, Ireland: Association for Computational Linguistics: 275–284. doi:10.18653/v1/2022.bionlp-1.26.
↑ Artstein, Ron (2017), "Inter-annotator Agreement", in Ide, Nancy; Pustejovsky, James (eds.), Handbook of Linguistic Annotation, Dordrecht: Springer Netherlands, pp. 297–313, doi:10.1007/978-94-024-0881-2_11, ISBN 978-94-024-0881-2, retrieved 2025-02-22
↑ Nie, Yixin; Zhou, Xiang; Bansal, Mohit (November 2020). "What Can We Learn from Collective Human Opinions on Natural Language Inference Data?". In Webber, Bonnie; Cohn, Trevor; He, Yulan; Liu, Yang (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics. pp. 9131–9143. doi:10.18653/v1/2020.emnlp-main.734.
↑ Pavlick, Ellie; Kwiatkowski, Tom (November 2019). "Inherent Disagreements in Human Textual Inferences". Transactions of the Association for Computational Linguistics. 7: 677–694. doi:10.1162/tacl_a_00293. ISSN 2307-387X.
↑ Gururangan, Suchin; Swayamdipta, Swabha; Levy, Omer; Schwartz, Roy; Bowman, Samuel R.; Smith, Noah A. (2018-04-16). "Annotation Artifacts in Natural Language Inference Data". arXiv:1803.02324 [cs.CL].
↑ Deng, Chunyuan; Zhao, Yilun; Tang, Xiangru; Gerstein, Mark; Cohan, Arman (June 2024). "Investigating Data Contamination in Modern Benchmarks for Large Language Models". In Duh, Kevin; Gomez, Helena; Bethard, Steven (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City, Mexico: Association for Computational Linguistics. pp. 8706–8719. arXiv:2311.09783. doi:10.18653/v1/2024.naacl-long.482.
↑ LI, Yanyang (2025-02-17), lyy1994/awesome-data-contamination, retrieved 2025-02-22
↑ Shannon, C. E. (1951). "Prediction and Entropy of Printed English". Bell System Technical Journal. 30 (1): 50–64. Bibcode:1951BSTJ...30...50S. doi:10.1002/j.1538-7305.1951.tb01366.x. ISSN 1538-7305.
↑ Radford, Alec; Wu, Jeffrey; Child, Rewon; Luan, David; Amodei, Dario; Sutskever, Ilya (February 14, 2019). "Better language models and their implications". OpenAI.
↑ Magnusson, Ian; Bhagia, Akshita; Hofmann, Valentin; Soldaini, Luca; Jha, Ananya Harsh; Tafjord, Oyvind; Schwenk, Dustin; Walsh, Evan Pete; Elazar, Yanai (2024-12-07). "Paloma: A Benchmark for Evaluating Language Model Fit". arXiv:2312.10523 [cs.CL].
↑ Davis, Ernest (2023-10-23). "Benchmarks for Automated Commonsense Reasoning: A Survey". ACM Comput. Surv. 56 (4): 81:1–81:41. arXiv:2302.04752. doi:10.1145/3615355. ISSN 0360-0300.
↑ Chen, Danqi; Bolton, Jason; Manning, Christopher D. (2016-08-08). "A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task". arXiv:1606.02858 [cs.CL].
↑ "The Corpus of Linguistic Acceptability (CoLA)". nyu-mll.github.io. Archived from the original on 2025-03-11. Retrieved 2025-04-19.
↑ Warstadt, Alex; Singh, Amanpreet; Bowman, Samuel R. (November 2019). "Neural Network Acceptability Judgments". Transactions of the Association for Computational Linguistics. 7: 625–641. arXiv:1805.12471. doi:10.1162/tacl_a_00290. ISSN 2307-387X.
↑ Zellers, Rowan; Holtzman, Ari; Bisk, Yonatan; Farhadi, Ali; Choi, Yejin (2019-05-19). "HellaSwag: Can a Machine Really Finish Your Sentence?". arXiv:1905.07830 [cs.CL].
↑ "HellaSwag". rowanzellers.com. Retrieved 2025-02-06.
↑ Paperno, Denis; Kruszewski, Germán; Lazaridou, Angeliki; Pham, Quan Ngoc; Bernardi, Raffaella; Pezzelle, Sandro; Baroni, Marco; Boleda, Gemma; Fernández, Raquel (2016-06-20). "The LAMBADA dataset: Word prediction requiring a broad discourse context". arXiv:1606.06031 [cs.CL].
↑ Williams, Adina; Nangia, Nikita; Bowman, Samuel R. (2018-02-19). "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference". arXiv:1704.05426 [cs.CL].
↑ Lai, Guokun; Xie, Qizhe; Liu, Hanxiao; Yang, Yiming; Hovy, Eduard (2017-12-05). "RACE: Large-scale ReAding Comprehension Dataset From Examinations". arXiv:1704.04683 [cs.CL].
↑ Bowman, Samuel R.; Angeli, Gabor; Potts, Christopher; Manning, Christopher D. (September 2015). "A large annotated corpus for learning natural language inference". In Màrquez, Lluís; Callison-Burch, Chris; Su, Jian (eds.). Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics. pp. 632–642. arXiv:1508.05326. doi:10.18653/v1/D15-1075.
↑ "The Stanford Natural Language Processing Group". nlp.stanford.edu. Retrieved 2025-02-22.
↑ Zellers, Rowan; Bisk, Yonatan; Schwartz, Roy; Choi, Yejin (2018-08-16). "SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference". arXiv:1808.05326 [cs.CL].
↑ Levesque, Hector; Davis, Ernest; Morgenstern, Leora (2012). The Winograd Schema Challenge. Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning.
↑ Kocijan, Vid; Davis, Ernest; Lukasiewicz, Thomas; Marcus, Gary; Morgenstern, Leora (2023-07-11). "The defeat of the Winograd Schema Challenge". Artificial Intelligence. 325 103971. arXiv:2201.02387. doi:10.1016/j.artint.2023.103971. ISSN 0004-3702. S2CID 245827747.
↑ Sakaguchi, Keisuke; Le Bras, Ronan; Bhagavatula, Chandra; Choi, Yejin (2019). "WinoGrande: An Adversarial Winograd Schema Challenge at Scale". arXiv:1907.10641 [cs.CL].
↑ Bojar, Ondřej; Buck, Christian; Federmann, Christian; Haddow, Barry; Koehn, Philipp; Leveling, Johannes; Monz, Christof; Pecina, Pavel; Post, Matt; Saint-Amand, Herve; Soricut, Radu; Specia, Lucia; Tamchyna, Aleš (June 2014). Bojar, Ondřej; Buck, Christian; Federmann, Christian; Haddow, Barry; Koehn, Philipp; Monz, Christof; Post, Matt; Specia, Lucia (eds.). "Findings of the 2014 Workshop on Statistical Machine Translation". Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, Maryland, USA: Association for Computational Linguistics: 12–58. doi:10.3115/v1/W14-3302. hdl:20.500.11820/789fbc29-61e0-4529-af4a-819461c57a8f.
↑ Daum, Shilo; Shapira, Tal; Bremler-Barr, Anat; Hay, David (2024). "Non-uniformity is All You Need: Efficient and Timely Encrypted Traffic Classification with ECHO". arXiv:2406.01852 [cs.NI].
↑ Zhou, Jeffrey; Lu, Tianjian; Mishra, Swaroop; Brahma, Siddhartha; Basu, Sujoy; Luan, Yi; Zhou, Denny; Hou, Le (2023-11-14). "Instruction-Following Evaluation for Large Language Models". arXiv:2311.07911 [cs.CL].
1 2 Zheng, Lianmin; Chiang, Wei-Lin; Sheng, Ying; Zhuang, Siyuan; Wu, Zhanghao; Zhuang, Yonghao; Lin, Zi; Li, Zhuohan; Li, Dacheng (2023-12-24). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena". arXiv:2306.05685 [cs.CL].
↑ lmarena/arena-hard-auto, Arena, 2026-09-29, retrieved 2026-10-04
↑ Sirdeshmukh, Ved; Deshpande, Kaustubh; Mols, Johannes; Jin, Lifeng; Cardona, Ed-Yeremai; Lee, Dean; Kritz, Jeremy; Primack, Willow; Yue, Summer; Xing, Chen (2025). "MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMS". arXiv:2501.17399 [cs.CL].
↑ Mishra, Swaroop; Khashabi, Daniel; Baral, Chitta; Hajishirzi, Hannaneh (2022-03-14). "Cross-Task Generalization via Natural Language Crowdsourcing Instructions". arXiv:2104.08773 [cs.CL].
↑ Wang, Yizhong; Mishra, Swaroop; Alipoormolabashi, Pegah; Kordi, Yeganeh; Mirzaei, Amirreza; Arunkumar, Anjana; Ashok, Arjun; Dhanasekaran, Arut Selvan; Naik, Atharva (2022-10-24). "Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks". arXiv:2204.07705 [cs.CL].
↑ Minev, Z. K.; Mundhada, S. O.; Shankar, S.; Reinhold, P.; Gutiérrez-Jáuregui, R.; Schoelkopf, R. J.; Mirrahimi, M.; Carmichael, H. J.; Devoret, M. H. (2019). "To catch and reverse a quantum jump mid-flight". Nature. 570 (7760): 200–204. arXiv:1803.00545. Bibcode:2019Natur.570..200M. doi:10.1038/s41586-019-1287-z. PMID 31160725.
↑ Masry, Ahmed; Do, Xuan Long; Tan, Jia Qing; Joty, Shafiq; Hoque, Enamul (May 2022). "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning". In Muresan, Smaranda; Nakov, Preslav; Villavicencio, Aline (eds.). Findings of the Association for Computational Linguistics: ACL 2022. Dublin, Ireland: Association for Computational Linguistics. pp. 2263–2279. arXiv:2203.10244. doi:10.18653/v1/2022.findings-acl.177.
↑ Reddy, Siva; Chen, Danqi; Manning, Christopher D. (2019-05-01). "CoQA: A Conversational Question Answering Challenge". Transactions of the Association for Computational Linguistics. 7: 249–266. arXiv:1808.07042. doi:10.1162/tacl_a_00266. ISSN 2307-387X.
↑ "Industry Documents Library". industrydocuments.ucsf.edu. Retrieved 2025-04-05.
↑ "DocVQA". www.docvqa.org. Retrieved 2025-04-05.
↑ Mathew, Minesh; Karatzas, Dimosthenis; Jawahar, C. V. (2021). "DocVQA: A Dataset for VQA on Document Images". pp. 2200–2209.
↑ Mendoza-Arenas, J. J.; Gómez-Ruiz, F. J.; Rodríguez, F. J.; Quiroga, L. (2019). "Enhancing violations of Leggett-Garg inequalities in nonequilibrium correlated many-body systems by interactions and decoherence". Scientific Reports. 9 (1): 17772. arXiv:1903.00016. Bibcode:2019NatSR...917772M. doi:10.1038/s41598-019-54121-1. PMC 6882789. PMID 31780693.
↑ Khrennikov, Andrei; Ozawa, Masanao; Benninger, Felix; Shor, Oded (2025). "Coupling quantum-like cognition with the neuronal networks within generalized probability theory". Journal of Mathematical Psychology. 125 102923. arXiv:2411.00036. doi:10.1016/j.jmp.2025.102923.
↑ Yang, Zhilin; Qi, Peng; Zhang, Saizheng; Bengio, Yoshua; Cohen, William; Salakhutdinov, Ruslan; Manning, Christopher D. (October 2018). Riloff, Ellen; Chiang, David; Hockenmaier, Julia; Tsujii, Jun'ichi (eds.). "HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering". Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics: 2369–2380. doi:10.18653/v1/D18-1259.
↑ Richardson, Matthew; Burges, Christopher J.C.; Renshaw, Erin (October 2013). "MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text". In Yarowsky, David; Baldwin, Timothy; Korhonen, Anna; Livescu, Karen; Bethard, Steven (eds.). Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics. pp. 193–203. doi:10.18653/v1/D13-1020.
↑ Kwiatkowski, Tom; Palomaki, Jennimaria; Redfield, Olivia; Collins, Michael; Parikh, Ankur; Alberti, Chris; Epstein, Danielle; Polosukhin, Illia; Devlin, Jacob; Lee, Kenton; Toutanova, Kristina; Jones, Llion; Kelcey, Matthew; Chang, Ming-Wei; Dai, Andrew M. (2019-08-01). "Natural Questions: A Benchmark for Question Answering Research". Transactions of the Association for Computational Linguistics. 7: 453–466. doi:10.1162/tacl_a_00276. ISSN 2307-387X.
↑ Enns, John (2018). "Multiplicities in the ordinary part of mod $p$ cohomology for $\mathrm{GL}_n(\mathbb{Q}_p)$". arXiv:1809.00278 [math.NT].
↑ Du Toit, E.J.; o'Brien, M.R.; Vann, R.G.L. (2017). "A Kinetic Study of Microwave Start-up of Tokamak Plasmas". EPJ Web of Conferences. 147: 01002. arXiv:1704.00517. Bibcode:2017EPJWC.14701002D. doi:10.1051/epjconf/201714701002.
↑ Baik, Jinho; Barraquand, Guillaume; Corwin, Ivan; Suidan, Toufic (2018). "Pfaffian Schur processes and last passage percolation in a half-quadrant". The Annals of Probability. 46 (6). arXiv:1606.00525. doi:10.1214/17-AOP1226.
↑ Wallis, Ben (2018). "Closed ideals of operators acting on some families of sequence spaces". arXiv:1806.00382 [math.FA].
↑ Geva, Mor; Khashabi, Daniel; Segal, Elad; Khot, Tushar; Roth, Dan; Berant, Jonathan (2021-04-26). "Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies". Transactions of the Association for Computational Linguistics. 9: 346–361. doi:10.1162/tacl_a_00370. ISSN 2307-387X.
↑ Berant, Jonathan; Chou, Andrew; Frostig, Roy; Liang, Percy (October 2013). "Semantic Parsing on Freebase from Question-Answer Pairs". In Yarowsky, David; Baldwin, Timothy; Korhonen, Anna; Livescu, Karen; Bethard, Steven (eds.). Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics. pp. 1533–1544. doi:10.18653/v1/D13-1160.
↑ "C-Eval: 一个适用于大语言模型的多层次多学科中文评估套件". cevalbenchmark.com. Retrieved 2025-02-25.
↑ Jin, Di; Pan, Eileen; Oufattole, Nassim; Weng, Wei-Hung; Fang, Hanyi; Szolovits, Peter (January 2021). "What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams". Applied Sciences. 11 (14): 6421. doi:10.3390/app11146421. hdl:1721.1/136684.2. ISSN 2076-3417.
↑ Majumdar, Arjun; Ajay, Anurag; Zhang, Xiaohan; Putta, Pranav; Yenamandra, Sriram; Henaff, Mikael; Silwal, Sneha; Mcvay, Paul; Maksymets, Oleksandr; Arnaud, Sergio; Yadav, Karmesh; Li, Qiyang; Newman, Ben; Sharma, Mohit; Berges, Vincent (2024). "OpenEQA: Embodied Question Answering in the Era of Foundation Models". 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16488–16498. doi:10.1109/CVPR52733.2024.01560. ISBN 979-8-3503-5300-6.
↑ Bisk, Yonatan; Zellers, Rowan; Bras, Ronan Le; Gao, Jianfeng; Choi, Yejin (2020-04-03). "PIQA: Reasoning about Physical Commonsense in Natural Language". Proceedings of the AAAI Conference on Artificial Intelligence. 34 (5): 7432–7439. arXiv:1911.11641. doi:10.1609/aaai.v34i05.6239. ISSN 2374-3468.
↑ "Grok-1.5 Vision Preview | xAI". x.ai. Retrieved 2025-03-12.
↑ Lu, Pan; Mishra, Swaroop; Xia, Tanglin; Qiu, Liang; Chang, Kai-Wei; Zhu, Song-Chun; Tafjord, Oyvind; Clark, Peter; Kalyan, Ashwin (2022-12-06). "Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering". Advances in Neural Information Processing Systems. 35: 2507–2521. arXiv:2209.09513. doi:10.52202/068431-0182. ISBN 978-1-7138-7108-8.{{cite journal}}: CS1 maint: periodical has ISBN (link)
↑ Wei, Jason; Karina, Nguyen; Chung, Hyung Won; Jiao, Yunxin Joy; Papay, Spencer; Glaese, Amelia; Schulman, John; Fedus, William (2024-11-07), Measuring short-form factuality in large language models, arXiv:2411.04368
↑ "Introducing SimpleQA". openai.com. Retrieved 2025-10-02.
↑ Matias, José; Oliveira, Julio P.C.; Le Roux, Galo A.C.; Jäschke, Johannes (2022). "Steady-state real-time optimization using transient measurements on an experimental rig". Journal of Process Control. 115: 181–196. arXiv:2109.00795. doi:10.1016/j.jprocont.2022.04.015.
↑ Lovesey, S. W. (2022). "Polar magnetization unveiled by polarized neutron diffraction". Physical Review B. 106 (6) 064415. arXiv:2206.00461. Bibcode:2022PhRvB.106f4415L. doi:10.1103/PhysRevB.106.064415.
↑ Ddamulira, Mahadi; Emong, Paul; Geoffrey Ismail Mirumbe (2022). "Members of Narayana's cow sequence that are concatenations of two repdigits". arXiv:2210.00926 [math.NT].
↑ Kazemi, Mehran; Fatemi, Bahare; Bansal, Hritik; Palowitch, John; Anastasiou, Chrysovalantis; Sanket Vaibhav Mehta; Jain, Lalit K.; Aglietti, Virginia; Jindal, Disha; Chen, Peter; Dikkala, Nishanth; Tyen, Gladys; Liu, Xin; Shalit, Uri; Chiappa, Silvia; Olszewska, Kate; Tay, Yi; Tran, Vinh Q.; Le, Quoc V.; Firat, Orhan (2025). "BIG-Bench Extra Hard". arXiv:2502.19187 [cs.CL].
↑ Wang, Alex; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omer; Bowman, Samuel R. (2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding". arXiv:1804.07461 [cs.CL].
↑ "GLUE Benchmark". gluebenchmark.com. Retrieved 2019-02-25.
↑ Herzig, Florian; Kozioł, Karol; Vignéras, Marie-France (2020). "On the Existence of Admissible Supersingular Representations of -Adic Reductive Groups". Forum of Mathematics, Sigma. 8 e2. arXiv:1905.00053. doi:10.1017/fms.2019.50.
↑ "Holistic Evaluation of Language Models (HELM)". crfm.stanford.edu. Retrieved 2026-02-06.
↑ Hernandez, A.; Woo, S.; Corrales, H.; Parra, I.; Kim, E.; Llorca, D. F.; Sotelo, M. A. (2020). "3D-DEEP: 3-Dimensional Deep-learning based on elevation patterns for road scene interpretation". 2020 IEEE Intelligent Vehicles Symposium (IV). pp. 892–898. arXiv:2009.00330. doi:10.1109/IV47402.2020.9304601. ISBN 978-1-7281-6673-5.
↑ Arjomandbigdeli, Ali; Mata, Andrew; Bak, Stanley (2024). "Verification of Neural Network Control Systems in Continuous Time". AI Verification. Lecture Notes in Computer Science. Vol. 14846. pp. 100–115. arXiv:2406.00157. doi:10.1007/978-3-031-65112-0_5. ISBN 978-3-031-65111-3.
↑ Zimmerman, Charlotte; Olsho, Alexis; Loverude, Michael; Suzanne White Brahmia (2023). "Expert covariational reasoning resources in physics graphing tasks". arXiv:2306.00921 [physics.ed-ph].
↑ "openai/MMMLU · Datasets at Hugging Face". huggingface.co. 2024-10-22. Retrieved 2025-02-28.
↑ "ECI Documentation – Overview". Epoch AI. Retrieved 2026-09-24.
↑ "Intelligence Benchmarking | Artificial Analysis". artificialanalysis.ai. Retrieved 2026-09-24.
↑ "MMMU". mmmu-benchmark.github.io. Retrieved 2025-02-28.
↑ Ates, Halim Cagri; Bhargava, Shruti; Li, Site; Lu, Jiarui; Maddula, Siddhardha; Moniz, Joel Ruben Antony; Nalamalapu, Anil Kumar; Nguyen, Roman Hoang; Ozyildirim, Melis; Patel, Alkesh; Piraviperumal, Dhivya; Renkens, Vincent; Samal, Ankit; Tran, Thy; Tseng, Bo-Hsiang; Yu, Hong; Zhang, Yuan; Zou, Shirley (2023). "MARRS: Multimodal Reference Resolution System". Proceedings of the Sixth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2023). pp. 51–58. arXiv:2311.01650. doi:10.18653/v1/2023.crac-main.7.
↑ Ma, Yingjie; Guo, Jing; Maloney, Andrew; Braatz, Richard (2025). "Quasi-steady-state approach for efficient multiscale simulation and optimization of mAb glycosylation in CHO cell culture". Chemical Engineering Science. 318 122162. arXiv:2409.00281. doi:10.1016/j.ces.2025.122162.
↑ "MMT-Bench". mmt-bench.github.io. Retrieved 2025-07-12.
↑ Ouyang, Linke; Qu, Yuan; Zhou, Hongbin; Zhu, Jiawei (2024). "OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations". arXiv:2412.07626 [cs.CV].
↑ Padlewski, Piotr; Bain, Max; Henderson, Matthew; Zhu, Zhongkai; Relan, Nishant; Pham, Hai; Ong, Donovan; Aleksiev, Kaloyan; Ormazabal, Aitor; Phua, Samuel; Yeo, Ethan; Lamprecht, Eugenie; Liu, Qi; Wang, Yuqi; Chen, Eric; Fu, Deyu; Li, Lei; Zheng, Che; Cyprien de Masson d'Autume; Yogatama, Dani; Artetxe, Mikel; Tay, Yi (2024). "Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models". arXiv:2405.02287 [cs.CL].
↑ Hu, Kairui; Wu, Penghao; Pu, Fanyi; Xiao, Wang; Zhang, Yuanhan; Yue, Xiang; Li, Bo; Liu, Ziwei (2025). "Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos". arXiv:2501.13826 [cs.CV].
↑ "Video-MMMU". videommmu.github.io. Retrieved 2025-06-07.
↑ "FACTS Grounding: A new benchmark for evaluating the factuality of large language models". Google DeepMind. 2024-12-17. Retrieved 2025-06-07.
↑ Jacovi, Alon; Wang, Andrew; Alberti, Chris; Tao, Connie; Lipovetz, Jon; Olszewska, Kate; Haas, Lukas; Liu, Michelle; Keating, Nate; Bloniarz, Adam; Saroufim, Carl; Fry, Corey; Marcus, Dror; Kukliansky, Doron; Gaurav Singh Tomar; Swirhun, James; Xing, Jinwei; Wang, Lily; Gurumurthy, Madhu; Aaron, Michael; Ambar, Moran; Fellinger, Rachana; Wang, Rui; Zhang, Zizhao; Goldshtein, Sasha; Das, Dipanjan (2025). "The FACTS Grounding Leaderboard: Benchmarking LLMS' Ability to Ground Responses to Long-Form Input". arXiv:2501.03200 [cs.CL].
↑ Zhang, Xinrong; Chen, Yingfa; Hu, Shengding; Xu, Zihang; Chen, Junhao; Moo Khai Hao; Han, Xu; Zhen Leng Thai; Wang, Shuo; Liu, Zhiyuan; Sun, Maosong (2024). "$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens". arXiv:2402.13718 [cs.CL].
↑ An, Chenxin; Gong, Shansan; Zhong, Ming; Zhao, Xingjian; Li, Mukai; Zhang, Jun; Kong, Lingpeng; Qiu, Xipeng (August 2024). Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek (eds.). "L-Eval: Instituting Standardized Evaluation for Long Context Language Models". Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics: 14388–14411. arXiv:2307.11088. doi:10.18653/v1/2024.acl-long.776.
↑ Lee, Jinhyuk; Chen, Anthony; Dai, Zhuyun; Dua, Dheeru; Devendra Singh Sachan; Boratko, Michael; Luan, Yi; Arnold, Sébastien M. R.; Perot, Vincent; Dalmia, Siddharth; Hu, Hexiang; Lin, Xudong; Pasupat, Panupong; Amini, Aida; Cole, Jeremy R.; Riedel, Sebastian; Naim, Iftekhar; Chang, Ming-Wei; Guu, Kelvin (2024). "Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?". arXiv:2406.13121 [cs.CL].
↑ Li, Tianle; Zhang, Ge; Quy Duc Do; Yue, Xiang; Chen, Wenhu (2024). "Long-context LLMS Struggle with Long In-context Learning". arXiv:2404.02060 [cs.CL].
↑ "LongBench v2". longbench2.github.io. Retrieved 2025-02-21.
↑ Bai, Yushi; Tu, Shangqing; Zhang, Jiajie; Peng, Hao; Wang, Xiaozhi; Lv, Xin; Cao, Shulin; Xu, Jiazheng; Hou, Lei; Dong, Yuxiao; Tang, Jie; Li, Juanzi (2024). "LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks". arXiv:2412.15204 [cs.CL].
↑ Tay, Yi; Dehghani, Mostafa; Abnar, Samira; Shen, Yikang; Bahri, Dara; Pham, Philip; Rao, Jinfeng; Yang, Liu; Ruder, Sebastian; Metzler, Donald (2020). "Long Range Arena: A Benchmark for Efficient Transformers". arXiv:2011.04006 [cs.LG].
↑ Vodrahalli, Kiran; Ontanon, Santiago; Tripuraneni, Nilesh; Xu, Kelvin; Jain, Sanil; Shivanna, Rakesh; Hui, Jeffrey; Dikkala, Nishanth; Kazemi, Mehran (2024-09-20). "Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries". arXiv:2409.12640 [cs.CL].
↑ Visser, Eline (2022). A grammar of Kalamang. Language Science Press. ISBN 978-3-96110-343-0.
↑ Visser, Eline (2021-09-24), "dictionaria/kalamang: Kalamang dictionary", Zenodo, doi:10.5281/ZENODO.5526419, retrieved 2025-04-05
↑ Tanzer, Garrett; Suzgun, Mirac; Visser, Eline; Jurafsky, Dan; Melas-Kyriazi, Luke (2023). "A Benchmark for Learning to Translate a New Language from One Grammar Book". arXiv:2309.16575 [cs.CL].
↑ @GregKamradt (November 8, 2023). "Pressure Testing GPT-4-128K With Long Context Recall..." (Tweet) – via X (formerly Twitter).
↑ Modarressi, Ali; Deilamsalehy, Hanieh; Dernoncourt, Franck; Bui, Trung; Rossi, Ryan A.; Yoon, Seunghyun; Schütze, Hinrich (2025). "NoLiMa: Long-Context Evaluation Beyond Literal Matching". arXiv:2502.05167 [cs.CL].
↑ Hsieh, Cheng-Ping; Sun, Simeng; Kriman, Samuel; Acharya, Shantanu; Rekesh, Dima; Jia, Fei; Zhang, Yang; Ginsburg, Boris (2024). "RULER: What's the Real Context Size of Your Long-Context Language Models?". arXiv:2404.06654 [cs.CL].
↑ Shaham, Uri; Ivgi, Maor; Efrat, Avia; Berant, Jonathan; Levy, Omer (2023). "ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding". arXiv:2305.14196 [cs.CL].
↑ Aider-AI/polyglot-benchmark, Aider AI, 2025-03-29, retrieved 2025-03-30
↑ Hendrycks, Dan; Basart, Steven; Kadavath, Saurav; Mazeika, Mantas; Arora, Akul; Guo, Ethan; Burns, Collin; Puranik, Samir; He, Horace; Song, Dawn; Steinhardt, Jacob (2021). "Measuring Coding Challenge Competence with APPS". arXiv:2105.09938 [cs.SE].
↑ Zhuo, Terry Yue; Chien, Vu Minh; Chim, Jenny; Hu, Han; Yu, Wenhao; Widyasari, Ratnadira; Yusuf, Imam Nur Bani; Zhan, Haolan; He, Junda; Paul, Indraneil; Brunner, Simon; Gong, Chen; Hoang, James; Zebaze, Armel Randy; Hong, Xiaoheng (2024-10-04). "BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions". Iclr 2025. arXiv:2406.15877.
↑ "BigCodeBench Leaderboard". bigcode-bench.github.io. Retrieved 2025-04-09.
↑ "Berkeley Function Calling Leaderboard". gorilla.cs.berkeley.edu. Retrieved 2025-03-11.
↑ "CodeElo". codeelo-bench.github.io. Retrieved 2025-02-13.
↑ Lai, Yuhang; Li, Chengxi; Wang, Yiming; Zhang, Tianyi; Zhong, Ruiqi; Zettlemoyer, Luke; Scott Wen-tau Yih; Fried, Daniel; Wang, Sida; Yu, Tao (2022). "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation". arXiv:2211.11501 [cs.SE].
↑ "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation". ds1000-code-gen.github.io. Retrieved 2025-03-11.
↑ Jing, Liqiang; Huang, Zhehui; Wang, Xiaoyang; Yao, Wenlin; Yu, Wenhao; Ma, Kaixin; Zhang, Hongming; Du, Xinya; Yu, Dong (2024). "DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?". arXiv:2409.07703 [cs.AI].
1 2 Austin, Jacob; Odena, Augustus; Nye, Maxwell; Bosma, Maarten; Michalewski, Henryk; Dohan, David; Jiang, Ellen; Cai, Carrie; Terry, Michael; Le, Quoc; Sutton, Charles (2021). "Program Synthesis with Large Language Models". arXiv:2108.07732 [cs.PL].
↑ Kushman, Nate; Artzi, Yoav; Zettlemoyer, Luke; Barzilay, Regina (June 2014). Toutanova, Kristina; Wu, Hua (eds.). "Learning to Automatically Solve Algebra Word Problems". Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Baltimore, Maryland: Association for Computational Linguistics: 271–281. doi:10.3115/v1/P14-1026.
↑ Huang, Danqing; Shi, Shuming; Lin, Chin-Yew; Yin, Jian; Ma, Wei-Ying (August 2016). Erk, Katrin; Smith, Noah A. (eds.). "How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation". Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics: 887–896. doi:10.18653/v1/P16-1084.
↑ Ling, Wang; Yogatama, Dani; Dyer, Chris; Blunsom, Phil (July 2017). Barzilay, Regina; Kan, Min-Yen (eds.). "Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems". Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics: 158–167. arXiv:1705.04146. doi:10.18653/v1/P17-1015.
↑ Glazer, Elliot; Erdil, Ege; Besiroglu, Tamay; Chicharro, Diego; et al. (2024). "FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI". arXiv:2411.04872 [cs.AI].
↑ "About FrontierMath". Epoch AI. Retrieved 2025-10-02.
↑ AI, Epoch (2026-01-27). "Introducing FrontierMath: Open Problems". Epoch AI. Retrieved 2026-03-20.
↑ "FrontierMath: Open Problems - Unsolved Mathematical Challenges". Epoch AI. Retrieved 2026-03-20.
↑ Zhang, Hugh; Da, Jeff; Lee, Dean; Robinson, Vaughn; et al. (2024). "A Careful Examination of Large Language Model Performance on Grade School Arithmetic". arXiv:2405.00332 [cs.CL].
↑ Cobbe, Karl; Kosaraju, Vineet; Bavarian, Mohammad; Chen, Mark; et al. (2021). "Training Verifiers to Solve Math Word Problems". arXiv:2110.14168 [cs.LG].
↑ "madrylab/gsm8k-platinum · Datasets at Hugging Face". huggingface.co. Retrieved 2025-03-07.
↑ Luong, Thang; Hwang, Dawsen; Nguyen, Hoang H; Ghiasi, Golnaz; et al. (November 2025). Christodoulopoulos, Christos; Chakraborty, Tanmoy; Rose, Carolyn; Peng, Violet (eds.). "Towards Robust Mathematical Reasoning". Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics: 35406–35430. doi:10.18653/v1/2025.emnlp-main.1794. ISBN 979-8-89176-332-6.{{cite journal}}: CS1 maint: periodical has ISBN (link)
↑ Luong, Thang; Hwang, Dawsen; Nguyen, Hoang H.; Ghiasi, Golnaz; et al., "Towards Robust Mathematical Reasoning", Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, retrieved 2025-11-08 – via IMO-Bench
↑ Hendrycks, Dan; Burns, Collin; Kadavath, Saurav; Arora, Akul; Basart, Steven; Tang, Eric; Song, Dawn; Steinhardt, Jacob (2021). "Measuring Mathematical Problem Solving with the MATH Dataset". arXiv:2103.03874 [cs.LG].
↑ "MATH-Perturb". math-perturb.github.io. Retrieved 2025-04-09.
↑ Wang, Yan; Liu, Xiaojiang; Shi, Shuming (September 2017). "Deep Neural Solver for Math Word Problems". In Palmer, Martha; Hwa, Rebecca; Riedel, Sebastian (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics. pp. 845–854. doi:10.18653/v1/D17-1088.
↑ "MathArena.ai". matharena.ai. Retrieved 2025-02-22.
↑ Liu, Hongwei; Zheng, Zilong; Qiao, Yuxuan; Duan, Haodong; et al. (2024). "MathBench: Evaluating the Theory and Application Proficiency of LLMS with a Hierarchical Mathematics Benchmark". arXiv:2405.12209 [cs.CL].
↑ math-eval (2025-01-26), math-eval/MathEval, retrieved 2025-01-27
↑ Amini, Aida; Gabriel, Saadia; Lin, Peter; Koncel-Kedziorski, Rik; Choi, Yejin; Hajishirzi, Hannaneh (2019). "MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms". arXiv:1905.13319 [cs.CL].
↑ openai/miniF2F, OpenAI, 2025-02-01, retrieved 2025-02-03
↑ Gao, Bofei; Song, Feifan; Yang, Zhe; Cai, Zefan; et al. (2024). "Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models". arXiv:2410.07985 [cs.CL].
↑ Azerbayev, Zhangir; Piotrowski, Bartosz; Schoelkopf, Hailey; Ayers, Edward W.; Radev, Dragomir; Avigad, Jeremy (2023). "ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics". arXiv:2302.12433 [cs.CL].
↑ Azerbayev, Zhangir (2025-04-02), zhangir-azerbayev/ProofNet, retrieved 2025-04-03
↑ deepseek-ai/DeepSeek-Prover-V1.5, DeepSeek, 2025-04-01, retrieved 2025-04-03
↑ Tsoukalas, George; Lee, Jasper; Jennings, John; Xin, Jimmy; Ding, Michelle; Jennings, Michael; Thakur, Amitayush; Chaudhuri, Swarat (2024). "PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition". arXiv:2407.11214 [cs.AI].
↑ "PutnamBench: A Multilingual Mathematics Benchmark for Formal Theorem-Proving". trishullab.github.io. Retrieved 2025-04-02.
↑ Garre, Suhaas; Knutsen, Erik; Mehta, Sushant; Chen, Edwin (2026). "Riemann-Bench: A Benchmark for Moonshot Mathematics". arXiv:2604.06802 [cs.AI].
↑ "Riemann-bench: A Benchmark for Moonshot Mathematics". Surge AI. 2026-03-24. Retrieved 2026-10-04.
↑ Chen, Wenhu; Yin, Ming; Ku, Max; Lu, Pan; Wan, Yixin; Ma, Xueguang; Xu, Jianyu; Wang, Xinyi; Xia, Tony (December 2023). "TheoremQA: A Theorem-driven Question Answering Dataset". In Bouamor, Houda; Pino, Juan; Bali, Kalika (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics. pp. 7889–7901. arXiv:2305.12524. doi:10.18653/v1/2023.emnlp-main.489.
↑ Chernyshev, Konstantin; Polshkov, Vitaliy; Artemova, Ekaterina; Myasnikov, Alex; Stepanov, Vlad; Miasnikov, Alexei; Tilga, Sergei (2024). "U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMS". arXiv:2412.03205 [cs.CL].
↑ Cui, Ruixiang (2025-02-03), ruixiangcui/AGIEval, retrieved 2025-02-03
↑ "Learning to reason with LLMs". openai.com. September 12, 2024. Retrieved 2025-02-27.
↑ Rein, David; Betty Li Hou; Asa Cooper Stickland; Petty, Jackson; Richard Yuanzhe Pang; Dirani, Julien; Michael, Julian; Bowman, Samuel R. (2023). "GPQA: A Graduate-Level Google-Proof Q&A Benchmark". arXiv:2311.12022 [cs.AI].
↑ Rein, I. David (2025-08-24), idavidrein/gpqa, retrieved 2025-08-25
↑ Rein, David (2024-05-15). "Can Good Benchmarks Contain Mistakes? – NYU Alignment Research Group". Retrieved 2025-09-15.
↑ Team, M-A-P; et al. (2025). "SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines". arXiv:2502.14739 [cs.CL].
↑ Phan, Long; Gatti, Alice; Li, Nathaniel; Khoja, Adam; Kim, Ryan; Ren, Richard; Hausenloy, Jason; Zhang, Oliver; Mazeika, Mantas; Hendrycks, Dan; Han, Ziwen; Hu, Josephina; Zhang, Hugh; Zhang, Chen Bo Calvin; Shaaban, Mohamed (2026-01-28). "A benchmark of expert-level academic questions to assess AI capabilities". Nature. 649 (8099): 1139–1146. Bibcode:2026Natur.649.1139C. doi:10.1038/s41586-025-09962-4. ISSN 1476-4687. PMC 12851929. PMID 41606155.
↑ "Humanity's Last Exam". lastexam.ai. Retrieved 2025-02-02.
↑ Safety, Center for AI; AI, Scale (2026-09-22). "Introducing HLE-Diamond". Humanity's Last Exam. Retrieved 2026-09-24.
↑ "LiveBench". livebench.ai. Retrieved 2025-01-27.
↑ "MathVista: Evaluating Math Reasoning in Visual Contexts". mathvista.github.io. Retrieved 2025-03-07.
↑ "OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI". gair-nlp.github.io. Retrieved 2025-02-03.
↑ "PhilosophyBench | AI and Philosophy Research at Stanford". PhilosophyBench. Retrieved 2026-10-14.
↑ He, Chaoqun; Luo, Renjie; Bai, Yuzhuo; Hu, Shengding; Zhen Leng Thai; Shen, Junhao; Hu, Jinyi; Han, Xu; Huang, Yujie; Zhang, Yuxiang; Liu, Jie; Qi, Lei; Liu, Zhiyuan; Sun, Maosong (2024). "OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems". arXiv:2402.14008 [cs.CL].
↑ "SimpleBench". simple-bench.com. Retrieved 2025-04-09.
↑ Hendrycks, Dan; Burns, Collin; Basart, Steven; Critch, Andrew; Li, Jerry; Song, Dawn; Steinhardt, Jacob (2020). "Aligning AI With Shared Human Values". arXiv:2008.02275 [cs.CY].
↑ Santurkar, Shibani; Durmus, Esin; Ladhak, Faisal; Lee, Cinoo; Liang, Percy; Hashimoto, Tatsunori (July 2023). "Whose Opinions Do Language Models Reflect?". Proceedings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research. Vol. 202. PMLR. pp. 29971–30004. arXiv:2303.17548.
↑ Durmus, Esin; Nguyen, Karina; Liao, Thomas I.; Schiefer, Nicholas; et al. (2023). "Towards Measuring the Representation of Subjective Global Opinions in Language Models". arXiv:2306.16388 [cs.CL].
↑ Ren, Yuanyi; Ye, Haoran; Fang, Hanjun; Zhang, Xin; Song, Guojie (August 2024). "ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models". In Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics. pp. 2015–2040. arXiv:2406.04214.

External links

LLM Stats – Interactive open-source database comparing the performance of different LLMs.

[1] Liang, Percy; Bommasani, Rishi; Lee, Tony; Tsipras, Dimitris; Soylu, Dilara; Yasunaga, Michihiro; Zhang, Yian; Narayanan, Deepak; Wu, Yuhuai (2023-10-01). "Holistic Evaluation of Language Models". Annals of the New York Academy of Sciences. 1525 (1): 140–146. arXiv:2211.09110. Bibcode:2023NYASA1525..140B. doi:10.1111/nyas.15007. PMID 37230490.

[2] Chang, Yupeng; Wang, Xu; Wang, Jindong; Wu, Yuan; Yang, Linyi; Zhu, Kaijie; Chen, Hao; Yi, Xiaoyuan; Wang, Cunxiang; Wang, Yidong; Ye, Wei; Zhang, Yue; Chang, Yi; Yu, Philip S.; Yang, Qiang (2024-06-30). "A Survey on Evaluation of Large Language Models". ACM Transactions on Intelligent Systems and Technology. 15 (3): 1–45. doi:10.1145/3641289. ISSN 2157-6904.

[3] Chen, Danqi; Yih, Wen-tau (July 2020). Savary, Agata; Zhang, Yue (eds.). "Open-Domain Question Answering". Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts. Online: Association for Computational Linguistics: 34–37. doi:10.18653/v1/2020.acl-tutorials.8.

[4] Weng, Lilian (2020-10-29). "How to Build an Open-Domain Question Answering System?". lilianweng.github.io. Retrieved 2025-03-05.

[:5-5] 1 2 Radford, Alec; Wu, Jeffrey; Child, Rewon; Luan, David; Amodei, Dario; Sutskever, Ilya (February 14, 2019). "Language Models are Unsupervised Multitask Learners" (PDF). OpenAI.

[6] "English Gigaword Fifth Edition". Linguistic Data Consortium. June 17, 2011. Retrieved 2025-05-17.

[:4-7] 1 2 Chelba, Ciprian; Mikolov, Tomas; Schuster, Mike; Ge, Qi; Brants, Thorsten; Koehn, Phillipp; Robinson, Tony (2013). "One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling". arXiv:1312.3005 [cs.CL].

[:1-8] 1 2 Dehghani, Mostafa; Tay, Yi; Gritsenko, Alexey A.; Zhao, Zhe; Houlsby, Neil; Diaz, Fernando; Metzler, Donald; Vinyals, Oriol (2021-07-14). "The Benchmark Lottery". arXiv:2107.07002 [cs.LG].

[9] DeepSeek-AI; Guo, Daya; Yang, Dejian; Zhang, Haowei; Song, Junxiao; Zhang, Ruoyu; Xu, Runxin; Zhu, Qihao; Ma, Shirong (2025-01-22). "DeepSeek-R1 incentivizes reasoning in LLMS through reinforcement learning". Nature. 645 (8081): 633–638. arXiv:2501.12948. Bibcode:2025Natur.645..633G. doi:10.1038/s41586-025-09422-z. PMC 12443585. PMID 40962978.

[:42-10] 1 2 Chen, Mark; Tworek, Jerry; Jun, Heewoo; Yuan, Qiming; Pinto, Henrique Ponde de Oliveira; Kaplan, Jared; Edwards, Harri; Burda, Yuri; Joseph, Nicholas (2021-07-14). "Evaluating Large Language Models Trained on Code". arXiv:2107.03374 [cs.LG].

[11] Vedantam, Ramakrishna; Lawrence Zitnick, C.; Parikh, Devi (2015). "CIDEr: Consensus-based image description evaluation". 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4566–4575. arXiv:1411.5726. doi:10.1109/CVPR.2015.7299087. ISBN 978-1-4673-6964-0.

[12] Anderson, Peter; Fernando, Basura; Johnson, Mark; Gould, Stephen (2016). "SPICE: Semantic Propositional Image Caption Evaluation". In Leibe, Bastian; Matas, Jiri; Sebe, Nicu; Welling, Max (eds.). Computer Vision – ECCV 2016. Lecture Notes in Computer Science. Vol. 9909. Cham: Springer International Publishing. pp. 382–398. arXiv:1607.08822. doi:10.1007/978-3-319-46454-1_24. ISBN 978-3-319-46454-1.

[13] Northcutt, Curtis G.; Athalye, Anish; Mueller, Jonas (2021-11-07). "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks". arXiv:2103.14749 [stat.ML].

[14] Richie, Russell; Grover, Sachin; Tsui, Fuchiang (Rich) (May 2022). Demner-Fushman, Dina; Cohen, Kevin Bretonnel; Ananiadou, Sophia; Tsujii, Junichi (eds.). "Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations". Proceedings of the 21st Workshop on Biomedical Language Processing. Dublin, Ireland: Association for Computational Linguistics: 275–284. doi:10.18653/v1/2022.bionlp-1.26.

[15] Artstein, Ron (2017), "Inter-annotator Agreement", in Ide, Nancy; Pustejovsky, James (eds.), Handbook of Linguistic Annotation, Dordrecht: Springer Netherlands, pp. 297–313, doi:10.1007/978-94-024-0881-2_11, ISBN 978-94-024-0881-2, retrieved 2025-02-22

[16] Nie, Yixin; Zhou, Xiang; Bansal, Mohit (November 2020). "What Can We Learn from Collective Human Opinions on Natural Language Inference Data?". In Webber, Bonnie; Cohn, Trevor; He, Yulan; Liu, Yang (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics. pp. 9131–9143. doi:10.18653/v1/2020.emnlp-main.734.

[17] Pavlick, Ellie; Kwiatkowski, Tom (November 2019). "Inherent Disagreements in Human Textual Inferences". Transactions of the Association for Computational Linguistics. 7: 677–694. doi:10.1162/tacl_a_00293. ISSN 2307-387X.

[18] Gururangan, Suchin; Swayamdipta, Swabha; Levy, Omer; Schwartz, Roy; Bowman, Samuel R.; Smith, Noah A. (2018-04-16). "Annotation Artifacts in Natural Language Inference Data". arXiv:1803.02324 [cs.CL].

[19] Deng, Chunyuan; Zhao, Yilun; Tang, Xiangru; Gerstein, Mark; Cohan, Arman (June 2024). "Investigating Data Contamination in Modern Benchmarks for Large Language Models". In Duh, Kevin; Gomez, Helena; Bethard, Steven (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City, Mexico: Association for Computational Linguistics. pp. 8706–8719. arXiv:2311.09783. doi:10.18653/v1/2024.naacl-long.482.

[20] LI, Yanyang (2025-02-17), lyy1994/awesome-data-contamination, retrieved 2025-02-22

[21] Shannon, C. E. (1951). "Prediction and Entropy of Printed English". Bell System Technical Journal. 30 (1): 50–64. Bibcode:1951BSTJ...30...50S. doi:10.1002/j.1538-7305.1951.tb01366.x. ISSN 1538-7305.

[22] Radford, Alec; Wu, Jeffrey; Child, Rewon; Luan, David; Amodei, Dario; Sutskever, Ilya (February 14, 2019). "Better language models and their implications". OpenAI.

[23] Magnusson, Ian; Bhagia, Akshita; Hofmann, Valentin; Soldaini, Luca; Jha, Ananya Harsh; Tafjord, Oyvind; Schwenk, Dustin; Walsh, Evan Pete; Elazar, Yanai (2024-12-07). "Paloma: A Benchmark for Evaluating Language Model Fit". arXiv:2312.10523 [cs.CL].

[24] Davis, Ernest (2023-10-23). "Benchmarks for Automated Commonsense Reasoning: A Survey". ACM Comput. Surv. 56 (4): 81:1–81:41. arXiv:2302.04752. doi:10.1145/3615355. ISSN 0360-0300.

[25] Chen, Danqi; Bolton, Jason; Manning, Christopher D. (2016-08-08). "A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task". arXiv:1606.02858 [cs.CL].

[26] "The Corpus of Linguistic Acceptability (CoLA)". nyu-mll.github.io. Archived from the original on 2025-03-11. Retrieved 2025-04-19.

[27] Warstadt, Alex; Singh, Amanpreet; Bowman, Samuel R. (November 2019). "Neural Network Acceptability Judgments". Transactions of the Association for Computational Linguistics. 7: 625–641. arXiv:1805.12471. doi:10.1162/tacl_a_00290. ISSN 2307-387X.

[28] Zellers, Rowan; Holtzman, Ari; Bisk, Yonatan; Farhadi, Ali; Choi, Yejin (2019-05-19). "HellaSwag: Can a Machine Really Finish Your Sentence?". arXiv:1905.07830 [cs.CL].

[29] "HellaSwag". rowanzellers.com. Retrieved 2025-02-06.

[30] Paperno, Denis; Kruszewski, Germán; Lazaridou, Angeliki; Pham, Quan Ngoc; Bernardi, Raffaella; Pezzelle, Sandro; Baroni, Marco; Boleda, Gemma; Fernández, Raquel (2016-06-20). "The LAMBADA dataset: Word prediction requiring a broad discourse context". arXiv:1606.06031 [cs.CL].

[31] Williams, Adina; Nangia, Nikita; Bowman, Samuel R. (2018-02-19). "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference". arXiv:1704.05426 [cs.CL].

[32] Lai, Guokun; Xie, Qizhe; Liu, Hanxiao; Yang, Yiming; Hovy, Eduard (2017-12-05). "RACE: Large-scale ReAding Comprehension Dataset From Examinations". arXiv:1704.04683 [cs.CL].

[33] Bowman, Samuel R.; Angeli, Gabor; Potts, Christopher; Manning, Christopher D. (September 2015). "A large annotated corpus for learning natural language inference". In Màrquez, Lluís; Callison-Burch, Chris; Su, Jian (eds.). Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics. pp. 632–642. arXiv:1508.05326. doi:10.18653/v1/D15-1075.

[34] "The Stanford Natural Language Processing Group". nlp.stanford.edu. Retrieved 2025-02-22.

[35] Zellers, Rowan; Bisk, Yonatan; Schwartz, Roy; Choi, Yejin (2018-08-16). "SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference". arXiv:1808.05326 [cs.CL].

[Hector-36] Levesque, Hector; Davis, Ernest; Morgenstern, Leora (2012). The Winograd Schema Challenge. Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning.

[37] Kocijan, Vid; Davis, Ernest; Lukasiewicz, Thomas; Marcus, Gary; Morgenstern, Leora (2023-07-11). "The defeat of the Winograd Schema Challenge". Artificial Intelligence. 325 103971. arXiv:2201.02387. doi:10.1016/j.artint.2023.103971. ISSN 0004-3702. S2CID 245827747.

[Sakaguchi-38] Sakaguchi, Keisuke; Le Bras, Ronan; Bhagavatula, Chandra; Choi, Yejin (2019). "WinoGrande: An Adversarial Winograd Schema Challenge at Scale". arXiv:1907.10641 [cs.CL].

[39] Bojar, Ondřej; Buck, Christian; Federmann, Christian; Haddow, Barry; Koehn, Philipp; Leveling, Johannes; Monz, Christof; Pecina, Pavel; Post, Matt; Saint-Amand, Herve; Soricut, Radu; Specia, Lucia; Tamchyna, Aleš (June 2014). Bojar, Ondřej; Buck, Christian; Federmann, Christian; Haddow, Barry; Koehn, Philipp; Monz, Christof; Post, Matt; Specia, Lucia (eds.). "Findings of the 2014 Workshop on Statistical Machine Translation". Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, Maryland, USA: Association for Computational Linguistics: 12–58. doi:10.3115/v1/W14-3302. hdl:20.500.11820/789fbc29-61e0-4529-af4a-819461c57a8f.

[40] Daum, Shilo; Shapira, Tal; Bremler-Barr, Anat; Hay, David (2024). "Non-uniformity is All You Need: Efficient and Timely Encrypted Traffic Classification with ECHO". arXiv:2406.01852 [cs.NI].

[41] Zhou, Jeffrey; Lu, Tianjian; Mishra, Swaroop; Brahma, Siddhartha; Basu, Sujoy; Luan, Yi; Zhou, Denny; Hou, Le (2023-11-14). "Instruction-Following Evaluation for Large Language Models". arXiv:2311.07911 [cs.CL].

[:3-42] 1 2 Zheng, Lianmin; Chiang, Wei-Lin; Sheng, Ying; Zhuang, Siyuan; Wu, Zhanghao; Zhuang, Yonghao; Lin, Zi; Li, Zhuohan; Li, Dacheng (2023-12-24). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena". arXiv:2306.05685 [cs.CL].

[43] lmarena/arena-hard-auto, Arena, 2026-09-29, retrieved 2026-10-04

[44] Sirdeshmukh, Ved; Deshpande, Kaustubh; Mols, Johannes; Jin, Lifeng; Cardona, Ed-Yeremai; Lee, Dean; Kritz, Jeremy; Primack, Willow; Yue, Summer; Xing, Chen (2025). "MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMS". arXiv:2501.17399 [cs.CL].

[45] Mishra, Swaroop; Khashabi, Daniel; Baral, Chitta; Hajishirzi, Hannaneh (2022-03-14). "Cross-Task Generalization via Natural Language Crowdsourcing Instructions". arXiv:2104.08773 [cs.CL].

[46] Wang, Yizhong; Mishra, Swaroop; Alipoormolabashi, Pegah; Kordi, Yeganeh; Mirzaei, Amirreza; Arunkumar, Anjana; Ashok, Arjun; Dhanasekaran, Arut Selvan; Naik, Atharva (2022-10-24). "Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks". arXiv:2204.07705 [cs.CL].

[47] Minev, Z. K.; Mundhada, S. O.; Shankar, S.; Reinhold, P.; Gutiérrez-Jáuregui, R.; Schoelkopf, R. J.; Mirrahimi, M.; Carmichael, H. J.; Devoret, M. H. (2019). "To catch and reverse a quantum jump mid-flight". Nature. 570 (7760): 200–204. arXiv:1803.00545. Bibcode:2019Natur.570..200M. doi:10.1038/s41586-019-1287-z. PMID 31160725.

[48] Masry, Ahmed; Do, Xuan Long; Tan, Jia Qing; Joty, Shafiq; Hoque, Enamul (May 2022). "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning". In Muresan, Smaranda; Nakov, Preslav; Villavicencio, Aline (eds.). Findings of the Association for Computational Linguistics: ACL 2022. Dublin, Ireland: Association for Computational Linguistics. pp. 2263–2279. arXiv:2203.10244. doi:10.18653/v1/2022.findings-acl.177.

[49] Reddy, Siva; Chen, Danqi; Manning, Christopher D. (2019-05-01). "CoQA: A Conversational Question Answering Challenge". Transactions of the Association for Computational Linguistics. 7: 249–266. arXiv:1808.07042. doi:10.1162/tacl_a_00266. ISSN 2307-387X.

[50] "Industry Documents Library". industrydocuments.ucsf.edu. Retrieved 2025-04-05.

[51] "DocVQA". www.docvqa.org. Retrieved 2025-04-05.

[52] Mathew, Minesh; Karatzas, Dimosthenis; Jawahar, C. V. (2021). "DocVQA: A Dataset for VQA on Document Images". pp. 2200–2209.

[53] Mendoza-Arenas, J. J.; Gómez-Ruiz, F. J.; Rodríguez, F. J.; Quiroga, L. (2019). "Enhancing violations of Leggett-Garg inequalities in nonequilibrium correlated many-body systems by interactions and decoherence". Scientific Reports. 9 (1): 17772. arXiv:1903.00016. Bibcode:2019NatSR...917772M. doi:10.1038/s41598-019-54121-1. PMC 6882789. PMID 31780693.

[54] Khrennikov, Andrei; Ozawa, Masanao; Benninger, Felix; Shor, Oded (2025). "Coupling quantum-like cognition with the neuronal networks within generalized probability theory". Journal of Mathematical Psychology. 125 102923. arXiv:2411.00036. doi:10.1016/j.jmp.2025.102923.

[55] Yang, Zhilin; Qi, Peng; Zhang, Saizheng; Bengio, Yoshua; Cohen, William; Salakhutdinov, Ruslan; Manning, Christopher D. (October 2018). Riloff, Ellen; Chiang, David; Hockenmaier, Julia; Tsujii, Jun'ichi (eds.). "HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering". Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics: 2369–2380. doi:10.18653/v1/D18-1259.

[56] Richardson, Matthew; Burges, Christopher J.C.; Renshaw, Erin (October 2013). "MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text". In Yarowsky, David; Baldwin, Timothy; Korhonen, Anna; Livescu, Karen; Bethard, Steven (eds.). Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics. pp. 193–203. doi:10.18653/v1/D13-1020.

[57] Kwiatkowski, Tom; Palomaki, Jennimaria; Redfield, Olivia; Collins, Michael; Parikh, Ankur; Alberti, Chris; Epstein, Danielle; Polosukhin, Illia; Devlin, Jacob; Lee, Kenton; Toutanova, Kristina; Jones, Llion; Kelcey, Matthew; Chang, Ming-Wei; Dai, Andrew M. (2019-08-01). "Natural Questions: A Benchmark for Question Answering Research". Transactions of the Association for Computational Linguistics. 7: 453–466. doi:10.1162/tacl_a_00276. ISSN 2307-387X.

[58] Enns, John (2018). "Multiplicities in the ordinary part of mod $p$ cohomology for $\mathrm{GL}_n(\mathbb{Q}_p)$". arXiv:1809.00278 [math.NT].

[59] Du Toit, E.J.; o'Brien, M.R.; Vann, R.G.L. (2017). "A Kinetic Study of Microwave Start-up of Tokamak Plasmas". EPJ Web of Conferences. 147: 01002. arXiv:1704.00517. Bibcode:2017EPJWC.14701002D. doi:10.1051/epjconf/201714701002.

[60] Baik, Jinho; Barraquand, Guillaume; Corwin, Ivan; Suidan, Toufic (2018). "Pfaffian Schur processes and last passage percolation in a half-quadrant". The Annals of Probability. 46 (6). arXiv:1606.00525. doi:10.1214/17-AOP1226.

[61] Wallis, Ben (2018). "Closed ideals of operators acting on some families of sequence spaces". arXiv:1806.00382 [math.FA].

[62] Geva, Mor; Khashabi, Daniel; Segal, Elad; Khot, Tushar; Roth, Dan; Berant, Jonathan (2021-04-26). "Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies". Transactions of the Association for Computational Linguistics. 9: 346–361. doi:10.1162/tacl_a_00370. ISSN 2307-387X.

[63] Berant, Jonathan; Chou, Andrew; Frostig, Roy; Liang, Percy (October 2013). "Semantic Parsing on Freebase from Question-Answer Pairs". In Yarowsky, David; Baldwin, Timothy; Korhonen, Anna; Livescu, Karen; Bethard, Steven (eds.). Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics. pp. 1533–1544. doi:10.18653/v1/D13-1160.

[64] "C-Eval: 一个适用于大语言模型的多层次多学科中文评估套件". cevalbenchmark.com. Retrieved 2025-02-25.

[65] Jin, Di; Pan, Eileen; Oufattole, Nassim; Weng, Wei-Hung; Fang, Hanyi; Szolovits, Peter (January 2021). "What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams". Applied Sciences. 11 (14): 6421. doi:10.3390/app11146421. hdl:1721.1/136684.2. ISSN 2076-3417.

[66] Majumdar, Arjun; Ajay, Anurag; Zhang, Xiaohan; Putta, Pranav; Yenamandra, Sriram; Henaff, Mikael; Silwal, Sneha; Mcvay, Paul; Maksymets, Oleksandr; Arnaud, Sergio; Yadav, Karmesh; Li, Qiyang; Newman, Ben; Sharma, Mohit; Berges, Vincent (2024). "OpenEQA: Embodied Question Answering in the Era of Foundation Models". 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16488–16498. doi:10.1109/CVPR52733.2024.01560. ISBN 979-8-3503-5300-6.

[67] Bisk, Yonatan; Zellers, Rowan; Bras, Ronan Le; Gao, Jianfeng; Choi, Yejin (2020-04-03). "PIQA: Reasoning about Physical Commonsense in Natural Language". Proceedings of the AAAI Conference on Artificial Intelligence. 34 (5): 7432–7439. arXiv:1911.11641. doi:10.1609/aaai.v34i05.6239. ISSN 2374-3468.

[68] "Grok-1.5 Vision Preview | xAI". x.ai. Retrieved 2025-03-12.

[69] Lu, Pan; Mishra, Swaroop; Xia, Tanglin; Qiu, Liang; Chang, Kai-Wei; Zhu, Song-Chun; Tafjord, Oyvind; Clark, Peter; Kalyan, Ashwin (2022-12-06). "Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering". Advances in Neural Information Processing Systems. 35: 2507–2521. arXiv:2209.09513. doi:10.52202/068431-0182. ISBN 978-1-7138-7108-8.{{cite journal}}: CS1 maint: periodical has ISBN (link)

[70] Wei, Jason; Karina, Nguyen; Chung, Hyung Won; Jiao, Yunxin Joy; Papay, Spencer; Glaese, Amelia; Schulman, John; Fedus, William (2024-11-07), Measuring short-form factuality in large language models, arXiv:2411.04368

[71] "Introducing SimpleQA". openai.com. Retrieved 2025-10-02.

[72] Matias, José; Oliveira, Julio P.C.; Le Roux, Galo A.C.; Jäschke, Johannes (2022). "Steady-state real-time optimization using transient measurements on an experimental rig". Journal of Process Control. 115: 181–196. arXiv:2109.00795. doi:10.1016/j.jprocont.2022.04.015.

[73] Lovesey, S. W. (2022). "Polar magnetization unveiled by polarized neutron diffraction". Physical Review B. 106 (6) 064415. arXiv:2206.00461. Bibcode:2022PhRvB.106f4415L. doi:10.1103/PhysRevB.106.064415.

[74] Ddamulira, Mahadi; Emong, Paul; Geoffrey Ismail Mirumbe (2022). "Members of Narayana's cow sequence that are concatenations of two repdigits". arXiv:2210.00926 [math.NT].

[75] Kazemi, Mehran; Fatemi, Bahare; Bansal, Hritik; Palowitch, John; Anastasiou, Chrysovalantis; Sanket Vaibhav Mehta; Jain, Lalit K.; Aglietti, Virginia; Jindal, Disha; Chen, Peter; Dikkala, Nishanth; Tyen, Gladys; Liu, Xin; Shalit, Uri; Chiappa, Silvia; Olszewska, Kate; Tay, Yi; Tran, Vinh Q.; Le, Quoc V.; Firat, Orhan (2025). "BIG-Bench Extra Hard". arXiv:2502.19187 [cs.CL].

[76] Wang, Alex; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omer; Bowman, Samuel R. (2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding". arXiv:1804.07461 [cs.CL].

[77] "GLUE Benchmark". gluebenchmark.com. Retrieved 2019-02-25.

[78] Herzig, Florian; Kozioł, Karol; Vignéras, Marie-France (2020). "On the Existence of Admissible Supersingular Representations of -Adic Reductive Groups". Forum of Mathematics, Sigma. 8 e2. arXiv:1905.00053. doi:10.1017/fms.2019.50.

[79] "Holistic Evaluation of Language Models (HELM)". crfm.stanford.edu. Retrieved 2026-02-06.

[80] Hernandez, A.; Woo, S.; Corrales, H.; Parra, I.; Kim, E.; Llorca, D. F.; Sotelo, M. A. (2020). "3D-DEEP: 3-Dimensional Deep-learning based on elevation patterns for road scene interpretation". 2020 IEEE Intelligent Vehicles Symposium (IV). pp. 892–898. arXiv:2009.00330. doi:10.1109/IV47402.2020.9304601. ISBN 978-1-7281-6673-5.

[81] Arjomandbigdeli, Ali; Mata, Andrew; Bak, Stanley (2024). "Verification of Neural Network Control Systems in Continuous Time". AI Verification. Lecture Notes in Computer Science. Vol. 14846. pp. 100–115. arXiv:2406.00157. doi:10.1007/978-3-031-65112-0_5. ISBN 978-3-031-65111-3.

[82] Zimmerman, Charlotte; Olsho, Alexis; Loverude, Michael; Suzanne White Brahmia (2023). "Expert covariational reasoning resources in physics graphing tasks". arXiv:2306.00921 [physics.ed-ph].

[83] "openai/MMMLU · Datasets at Hugging Face". huggingface.co. 2024-10-22. Retrieved 2025-02-28.

[84] "ECI Documentation – Overview". Epoch AI. Retrieved 2026-09-24.

[85] "Intelligence Benchmarking | Artificial Analysis". artificialanalysis.ai. Retrieved 2026-09-24.

[86] "MMMU". mmmu-benchmark.github.io. Retrieved 2025-02-28.

[87] Ates, Halim Cagri; Bhargava, Shruti; Li, Site; Lu, Jiarui; Maddula, Siddhardha; Moniz, Joel Ruben Antony; Nalamalapu, Anil Kumar; Nguyen, Roman Hoang; Ozyildirim, Melis; Patel, Alkesh; Piraviperumal, Dhivya; Renkens, Vincent; Samal, Ankit; Tran, Thy; Tseng, Bo-Hsiang; Yu, Hong; Zhang, Yuan; Zou, Shirley (2023). "MARRS: Multimodal Reference Resolution System". Proceedings of the Sixth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2023). pp. 51–58. arXiv:2311.01650. doi:10.18653/v1/2023.crac-main.7.

[88] Ma, Yingjie; Guo, Jing; Maloney, Andrew; Braatz, Richard (2025). "Quasi-steady-state approach for efficient multiscale simulation and optimization of mAb glycosylation in CHO cell culture". Chemical Engineering Science. 318 122162. arXiv:2409.00281. doi:10.1016/j.ces.2025.122162.

[89] "MMT-Bench". mmt-bench.github.io. Retrieved 2025-07-12.

[90] Ouyang, Linke; Qu, Yuan; Zhou, Hongbin; Zhu, Jiawei (2024). "OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations". arXiv:2412.07626 [cs.CV].

[91] Padlewski, Piotr; Bain, Max; Henderson, Matthew; Zhu, Zhongkai; Relan, Nishant; Pham, Hai; Ong, Donovan; Aleksiev, Kaloyan; Ormazabal, Aitor; Phua, Samuel; Yeo, Ethan; Lamprecht, Eugenie; Liu, Qi; Wang, Yuqi; Chen, Eric; Fu, Deyu; Li, Lei; Zheng, Che; Cyprien de Masson d'Autume; Yogatama, Dani; Artetxe, Mikel; Tay, Yi (2024). "Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models". arXiv:2405.02287 [cs.CL].

[92] Hu, Kairui; Wu, Penghao; Pu, Fanyi; Xiao, Wang; Zhang, Yuanhan; Yue, Xiang; Li, Bo; Liu, Ziwei (2025). "Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos". arXiv:2501.13826 [cs.CV].

[93] "Video-MMMU". videommmu.github.io. Retrieved 2025-06-07.

[94] "FACTS Grounding: A new benchmark for evaluating the factuality of large language models". Google DeepMind. 2024-12-17. Retrieved 2025-06-07.

[95] Jacovi, Alon; Wang, Andrew; Alberti, Chris; Tao, Connie; Lipovetz, Jon; Olszewska, Kate; Haas, Lukas; Liu, Michelle; Keating, Nate; Bloniarz, Adam; Saroufim, Carl; Fry, Corey; Marcus, Dror; Kukliansky, Doron; Gaurav Singh Tomar; Swirhun, James; Xing, Jinwei; Wang, Lily; Gurumurthy, Madhu; Aaron, Michael; Ambar, Moran; Fellinger, Rachana; Wang, Rui; Zhang, Zizhao; Goldshtein, Sasha; Das, Dipanjan (2025). "The FACTS Grounding Leaderboard: Benchmarking LLMS' Ability to Ground Responses to Long-Form Input". arXiv:2501.03200 [cs.CL].

[96] Zhang, Xinrong; Chen, Yingfa; Hu, Shengding; Xu, Zihang; Chen, Junhao; Moo Khai Hao; Han, Xu; Zhen Leng Thai; Wang, Shuo; Liu, Zhiyuan; Sun, Maosong (2024). "$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens". arXiv:2402.13718 [cs.CL].

[97] An, Chenxin; Gong, Shansan; Zhong, Ming; Zhao, Xingjian; Li, Mukai; Zhang, Jun; Kong, Lingpeng; Qiu, Xipeng (August 2024). Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek (eds.). "L-Eval: Instituting Standardized Evaluation for Long Context Language Models". Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics: 14388–14411. arXiv:2307.11088. doi:10.18653/v1/2024.acl-long.776.

[98] Lee, Jinhyuk; Chen, Anthony; Dai, Zhuyun; Dua, Dheeru; Devendra Singh Sachan; Boratko, Michael; Luan, Yi; Arnold, Sébastien M. R.; Perot, Vincent; Dalmia, Siddharth; Hu, Hexiang; Lin, Xudong; Pasupat, Panupong; Amini, Aida; Cole, Jeremy R.; Riedel, Sebastian; Naim, Iftekhar; Chang, Ming-Wei; Guu, Kelvin (2024). "Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?". arXiv:2406.13121 [cs.CL].

[99] Li, Tianle; Zhang, Ge; Quy Duc Do; Yue, Xiang; Chen, Wenhu (2024). "Long-context LLMS Struggle with Long In-context Learning". arXiv:2404.02060 [cs.CL].

[100] "LongBench v2". longbench2.github.io. Retrieved 2025-02-21.

[101] Bai, Yushi; Tu, Shangqing; Zhang, Jiajie; Peng, Hao; Wang, Xiaozhi; Lv, Xin; Cao, Shulin; Xu, Jiazheng; Hou, Lei; Dong, Yuxiao; Tang, Jie; Li, Juanzi (2024). "LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks". arXiv:2412.15204 [cs.CL].

[102] Tay, Yi; Dehghani, Mostafa; Abnar, Samira; Shen, Yikang; Bahri, Dara; Pham, Philip; Rao, Jinfeng; Yang, Liu; Ruder, Sebastian; Metzler, Donald (2020). "Long Range Arena: A Benchmark for Efficient Transformers". arXiv:2011.04006 [cs.LG].

[103] Vodrahalli, Kiran; Ontanon, Santiago; Tripuraneni, Nilesh; Xu, Kelvin; Jain, Sanil; Shivanna, Rakesh; Hui, Jeffrey; Dikkala, Nishanth; Kazemi, Mehran (2024-09-20). "Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries". arXiv:2409.12640 [cs.CL].

[104] Visser, Eline (2022). A grammar of Kalamang. Language Science Press. ISBN 978-3-96110-343-0.

[105] Visser, Eline (2021-09-24), "dictionaria/kalamang: Kalamang dictionary", Zenodo, doi:10.5281/ZENODO.5526419, retrieved 2025-04-05

[106] Tanzer, Garrett; Suzgun, Mirac; Visser, Eline; Jurafsky, Dan; Melas-Kyriazi, Luke (2023). "A Benchmark for Learning to Translate a New Language from One Grammar Book". arXiv:2309.16575 [cs.CL].

[107] @GregKamradt (November 8, 2023). "Pressure Testing GPT-4-128K With Long Context Recall..." (Tweet) – via X (formerly Twitter).

[108] Modarressi, Ali; Deilamsalehy, Hanieh; Dernoncourt, Franck; Bui, Trung; Rossi, Ryan A.; Yoon, Seunghyun; Schütze, Hinrich (2025). "NoLiMa: Long-Context Evaluation Beyond Literal Matching". arXiv:2502.05167 [cs.CL].

[109] Hsieh, Cheng-Ping; Sun, Simeng; Kriman, Samuel; Acharya, Shantanu; Rekesh, Dima; Jia, Fei; Zhang, Yang; Ginsburg, Boris (2024). "RULER: What's the Real Context Size of Your Long-Context Language Models?". arXiv:2404.06654 [cs.CL].

[110] Shaham, Uri; Ivgi, Maor; Efrat, Avia; Berant, Jonathan; Levy, Omer (2023). "ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding". arXiv:2305.14196 [cs.CL].

[111] Aider-AI/polyglot-benchmark, Aider AI, 2025-03-29, retrieved 2025-03-30

[112] Hendrycks, Dan; Basart, Steven; Kadavath, Saurav; Mazeika, Mantas; Arora, Akul; Guo, Ethan; Burns, Collin; Puranik, Samir; He, Horace; Song, Dawn; Steinhardt, Jacob (2021). "Measuring Coding Challenge Competence with APPS". arXiv:2105.09938 [cs.SE].

[113] Zhuo, Terry Yue; Chien, Vu Minh; Chim, Jenny; Hu, Han; Yu, Wenhao; Widyasari, Ratnadira; Yusuf, Imam Nur Bani; Zhan, Haolan; He, Junda; Paul, Indraneil; Brunner, Simon; Gong, Chen; Hoang, James; Zebaze, Armel Randy; Hong, Xiaoheng (2024-10-04). "BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions". Iclr 2025. arXiv:2406.15877.

[114] "BigCodeBench Leaderboard". bigcode-bench.github.io. Retrieved 2025-04-09.

[115] "Berkeley Function Calling Leaderboard". gorilla.cs.berkeley.edu. Retrieved 2025-03-11.

[116] "CodeElo". codeelo-bench.github.io. Retrieved 2025-02-13.

[117] Lai, Yuhang; Li, Chengxi; Wang, Yiming; Zhang, Tianyi; Zhong, Ruiqi; Zettlemoyer, Luke; Scott Wen-tau Yih; Fried, Daniel; Wang, Sida; Yu, Tao (2022). "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation". arXiv:2211.11501 [cs.SE].

[118] "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation". ds1000-code-gen.github.io. Retrieved 2025-03-11.

[119] Jing, Liqiang; Huang, Zhehui; Wang, Xiaoyang; Yao, Wenlin; Yu, Wenhao; Ma, Kaixin; Zhang, Hongming; Du, Xinya; Yu, Dong (2024). "DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?". arXiv:2409.07703 [cs.AI].

[:0-120] 1 2 Austin, Jacob; Odena, Augustus; Nye, Maxwell; Bosma, Maarten; Michalewski, Henryk; Dohan, David; Jiang, Ellen; Cai, Carrie; Terry, Michael; Le, Quoc; Sutton, Charles (2021). "Program Synthesis with Large Language Models". arXiv:2108.07732 [cs.PL].

[121] Kushman, Nate; Artzi, Yoav; Zettlemoyer, Luke; Barzilay, Regina (June 2014). Toutanova, Kristina; Wu, Hua (eds.). "Learning to Automatically Solve Algebra Word Problems". Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Baltimore, Maryland: Association for Computational Linguistics: 271–281. doi:10.3115/v1/P14-1026.

[122] Huang, Danqing; Shi, Shuming; Lin, Chin-Yew; Yin, Jian; Ma, Wei-Ying (August 2016). Erk, Katrin; Smith, Noah A. (eds.). "How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation". Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics: 887–896. doi:10.18653/v1/P16-1084.

[123] Ling, Wang; Yogatama, Dani; Dyer, Chris; Blunsom, Phil (July 2017). Barzilay, Regina; Kan, Min-Yen (eds.). "Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems". Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics: 158–167. arXiv:1705.04146. doi:10.18653/v1/P17-1015.

[124] Glazer, Elliot; Erdil, Ege; Besiroglu, Tamay; Chicharro, Diego; et al. (2024). "FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI". arXiv:2411.04872 [cs.AI].

[125] "About FrontierMath". Epoch AI. Retrieved 2025-10-02.

[126] AI, Epoch (2026-01-27). "Introducing FrontierMath: Open Problems". Epoch AI. Retrieved 2026-03-20.

[127] "FrontierMath: Open Problems - Unsolved Mathematical Challenges". Epoch AI. Retrieved 2026-03-20.

[128] Zhang, Hugh; Da, Jeff; Lee, Dean; Robinson, Vaughn; et al. (2024). "A Careful Examination of Large Language Model Performance on Grade School Arithmetic". arXiv:2405.00332 [cs.CL].

[:2-129] Cobbe, Karl; Kosaraju, Vineet; Bavarian, Mohammad; Chen, Mark; et al. (2021). "Training Verifiers to Solve Math Word Problems". arXiv:2110.14168 [cs.LG].

[130] "madrylab/gsm8k-platinum · Datasets at Hugging Face". huggingface.co. Retrieved 2025-03-07.

[131] Luong, Thang; Hwang, Dawsen; Nguyen, Hoang H; Ghiasi, Golnaz; et al. (November 2025). Christodoulopoulos, Christos; Chakraborty, Tanmoy; Rose, Carolyn; Peng, Violet (eds.). "Towards Robust Mathematical Reasoning". Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics: 35406–35430. doi:10.18653/v1/2025.emnlp-main.1794. ISBN 979-8-89176-332-6.{{cite journal}}: CS1 maint: periodical has ISBN (link)

[132] Luong, Thang; Hwang, Dawsen; Nguyen, Hoang H.; Ghiasi, Golnaz; et al., "Towards Robust Mathematical Reasoning", Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, retrieved 2025-11-08 – via IMO-Bench

[133] Hendrycks, Dan; Burns, Collin; Kadavath, Saurav; Arora, Akul; Basart, Steven; Tang, Eric; Song, Dawn; Steinhardt, Jacob (2021). "Measuring Mathematical Problem Solving with the MATH Dataset". arXiv:2103.03874 [cs.LG].

[134] "MATH-Perturb". math-perturb.github.io. Retrieved 2025-04-09.

[135] Wang, Yan; Liu, Xiaojiang; Shi, Shuming (September 2017). "Deep Neural Solver for Math Word Problems". In Palmer, Martha; Hwa, Rebecca; Riedel, Sebastian (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics. pp. 845–854. doi:10.18653/v1/D17-1088.

[136] "MathArena.ai". matharena.ai. Retrieved 2025-02-22.

[137] Liu, Hongwei; Zheng, Zilong; Qiao, Yuxuan; Duan, Haodong; et al. (2024). "MathBench: Evaluating the Theory and Application Proficiency of LLMS with a Hierarchical Mathematics Benchmark". arXiv:2405.12209 [cs.CL].

[138] th-eval (2025-01-26), math-eval/MathEval, retrieved 2025-01-27

[139] Amini, Aida; Gabriel, Saadia; Lin, Peter; Koncel-Kedziorski, Rik; Choi, Yejin; Hajishirzi, Hannaneh (2019). "MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms". arXiv:1905.13319 [cs.CL].

[140] openai/miniF2F, OpenAI, 2025-02-01, retrieved 2025-02-03

[141] Gao, Bofei; Song, Feifan; Yang, Zhe; Cai, Zefan; et al. (2024). "Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models". arXiv:2410.07985 [cs.CL].

[142] Azerbayev, Zhangir; Piotrowski, Bartosz; Schoelkopf, Hailey; Ayers, Edward W.; Radev, Dragomir; Avigad, Jeremy (2023). "ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics". arXiv:2302.12433 [cs.CL].

[143] Azerbayev, Zhangir (2025-04-02), zhangir-azerbayev/ProofNet, retrieved 2025-04-03

[144] deepseek-ai/DeepSeek-Prover-V1.5, DeepSeek, 2025-04-01, retrieved 2025-04-03

[145] Tsoukalas, George; Lee, Jasper; Jennings, John; Xin, Jimmy; Ding, Michelle; Jennings, Michael; Thakur, Amitayush; Chaudhuri, Swarat (2024). "PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition". arXiv:2407.11214 [cs.AI].

[146] "PutnamBench: A Multilingual Mathematics Benchmark for Formal Theorem-Proving". trishullab.github.io. Retrieved 2025-04-02.

[147] Garre, Suhaas; Knutsen, Erik; Mehta, Sushant; Chen, Edwin (2026). "Riemann-Bench: A Benchmark for Moonshot Mathematics". arXiv:2604.06802 [cs.AI].

[148] "Riemann-bench: A Benchmark for Moonshot Mathematics". Surge AI. 2026-03-24. Retrieved 2026-10-04.

[149] Chen, Wenhu; Yin, Ming; Ku, Max; Lu, Pan; Wan, Yixin; Ma, Xueguang; Xu, Jianyu; Wang, Xinyi; Xia, Tony (December 2023). "TheoremQA: A Theorem-driven Question Answering Dataset". In Bouamor, Houda; Pino, Juan; Bali, Kalika (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics. pp. 7889–7901. arXiv:2305.12524. doi:10.18653/v1/2023.emnlp-main.489.

[150] Chernyshev, Konstantin; Polshkov, Vitaliy; Artemova, Ekaterina; Myasnikov, Alex; Stepanov, Vlad; Miasnikov, Alexei; Tilga, Sergei (2024). "U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMS". arXiv:2412.03205 [cs.CL].

[151] Cui, Ruixiang (2025-02-03), ruixiangcui/AGIEval, retrieved 2025-02-03

[152] "Learning to reason with LLMs". openai.com. September 12, 2024. Retrieved 2025-02-27.

[153] Rein, David; Betty Li Hou; Asa Cooper Stickland; Petty, Jackson; Richard Yuanzhe Pang; Dirani, Julien; Michael, Julian; Bowman, Samuel R. (2023). "GPQA: A Graduate-Level Google-Proof Q&A Benchmark". arXiv:2311.12022 [cs.AI].

[154] Rein, I. David (2025-08-24), idavidrein/gpqa, retrieved 2025-08-25

[155] Rein, David (2024-05-15). "Can Good Benchmarks Contain Mistakes? – NYU Alignment Research Group". Retrieved 2025-09-15.

[156] Team, M-A-P; et al. (2025). "SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines". arXiv:2502.14739 [cs.CL].

[157] Phan, Long; Gatti, Alice; Li, Nathaniel; Khoja, Adam; Kim, Ryan; Ren, Richard; Hausenloy, Jason; Zhang, Oliver; Mazeika, Mantas; Hendrycks, Dan; Han, Ziwen; Hu, Josephina; Zhang, Hugh; Zhang, Chen Bo Calvin; Shaaban, Mohamed (2026-01-28). "A benchmark of expert-level academic questions to assess AI capabilities". Nature. 649 (8099): 1139–1146. Bibcode:2026Natur.649.1139C. doi:10.1038/s41586-025-09962-4. ISSN 1476-4687. PMC 12851929. PMID 41606155.

[158] "Humanity's Last Exam". lastexam.ai. Retrieved 2025-02-02.

[159] Safety, Center for AI; AI, Scale (2026-09-22). "Introducing HLE-Diamond". Humanity's Last Exam. Retrieved 2026-09-24.

[160] "LiveBench". livebench.ai. Retrieved 2025-01-27.

[161] "MathVista: Evaluating Math Reasoning in Visual Contexts". mathvista.github.io. Retrieved 2025-03-07.

[162] "OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI". gair-nlp.github.io. Retrieved 2025-02-03.

[163] "PhilosophyBench | AI and Philosophy Research at Stanford". PhilosophyBench. Retrieved 2026-10-14.

[164] He, Chaoqun; Luo, Renjie; Bai, Yuzhuo; Hu, Shengding; Zhen Leng Thai; Shen, Junhao; Hu, Jinyi; Han, Xu; Huang, Yujie; Zhang, Yuxiang; Liu, Jie; Qi, Lei; Liu, Zhiyuan; Sun, Maosong (2024). "OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems". arXiv:2402.14008 [cs.CL].

[165] "SimpleBench". simple-bench.com. Retrieved 2025-04-09.

[166] Hendrycks, Dan; Burns, Collin; Basart, Steven; Critch, Andrew; Li, Jerry; Song, Dawn; Steinhardt, Jacob (2020). "Aligning AI With Shared Human Values". arXiv:2008.02275 [cs.CY].

[167] Santurkar, Shibani; Durmus, Esin; Ladhak, Faisal; Lee, Cinoo; Liang, Percy; Hashimoto, Tatsunori (July 2023). "Whose Opinions Do Language Models Reflect?". Proceedings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research. Vol. 202. PMLR. pp. 29971–30004. arXiv:2303.17548.

[168] Durmus, Esin; Nguyen, Karina; Liao, Thomas I.; Schiefer, Nicholas; et al. (2023). "Towards Measuring the Representation of Subjective Global Opinions in Language Models". arXiv:2306.16388 [cs.CL].

[169] Ren, Yuanyi; Ye, Haoran; Fang, Hanjun; Zhang, Xin; Song, Guojie (August 2024). "ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models". In Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics. pp. 2015–2040. arXiv:2406.04214.

v t e Large language models (LLMs)
List of LLMs AI Companies Benchmarks List of chatbots Foundation model Generative AI
Concepts	Language model NLP NLG Computational linguistics Foundation model Small language model Reasoning model Generative pre-trained transformer Transformer Attention KV cache Context window Tokenization Word embedding Parameter Hyperparameter Autoregression Mixture of experts (MoE) Inference Model compression Knowledge distillation Speculative decoding PagedAttention Neural scaling law Multimodality Open weights
Training, prompting, and alignment	Self-supervised learning Supervised learning Fine-tuning Instruction tuning RLHF Constitutional AI AI alignment AI safety Mechanistic interpretability Prompt engineering In-context learning Chain-of-thought prompting RAG Prompt injection Adversarial machine learning Hallucination Stochastic parrot Glitch token Generative engine optimization
Models	Apertus BERT BLOOM Chinchilla Claude DBRX DeepSeek-LLM Gemini Gemma GloVe GLM GPT GPT-J PanGu Granite Hy Inkling Jais LaMDA Laguna S Llama MiMo Minerva Mistral and Mixtral Muse Spark Muse Glimmer Nemotron PaLM Phi Qwen Seq2seq T5 Vicuna Word2vec XLNet
Chatbots and assistants	Amazon Q ChatGPT Character.ai Claude Claude Code DeepSeek Doubao Ernie Bot Gemini Grok Kimi Kimi Code Lumo Microsoft Copilot Meta AI Perplexity AI Sparrow You.com
Agents, coding, and applications	AI agent Intelligent agent AutoGPT CrewAI LangChain Manus Model Context Protocol Agent2Agent OpenAI Codex Vibe coding Code generation Question answering Machine translation Text summarization Chatbot Virtual assistant
Software	PyTorch TensorFlow Hugging Face llama.cpp LM Studio Ollama SGLang TensorRT-LLM vLLM ONNX OpenVINO Vector database ChromaDB Deep learning software Open-source AI software
Hardware and infrastructure	AI data center CUDA GPU High Bandwidth Memory Neural processing unit TPU
Benchmarks, evaluation, and detection	Language model benchmark AI agent benchmark MMLU Humanity's Last Exam LMArena LLM-as-a-Judge Perplexity metric GPTZero Artificial intelligence content detection Undetectable.ai
Datasets and data	Data set Text corpus Common Crawl The Pile Web scraping Synthetic data Training, validation, and test data sets
Organizations	AI21 Labs Alibaba Anthropic Baidu Cohere DeepSeek EleutherAI Google DeepMind Hugging Face Meta AI Microsoft AI Mistral AI MiniMax Moonshot AI Nvidia OpenAI OpenRouter Sarvam AI StepFun Technology Innovation Institute Thinking Machines Lab xAI
People	Sam Altman Dario Amodei Yoshua Bengio Aidan Gomez Demis Hassabis Geoffrey Hinton Andrej Karpathy Yann LeCun Percy Liang Liang Wenfeng Christopher D. Manning Arthur Mensch Mira Murati Alec Radford Noam Shazeer Ilya Sutskever Ashish Vaswani Andrew Ng
Social, economic, and governance	AI anthropomorphism AI arms race AI boom AI bubble AI-induced psychosis AI slop Competition Copyright Deaths linked to chatbots Dependency (GAID) Environmental impact Regulation Ethics Existential risk In education In healthcare Workplace impact
Category:Large language models