BIG-bench (Beyond the Imitation Game Benchmark)
Introduced by Srivastava et al. in Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
The Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities. Big-bench include more than 200 tasks.
Benchmarks
| Trend | Task | Dataset Variant | Best Model | Paper | Code |
|---|---|---|---|---|---|
|
BBH-nlp
|
Flan-PaLM 540B
|
|||
|
BIG-bench (Reasoning About Colored Objects)
|
PaLM 2
|
|||
|
BIG-bench (Penguins In A Table)
|
PaLM 2
|
|||
|
BIG-bench (Navigate)
|
PaLM 2
|
|||
|
BIG-bench (Movie Recommendation)
|
PaLM 2
|
|||
|
BIG-bench (Hyperbaton)
|
Bloomberg GPT
|
|||
|
BIG-bench (Formal Fallacies Syllogisms Negation)
|
PaLM 2
|
|||
|
BIG-bench (Date Understanding)
|
PaLM 2
|
|||
|
BIG-bench (Causal Judgment)
|
PaLM 2
|
|||
|
BIG-bench (Disambiguation QA)
|
PaLM 2
|
|||
|
BIG-bench (Temporal Sequences)
|
PaLM 2
|
|||
|
BIG-bench (Ruin Names)
|
PaLM 2
|
|||
|
BIG-bench (Sports Understanding)
|
PaLM 2
|
|||
|
BIG-bench (SNARKS)
|
PaLM 2
|
|||
|
BBH-alg
|
code-davinci-002 175B
|
|||
|
BIG-bench (Anachronisms)
|
Chinchilla-70B
|
|||
|
BIG-bench (Novel Concepts)
|
PaLM-540B
|
|||
|
BIG-bench
|
Orca 2-13B
|
|||
|
BIG-bench (Logic Grid Puzzle)
|
Chinchilla-70B
|
|||
|
BIG-bench (Winowhy)
|
PaLM-540B
|
|||
|
BIG-bench (StrategyQA)
|
PaLM-540B
|
|||
|
BIG-bench (Hindu Knowledge)
|
PaLM-540B
|
|||
|
Big-bench Lite
|
PaLM 62B
|
|||
|
BIG-bench-lite
|
GLM-130B
|
|||
|
BIG-bench (Known Unknowns)
|
PaLM-540B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench (Logical Fallacy Detection)
|
Chinchilla-70B
|
|||
|
BIG-bench (Logical Sequence)
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Chinchilla-70B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench (Ruin Names)
|
CoT-T5 11B
|
|||
|
BIG-bench (SNARKS)
|
CoT-T5 11B
|
|||
|
Big-bench Hard
|
CoT-T5 11B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
GAL 30B
|
|||
|
BIG-bench
|
GAL 120B
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
GAL 120B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
GAL 30B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
GAL 120B
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
GAL 120B
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Chinchilla
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench
|
Gopher-280B
|
|||
|
BIG-bench (Hyperbaton)
|
CoT-T5 11B
|
|||
|
BIG-bench (Navigate)
|
CoT-T5 11B
|
Papers
| Paper | Code | Results | Date | Stars |
|---|
Dataset Loaders
No data loaders found. You can
submit your data loader here.
No data loaders found. You can
submit your data loader here.
Tasks
-
Language Modelling
-
Common Sense Reasoning
-
-
Multiple Choice Question Answering (MCQA)
-
Logical Reasoning
-
Word Sense Disambiguation
-
Sarcasm Detection
-
General Knowledge
-
Multi-task Language Understanding
-
Intent Recognition
-
Riddle Sense
-
Natural Questions
-
BIG-bench Machine Learning
-
Analogical Similarity
-
Identify Odd Metapor
-
Odd One Out
-
Crash Blossom
-
Auto Debugging
-
Crass AI
-
Discourse Marker Prediction
-
Empirical Judgments
-
Irony Identification
-
Timedial
-
Understanding Fables
-
Dark Humor Detection
-
Business Ethics
-
Moral Disputes
-
Moral Permissibility
-
Moral Scenarios
-
FEVER (2-way)
-
FEVER (3-way)
-
Misconceptions
-
Sentence Ambiguity
-
Global Facts
-
Miscellaneous
-
Similarities Abstraction
-
TriviaQA
-
High School European History
-
High School US History
-
High School World History
-
International Law
-
Jurisprudence
-
Logical Fallacies
-
Management
-
Marketing
-
Philosophy
-
Prehistory
-
Professional Law
-
World Religions
-
Analytic Entailment
-
Entailed Polarity
-
Epistemic Reasoning
-
Evaluating Information Essentiality
-
Logical Args
-
Metaphor Boolean
-
Physical Intuition
-
Presuppositions As NLI
-
Abstract Algebra
-
College Mathematics
-
Elementary Mathematics
-
Formal Logic
-
High School Mathematics
-
Mathematical Induction
-
Professional Accounting
-
Anatomy
-
Clinical Knowledge
-
College Medicine
-
Human Aging
-
Human Organs Senses Multiple Choice
-
Medical Genetics
-
Nutrition
-
Professional Medicine
-
Virology
-
English Proverbs
-
Fantasy Reasoning
-
Figure Of Speech Detection
-
GRE Reading Comprehension
-
Implicatures
-
Implicit Relations
-
LAMBADA
-
Movie Dialog Same Or Different
-
Nonsense Words Grammar
-
Phrase Relatedness
-
Question Selection
-
RACE-h
-
RACE-m
-
Astronomy
-
College Biology
-
College Chemistry
-
College Computer Science
-
College Physics
-
Computer Security
-
Conceptual Physics
-
Electrical Engineering
-
High School Biology
-
High School Chemistry
-
High School Computer Science
-
High School Physics
-
High School Statistics
-
Physics MC
-
Econometrics
-
High School Geography
-
High School Government and Politics
-
High School Macroeconomics
-
High School Microeconomics
-
High School Psychology
-
Human Sexuality
-
Professional Psychology
-
Public Relations
-
Security Studies
-
Sociology
-
US Foreign Policy
-
Memorization
Similar Datasets
-
Language Modelling
-
Common Sense Reasoning
-
-
Multiple Choice Question Answering (MCQA)
-
Logical Reasoning
-
Word Sense Disambiguation
-
Sarcasm Detection
-
General Knowledge
-
Multi-task Language Understanding
-
Intent Recognition
-
Riddle Sense
-
Natural Questions
-
BIG-bench Machine Learning
-
Analogical Similarity
-
Identify Odd Metapor
-
Odd One Out
-
Crash Blossom
-
Auto Debugging
-
Crass AI
-
Discourse Marker Prediction
-
Empirical Judgments
-
Irony Identification
-
Timedial
-
Understanding Fables
-
Dark Humor Detection
-
Business Ethics
-
Moral Disputes
-
Moral Permissibility
-
Moral Scenarios
-
FEVER (2-way)
-
FEVER (3-way)
-
Misconceptions
-
Sentence Ambiguity
-
Global Facts
-
Miscellaneous
-
Similarities Abstraction
-
TriviaQA
-
High School European History
-
High School US History
-
High School World History
-
International Law
-
Jurisprudence
-
Logical Fallacies
-
Management
-
Marketing
-
Philosophy
-
Prehistory
-
Professional Law
-
World Religions
-
Analytic Entailment
-
Entailed Polarity
-
Epistemic Reasoning
-
Evaluating Information Essentiality
-
Logical Args
-
Metaphor Boolean
-
Physical Intuition
-
Presuppositions As NLI
-
Abstract Algebra
-
College Mathematics
-
Elementary Mathematics
-
Formal Logic
-
High School Mathematics
-
Mathematical Induction
-
Professional Accounting
-
Anatomy
-
Clinical Knowledge
-
College Medicine
-
Human Aging
-
Human Organs Senses Multiple Choice
-
Medical Genetics
-
Nutrition
-
Professional Medicine
-
Virology
-
English Proverbs
-
Fantasy Reasoning
-
Figure Of Speech Detection
-
GRE Reading Comprehension
-
Implicatures
-
Implicit Relations
-
LAMBADA
-
Movie Dialog Same Or Different
-
Nonsense Words Grammar
-
Phrase Relatedness
-
Question Selection
-
RACE-h
-
RACE-m
-
Astronomy
-
College Biology
-
College Chemistry
-
College Computer Science
-
College Physics
-
Computer Security
-
Conceptual Physics
-
Electrical Engineering
-
High School Biology
-
High School Chemistry
-
High School Computer Science
-
High School Physics
-
High School Statistics
-
Physics MC
-
Econometrics
-
High School Geography
-
High School Government and Politics
-
High School Macroeconomics
-
High School Microeconomics
-
High School Psychology
-
Human Sexuality
-
Professional Psychology
-
Public Relations
-
Security Studies
-
Sociology
-
US Foreign Policy
-
Memorization
