TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Natural Language Inference	ANLI test	RoBERTa (Large)	A1	72.4	# 5
Natural Language Inference	ANLI test	RoBERTa (Large)	A2	49.8	# 14
Natural Language Inference	ANLI test	RoBERTa (Large)	A3	44.4	# 16
Text Classification	arXiv-10	RoBERTa	Accuracy	0.779	# 2
Stock Market Prediction	Astock	RoBERTa WWM Ext (News+Factors)	Accuray	62.49	# 11
Stock Market Prediction	Astock	RoBERTa WWM Ext (News+Factors)	F1-score	62.54	# 11
Stock Market Prediction	Astock	RoBERTa WWM Ext (News+Factors)	Recall	62.51	# 11
Stock Market Prediction	Astock	RoBERTa WWM Ext (News+Factors)	Precision	62.59	# 11
Stock Market Prediction	Astock	RoBERTa WWM Ext (News)	Accuray	61.34	# 12
Stock Market Prediction	Astock	RoBERTa WWM Ext (News)	F1-score	61.48	# 12
Stock Market Prediction	Astock	RoBERTa WWM Ext (News)	Recall	61.32	# 12
Stock Market Prediction	Astock	RoBERTa WWM Ext (News)	Precision	61.97	# 12
Linguistic Acceptability	CoLA	RoBERTa (ensemble)	Accuracy	67.8%	# 20
Common Sense Reasoning	CommonsenseQA	RoBERTa-Large 355M	Accuracy	72.1	# 18
Sentence Completion	HellaSwag	RoBERTa-Large 355M	Accuracy	81.7	# 36
Sentence Completion	HellaSwag	RoBERTa-Large Ensemble	Accuracy	85.5	# 19
Type prediction	ManyTypes4TypeScript	RoBERTa	Average Accuracy	59.84	# 7
Type prediction	ManyTypes4TypeScript	RoBERTa	Average Precision	57.45	# 5
Type prediction	ManyTypes4TypeScript	RoBERTa	Average Recall	57.62	# 5
Type prediction	ManyTypes4TypeScript	RoBERTa	Average F1	57.54	# 5
Multi-task Language Understanding	MMLU	RoBERTa-base 125M (fine-tuned)	Average (%)	27.9	# 94
Semantic Textual Similarity	MRPC	RoBERTa (ensemble)	Accuracy	92.3%	# 3
Natural Language Inference	MultiNLI	RoBERTa	Matched	90.8	# 8
Natural Language Inference	MultiNLI	RoBERTa (ensemble)	Mismatched	90.2	# 7
Only Connect Walls Dataset Task 1 (Grouping)	OCW	RoBERTa (LARGE)	# Correct Groups	29 ± 3	# 21
Only Connect Walls Dataset Task 1 (Grouping)	OCW	RoBERTa (LARGE)	Fowlkes Mallows Score (FMS)	26.7 ± .2	# 19
Only Connect Walls Dataset Task 1 (Grouping)	OCW	RoBERTa (LARGE)	Adjusted Rand Index (ARI)	8.4 ± .3	# 19
Only Connect Walls Dataset Task 1 (Grouping)	OCW	RoBERTa (LARGE)	Adjusted Mutual Information (AMI)	9.4 ± .4	# 20
Only Connect Walls Dataset Task 1 (Grouping)	OCW	RoBERTa (LARGE)	# Solved Walls	0 ± 0	# 10
Only Connect Walls Dataset Task 1 (Grouping)	OCW	RoBERTa (LARGE)	Wasserstein Distance (WD)	88.4 ± .4	# 1
Question Answering	PIQA	RoBERTa-Large 355M	Accuracy	79.4	# 32
Natural Language Inference	QNLI	RoBERTa (ensemble)	Accuracy	98.9%	# 5
Question Answering	Quora Question Pairs	RoBERTa	Accuracy	90.2%	# 6
Reading Comprehension	RACE	RoBERTa	Accuracy	83.2	# 7
Reading Comprehension	RACE	RoBERTa	Accuracy (High)	81.3	# 6
Reading Comprehension	RACE	RoBERTa	Accuracy (Middle)	86.5	# 6
Natural Language Inference	RTE	RoBERTa	Accuracy	88.2%	# 18
Natural Language Inference	RTE	RoBERTa (ensemble)	Accuracy	88.2%	# 18
Document Image Classification	RVL-CDIP	Roberta base	Accuracy	90.06	# 30
Document Image Classification	RVL-CDIP	Roberta base	Parameters	125M	# 19
Question Answering	SIQA	RoBERTa-Large 355M (fine-tuned)	Accuracy	76.7	# 8
Question Answering	SQuAD2.0	RoBERTa (single model)	EM	86.820	# 90
Question Answering	SQuAD2.0	RoBERTa (single model)	F1	89.795	# 91
Question Answering	SQuAD2.0 dev	RoBERTa (no data aug)	F1	89.4	# 3
Question Answering	SQuAD2.0 dev	RoBERTa (no data aug)	EM	86.5	# 3
Sentiment Analysis	SST-2 Binary classification	RoBERTa (ensemble)	Accuracy	96.7	# 11
Semantic Textual Similarity	STS Benchmark	RoBERTa	Pearson Correlation	0.922	# 7
Common Sense Reasoning	SWAG	RoBERTa	Test	89.9	# 2
Natural Language Inference	WNLI	RoBERTa (ensemble)	Accuracy	89	# 8

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/task-1-grouping-on-ocw)](https://paperswithcode.com/sota/task-1-grouping-on-ocw?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/text-classification-on-arxiv-10)](https://paperswithcode.com/sota/text-classification-on-arxiv-10?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/common-sense-reasoning-on-swag)](https://paperswithcode.com/sota/common-sense-reasoning-on-swag?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/semantic-textual-similarity-on-mrpc)](https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/question-answering-on-squad20-dev)](https://paperswithcode.com/sota/question-answering-on-squad20-dev?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/natural-language-inference-on-anli-test)](https://paperswithcode.com/sota/natural-language-inference-on-anli-test?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/natural-language-inference-on-qnli)](https://paperswithcode.com/sota/natural-language-inference-on-qnli?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/question-answering-on-quora-question-pairs)](https://paperswithcode.com/sota/question-answering-on-quora-question-pairs?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/type-prediction-on-manytypes4typescript)](https://paperswithcode.com/sota/type-prediction-on-manytypes4typescript?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/natural-language-inference-on-multinli)](https://paperswithcode.com/sota/natural-language-inference-on-multinli?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/reading-comprehension-on-race)](https://paperswithcode.com/sota/reading-comprehension-on-race?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/semantic-textual-similarity-on-sts-benchmark)](https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/question-answering-on-social-iqa)](https://paperswithcode.com/sota/question-answering-on-social-iqa?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/natural-language-inference-on-wnli)](https://paperswithcode.com/sota/natural-language-inference-on-wnli?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/stock-market-prediction-on-astock)](https://paperswithcode.com/sota/stock-market-prediction-on-astock?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/sentiment-analysis-on-sst-2-binary)](https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/common-sense-reasoning-on-commonsenseqa)](https://paperswithcode.com/sota/common-sense-reasoning-on-commonsenseqa?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/natural-language-inference-on-rte)](https://paperswithcode.com/sota/natural-language-inference-on-rte?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/sentence-completion-on-hellaswag)](https://paperswithcode.com/sota/sentence-completion-on-hellaswag?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/linguistic-acceptability-on-cola)](https://paperswithcode.com/sota/linguistic-acceptability-on-cola?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/document-image-classification-on-rvl-cdip)](https://paperswithcode.com/sota/document-image-classification-on-rvl-cdip?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/question-answering-on-piqa)](https://paperswithcode.com/sota/question-answering-on-piqa?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/question-answering-on-squad20)](https://paperswithcode.com/sota/question-answering-on-squad20?p=roberta-a-robustly-optimized-bert-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/roberta-a-robustly-optimized-bert-pretraining/multi-task-language-understanding-on-mmlu)](https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu?p=roberta-a-robustly-optimized-bert-pretraining)`

RoBERTa: A Robustly Optimized BERT Pretraining Approach

26 Jul 2019 · Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov ·

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

PDF Abstract

Code

Add Remove Mark official

pytorch/fairseq official

↳ Quickstart in

Colab

PyTorch Hub

29,251

huggingface/transformers

124,984

PaddlePaddle/PaddleNLP

11,420

salesforce/codet5

2,593

brightmart/roberta_zh

2,525

See all 59 implementations

Tasks

Add Remove

Common Sense Reasoning

Document Image Classification

Language Modelling

Lexical Simplification

Linguistic Acceptability

Multi-task Language Understanding

Natural Language Inference

Only Connect Walls Dataset Task 1 (Grouping)

Question Answering

Reading Comprehension

Riddle Sense

Semantic Textual Similarity

Sentence Completion

Sentiment Analysis

Stock Market Prediction

Text Classification

Type prediction

Datasets

GLUE

SST

SQuAD

MultiNLI SST-2

QNLI

MRPC

MMLU

CoLA

HellaSwag

PIQA

RACE

CommonsenseQA

WebText

ANLI

SWAG

RVL-CDIP

Quora

Quora Question Pairs RTE STS Benchmark

SIQA WNLI

OCW Astock ManyTypes4TypeScript

arXiv-10

Results from the Paper

Edit

Ranked #1 on Only Connect Walls Dataset Task 1 (Grouping) on OCW (Wasserstein Distance (WD) metric, using extra training data)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Natural Language Inference	ANLI test	RoBERTa (Large)	A1	72.4	# 5	Compare
			A2	49.8	# 14	Compare
			A3	44.4	# 16	Compare
Text Classification	arXiv-10	RoBERTa	Accuracy	0.779	# 2	Compare
Stock Market Prediction	Astock	RoBERTa WWM Ext (News+Factors)	Accuray	62.49	# 11	Compare
			F1-score	62.54	# 11	Compare
			Recall	62.51	# 11	Compare
			Precision	62.59	# 11	Compare
Stock Market Prediction	Astock	RoBERTa WWM Ext (News)	Accuray	61.34	# 12	Compare
			F1-score	61.48	# 12	Compare
			Recall	61.32	# 12	Compare
			Precision	61.97	# 12	Compare
Linguistic Acceptability	CoLA	RoBERTa (ensemble)	Accuracy	67.8%	# 20	Compare
Sentence Completion	HellaSwag	RoBERTa-Large 355M	Accuracy	81.7	# 36	Compare
Type prediction	ManyTypes4TypeScript	RoBERTa	Average Accuracy	59.84	# 7	Compare
			Average Precision	57.45	# 5	Compare
			Average Recall	57.62	# 5	Compare
			Average F1	57.54	# 5	Compare
Multi-task Language Understanding	MMLU	RoBERTa-base 125M (fine-tuned)	Average (%)	27.9	# 94	Compare
Semantic Textual Similarity	MRPC	RoBERTa (ensemble)	Accuracy	92.3%	# 3	Compare
Natural Language Inference	MultiNLI	RoBERTa	Matched	90.8	# 8	Compare
Natural Language Inference	MultiNLI	RoBERTa (ensemble)	Mismatched	90.2	# 7	Compare
Only Connect Walls Dataset Task 1 (Grouping)	OCW	RoBERTa (LARGE)	# Correct Groups	29 ± 3	# 21	Compare
			Fowlkes Mallows Score (FMS)	26.7 ± .2	# 19	Compare
			Adjusted Rand Index (ARI)	8.4 ± .3	# 19	Compare
			Adjusted Mutual Information (AMI)	9.4 ± .4	# 20	Compare
			# Solved Walls	0 ± 0	# 10	Compare
			Wasserstein Distance (WD)	88.4 ± .4	# 1	Compare
Natural Language Inference	QNLI	RoBERTa (ensemble)	Accuracy	98.9%	# 5	Compare
Question Answering	Quora Question Pairs	RoBERTa	Accuracy	90.2%	# 6	Compare
Reading Comprehension	RACE	RoBERTa	Accuracy	83.2	# 7	Compare
			Accuracy (High)	81.3	# 6	Compare
			Accuracy (Middle)	86.5	# 6	Compare
Natural Language Inference	RTE	RoBERTa	Accuracy	88.2%	# 18	Compare
Natural Language Inference	RTE	RoBERTa (ensemble)	Accuracy	88.2%	# 18	Compare
Question Answering	SQuAD2.0	RoBERTa (single model)	EM	86.820	# 90	Compare
Question Answering	SQuAD2.0	RoBERTa (single model)	F1	89.795	# 91	Compare
Question Answering	SQuAD2.0 dev	RoBERTa (no data aug)	F1	89.4	# 3	Compare
Question Answering	SQuAD2.0 dev	RoBERTa (no data aug)	EM	86.5	# 3	Compare
Sentiment Analysis	SST-2 Binary classification	RoBERTa (ensemble)	Accuracy	96.7	# 11	Compare
Semantic Textual Similarity	STS Benchmark	RoBERTa	Pearson Correlation	0.922	# 7	Compare
Common Sense Reasoning	SWAG	RoBERTa	Test	89.9	# 2	Compare
Natural Language Inference	WNLI	RoBERTa (ensemble)	Accuracy	89	# 8	Compare

Results from Other Papers

Task	Dataset	Model	Metric Name	Metric Value	Rank	Compare
Common Sense Reasoning	CommonsenseQA	RoBERTa-Large 355M	Accuracy	72.1	# 18	See all
Sentence Completion	HellaSwag	RoBERTa-Large Ensemble	Accuracy	85.5	# 19	See all
Question Answering	PIQA	RoBERTa-Large 355M	Accuracy	79.4	# 32	See all
Question Answering	SIQA	RoBERTa-Large 355M (fine-tuned)	Accuracy	76.7	# 8	See all
Document Image Classification	RVL-CDIP	Roberta base	Accuracy	90.06	# 30	See all
Document Image Classification	RVL-CDIP	Roberta base	Parameters	125M	# 19	See all

Methods

Add Remove

Adam • Attention Dropout • BERT • Dense Connections • Dropout • GELU • Layer Normalization • Linear Layer • Linear Warmup With Linear Decay • Multi-Head Attention • Residual Connection • RoBERTa • Scaled Dot-Product Attention • Softmax • Weight Decay • WordPiece

Edit Social Preview

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit