Language Modelling
4482 papers with code • 51 benchmarks • 157 datasets
Language Modeling is the task of predicting the next word or character in a document. This technique can be used to train language models that can further be applied to a wide range of natural language tasks like text generation, text classification, and question answering.
Historically, language modelling was done with N-gram language models (which still have niche uses), but since the 2010s neural language models took over, and starting from the 2020s SOTA was achieved exclusively with large language models (LLMs).
A model's language modeling capability is measured using cross-entropy and perplexity. Some datasets to evaluate language modeling are WikiText-103, One Billion Word, Text8, C4, The Pile, among others.
Some notable state-of-the-art language models include:
Check below for all state-of-the-art models.
Here are some additional readings to go deeper on the task:
- Language Modeling - Lena Voita
( Image credit: Exploring the Limits of Language Modeling )
Benchmarks
These leaderboards are used to track progress in Language Modelling
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Language Modelling models and implementationsDatasets
IMDb Movie Reviews
Pubmed
Penn Treebank
WikiText-2
C4
WikiText-103
BookCorpus
The Pile
BIG-bench
Subtasks
Most implemented papers
Listen, Attend and Spell
Alexander-H-Liu/End-to-end-ASR-Pytorch
•
•
Unlike traditional DNN-HMM models, this model learns all the components of a speech recognizer jointly.
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
google-research/bert
•
•
ICLR 2020
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
kimiyoung/transformer-xl
•
•
ACL 2019
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
locuslab/TCN
•
•
Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
huggingface/transformers
•
•
NeurIPS 2019
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging.
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
On LibriSpeech, we achieve 6. 8% WER on test-other without the use of a language model, and 5. 8% WER with shallow fusion with a language model.
Efficient Neural Architecture Search via Parameter Sharing
google-research/google-research
•
•
The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set.
Unsupervised Cross-lingual Representation Learning at Scale
facebookresearch/XLM
•
•
ACL 2020
We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale.
Matching Networks for One Shot Learning
oscarknagg/few-shot
•
•
NeurIPS 2016
Our algorithm improves one-shot accuracy on ImageNet from 87. 6% to 93. 2% and from 88. 0% to 93. 8% on Omniglot compared to competing approaches.
Conformer: Convolution-augmented Transformer for Speech Recognition
Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs).
