Transfer Learning
2824 papers with code • 7 benchmarks • 14 datasets
Transfer Learning is a machine learning technique where a model trained on one task is re-purposed and fine-tuned for a related, but different task. The idea behind transfer learning is to leverage the knowledge learned from a pre-trained model to solve a new, but related problem. This can be useful in situations where there is limited data available to train a new model from scratch, or when the new task is similar enough to the original task that the pre-trained model can be adapted to the new problem with only minor modifications.
( Image credit: Subodh Malgonde )
Benchmarks
These leaderboards are used to track progress in Transfer Learning
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Transfer Learning models and implementationsDatasets
KITTI
Office-Home
Sims4Action
MFRC
CARLANE Benchmark
BanglaLekha-Isolated
SpaRTUN
fluocells
Retinal Fundus MultiDisease Image Dataset (RFMiD)
Subtasks
Most implemented papers
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available.
Universal Language Model Fine-tuning for Text Classification
Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
UKPLab/sentence-transformers
•
•
IJCNLP 2019
However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10, 000 sentences requires about 50 million inference computations (~65 hours) with BERT.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
huggingface/transformers
•
•
arXiv 2019
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP).
High Quality Monocular Depth Estimation via Transfer Learning
Accurate depth estimation from images is a fundamental task in many applications including scene understanding and reconstruction.
ResNeSt: Split-Attention Networks
It is well known that featuremap attention and multi-path representation are important for visual recognition.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
huggingface/transformers
•
•
NeurIPS 2019
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging.
Bag of Tricks for Image Classification with Convolutional Neural Networks
Much of the recent progress made in image classification research can be credited to training procedure refinements, such as changes in data augmentations and optimization methods.
Universal Sentence Encoder
facebookresearch/InferSent
•
•
For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance.
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
facebookresearch/InferSent
•
•
EMNLP 2017
Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features.
