Referring Expression Segmentation
68 papers with code • 25 benchmarks • 11 datasets
The task aims at labeling the pixels of an image or video that represent an object instance referred by a linguistic expression. In particular, the referring expression (RE) must allow the identification of an individual object in a discourse or scene (the referent). REs unambiguously identify the target instance.
Benchmarks
These leaderboards are used to track progress in Referring Expression Segmentation
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
Datasets
RefCOCO
DAVIS 2017
JHMDB
Referring Expressions for DAVIS 2016 & 2017
Refer-YouTube-VOS
Google Refexp
A2D Sentences
PhraseCut
A2Dre
Most implemented papers
Segmentation from Natural Language Expressions
To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information.
Image Segmentation Using Text and Image Prompts
After training on an extended version of the PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on an additional image expressing the query.
CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions
ruotianluo/iep-ref
•
•
CVPR 2019
Yet there has been evidence that current benchmark datasets suffer from bias, and current state-of-the-art models cannot be easily evaluated on their intermediate reasoning process.
MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
ashkamath/mdetr
•
•
We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting.
RefVOS: A Closer Look at Referring Expressions for Video Object Segmentation
The task of video object segmentation with referring expressions (language-guided VOS) is to, given a linguistic phrase and a video, generate binary masks for the object to which the phrase refers.
SynthRef: Generation of Synthetic Referring Expressions for Object Segmentation
Recent advances in deep learning have brought significant progress in visual grounding tasks such as language-guided video object segmentation.
End-to-End Referring Video Object Segmentation with Multimodal Transformers
Due to the complex nature of this multimodal task, which combines text reasoning, video understanding, instance segmentation and tracking, existing approaches typically rely on sophisticated pipelines in order to tackle it.
Unleashing Text-to-Image Diffusion Models for Visual Perception
wl-zhao/VPD
•
•
ICCV 2023
In this paper, we propose VPD (Visual Perception with a pre-trained Diffusion model), a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks.
GRES: Generalized Referring Expression Segmentation
henghuiding/ReLA
•
•
CVPR 2023
Existing classic RES datasets and methods commonly support single-target expressions only, i. e., one expression refers to one target object.
UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces
We evaluate our unified models on various benchmarks.
