Video Question Answering
153 papers with code • 20 benchmarks • 32 datasets
Video Question Answering (VideoQA) aims to answer natural language questions according to the given videos. Given a video and a question in natural language, the model produces accurate answers according to the content of the video.
Benchmarks
These leaderboards are used to track progress in Video Question Answering
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Video Question Answering models and implementations
HowTo100M
TVQA
ActivityNet-QA
MovieQA
TGIF-QA
NExT-QA
MSRVTT-QA
TVQA+
TGIF
VALUE
Most implemented papers
Is Space-Time Attention All You Need for Video Understanding?
facebookresearch/TimeSformer
•
•
We present a convolution-free approach to video classification built exclusively on self-attention over space and time.
Visual Instruction Tuning
haotian-liu/LLaVA
•
•
NeurIPS 2023
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field.
Flamingo: a Visual Language Model for Few-Shot Learning
mlfoundations/open_flamingo
•
•
DeepMind 2022
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research.
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts.
TVQA: Localized, Compositional Video Question Answering
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.
Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations
jpthu17/emcl
•
•
Most video-and-language representation learning approaches employ contrastive learning, e. g., CLIP, to project the video and text features into a common latent space according to the semantic similarities of text-video pairs.
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement.
Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning
jpthu17/HBI
•
•
CVPR 2023
Contrastive learning-based video-language representation learning approaches, e. g., CLIP, have achieved outstanding performance, which pursue semantic interaction upon pre-defined video-text pairs.
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM.
Exploring Models and Data for Image Question Answering
A suite of baseline results on this new dataset are also presented.
