Search Results for author:
Found 84 papers, 58 papers with code
Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works
These results strongly validate the character understanding capability of LLMs.
Character is Destiny: Can Large Language Models Simulate Persona-Driven Decisions in Role-Playing?
Can Large Language Models substitute humans in making important decisions?
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models
We present InstantMesh, a feed-forward framework for instant 3D mesh generation from a single image, featuring state-of-the-art generation quality and significant training scalability.
SurveyAgent: A Conversational System for Personalized and Efficient Research Survey
In the rapidly advancing research fields such as AI, managing and staying abreast of the latest scientific literature has become a significant challenge for researchers.
SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model
Controllable spherical panoramic image generation holds substantial applicative potential across a variety of domains. However, it remains a challenging task due to the inherent spherical distortion and geometry characteristics, resulting in low-quality content generation. In this paper, we introduce a novel framework of SphereDiffusion to address these unique challenges, for better generating high-quality and precisely controllable spherical panoramic images. For the spherical distortion characteristic, we embed the semantics of the distorted object with text encoding, then explicitly construct the relationship with text-object correspondence to better use the pre-trained knowledge of the planar images. Meanwhile, we employ a deformable technique to mitigate the semantic deviation in latent space caused by spherical distortion. For the spherical geometry characteristic, in virtue of spherical rotation invariance, we improve the data diversity and optimization objectives in the training process, enabling the model to better learn the spherical geometry characteristic. Furthermore, we enhance the denoising process of the diffusion model, enabling it to effectively use the learned geometric characteristic to ensure the boundary continuity of the generated images. With these specific techniques, experiments on Structured3D dataset show that SphereDiffusion significantly improves the quality of controllable spherical image generation and relatively reduces around 35% FID on average.
BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion
Image inpainting, the process of restoring corrupted images, has seen significant advancements with the advent of diffusion models (DMs).
Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners
Thus, instead of training the giant models from scratch, we propose to bridge the existing strong models with a shared latent representation space.
Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation
Diffusion models have proven to be highly effective in image and video generation; however, they still face composition challenges when generating images of varying sizes due to single-scale training data.
DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing
Large-scale Text-to-Image (T2I) diffusion models have revolutionized image generation over the last few years.
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild
We introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up.
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model.
Ranked #1 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
Towards A Better Metric for Text-to-Video Generation
Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.
ConcEPT: Concept-Enhanced Pre-Training for Language Models
In this paper, we propose ConcEPT, which stands for Concept-Enhanced Pre-Training for language models, to infuse conceptual knowledge into PLMs.
SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
Both quantitative and qualitative results on this evaluation dataset indicate that our SmartEdit surpasses previous methods, paving the way for the practical application of complex instruction-based image editing.
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Recent advances in text-to-image generation have made remarkable progress in synthesizing realistic human photos conditioned on given text prompts.
Ranked #6 on
Diffusion Personalization Tuning Free
on AgeDB
Diffusion Personalization Tuning Free
Text-to-Image Generation
AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
For appearance control, we borrow intermediate latents and their features from the text-to-image (T2I) generation for ensuring the generated first frame is equal to the given generated image.
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion.
MagicStick: Controllable Video Editing via Control Handle Transformations
Yet succinct, our method is the first method to show the ability of video property editing from the pre-trained text-to-image model.
X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
To enhance the guidance ability of X-Adapter, we employ a null-text training strategy for the upgraded model.
StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter
To address these challenges, we introduce StyleCrafter, a generic method that enhances pre-trained T2V models with a style control adapter, enabling video generation in any style by providing a reference image.
Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sources
Pre-trained language models (PLMs) have established the new paradigm in the field of NLP.
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
The I2V model is designed to produce videos that strictly adhere to the content of the provided reference image, preserving its content, structure, and style.
Ranked #3 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models
As a result, our CustomNet ensures enhanced identity preservation and generates diverse, harmonious outputs.
InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales.
New Boolean satisfiability problem heuristic strategy: Minimal Positive Negative Product Strategy
This study presents a novel heuristic algorithm called the "Minimal Positive Negative Product Strategy" to guide the CDCL algorithm in solving the Boolean satisfiability problem.
FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling
With the availability of large-scale video datasets and the advances of diffusion models, text-driven video generation has achieved substantial progress.
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
Animating a still image offers an engaging visual experience.
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
For video generation, various open-sourced models and public-available services have been developed to generate high-quality videos.
Unifying Image Processing as Visual Prompting Question Answering
To address this issue, we propose a universal model for general image processing that covers image restoration, image enhancement, image feature extraction tasks, etc.
ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models
Our work also suggests that a pre-trained diffusion model trained on low-resolution images can be directly used for high-resolution visual generation without further tuning, which may provide insights for future research on ultra-high-resolution image and video synthesis.
Making LLaMA SEE and Draw with SEED Tokenizer
We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs.
Can Large Language Models Understand Real-World Complex Instructions?
To bridge this gap, we propose CELLO, a benchmark for evaluating LLMs' ability to follow complex instructions systematically.
HAT: Hybrid Attention Transformer for Image Restoration
In the training stage, we additionally adopt a same-task pre-training strategy to further exploit the potential of the model for further improvement.
StyleAdapter: A Single-Pass LoRA-Free Model for Stylized Image Generation
StyleAdapter can generate high-quality images that match the content of the prompts and adopt the style of the references (even for unseen styles) in a single pass, which is more flexible and efficient than previous methods.
KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases
Large language models (LLMs) have demonstrated impressive impact in the field of natural language processing, but they still struggle with several issues regarding, such as completeness, timeliness, faithfulness and adaptability.
GET3D--: Learning GET3D from Unconstrained Image Collections
While generative models have shown potential in creating 3D textured shapes from 2D images, their applicability in 3D industries is limited due to the lack of a well-defined camera distribution in real-world scenarios, resulting in low-quality shapes.
Planting a SEED of Vision in Large Language Model
Research on image tokenizers has previously reached an impasse, as frameworks employing quantized visual tokens have lost prominence due to subpar performance and convergence in multimodal comprehension (compared to BLIP-2, etc.)
Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure.
DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
Specifically, we construct classifier guidance based on the strong correspondence of intermediate features in the diffusion model.
DeSRA: Detect and Delete the Artifacts of GAN-based Real-World Super-Resolution Models
After detecting the artifact regions, we develop a finetune procedure to improve GAN-based SR models with a few samples, so that they can deal with similar types of artifacts in more unseen real data.
DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
This paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate thoughts into text.
InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions
Enhancing AI systems to perform tasks following human instructions can significantly boost productivity.
Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance
Our method, dubbed Make-Your-Video, involves joint-conditional video generation using a Latent Diffusion Model that is pre-trained for still image synthesis and then promoted for video generation with the introduction of temporal modules.
Inserting Anybody in Diffusion Models via Celeb Basis
Empowered by the proposed celeb basis, the new identity in our customized model showcases a better concept combination ability than previous personalization methods.
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community.
TaleCrafter: Interactive Story Visualization with Multiple Characters
Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable layout of objects in images.
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
Despite the success in large-scale text-to-image generation and text-conditioned image editing, existing methods still struggle to produce consistent generation and editing results.
Ranked #11 on
Text-based Image Editing
on PIE-Bench
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos
Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human.
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing
We also have a better zero-shot shape-aware editing ability based on the text-to-video model.
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
In this paper, we aim to ``dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly.
OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformer
Omnidirectional images (ODIs) have obtained lots of research interest for immersive experiences.
Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
Specifically, we first generate a high-quality 3D shape from the input text in the text-to-shape stage as a 3D shape prior.
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator.
Reference-based Image and Video Super-Resolution via C2-Matching
To tackle these challenges, we propose C2-Matching in this work, which performs explicit robust matching crossing transformation and resolution.
Mitigating Artifacts in Real-World Video Super-Resolution Models
To aggregate a new hidden state that contains fewer artifacts from the hidden state pool, we devise a Selective Cross Attention (SCA) module, in which the attention between input features and each hidden state is calculated.
MAPS-KB: A Million-scale Probabilistic Simile Knowledge Base
The ability to understand and generate similes is an imperative step to realize human-level AI.
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i. e., VQ tokenizers and generative transformers.
GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond
While most existing perceptual-oriented approaches attempt to generate realistic outputs through learning with adversarial loss, our method, Generative LatEnt bANk (GLEAN), goes beyond existing practices by directly leveraging rich and diverse priors encapsulated in a pre-trained GAN.
FaceFormer: Scale-aware Blind Face Restoration with Transformers
Blind face restoration usually encounters with diverse scale face inputs, especially in the real world.
Rethinking Alignment in Video Super-Resolution Transformers
In this paper, we rethink the role of alignment in VSR Transformers and make several counter-intuitive observations.
Ranked #2 on
Video Super-Resolution
on Vid4 - 4x upscaling
Language Models as Knowledge Embeddings
In this paper, we propose LMKE, which adopts Language Models to derive Knowledge Embeddings, aiming at both enriching representations of long-tail entities and solving problems of prior description-based methods.
Ranked #3 on
Link Prediction
on WN18RR
AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos
This paper studies the problem of real-world video super-resolution (VSR) for animation videos, and reveals three key improvements for practical animation VSR.
NTIRE 2022 Challenge on High Dynamic Range Imaging: Methods and Results
The challenge is composed of two tracks with an emphasis on fidelity and complexity constraints: In Track 1, participants are asked to optimize objective fidelity scores while imposing a low-complexity constraint (i. e. solutions can not exceed a given number of operations).
VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder
Equipped with the VQ codebook as a facial detail dictionary and the parallel decoder design, the proposed VQFR can largely enhance the restored quality of facial details while keeping the fidelity to previous methods.
RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization
Extensive experiments demonstrate that our simple RepSR is capable of achieving superior performance to previous SR re-parameterization methods among different model sizes.
Accelerating the Training of Video Super-Resolution Models
In this work, we show that it is possible to gradually train video models from small to large spatial/temporal sizes, i. e., in an easy-to-hard manner.
MM-RealSR: Metric Learning based Interactive Modulation for Real-World Super-Resolution
Instead of using known degradation levels as explicit supervision to the interactive mechanism, we propose a metric learning strategy to map the unquantifiable degradation levels in real-world scenarios to a metric space, which is trained in an unsupervised manner.
Activating More Pixels in Image Super-Resolution Transformer
In the training stage, we additionally adopt a same-task pre-training strategy to exploit the potential of the model for further improvement.
Ranked #1 on
Image Super-Resolution
on Set5 - 2x upscaling
NTIRE 2022 Challenge on Super-Resolution and Quality Enhancement of Compressed Video: Dataset, Methods and Results
This challenge includes three tracks.
Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning
We address this problem from a new perspective, by jointly considering colorization and temporal consistency in a unified framework.
Towards Vivid and Diverse Image Colorization with Generative Color Prior
Codes are available at https://github. com/ToTheBeginning/GCP-Colorization.
Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution
Unlike previous integral gradient methods, our FAIG aims at finding the most discriminative filters instead of input pixels/features for degradation removal in blind SR networks.
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
Though many attempts have been made in blind super-resolution to restore low-resolution images with unknown and complex degradations, they are still far from addressing general real-world degraded images.
Robust Reference-based Super-Resolution via C2-Matching
However, performing local transfer is difficult because of two gaps between input and reference images: the transformation gap (e. g. scale and rotation) and the resolution gap (e. g. HR and LR).
Towards Real-World Blind Face Restoration with Generative Facial Prior
Blind face restoration usually relies on facial priors, such as facial geometry prior or reference prior, to restore realistic and faithful details.
Ranked #2 on
Blind Face Restoration
on CelebA-Test
Positional Encoding as Spatial Inductive Bias in GANs
In this work, taking SinGAN and StyleGAN2 as examples, we show that such capability, to a large extent, is brought by the implicit positional encoding when using zero padding in the generators.
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
Video super-resolution (VSR) approaches tend to have more components than the image counterparts as they need to exploit the additional temporal dimension.
GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution
We show that pre-trained Generative Adversarial Networks (GANs), e. g., StyleGAN, can be used as a latent bank to improve the restoration quality of large-factor image super-resolution (SR).
Understanding Deformable Alignment in Video Super-Resolution
Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.
EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
In this work, we propose a novel Video Restoration framework with Enhanced Deformable networks, termed EDVR, to address these challenges.
Ranked #2 on
Deblurring
on REDS
Path-Restore: Learning Network Path Selection for Image Restoration
To leverage this, we propose Path-Restore, a multi-path CNN with a pathfinder that can dynamically select an appropriate route for each image region.
Deep Network Interpolation for Continuous Imagery Effect Transition
Deep convolutional neural network has demonstrated its capability of learning a deterministic mapping for the desired imagery effect.
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN).
Ranked #2 on
Face Hallucination
on FFHQ 512 x 512 - 16x upscaling
Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform
In this paper, we show that it is possible to recover textures faithful to semantic classes.
Ranked #55 on
Image Super-Resolution
on BSD100 - 4x upscaling
