Search Results for author:
Found 28 papers, 19 papers with code
SnapKV: LLM Knows What You are Looking for Before Generation
Specifically, SnapKV achieves a consistent decoding speed with a 3. 6x increase in generation speed and an 8. 2x enhancement in memory efficiency compared to baseline when processing inputs of 16K tokens.
JetMoE: Reaching Llama2 Performance with 0.1M Dollars
Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence.
Accelerating Greedy Coordinate Gradient via Probe Sampling
Safety of Large Language Models (LLMs) has become a central issue given their rapid progress and wide applications.
DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models
To overcome this dilemma, we observe the high similarity between the input from adjacent diffusion steps and propose displaced patch parallelism, which takes advantage of the sequential nature of the diffusion process by reusing the pre-computed feature maps from the previous timestep to provide context for the current step.
BitDelta: Your Fine-Tune May Only Be Worth One Bit
Large Language Models (LLMs) are typically trained in two phases: pre-training on large internet-scale datasets, and fine-tuning for downstream tasks.
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
We present two levels of fine-tuning procedures for Medusa to meet the needs of different use cases: Medusa-1: Medusa is directly fine-tuned on top of a frozen backbone LLM, enabling lossless inference acceleration.
REST: Retrieval-Based Speculative Decoding
We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation.
Scaling In-Context Demonstrations with Structured Attention
However, their capabilities of in-context learning are limited by the model architecture: 1) the use of demonstrations is constrained by a maximum sentence length due to positional embeddings; 2) the quadratic complexity of attention hinders users from using more demonstrations efficiently; 3) LLMs are shown to be sensitive to the order of the demonstrations.
Reward Collapse in Aligning Large Language Models
This insight allows us to derive closed-form expressions for the reward distribution associated with a set of utility functions in an asymptotic regime.
Large Language Models as Tool Makers
Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks.
What Makes Convolutional Models Great on Long Sequence Modeling?
We focus on the structure of the convolution kernel and identify two critical but intuitive principles enjoyed by S4 that are sufficient to make up an effective global convolutional model: 1) The parameterization of the convolutional kernel needs to be efficient in the sense that the number of parameters should scale sub-linearly with sequence length.
Ranked #6 on
Long-range modeling
on LRA
Is Vertical Logistic Regression Privacy-Preserving? A Comprehensive Privacy Analysis and Beyond
We provide a comprehensive and rigorous privacy analysis of VLR in a class of open-source Federated Learning frameworks, where the protocols might differ between one another, yet a procedure of obtaining local gradients is implicitly shared.
Do Transformers Really Perform Badly for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Since in many state-of-the-art models, relative positional encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.
First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
In this technical report, we present our solution of KDD Cup 2021 OGB Large-Scale Challenge - PCQM4M-LSC Track.
Do Transformers Really Perform Bad for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Ranked #1 on
Graph Regression
on PCQM4M-LSC
Towards a Theoretical Framework of Out-of-Distribution Generalization
We also introduce a new concept of expansion function, which characterizes to what extent the variance is amplified in the test domains over the training domains, and therefore give a quantitative meaning of invariant features.
A Theory of Label Propagation for Subpopulation Shift
In this work, we propose a provably effective framework for domain adaptation based on label propagation.
Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
This directly provides a rigorous guarantee of the certified robustness based on the margin of prediction outputs.
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
In this paper, we conduct sanity checks for the above beliefs on several recent unstructured pruning methods and surprisingly find that: (1) A set of methods which aims to find good subnetworks of the randomly-initialized network (which we call "initial tickets"), hardly exploits any information from the training data; (2) For the pruned networks obtained by these methods, randomly changing the preserved weights in each layer, while keeping the total number of preserved weights unchanged per layer, does not affect the final performance.
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets.
Ranked #25 on
Graph Property Prediction
on ogbg-molhiv
RANDOM MASK: Towards Robust Convolutional Neural Networks
We next investigate the adversarial examples which 'fool' a CNN with Random Mask.
Locally Differentially Private (Contextual) Bandits Learning
We study locally differentially private (LDP) bandits learning in this paper.
Defective Convolutional Networks
Robustness of convolutional neural networks (CNNs) has gained in importance on account of adversarial examples, i. e., inputs added as well-designed perturbations that are imperceptible to humans but can cause the model to predict incorrectly.
Defective Convolutional Layers Learn Robust CNNs
We first show that predictions made by the defective CNN are less dependent on textural information, but more on shape information, and further find that adversarial examples generated by the defective CNN appear to have semantic shapes.
Convergence of Adversarial Training in Overparametrized Neural Networks
Neural networks are vulnerable to adversarial examples, i. e. inputs that are imperceptibly perturbed from natural data and yet incorrectly classified by the network.
Adversarially Robust Generalization Just Requires More Unlabeled Data
Neural network robustness has recently been highlighted by the existence of adversarial examples.
Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks.
