Zero-shot Audio Classification
6 papers with code • 2 benchmarks • 2 datasets
Benchmarks
These leaderboards are used to track progress in Zero-shot Audio Classification
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
AudioSet
VGG-Sound
Most implemented papers
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
pku-yuangroup/languagebind
•
•
We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M.
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions.
Sound-Guided Semantic Image Manipulation
kuai-lab/sound-guided-semantic-image-manipulation
•
•
CVPR 2022
Our audio encoder is trained to produce a latent representation from an audio input, which is forced to be aligned with image and text representations in the multi-modal embedding space.
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer
zhaoyanpeng/vipant
•
•
NAACL 2022
In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2. 2\% R@1.
ImageBind: One Embedding Space To Bind Them All
facebookresearch/imagebind
•
•
CVPR 2023
We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together.
Investigating the Emergent Audio Classification Ability of ASR Foundation Models
julirao/whisper_audio_classification
•
•
Text and vision foundation models can perform many tasks in a zero-shot setting, a desirable property that enables these systems to be applied in general and low-resource settings.
