A Large-scale Dataset for Audio-Language Representation Learning

Luoyi Sun, Xuenan Xu, Mengyue Wu, Weidi Xie

[project page] [Arxiv] [Dataset]

Environments

conda create env -n acd python=3.10
conda activate acd
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt
wandb login ***

Training

Modify the paths in data.py.

The training command is:

cd ./laion_clap/evaluate

CUDA_VISIBLE_DEVICES=0  python eval_linear_probe-acd.py   
--save-frequency 5  
--save-top-performance 3  
--save-most-recent    
--dataset-type="webdataset"   
--precision="fp32"   
--warmup 1000    
--batch-size=256    
--lr=1e-4    
--wd=0.1    
--epochs=15  
--workers=8   
--use-bn-sync        
--amodel HTSAT-tiny    
--tmodel roberta     
--report-to "wandb"     
--wandb-notes "acd"     
--datasetnames "dataset"     
--datasetinfos "train"     
--seed 3407        
--logs ./logs/ckpt     
--gather-with-grad     
--lp-loss="bce"     
--lp-metrics="map"     
--lp-lr=1e-4     
--openai-model-cache-dir ./transformers_cache     
--pretrained="./pretrain-ckpt"     
--data-filling "repeatpad"    
--data-truncating "rand_trunc"     
--optimizer "adam"

[Optional] Evaluating Our Pre-trained Model

We also provide pre-trained models. The model with all training logs can be downloaded from HuggingFace.

Citation

If you are using our code, please consider citing our paper.

@inproceedings{sun2024auto,
  title={Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning},
  author={Sun, Luoyi and Xu, Xuenan and Wu, Mengyue and Xie, Weidi},
  booktitle={ACM Multimedia 2024}
}

Contact

If you have any question, please feel free to contact loiesun411@gmail.com.

Aknowledgement

This repo benefits from CLAP. Thanks for their wonderful works.