Skip to content

Commit

Permalink
init commit
Browse files Browse the repository at this point in the history
  • Loading branch information
zeyun-zhong committed Dec 23, 2022
1 parent d8f9f7a commit 0013afb
Show file tree
Hide file tree
Showing 117 changed files with 183,417 additions and 2 deletions.
128 changes: 126 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,126 @@
# Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation (WACV2023)
Code will be available as soon as possible.
# Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation (WACV 2023)
[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/anticipative-feature-fusion-transformer-for/action-anticipation-on-epic-kitchens-100)](https://paperswithcode.com/sota/action-anticipation-on-epic-kitchens-100?p=anticipative-feature-fusion-transformer-for) <br/>

This repository contains the official source code and data for
our [AFFT](https://arxiv.org/abs/2210.12649) paper.
If you find our code or paper useful, please consider citing:

Z. Zhong, D. Schneider, M. Voit, R. Stiefelhagen and J. Beyerer.
**Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation.**
arXiv preprint arXiv:2210.12649, 2022.

```bibtex
@article{zhong2022afft,
title={Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation},
author={Zhong, Zeyun and Schneider, David and Voit, Michael and Stiefelhagen, Rainer and Beyerer, J{\"u}rgen},
journal={arXiv preprint arXiv:2210.12649},
year={2022}
}
```

<div style="text-align:center">
<img src="fuser.png" alt="" width="1000"/>
</div>

## Installation
First clone the repo and set up the required packages in a conda environment.

```bash
$ git clone https://github.com/zeyun-zhong/AFFT.git
$ conda env create -f environment.yaml python=3.7
$ conda activate afft
```

## Download Data
### Dataset features

AFFT works on pre-extracted features, so you will need to download the features first. You can
download the TSN-features from RULSTM for [EK100](https://github.com/fpv-iplab/rulstm/blob/master/RULSTM/scripts/download_data_ek100_full.sh)
and for [EGTEA Gaze+](https://iplab.dmi.unict.it/sharing/rulstm/features/egtea.zip).
The RGB-Swin and audio features will be released soon.

Please make sure that your data structure follows the structure shown below. Note that
`dataset_root_dir` in [config.yaml](conf/config.yaml) should be changed to your specific data path.

```
Dataset root path (e.g., /home/user/datasets)
├── epickitchens100
│ └── features
│ │── rgb
│ │ └── data.mdb
│ │── rgb_omnivore
│ │ └── data.mdb
│ │── obj
│ │ └── data.mdb
│ │── audio
│ │ └── data.mdb
│ └── flow
│ └── data.mdb
└── egtea
└── features
│── TSN-C_3_egtea_action_CE_s1_rgb_model_best_fcfull_hd
│ └── data.mdb
│── TSN-C_3_egtea_action_CE_s1_flow_model_best_fcfull_hd
│ └── data.mdb
│── TSN-C_3_egtea_action_CE_s2_rgb_model_best_fcfull_hd
│ └── data.mdb
│── TSN-C_3_egtea_action_CE_s2_flow_model_best_fcfull_hd
│ └── data.mdb
│── TSN-C_3_egtea_action_CE_s3_rgb_model_best_fcfull_hd
│ └── data.mdb
└── TSN-C_3_egtea_action_CE_s3_flow_model_best_fcfull_hd
└── data.mdb
```

If you use a different organization, you would need to edit `rulstm_feats_dir` in [EK100-common](conf/dataset/epic_kitchens100/common.yaml)
and [EGTEA-common](conf/dataset/egtea/common.yaml).

### Model Zoo

| Dataset | Modalities | Performance <br/> (Actions) | Config | Model |
|---------|:---------------------------------------------------------|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|-----------|
| EK100 | R-Swin, O, AU, F <br/> R-TSN, O, AU, F <br/> R-TSN, O, F | 18.5 (MT5R) <br/> 17.0 (MT5R) <br/> 16.4 (MT5R) | `expts/01_SA-Fuser_ek100_val_Swin.txt` <br/> `expts/01_SA-Fuser_ek100_val_TSN.txt` <br/> `expts/01_SA-Fuser_ek100_val_TSN_wo_audio.txt` | [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/Elr_Rj8NyopLuxIsMsw_nh4BAbhqWwSt3oC6vdRjngx3rw?e=fgvULq) <br/> [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/Eur99IeJJepNgRtlx8JVjzgBg2c94kDtb8iQXBpS8lINqg?e=CRgROp) <br/> [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/EmpcaPNHINZPvpGDRtHDLksB4UGMnJW99x45la6f_Ke0DA?e=EwBXWV) |
| EGTEA | RGB-TSN, Flow | 42.5 (Top-1) | `expts/02_ek100_avt_tsn.txt` | [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/Euw6iM-bGOtAhSq1w4D80KsBnEUIphx85wtYHKicRvFgcA?e=8U3fdF) |


## Training
Recall that `dataset_root_dir` in [config.yaml](conf/config.yaml) should be changed to your specific path.
>### EpicKitchens-100
> ```bash
> python run.py -c expts/01_SA-Fuser_ek100_train.txt --mode train --nproc_per_node 2
>```
>### EGTEA Gaze+
> ```bash
> python run.py -c expts/06_SA-Fuser_egtea_train.txt --mode train --nproc_per_node 2
>```
## Validation
>### EpicKitchens-100
> ```bash
> python run.py -c expts/01_SA-Fuser_ek100_val_TSN_wo_audio.txt --mode test --nproc_per_node 1
>```
>### EGTEA Gaze+
> ```bash
> python run.py -c expts/06_SA-Fuser_egtea_val.txt --mode test --nproc_per_node 1
>```
## Test / Challenge (EK100)
> ```bash
> # save logits
> python run.py -c expts/01_SA-Fuser_ek100_test_TSN_wo_audio.txt --mode test --nproc_per_node 1
>
> # generate test / challenge file
> python challenge.py --prefix_h5 test --models fusion_ek100_tsn_wo_audio_4h_18s --weights 1.
>```
## License
This codebase is released under the license terms specified in the [LICENSE](LICENSE) file. Any imported libraries, datasets or other code follows the license terms set by respective authors.
## Acknowledgements
Many thanks to [Rohit Girdhar](https://github.com/facebookresearch/AVT) and [Antonino Furnari](https://github.com/fpv-iplab/rulstm) for providing their code and data.
Binary file added annotations/.DS_Store
Binary file not shown.
106 changes: 106 additions & 0 deletions annotations/egtea/actions.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
0, 0_0, Inspect/Read_recipe
1, 1_1, Open_fridge
2, 2_2, Take_eating:utensil
3, 3_3, Cut_tomato
4, 4_4, Turn on_faucet
5, 5_2, Put_eating:utensil
6, 1_5, Open_cabinet
7, 2_6, Take_condiment:container
8, 3_7, Cut_cucumber
9, 6_8, Operate_stove
10, 7_1, Close_fridge
11, 3_9, Cut_carrot
12, 5_6, Put_condiment:container
13, 3_10, Cut_onion
14, 1_11, Open_drawer
15, 2_12, Take_plate
16, 2_13, Take_bowl
17, 5_13, Put_bowl
18, 5_14, Put_trash
19, 5_12, Put_plate
20, 3_15, Cut_bell:pepper
21, 5_16, Put_cooking:utensil
22, 2_17, Take_paper:towel
23, 8_18, Move Around_bacon
24, 1_6, Open_condiment:container
25, 9_2, Wash_eating:utensil
26, 10_19, Spread_condiment
27, 11_4, Turn off_faucet
28, 5_20, Put_pan
29, 2_16, Take_cooking:utensil
30, 5_21, Put_lettuce
31, 8_22, Move Around_patty
32, 5_23, Put_pot
33, 7_5, Close_cabinet
34, 5_24, Put_bread
35, 2_24, Take_bread
36, 7_6, Close_condiment:container
37, 1_25, Open_fridge:drawer
38, 9_26, Wash_hand
39, 5_3, Put_tomato
40, 2_27, Take_seasoning:container
41, 2_28, Take_cup
42, 12_21, Divide/Pull Apart_lettuce
43, 5_28, Put_cup
44, 2_23, Take_pot
45, 13_29, Clean/Wipe_counter
46, 2_30, Take_bread:container
47, 2_3, Take_tomato
48, 2_20, Take_pan
49, 8_20, Move Around_pan
50, 9_31, Wash_cutting:board
51, 5_30, Put_bread:container
52, 2_32, Take_sponge
53, 2_21, Take_lettuce
54, 2_10, Take_onion
55, 5_32, Put_sponge
56, 12_17, Divide/Pull Apart_paper:towel
57, 1_33, Open_dishwasher
58, 2_34, Take_cheese:container
59, 2_35, Take_oil:container
60, 5_27, Put_seasoning:container
61, 2_7, Take_cucumber
62, 9_20, Wash_pan
63, 2_15, Take_bell:pepper
64, 12_10, Divide/Pull Apart_onion
65, 5_31, Put_cutting:board
66, 14_36, Mix_mixture
67, 2_37, Take_tomato:container
68, 5_38, Put_cheese
69, 8_2, Move Around_eating:utensil
70, 5_15, Put_bell:pepper
71, 15_39, Pour_oil
72, 2_40, Take_pasta:container
73, 3_21, Cut_lettuce
74, 5_37, Put_tomato:container
75, 9_13, Wash_bowl
76, 3_41, Cut_olive
77, 7_11, Close_drawer
78, 15_19, Pour_condiment
79, 9_23, Wash_pot
80, 14_42, Mix_pasta
81, 1_30, Open_bread:container
82, 2_43, Take_grocery:bag
83, 2_38, Take_cheese
84, 15_44, Pour_seasoning
85, 14_45, Mix_egg
86, 15_46, Pour_water
87, 5_17, Put_paper:towel
88, 5_7, Put_cucumber
89, 16_47, Compress_sandwich
90, 5_34, Put_cheese:container
91, 5_10, Put_onion
92, 17_45, Crack_egg
93, 2_31, Take_cutting:board
94, 1_35, Open_oil:container
95, 18_48, Squeeze_washing:liquid
96, 6_49, Operate_microwave
97, 7_25, Close_fridge:drawer
98, 9_50, Wash_strainer
99, 8_13, Move Around_bowl
100, 8_23, Move Around_pot
101, 5_43, Put_grocery:bag
102, 2_45, Take_egg
103, 1_34, Open_cheese:container
104, 7_35, Close_oil:container
105, 5_35, Put_oil:container
Loading

0 comments on commit 0013afb

Please sign in to comment.