-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
d8f9f7a
commit 0013afb
Showing
117 changed files
with
183,417 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,126 @@ | ||
# Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation (WACV2023) | ||
Code will be available as soon as possible. | ||
# Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation (WACV 2023) | ||
[](https://paperswithcode.com/sota/action-anticipation-on-epic-kitchens-100?p=anticipative-feature-fusion-transformer-for) <br/> | ||
|
||
This repository contains the official source code and data for | ||
our [AFFT](https://arxiv.org/abs/2210.12649) paper. | ||
If you find our code or paper useful, please consider citing: | ||
|
||
Z. Zhong, D. Schneider, M. Voit, R. Stiefelhagen and J. Beyerer. | ||
**Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation.** | ||
arXiv preprint arXiv:2210.12649, 2022. | ||
|
||
```bibtex | ||
@article{zhong2022afft, | ||
title={Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation}, | ||
author={Zhong, Zeyun and Schneider, David and Voit, Michael and Stiefelhagen, Rainer and Beyerer, J{\"u}rgen}, | ||
journal={arXiv preprint arXiv:2210.12649}, | ||
year={2022} | ||
} | ||
``` | ||
|
||
<div style="text-align:center"> | ||
<img src="fuser.png" alt="" width="1000"/> | ||
</div> | ||
|
||
## Installation | ||
First clone the repo and set up the required packages in a conda environment. | ||
|
||
```bash | ||
$ git clone https://github.com/zeyun-zhong/AFFT.git | ||
$ conda env create -f environment.yaml python=3.7 | ||
$ conda activate afft | ||
``` | ||
|
||
## Download Data | ||
### Dataset features | ||
|
||
AFFT works on pre-extracted features, so you will need to download the features first. You can | ||
download the TSN-features from RULSTM for [EK100](https://github.com/fpv-iplab/rulstm/blob/master/RULSTM/scripts/download_data_ek100_full.sh) | ||
and for [EGTEA Gaze+](https://iplab.dmi.unict.it/sharing/rulstm/features/egtea.zip). | ||
The RGB-Swin and audio features will be released soon. | ||
|
||
Please make sure that your data structure follows the structure shown below. Note that | ||
`dataset_root_dir` in [config.yaml](conf/config.yaml) should be changed to your specific data path. | ||
|
||
``` | ||
Dataset root path (e.g., /home/user/datasets) | ||
├── epickitchens100 | ||
│ └── features | ||
│ │── rgb | ||
│ │ └── data.mdb | ||
│ │── rgb_omnivore | ||
│ │ └── data.mdb | ||
│ │── obj | ||
│ │ └── data.mdb | ||
│ │── audio | ||
│ │ └── data.mdb | ||
│ └── flow | ||
│ └── data.mdb | ||
└── egtea | ||
└── features | ||
│── TSN-C_3_egtea_action_CE_s1_rgb_model_best_fcfull_hd | ||
│ └── data.mdb | ||
│── TSN-C_3_egtea_action_CE_s1_flow_model_best_fcfull_hd | ||
│ └── data.mdb | ||
│── TSN-C_3_egtea_action_CE_s2_rgb_model_best_fcfull_hd | ||
│ └── data.mdb | ||
│── TSN-C_3_egtea_action_CE_s2_flow_model_best_fcfull_hd | ||
│ └── data.mdb | ||
│── TSN-C_3_egtea_action_CE_s3_rgb_model_best_fcfull_hd | ||
│ └── data.mdb | ||
└── TSN-C_3_egtea_action_CE_s3_flow_model_best_fcfull_hd | ||
└── data.mdb | ||
``` | ||
|
||
If you use a different organization, you would need to edit `rulstm_feats_dir` in [EK100-common](conf/dataset/epic_kitchens100/common.yaml) | ||
and [EGTEA-common](conf/dataset/egtea/common.yaml). | ||
|
||
### Model Zoo | ||
|
||
| Dataset | Modalities | Performance <br/> (Actions) | Config | Model | | ||
|---------|:---------------------------------------------------------|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|-----------| | ||
| EK100 | R-Swin, O, AU, F <br/> R-TSN, O, AU, F <br/> R-TSN, O, F | 18.5 (MT5R) <br/> 17.0 (MT5R) <br/> 16.4 (MT5R) | `expts/01_SA-Fuser_ek100_val_Swin.txt` <br/> `expts/01_SA-Fuser_ek100_val_TSN.txt` <br/> `expts/01_SA-Fuser_ek100_val_TSN_wo_audio.txt` | [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/Elr_Rj8NyopLuxIsMsw_nh4BAbhqWwSt3oC6vdRjngx3rw?e=fgvULq) <br/> [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/Eur99IeJJepNgRtlx8JVjzgBg2c94kDtb8iQXBpS8lINqg?e=CRgROp) <br/> [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/EmpcaPNHINZPvpGDRtHDLksB4UGMnJW99x45la6f_Ke0DA?e=EwBXWV) | | ||
| EGTEA | RGB-TSN, Flow | 42.5 (Top-1) | `expts/02_ek100_avt_tsn.txt` | [link](https://fraunhofer-my.sharepoint.com/:f:/g/personal/zeyun_zhong_iosb_fraunhofer_de/Euw6iM-bGOtAhSq1w4D80KsBnEUIphx85wtYHKicRvFgcA?e=8U3fdF) | | ||
|
||
|
||
## Training | ||
Recall that `dataset_root_dir` in [config.yaml](conf/config.yaml) should be changed to your specific path. | ||
>### EpicKitchens-100 | ||
> ```bash | ||
> python run.py -c expts/01_SA-Fuser_ek100_train.txt --mode train --nproc_per_node 2 | ||
>``` | ||
>### EGTEA Gaze+ | ||
> ```bash | ||
> python run.py -c expts/06_SA-Fuser_egtea_train.txt --mode train --nproc_per_node 2 | ||
>``` | ||
## Validation | ||
>### EpicKitchens-100 | ||
> ```bash | ||
> python run.py -c expts/01_SA-Fuser_ek100_val_TSN_wo_audio.txt --mode test --nproc_per_node 1 | ||
>``` | ||
>### EGTEA Gaze+ | ||
> ```bash | ||
> python run.py -c expts/06_SA-Fuser_egtea_val.txt --mode test --nproc_per_node 1 | ||
>``` | ||
## Test / Challenge (EK100) | ||
> ```bash | ||
> # save logits | ||
> python run.py -c expts/01_SA-Fuser_ek100_test_TSN_wo_audio.txt --mode test --nproc_per_node 1 | ||
> | ||
> # generate test / challenge file | ||
> python challenge.py --prefix_h5 test --models fusion_ek100_tsn_wo_audio_4h_18s --weights 1. | ||
>``` | ||
## License | ||
This codebase is released under the license terms specified in the [LICENSE](LICENSE) file. Any imported libraries, datasets or other code follows the license terms set by respective authors. | ||
## Acknowledgements | ||
Many thanks to [Rohit Girdhar](https://github.com/facebookresearch/AVT) and [Antonino Furnari](https://github.com/fpv-iplab/rulstm) for providing their code and data. |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
0, 0_0, Inspect/Read_recipe | ||
1, 1_1, Open_fridge | ||
2, 2_2, Take_eating:utensil | ||
3, 3_3, Cut_tomato | ||
4, 4_4, Turn on_faucet | ||
5, 5_2, Put_eating:utensil | ||
6, 1_5, Open_cabinet | ||
7, 2_6, Take_condiment:container | ||
8, 3_7, Cut_cucumber | ||
9, 6_8, Operate_stove | ||
10, 7_1, Close_fridge | ||
11, 3_9, Cut_carrot | ||
12, 5_6, Put_condiment:container | ||
13, 3_10, Cut_onion | ||
14, 1_11, Open_drawer | ||
15, 2_12, Take_plate | ||
16, 2_13, Take_bowl | ||
17, 5_13, Put_bowl | ||
18, 5_14, Put_trash | ||
19, 5_12, Put_plate | ||
20, 3_15, Cut_bell:pepper | ||
21, 5_16, Put_cooking:utensil | ||
22, 2_17, Take_paper:towel | ||
23, 8_18, Move Around_bacon | ||
24, 1_6, Open_condiment:container | ||
25, 9_2, Wash_eating:utensil | ||
26, 10_19, Spread_condiment | ||
27, 11_4, Turn off_faucet | ||
28, 5_20, Put_pan | ||
29, 2_16, Take_cooking:utensil | ||
30, 5_21, Put_lettuce | ||
31, 8_22, Move Around_patty | ||
32, 5_23, Put_pot | ||
33, 7_5, Close_cabinet | ||
34, 5_24, Put_bread | ||
35, 2_24, Take_bread | ||
36, 7_6, Close_condiment:container | ||
37, 1_25, Open_fridge:drawer | ||
38, 9_26, Wash_hand | ||
39, 5_3, Put_tomato | ||
40, 2_27, Take_seasoning:container | ||
41, 2_28, Take_cup | ||
42, 12_21, Divide/Pull Apart_lettuce | ||
43, 5_28, Put_cup | ||
44, 2_23, Take_pot | ||
45, 13_29, Clean/Wipe_counter | ||
46, 2_30, Take_bread:container | ||
47, 2_3, Take_tomato | ||
48, 2_20, Take_pan | ||
49, 8_20, Move Around_pan | ||
50, 9_31, Wash_cutting:board | ||
51, 5_30, Put_bread:container | ||
52, 2_32, Take_sponge | ||
53, 2_21, Take_lettuce | ||
54, 2_10, Take_onion | ||
55, 5_32, Put_sponge | ||
56, 12_17, Divide/Pull Apart_paper:towel | ||
57, 1_33, Open_dishwasher | ||
58, 2_34, Take_cheese:container | ||
59, 2_35, Take_oil:container | ||
60, 5_27, Put_seasoning:container | ||
61, 2_7, Take_cucumber | ||
62, 9_20, Wash_pan | ||
63, 2_15, Take_bell:pepper | ||
64, 12_10, Divide/Pull Apart_onion | ||
65, 5_31, Put_cutting:board | ||
66, 14_36, Mix_mixture | ||
67, 2_37, Take_tomato:container | ||
68, 5_38, Put_cheese | ||
69, 8_2, Move Around_eating:utensil | ||
70, 5_15, Put_bell:pepper | ||
71, 15_39, Pour_oil | ||
72, 2_40, Take_pasta:container | ||
73, 3_21, Cut_lettuce | ||
74, 5_37, Put_tomato:container | ||
75, 9_13, Wash_bowl | ||
76, 3_41, Cut_olive | ||
77, 7_11, Close_drawer | ||
78, 15_19, Pour_condiment | ||
79, 9_23, Wash_pot | ||
80, 14_42, Mix_pasta | ||
81, 1_30, Open_bread:container | ||
82, 2_43, Take_grocery:bag | ||
83, 2_38, Take_cheese | ||
84, 15_44, Pour_seasoning | ||
85, 14_45, Mix_egg | ||
86, 15_46, Pour_water | ||
87, 5_17, Put_paper:towel | ||
88, 5_7, Put_cucumber | ||
89, 16_47, Compress_sandwich | ||
90, 5_34, Put_cheese:container | ||
91, 5_10, Put_onion | ||
92, 17_45, Crack_egg | ||
93, 2_31, Take_cutting:board | ||
94, 1_35, Open_oil:container | ||
95, 18_48, Squeeze_washing:liquid | ||
96, 6_49, Operate_microwave | ||
97, 7_25, Close_fridge:drawer | ||
98, 9_50, Wash_strainer | ||
99, 8_13, Move Around_bowl | ||
100, 8_23, Move Around_pot | ||
101, 5_43, Put_grocery:bag | ||
102, 2_45, Take_egg | ||
103, 1_34, Open_cheese:container | ||
104, 7_35, Close_oil:container | ||
105, 5_35, Put_oil:container |
Oops, something went wrong.