EgoOops annotations

Overview

This is the official repository for annotations of the EgoOops dataset proposed in EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos with Procedural Texts.

TODO

• [] Add the list of the video files

Annotations

meta/metadata.json: annotations of video-text alignment (time stamps and steps), mistake labels, descripion explaining errors

{
    "videos": [
    {
        "task_id": str,
        "video_id": str,
        "segments": [
        {
            "startTime": float (seconds),
            "endTime": float (seconds),
            "instruction": int (zero-based index),
            "labels": list[int] (zero-based index, an empty list means a correct step),
            "caption": str (an empty string means a correct step),
        },
        ...
        ],
    },
    ...
    ],
    "instructions": {
    "blacklight": list[str],
    ...
    }
}

meta/microqr/${task_id}.json: alignment between the number embedded in a QR code attached an object and the name of the object

meta/mistake_classes.json: a list of the names of the annotated mistake classes

Videos

Download the videos from our laboratory's web page

|-- blacklight
|   |-- S1800001.MP4
|   |-- S1800002.MP4
|   ...
|
|-- cardboard
|   ...
|
...

The conversion table of task names

The task names in the dataset are different from ones used in the paper. Refer to the table below.

Dataset	Paper
blacklight	ionic reaction experiments (IR)
cardboard	cardboard crafts (CB)
electronics	electrical circuits (EC)
ion	ionic reaction experiments (IR)
tsumiki	toy block building (BB)

License

EgoOops by Yuto Haneji is licensed under CC BY-SA 4.0

BibTeX

@misc{haneji2024egooopsdatasetmistakeaction,
      title={EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos with Procedural Texts},
      author={Yuto Haneji and Taichi Nishimura and Hirotaka Kameko and Keisuke Shirai and Tomoya Yoshida and Keiya Kajimura and Koki Yamamoto and Taiyu Cui and Tomohiro Nishimoto and Shinsuke Mori},
      year={2024},
      eprint={2410.05343},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.05343},
}