Skip to content

Home

Jump to bottom
Joao Paulo Papa edited this page Jun 16, 2017 · 36 revisions

Welcome! This homepage describes the source code for the implementation of OPFSumm.

Table of Contents

Introduction

OPFSumm is a video summarization approach based on the Optimum-Path Forest (OPF) classifier [1,2]. Basically, it aims at automatic generating static summaries from video sequences. The first work that employed OPF for static video summarization used the prototypes selected by the OPF clustering algorithm [3] as the keyframes to compose the video summary [4]. Later on, the supervised OPF was used for automatic genre classification in videos, achieving promising results [5]. Castelo-Fernández and Calderón-Ruiz also proposed an OPF-based approach for video summarization, in which scenes, shots and frames are identified at different levels of clustering processes [6].

In 2016, Martins et al. [7] proposed a new OPF-based video summarization approach that makes use of temporal information to improve the quality of the video summaries. In short, the main idea remains using the prototypes as the keyframes, but now we can split the video into different frame sets respecting their temporal information. Then, we can apply OPF clustering on each set and obtain a summary of that specific part of the video. Finally, we can merge all partial summaries (chronological order) to obtain the final summary.

In this document, we explain how to use OPFSumm for video summarization purposes. However, if you are looking for more information about the OPF classifier, please consult http://www.ic.unicamp.br/~afalcao/libopf or the [LibOPF](https://github.com/jppbsi/LibOPF) Github homepage.

File format

Since OPFSumm employs unsupervised OPF, one is required to use the OPF file format for unlabeled datasets, as follows: 

<# of samples> <# of labels> <# of features> 
<0> <label> <feature 1 from element 0> <feature 2 from element 0> ...
<1> <label> <feature 1 from element 1> <feature 2 from element 1> ...
.
.
<i> <label> <feature 1 from element i> <feature 2 from element i> ...
<i+1> <label> <feature 1 from element i+1> <feature 2 from element i+1> ...
.
.
<n-1> <label> <feature 1 from element n-1> <feature 2 from element n-1> ...

The first number of each line is a sample identifier (i.e., the frame number) numbered from 0 to n-1, where n stands for the number of frames. Although we have no labels, we shall set the number of labels to 1 in the header, but we can use 0 in the second column for the further lines.

Example: Suppose you have a dataset with 5 frames, being each one represented by a feature vector of size 4. Therefore, the OPF file format should looks like as follows: 

	5	1	4
	0	0	0.21	0.45	0.40	0.13
	1	0	0.22	0.43	0.36	0.10
	2	0	0.67	1.12	0.43	0.11
	3	0	0.60	1.11	0.97	0.56
	4	0	0.79	0.04	0.09	0.25

Notice the file must be binary with no blank spaces. This ASCII representation is just for the sake of explanation. You can use txt2opf tool available at LibOPF to convert a text file to a binary one.

Installation

OPFSumm source code is straightforward to be installed. You can just clone the repository and type make to build the library. The main source code directories and files are organized as follows: 

	/bin		: it has the opfsumm executable.	
	/include	: it contains both OPFSumm and OPF source codes (.h files)
	/src		: it contains both OPFSumm and OPF source codes (.c files)
	/src/util	: it has auxiliary algorithms used in OPF and OPFSumm codes
	Makefile	: on Unix systems, it is used to build the library

Using OPFSumm

In order to use OPFSumm, you need to set its initialization parameters, as follows: 

	Usage: opfsumm [1][2][3][4][5][6][7][8][9]

	[1]: Unlabeled dataset in the OPF file format;
	[2]: Kmax (maximum degree for the k-nn graph); --> Commonly set in [5-50]
	[3]: Alpha (it weights the temporal distance)  --> Used in the interval [0-1]
	[4]: Subset size in the interval [0-1]; --> Example: size = 0.2 means we will split the dataset into N small subsets of 20% of the original dataset size.
	[5]: Plateau type (0=height; 1=area; 2=volume); --> Allows to remove maxima from the pdf based on height, area or volume criteria (default = 0 (height).
	[6]: Value of parameter [5] (in 0-1); --> default = 0.
	[7]: Consider the pre-processing step (y=yes; n=no);  --> Removes meaningless frames such as unique color frames.
	[8]: Consider the post-processing step (y=yes; n=no); --> Removes redundant keyframes from the summary
	[9]: Precomputed distance file (*optional);

The OPFSumm outputs the k value that minimized the cut in the graph, as well as the number of obtained keyframes and the OPF classifier stored in the file classifier.opf. The keyframes selected from the video are also outputed in a .out file"

Here is an example of the OPFSumm output file, where each line corresponds to the number of the video frame selected by OPFSumm as a summary keyframe. 

	25	
	43	
	69
	127
	249
	500
	752
	756
	790
	803
	887

Toy Example

Here, we show how to run OPFSumm in a dataset, whose videos are represented as a frame collection in a binary OPF file. Basically, for each video frame, you must extract features to compose each line of the OPF file. Notice the number of features must match the one provided in the header.

In order to simulate OPFSumm over a video dataset, go to run-example folder, and you will find it structured as follows: 

	/dataset	--> folder containing all video samples;
	runOPFSumm.sh	--> basic script to run OPFSumm on the dataset folder.

Now, you can run the command ./runOPFSumm.sh 0.10 0.25 0 0 y y to execute OPFsumm. Notice these parameters are just for explanation purposes. You are free to adjust it whenever you need.

Note that runOPFSumm.sh receives all OPFSumm parameters, except the precomputed distance file (if you need it, please include a parameter call at the end of line 19 of the aforementioned shell script file).

Evaluation

The methodology established to evaluate OPFSumm video summaries incorporates the well-known F-measure as the main performance measure. Essentially, it is defined as follows: 

	F-measure = 2 * (Precision * Recall) / (Precision + Recall)

Open Video and Youtube

To evaluate summaries on Open Video and/or Youtube databases, please consult http://www.recod.ic.unicamp.br/~jurandy/pub/summary/, where you can find all the F-measure source codes and batch scripts.

After opening the link, download the file Summary.tar.gz and decompress its files and folders. You will find something like this: 

	/Databases	--> it has all the videos from Open Video and Youtube
	/Plots		--> it contains the graphical view from F-measure summaries comparison.
	/Results	--> it keeps the OPF output files to be evaluated		
	/Scripts	--> the main source code of F-measure and the batch scripts to evaluate the databases.

To run the evaluation step, just copy the OPF output files after running OPFSumm into the folder Results, and run the command /Scripts/maketable. It will simply evaluate the generated summaries against human summaries (ground-truth of Open Video and Youtube datasets) through CUS (Comparison of User Summaries) methodology (you can find more details about it here: https://sites.google.com/site/vsummsite/documentation).

SumMe

To evaluate summaries on SumMe video database, you can use the source code from evaluation/SumMe folder, which has the following structure: 

	/GT				--> ground-truth of SumMe videos
	/python				--> F-measure evaluation codes
	/summary			--> summaries evaluated are stored here
	car_over_camera.gch.dat.out	--> an example of OPFSumm output file (static summary)
	runEval.sh			--> script to batch evaluation

Now, go to evaluation folder and run the command ./runEval.sh. Inside the script, all the initialization parameters were set for keyframe evaluation (parameters -k and .key).

For a more detailed explanation about the SumMe benchmark, please refer to the following homepages:

  • https://www.vision.ee.ethz.ch/~gyglim/vsum/index.php
  • https://www.vision.ee.ethz.ch/~gyglim/vsum/index.php#benchmark

    • References


    [1] J. P. Papa, A. X. Falcão, and Celso T. N. Suzuki. Supervised pattern classification based on optimum-path forest. International Journal of Imaging Systems and Technology, 19(2):120-131, 2009.
    [2] J. P. Papa, A. X. Falcão, V. H. C. Albuquerque and J. M. R. da Silva Tavares. Efficient supervised optimum-path forest classification for large datasets. Pattern Recognition, 45(1):512-520, 2012.
    [3] L.M. Rocha, F.A.M. Cappabianco, and A.X. Falcão. Data clustering as an optimum-path forest problem with applications in image analysis. International Journal of Imaging Systems and Technology, 19(2):50-68, 2009.
    [4] G.B. Martins, L.C.S. Afonso, D. Osaku, J.G. Almeida and J.P. Papa. Static Video Summarization through Optimum-Path Forest Clustering. In: 19th Iberoamerican Congress on Pattern Recognition, Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, v. 8827, 893-900, 2014.
    [5] G.B. Martins, J.G. Almeida and J.P. Papa. Supervised Video Genre Classification Using Optimum-Path Forest. In: XX Iberoamerican Congress on Pattern Recognition. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, v. 9423, 735-742, 2015.
    [6] C. Castelo-Fernández and G. Calderón-Ruiz. Automatic Video Summarization Using the Optimum-Path Forest Unsupervised Classifier. In: XX Iberoamerican Congress on Pattern Recognition. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, v. 9423, 760-767, 2015.
    [7] G.B. Martins, J.P. Papa and J.G. Almeida. Temporal-and Spatial-Driven Video Summarization Using Optimum-Path Forest. In: 2016 29th SIBGRAPI Conference on Graphics, Patterns and Images, 335-339, 2016.

    Clone this wiki locally