Skip to content

Home

Jump to bottom
Joao Paulo Papa edited this page Jun 16, 2017 · 36 revisions

Welcome! This homepage describes the source code for the implementation of OPFsumm.

Table of Contents

Introduction

OPFsumm is a video summarization approach based on the Optimum-Path Forest (OPF) classifier [1,2]. Basically, it aims at automatic generating static summaries from video sequences. The first work that employed OPF for static video summarization used the prototypes selected by the OPF clustering algorithm [3] as the keyframes to compose the video summary [4]. Later on, the supervised OPF was used for automatic genre classification in videos, achieving promising results [5]. Castelo-Fernández and Calderón-Ruiz also proposed an OPF-based approach for video summarization, in which scenes, shots and frames are identified at different levels of clustering processes [6].

In 2016, Martins et al. [7] proposed a new OPF-based video summarization approach that makes use of temporal information to improve the quality of the video summaries. In short, the main idea remains using the prototypes as the keyframes, but now we can split the video into different frame sets respecting their temporal information. Then, we can apply OPF clustering on each set and obtain a summary of that specific part of the video. Finally, we can merge all partial summaries (chronological order) to obtain the final summary.

In this document, we explain how to use OPFsumm for video summarization purposes. However, if you are looking for more information about the OPF classifier, please consult http://www.ic.unicamp.br/~afalcao/libopf or the [LibOPF](https://github.com/jppbsi/LibOPF) Github homepage.

File format

Since OPFsumm employs unsupervised OPF, one is required to use the OPF file format for unlabeled datasets, as follows: 

<# of samples> <# of labels> <# of features> 
<0> <label> <feature 1 from element 0> <feature 2 from element 0> ...
<1> <label> <feature 1 from element 1> <feature 2 from element 1> ...
.
.
<i> <label> <feature 1 from element i> <feature 2 from element i> ...
<i+1> <label> <feature 1 from element i+1> <feature 2 from element i+1> ...
.
.
<n-1> <label> <feature 1 from element n-1> <feature 2 from element n-1> ...

The first number of each line is a sample identifier (i.e., the frame number) numbered from 0 to n-1, where n stands for the number of frames. Although we have no labels, we shall set the number of labels to 1 in the header, but we can use 0 in the second column for the further lines.

Example: Suppose you have a dataset with 5 frames, being each one represented by a feature vector of size 4. Therefore, the OPF file format should looks like as follows: 

	5	1	4
	0	0	0.21	0.45	0.40	0.13
	1	0	0.22	0.43	0.36	0.10
	2	0	0.67	1.12	0.43	0.11
	3	0	0.60	1.11	0.97	0.56
	4	0	0.79	0.04	0.09	0.25

Notice the file must be binary with no blank spaces. This ASCII representation is just for the sake of explanation. You can use txt2opf tool available at LibOPF to convert a text file to a binary one.

Installation

OPFsumm source code is straightforward to be installed. You can just clone the repository and type make to build the library. The main source code directories and files are organized as follows: 

	/bin		: it has the opfsumm executable.	
	/include	: it contains both OPFSumm and OPF source codes (.h files)
	/src		: it contains both OPFSumm and OPF source codes (.c files)
	/src/util	: it has auxiliary algorithms used in OPF and OPFSumm codes
	Makefile	: on Unix systems, it is used to build the library

Using OPFsumm

In order to use OPFsumm, you need to set its initialization parameters, as follows: 

	Usage: opfsumm [1][2][3][4][5][6][7][8][9]

	[1]: Unlabeled dataset in the OPF file format;
	[2]: Kmax (maximum degree for the k-nn graph); --> Commonly set in [5-50]
	[3]: Alpha (it weights the temporal distance)  --> Used in the interval [0-1]
	[4]: Subset size in the interval [0-1]; --> Example: size = 0.2 means we will split the dataset into N small subsets of 20% of the original dataset size.
	[5]: Plateau type (0=height; 1=area; 2=volume); --> Allows to remove maxima from the pdf based on height, area or volume criteria (default = 0 (height).
	[6]: Value of parameter [5] (in 0-1); --> default = 0.
	[7]: Consider the pre-processing step (y=yes; n=no);  --> Removes meaningless frames such as unique color frames.
	[8]: Consider the post-processing step (y=yes; n=no); --> Removes redundant keyframes from the summary
	[9]: Precomputed distance file (*optional);

The OPFsumm outputs the k value that minimized the cut in the graph, as well as the number of obtained keyframes and the OPF classifier stored in the file classifier.opf. The keyframes selected from the video are also outputed in a .out file"

Here is an example of the OPFsumm output file, where each line corresponds to the number of the video frame selected by OPFsumm as a summary keyframe. 

	25	
	43	
	69
	127
	249
	500
	752
	756
	790
	803
	887

Toy Example

Here, we will show how to run OPFSumm in an dataset, whose videos are represented as frame collection in a binary opf file. To simulate OPFSumm running into a video dataset, navigate to 'run-example' folder, and you will find it structured as follows: /dataset --> folder containing all video samples; runOPFSumm.sh --> basic script to batch execute OPFSumm on the dataset folder.

Just open a terminal window (Unix systems) and run the command './runOPFSumm.sh 0.10 0.25 0 0 y y' *.

  • These parameters are test examples. You are free to adjust it if you need to.
    • Note that 'runOPFSumm.sh' receives all OPFSumm parameters, except the precomputed distance file (if you need it, please include a parameter call at the end of line 19).

References


[1] J. P. Papa, A. X. Falcão, and Celso T. N. Suzuki. Supervised pattern classification based on optimum-path forest. International Journal of Imaging Systems and Technology, 19(2):120-131, 2009.
[2] J. P. Papa, A. X. Falcão, V. H. C. Albuquerque and J. M. R. da Silva Tavares. Efficient supervised optimum-path forest classification for large datasets. Pattern Recognition, 45(1):512-520, 2012.
[3] L.M. Rocha, F.A.M. Cappabianco, and A.X. Falcão. Data clustering as an optimum-path forest problem with applications in image analysis. International Journal of Imaging Systems and Technology, 19(2):50-68, 2009.
[4] G.B. Martins, L.C.S. Afonso, D. Osaku, J.G. Almeida and J.P. Papa. Static Video Summarization through Optimum-Path Forest Clustering. In: 19th Iberoamerican Congress on Pattern Recognition, Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, v. 8827, 893-900, 2014.
[5] G.B. Martins, J.G. Almeida and J.P. Papa. Supervised Video Genre Classification Using Optimum-Path Forest. In: XX Iberoamerican Congress on Pattern Recognition. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, v. 9423, 735-742, 2015.
[6] C. Castelo-Fernández and G. Calderón-Ruiz. Automatic Video Summarization Using the Optimum-Path Forest Unsupervised Classifier. In: XX Iberoamerican Congress on Pattern Recognition. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, v. 9423, 760-767, 2015.
[7] G.B. Martins, J.P. Papa and J.G. Almeida. Temporal-and Spatial-Driven Video Summarization Using Optimum-Path Forest. In: 2016 29th SIBGRAPI Conference on Graphics, Patterns and Images, 335-339, 2016.

Clone this wiki locally