fabio.tex

\documentclass{iucr}

\begin{document}

\title{FabIO: easy access to 2D X-ray detector images in Python}
\shorttitle{FabIO}

    \author[a]{Henning O.}{S$\o$rensen}
    \author[b]{Erik B.}{Knudsen}
    \author[c]{Jonathan P.}{Wright}
	\cauthor[c]{J\'er\^ome}{Kieffer}{jerome.kieffer@esrf,fr}{}
	\author[c]{Ga\"el}{Goret}
%	\author[c]{V. Armando}{Sol\'e}
    \aff[a]{Nano-Science Center, Department of Chemistry, University of Copenhagen,
            Universitetsparken 5, \city{Copenhagen}, \country{Denmark} }
    \aff[b]{Department of Physics, Technical University of Denmark,
            \city{Kongens Lyngby} \country{Denmark}}
    \aff[c]{European Synchrotron Radiation Facility, \city{Grenoble}, \country{France}}
	\shortauthor{S$\o$rensen et al.}

\maketitle

\section{Introduction}

One obstacle when writing software to analyse data collected from a
two-dimensional detector is to read the raw data into the program,
not least because the data can be stored in many different formats
depending on the instrument used.
To overcome this problem we decided to develop a general module,
FabIO (FABle I/O), to handle reading and writing of two-dimensional
data.
The codebase was initiated by merging parts of our fabian imageviewer
\cite{fabian} and ImageD11 \cite{ImageD11} peaksearch programs and has
been developed since 2007 as part of the FABLE \cite{fable} program suite
for analysis of 3DXRD microscopy data \cite{3dxrd}.
Over time the code has evolved to handle multi-frame image formats
as well as writing many of the file formats.
We believe FabIO is now ready for a wider audience and could save other
researchers from repeating the work involved in decoding a
binary file format.
The module is written in an object-oriented scripting language Python
\cite{python}.
Table \ref{format} shows the list of file formats that FabIO
can currently (ver. 0.1.0) read.
The software is already used in a number of applications including
FitAllB \cite{fitallb}, the FABLE graphical interface \cite{fable}, EDNA \cite{edna}
and the fast azimuthal integration library, pyFAI \cite{pyfai}.


\section{FabIO python module}

\subsection{Philosophy}

The intention behind this development was to create a python module which would enable easy reading
of 2D data images, from any detector, without having to worry about
the file format.
Therefore FabIO just needs a file name to open a file and it determines the
file format automatically and deals with gzip \cite{gzip} and bzip2
\cite{bzip2} compression transparently.
An object is returned which stores the image data in memory as a 2D
numpy array \cite{numpy}.
If the file contains header information (metadata), these are also read and stored
as strings in a python dictionary.
Methods are provided for reading the previous or next image in
a series of images as well as jumping to a specific file number.
For the user, these auxiliary methods are intended to be independent of
the image format (as far as is reasonably possible).
The software is very modular, this allow to includes new classes for handeling 
other data formats quite easily. 
FabIO and its source-code are freely available to everyone on-line \cite{fabio}, 
licensed as GNU General Public License version 3 (GPLv3). 

\subsection{Implementation}
The main language used in the development of FabIO is Python \cite{python}. However, for some image formats requiering time consuming algorithms, the Python code has been interfaced with a standard C layer (sometimes generated using cython \cite{cython}). 

Consequence of a constant concern for the speed of calculation, FabIO makes use of the numerical python module NumPy \cite{numpy}. Two other python modules can be optionaly used. 
An XML toolkit called, Lxml, for reading XML files used in EDNA \cite{edna}, and the Python Image Library (PIL \cite{pil}) which contains a useful representation for the display of images in GUIs based on the Python Tkinter library \cite{tkinter} but also for access to the many image processing operations in that library.

FabIO do not integrate any visualisation tool, but, images can be displayed very conveniently using the combination of the library matplotlib \cite{matplotlib} with the interactive python shell IPython \cite{ipython}.
 
Reading and writing procedures of the various TIFF \cite{tiff} formats are based on the
TiFFIO code of V. Armando Sol\'e from PyMCA \cite{pymca}.

In the Python shell the {\em fabio} module must be imported prior to reading an
image of one of the supported formats (see Table \ref{format}).
The {\em fabio.open} function creates an instance of the python class {\em fabioimage},
from the name of a file.
This instance stores the image data in {\em img.data} as a numpy array
Often the image file contains furthermore information than just
the intensities of the pixels, e.g. information about how the image is
stored and the instrument parameters at the time of the image acquisition.
Header information, if available, is available in {\em img.header} as a Python
dictionary.

Currently, FabIO does not attempt to interpret or translate metadata which
is found in the headers of various frame formats, but merely presents the
information as a string. Neverthless FabIO is capable of converting one
dataformat into another, taking care of the dataformat; for example float array
are converted to integer array if the output format only accepts integers.


\subsection{FabIO methods}

One the strengths of the implementation in an object oriented language as Python
is the possibility to combine functions (or methods) together with data which
are appropriate for specific formats.
In addition to the header information and image data, every fabioimage instance
(returned by fabio.open) provides methods which provide information about the image minimum, maximum and mean values.
There are methods which gives the file number, name etc. One of the most important methods which  actually changes for certain formats
is get next image in a sequence; these functions are next, previous, and
getframe.


\section{Installation}

As any Python module, FabIO can be installed from its sources available on sourceforge\cite{fabio}
but we advice to use the package installer provided for the most common
platforms: Windows, MacOSX and Linux. Moreover FabIO is part of common linux
distribution Ubuntu (since 11.10) and Debian7.

\subsection{Examples}

Opening an image:

\begin{verbatim}
import fabio     
im0 = fabio.open('Quartz_0100.tif') # Open image file
print(im0.data[1024,1024]) # check a pixel value
im1 = im0.next() # Open next image
im270 = im1.getframe(270) # Jump to file number
\end{verbatim}

Normalising the intensity to a value in the header

\begin{verbatim}
img = fabio.open('exampleimage0001.edf')
print img.header
{'ByteOrder': 'LowByteFirst',
 'DATE (scan begin)': 'Mon Jun 28 21:22:16 2010',
 'ESRFCurrent': '198.099',
...
}
# Normalise to beam current and save data
srcur = float(img.header['ESRFCurrent'])
img.data *= 200.0/srcur
img.write('normed_0001.edf')
\end{verbatim}

Interactive viewing

\begin{verbatim}
import pylab								# import pylab from matplotlib
pylab.imshow(img.data)					# display as an image
pylab.show()								# show window
\end{verbatim}

%\begin{verbatim}
%> import fabio
%> im = fabio.open(’Quartz_0100.tif’)	# Open image file
%> im.data								# Print image data
%> im = im.next()						# Open next image
%> im = im.getframe(270)				# Jump to file number.
%\end{verbatim}


%\begin{verbatim}
%> import fabio 								# import the fabio module
%> img = fabio.open(‘exampleimage0001.edf’) 	# Create a fabioimage instance from the file
%> img.header 								# prints all headers from the file as a python dict.
%{'ByteOrder': 'LowByteFirst',
% 'DATE (scan begin)': 'Mon Jun 28 21:22:16 2010',
% 'DataType': 'UnsignedShort',
% 'Detector': 'frelon4m (sn=29)',
% 'Dim_1': '2048',
% 'Dim_2': '2048',
% 'ESRFAutoTime': '62.2611',
% 'ESRFBeamLine': 'id11',
% 'ESRFCurrent': '200.099',
% 'ESRFFillMode': '7/8 multibunch',
% 'ESRFRefill': '38265',
% 'Experiment': 'ma558',
% 'Fz1': '20.465713',
% 'Fz2': '20.467421',
%……
% 'zb': '22.300000',
% 'zm': '1.239060'}
%> Fz1 = img.header[‘Fz1’]					# retrieve an individual value
%>
%> import pylab								# import pylab from matplotlib
%> pylab.imshow(img.data)					# display as an image
%> pylab.show()								# show window
%\end{verbatim}


\section{Future and perspectives}

Hierachical data format version 5 (HDF5\cite{hdf5}) is a data format which is
getting popular in the field of X-rays data storage. Mainly processed
data are stored in this format today but new detector should provide native
output in HDF5. Then, FabIO will interface those data using the python binding 
to HDF5 (h5py\cite{h5py}) for providing data via the same interface as of today.

Meanwhile FabIO will be upgraded to Python3; this change of version will affect
a lot of the internal code of FabIO as all string handling and file handling
changed between current Python2 and the new Python3. This change is already
ongoing as many part of native code in C was re-implemented in
Cython\cite{cython} to smoothen the transition as Cython is compatible with
Python3.

\subsection{Conclusion}

FabIO gives an easy way to read and write 2D images when using the
python computer language.
It has originally developed for X-ray diffraction but now gives
an easy way for scientists to access and manipulate
their data from a wide range of range of 2D detectors.
We welcome contributions to fruther improve the code and hope to add
further file formats in the future as well as port the existing code base to the emerging python3.


\subsection{acknowledgement}
We acknowledge Andy G\"otz and Kenneth Evans for extensive testing when including
the FabIO reader in the Fable ImageViewer.
We also thank V. Armando Sol\'e for assistance with his TiffIO reader and
Carsten Gundlach for implementation of FabIO at the beamlines i711 and i811, MAX IV and providing bug reports.
We finally acknowledge our colleagues who have reported bugs and helped to
improve FabIO.
Financial support was granted by the EU 6th Framework NEST/ADVENTURE project
TotalCryst.

\bibliographystyle{iucr}
\bibliography{biblio}


\appendix
\section{Various detector format supported by FabIO.}
% put in appendix it can be full width
\onecolumn
\begin{table}[h]

\caption{\label{format}List of file formats that FabIO can read and write}
\vspace{1mm}
\begin{center}
\begin{tabular}{llccc}
Python Module   & Detector		& Typical extension & Multi-image	& Write\\% & Notes\\
\hline % alphabetical order - all formats are equal :-)
adsc	   &   ADSC Quantum		&	.img	&	No	&	Yes		\\%Check\\
brukeri		&   Bruker formats		&	.sfrm	&	No	&	Yes		\\%TODO Complete\\
DM3			&						&	.dm3	&	?	&	?		\\%?\\
edf		    &   ESRF data format	&	.edf	&	Yes	&	Yes		\\%Several bit formats\\
EDNA\cite(edna}-XML		&	.xml	&	No		&	No	 \\%Internaly used in EDNA\cite{edna}\\
cbf		    &   CIF Binary Files	&	.cbf	&	No?	& 	Yes		\\%Used also by Pilatus \\
kcd	    	&   Nonius 	KappaCCD	&	.kccd	&	No 	&	No		\\%Multiples frames are merged are reading. now bruker\\
fit2dmask \cite{fit2d} 		&   .msk    &   No  &   Yes  \\
fit2dspreadsheet \cite{fit2d}	&   .spr    &   No  &   Yes    \\
GE		    &   General Electric	&	-		&	Yes	&	No?		\\%massivly multi-frame\\
HiPiC       & Hamamatsu CCDs 		&	.tif	&	No	&	No?	 	\\%Tiff based format\\
marccd		&   MarCCD/Mar165		&	.mccd	&	No	&	No?		\\%Tiff based format\\
mar345		&   Mar345				&	.mar3450		&	No	&	Yes		\\%Compressed format\\
OXD		    &   Oxford Diffraction 	&	.img	&	No	&	Yes		\\%now Agilent\\
pilatus	    & Dectris Pilatus Tiff		&	.tif	&	No?	&	No?		\\%Tiff based  format\\
PNM			&	.pnm	&	No	&	?		\\% pnm is a unix image format
TIFF		&	.tif	&	No	&	?		\\%can be compressed or not\\
\end{tabular}
\end{center}
\end{table}

\end{document}