Add an alternative normalization syntax

README.md

@@ -21,7 +21,7 @@ Installation
 
 1. Create a dedicated python virtual environment and populate it with required modules listed in file `requirements.txt`.
 
-2. Make sure that you have ImageJ/Fiji, ImageMagick, and ffmpeg installed. The module often works as a configurable glue-script that delegates tasks to these tools.
+2. Make sure that you have ImageJ/Fiji, ImageMagick, and ffmpeg installed. The module generally works as a configurable glue-script that delegates tasks to these tools.
 
 
 
@@ -111,7 +111,7 @@ Option `--stitch` (added by default, use `--no-stitch` to disable).
 
 Stitching of tiles is delegated to the standard ImageJ "Grid/Collection stitching" plugin. ImageJ is called in the background in a headless mode (additionally enforced by using Xvfb).
 
-The value of the tile-over-tile overlap needs to be given in the config file, in section Stitching. The Harmony setting of '0%' is in reality about 0.5% but in the config file you are expected to provide the nominal value given in Harmony. Please bear in mind that the ideal overlap in bright-field may turn out to be suboptimal in fluorescence channels.
+The value of the tile-over-tile overlap needs to be given in the configuration file, in section Stitching. The Harmony setting of '0%' is in reality about 0.5% but in the config file you are expected to provide the nominal value given in Harmony. Please bear in mind that the ideal overlap in bright-field may turn out to be suboptimal in fluorescence channels.
 
 Stitches retain the channel order of tiles.
 
@@ -132,7 +132,8 @@ Option `--remix` (added by default, use `--no-remix` to disable).
 Generation of so-called remixes involves: (i) affine adjustment of the dynamic range of pixel intensities, (ii) conversion from 16-bit to 8-bit depth with a simple scaling of pixel intensities, (iii) pseudo-coloring of individual channel images.
 
 
-Affine adjustments are given in normalization section of the config file, as a pair of numbers comprising a subtracted (background) intensity value and a multiplicative factor (usually > 1). The list and exact definition of overlaid channels and their respective pseudo-colors is provided in the config file in section Remixes. Available pseudo-colors are: gray, red, green, blue, cyan, magenta, yellow.
+Affine adjustments are given in the normalization section of the configuration file, either as a pair of numbers comprising a subtracted (background) intensity value and a multiplicative factor (usually > 1) or the range of intensity percentages (0% is just 0, whereas 100% corresponds to the maximum possible value of 2^16 - 1, because the 16-bit depth is assumed). The list and exact definition of overlaid channels and their respective pseudo-colors is provided in the config file in section Remixes. Available pseudo-colors are: gray, red, green, blue, cyan, magenta, yellow.
+
 
 Well information provided in and extracted from the plate layout definition may be used to annotate remixes by burning in textual information in top-left image corner. To annotate remixes, add option `--annotate-remixes-with-wells-info`.
 
@@ -152,7 +153,7 @@ In the case of time-lapse experiments, remixes of consecutive time points may be
 
 The frame rate is adjusted so that one hour of experiment real time is one second of the movie. This default can be changed in the config file (see the optional sections).
 
-Hint: To watch movies with zoom, you may use [VLC](https://videolan.com) and then in its menu Tools → Adjustments and Effects → Video effects tick the option 'Interactive zoom'; use the bird-view in the top-left corner to move around.
+Hint: To watch movies with zoom, you may use [VLC](https://videolan.com) and then in its menu Tools → Adjustments and Effects → Video effects tick the option 'Interactive zoom'; use the bird-view in the top-left corner to move around and a vertical slider just below to (awkwardly) change zoom. Alternatively, you may use [mpv](https://mpv.io), in which you move around and zoom in/out using keyboards shortcuts (and can loop the movie retaining the viewport settings).
 
 
 
@@ -177,7 +178,7 @@ Stitching:
 
 Normalization:
     dapi:    -670, *1.1
-    bf:     -6500, *1.9
+    bf:     10% ... 30%   # equivalent to: -6553, *5
     p65:    -2420, *2.9
     polyic: -5635, *7.8
 
@@ -239,7 +240,7 @@ to obtain BZ2-compressed archives of images, one archive per well.
 
 By default, internal compression is abrogated (TIFF files exported by Harmony are internally compressed with LZW, which prevents a more efficient external compression).
 
-Note on compression ratio: bzip2, although slow, was experimentally checked to give the best compression ratio, exceeding that of zip and even xz (at its "ultra" settings and extra-large dictionary), and is considered more suitable for long-term data storage than xz.
+Note on compression ratio: bzip2, although slow, was experimentally confirmed to give the best compression ratio, exceeding that of zip and even xz (at its "ultra" settings and extra-large dictionary), and is considered more suitable for long-term data storage than xz.
 
 
 ### Unarchiving
@@ -248,14 +249,14 @@ To extract all images, enter the Images folder containing per-well archives and
 ```
     find . -name 'well-*.tar.bz2' -exec tar xfj {} \;
 ```
-The images are decompressed directly in the Images folder (not in any well subfolder), recovering the original flat layout of files in folder Images.
+The images are decompressed directly in the Images folder (not in any well subfolder), recovering the original flat layout of image files in folder Images.
 
 
 
 Limitations
 ===========
 
-The module has never been used to process Z-stacks (except for a built-in capability to select best focused image in a Z stack).
+The module has never been used to process Z-stacks (except for a built-in capability to select the best focused image in a Z stack).
 
 
 

maestro.py

@@ -438,7 +438,10 @@ def stitch_tiles(
         zero_pixels_in_output_image_retries_count = 0
 
 
-    desc = f"[{well_id}] {STITCHES_FOLDER_BASE_NAME}: {'fijing:' if n_fields_x*n_fields_y>1 else 'copying:'}"
+    desc = (
+        f"[{well_id}] {STITCHES_FOLDER_BASE_NAME}: "
+        f"{'fijing:' if n_fields_x*n_fields_y>1 else 'copying:'}"
+    )
     for timepoint, tile_images_paths in tqdm(tile_images_paths_grouped_by_timepoint, desc=desc,
                                              **TQDM_STYLE):
 
@@ -506,7 +509,8 @@ def stitch_tiles(
 
                 if require_no_zero_pixels_in_output_image:
                     dst_file_path_s = str(dst_file_path.absolute())
-                    reading_ok, dst_img = cv2.imreadmulti(dst_file_path_s, flags=cv2.IMREAD_UNCHANGED)
+                    reading_ok, dst_img = cv2.imreadmulti(dst_file_path_s,
+                                                          flags=cv2.IMREAD_UNCHANGED)
                     assert reading_ok
                     assert len(dst_img) > 0
                     if (zero_pixel_count := sum(sum(dst_img).ravel() == 0)) > 0:
@@ -566,23 +570,34 @@ def _assemble_suffix(info: pd.DataFrame) -> str:
 
 
 
-def _derive_channels_composition(mixing_info: pd.DataFrame) -> dict:
+def _intensity_range_arithmetics(obs_norm: str, obs_name: str) -> Tuple[str, str]:
+    # 'Normalization' means manual adjustment of the dynamic range of pixel intensities.
+
+    if pd.isna(obs_norm):
+        print(f"Warning: Missing normalization for observable '{obs_name}'! Assuming -0, *1.")
+        return ('0', '1.')
+
+    normalization_re1 = re.compile(r'-(?P<sub>[0-9]+)?\ *,\ *\*(?P<mul>(\d+(\.\d*)?)|(\.\d+))?')
+    if normalization_re1_match := normalization_re1.match(obs_norm):
+        return tuple(map(normalization_re1_match.group, ['sub', 'mul']))
+
+    normalization_re2 = re.compile(r'((?P<begin>[0-9]+)%)?\ *...\ *((?P<end>[0-9]+)%)?')
+    if normalization_re2_match := normalization_re2.match(obs_norm):
+        begin, end = map(float, map(normalization_re2_match.group, ['begin', 'end']))
+        return tuple(map(str, [int((begin/100)*(2**16 - 1)), 100./(end - begin)]))
+
+    print("Cannot parse the normalization for observable '{obs_name}' ('obs_norm'?)! "
+          "Assuming -0, *1.")
+    return ('0', '1.')
+
 
-    # Here, 'normalization' means manual adjustment of the dynamic range of pixel intensities.
-    normalization_re = re.compile(r'-(?P<sub>[0-9]+)?\ *,\ *\*(?P<mul>(\d+(\.\d*)?)|(\.\d+))?')
+
+def _derive_channels_composition(mixing_info: pd.DataFrame) -> dict:
 
     # single_channel
     if len(mixing_info) == 1 and mixing_info.index[0] == 'gray':
         tiff_page, obs_norm = mixing_info[['TiffPage', 'Normalization']].iloc[0]
-        if not pd.isna(obs_norm):
-            normalization_re_match = normalization_re.match(obs_norm)
-            assert normalization_re_match is not None, "Cannot parse the normalization parameters"
-            sub, mul = map(normalization_re_match.group, ['sub', 'mul'])
-        else:
-            print(f"Warning: Missing normalization for '{mixing_info['Observable'].iloc[0]}'! "
-                  "Assuming -0, *1.")
-            print(mixing_info)
-            sub, mul = map(str, [0, 1.])
+        sub, mul = _intensity_range_arithmetics(obs_norm, mixing_info['Observable'].iloc[0])
         return {'gray': (tiff_page, (sub, mul))}
 
     # multi_channel
@@ -591,15 +606,7 @@ def _derive_channels_composition(mixing_info: pd.DataFrame) -> dict:
         composition[component] = []
         for color, (tiff_page, obs_norm) in mixing_info[['TiffPage', 'Normalization']].iterrows():
             if component in COLOR_COMPONENTS[color]:
-                if not pd.isna(obs_norm):
-                    normalization_re_match = normalization_re.match(obs_norm)
-                    assert normalization_re_match is not None
-                    sub, mul = map(normalization_re_match.group, ['sub', 'mul'])
-                else:
-                    print(f"Warning: Missing normalization for the {color} observable! "
-                          "Assuming -0, *1.")
-                    print(mixing_info)
-                    sub, mul = map(str, [0, 1.])
+                sub, mul = _intensity_range_arithmetics(obs_norm, color)
                 component_definition = (tiff_page, (sub, mul))
                 composition[component].append(component_definition)
     return composition
@@ -678,7 +685,9 @@ def remix_channels(
                 if n_source_channels == 0:
                     continue
                 assert n_source_channels < 3, \
-                    f"Too many source channels for the {component} component of the overlay!"
+                    f"Too many source channels for the *{
+                        {'R': 'red', 'G': 'green', 'B': 'blue'}[component]
+                    }* component of the overlay!"
 
                 used_component_channels += component
 
@@ -1155,7 +1164,8 @@ def show_wells(
 ) -> None:
 
     wells_info = extract_wells_info(operetta_export_folder_path)
-    wells_info_str ='\n'.join(wells_info['WellId']) if simple else wells_info.to_string(max_cols=100)
+    wells_info_str= '\n' .join(wells_info['WellId']) if simple else \
+        wells_info.to_string(max_cols=100)
     print(wells_info_str)
 
 
@@ -1166,7 +1176,8 @@ def show_channels(
 ) -> None:
 
     channels_info = extract_channels_info(operetta_export_folder_path)
-    channels_info_str = '\n'.join(channels_info['Name']) if simple else channels_info.to_string(max_colwidth=48)
+    channels_info_str = '\n'.join(channels_info['Name']) if simple else \
+        channels_info.to_string(max_colwidth=48)
     print(channels_info_str)
 
 
@@ -1263,7 +1274,8 @@ def trackerabilize(
         if not item.is_dir():
             continue
         folder = item
-        if folder.name.endswith(REMIXES_FOLDER_BASE_NAME) and folder.name != REMIXES_FOLDER_BASE_NAME:
+        if folder.name.endswith(REMIXES_FOLDER_BASE_NAME) and \
+                folder.name != REMIXES_FOLDER_BASE_NAME:
             remixes_folder = folder
 
             # source image files
@@ -1276,8 +1288,11 @@ def trackerabilize(
             observables = dict(enumerate(set(
                 single_channel_remix_match.group('observable')
                 for path in image_files_paths
-                if (single_channel_remix_match := single_channel_remix_re.search(path.name)) is not None
+                if (single_channel_remix_match := single_channel_remix_re.search(path.name)) \
+                        is not None
             )))
+            for obs in observables.values():
+                assert obs in colors.index, f"Observable '{obs}' has no assigned pseudocolor!"
 
             # create output folder
             well_id = remixes_folder.name.replace(f"-{REMIXES_FOLDER_BASE_NAME}", '')
@@ -1302,12 +1317,18 @@ def trackerabilize(
                 if delta_t is not None:
                     print(f"time_interval {delta_t.total_seconds()}", file=st_file)
 
-            print(f"Info: Images from '{remixes_folder.name}' symlinked in '{shuttletrackerable_folder.name}'.")
+            print(f"Info: Images from '{remixes_folder.name}' symlinked in "
+                  f"'{shuttletrackerable_folder.name}'.")
 
             processed_folders_count += 1
 
-    print("Info: Creating ShuttleTracker-viewable folders ... done")
-    if processed_folders_count != len(wells_info):
+    if processed_folders_count == len(wells_info):
+        print("Info: Creating ShuttleTracker-viewable folders ... done")
+    else:
+        if processed_folders_count > 0:
+            print("Info: Creating ShuttleTracker-viewable folders ... PARTLY done")
+        else:
+            print("Info: Creating ShuttleTracker-viewable folders ... NOT done")
         print(f"Warning: Processed folder count: {processed_folders_count}, "
               f"well count in metadata: {len(wells_info)}.")
 
@@ -1489,14 +1510,16 @@ def remaster(
         realtime_speedup = int(config['Movies']['realtime_speedup'].replace('x', ''))
     else:
         realtime_speedup = MOVIE_DEFAULT_REALTIME_SPEEDUP
-        print(f"Info: Assumed default movie realtime speedup: {MOVIE_DEFAULT_REALTIME_SPEEDUP}x.")
+        print("Info: Assumed the default movie real time speed-up: "
+              f"{MOVIE_DEFAULT_REALTIME_SPEEDUP}x.")
 
     # (optional) text annotation settings
     if 'Annotations' in config and 'font_size' in config['Annotations']:
         annotation_font_size = int(config['Annotations']['font_size'].replace('pt', ''))
     else:
         annotation_font_size = TEXT_ANNOTATIONS_DEFAULT_FONT_SIZE
-        print(f"Info: Assumed default text annotation font size: {TEXT_ANNOTATIONS_DEFAULT_FONT_SIZE} pt.")
+        print("Info: Assumed default text annotation font size: "
+              f"{TEXT_ANNOTATIONS_DEFAULT_FONT_SIZE} pt.")
 
 
     # selection of wells
@@ -1765,7 +1788,8 @@ def _check(
             empty_image_path = images_folder_path / 'empty.tiff'
             cv2.imwrite(str(empty_image_path.absolute()), empty_image.astype(np.uint16))
 
-            for image_path, image_shape in tqdm(image_shapes.items(), desc='Info: Fixing ...', **TQDM_STYLE):
+            for image_path, image_shape in tqdm(image_shapes.items(), desc='Info: Fixing ...',
+                                                **TQDM_STYLE):
                 if image_shape[0]*image_shape[1] in (0, 1):
                     image_path.rename(str(image_path.absolute()) + '.orig')
                     image_path.symlink_to(empty_image_path.name)