Potential bug: TaskLoader does not run when patching and using 'Gapfill' samlping strategy

[MartinSJRogers]

Description

The Taskloader function seems to freeze when asked to generate patched functions using the gapfill sampling strategy:

Reproduction steps

This first example runs fine

# Instantiate task loader
task_loader = TaskLoader(
    context = [modis_ds, amsr_ds] *5,
    target = modis_ds,
    context_delta_t = [-2, -2, -1, -1, 0, 0, 1,1,2,2], 
    target_delta_t = 0,
    #links = [(4, 0)],
)
for date in tqdm(dates, disable=not progress):
    tasks_per_date = task_loader(date, context_sampling=["all", "all", "all", "all", "all",
                                                                "all", "all", "all", "all", "all"],
                            target_sampling="all",
                            patch_strategy="sliding",
                            patch_size=0.5,
                            stride=0.25
            )
            for task in tasks_per_date:
                task.remove_context_nans().remove_target_nans()
            train_tasks.extend(tasks_per_date)

But if you call this example, nothing happens:

# Instantiate task loader
task_loader = TaskLoader(
    context = [modis_ds, amsr_ds] *5,
    target = modis_ds,
    context_delta_t = [-2, -2, -1, -1, 0, 0, 1,1,2,2], 
    target_delta_t = 0,
    links = [(4, 0)],
)
## Code runs down to here and then stops:
for date in tqdm(dates, disable=not progress):
     tasks_per_date = task_loader(date, context_sampling=["all", "all", "all", "all", "gapfill",
                                                                "all", "all", "all", "all", "all"],
                            target_sampling="gapfill",
                            patch_strategy="sliding",
                            patch_size=0.5,
                            stride=0.25
            )
            for task in tasks_per_date:
                task.remove_context_nans().remove_target_nans()
            train_tasks.extend(tasks_per_date)

Version

Patchwise_train fork, monotonic errors branch,

Screenshots

![DESCRIPTION](LINK.png)

OS

Windows

[davidwilby]

Could you maybe send your code (privately) so I can see whether this is relating to the datasets used? I haven't been able to replicate this behaviour so far, though using a different dataset I expect.

Have you run this code with a debugger? I'd like to check 1. Is it really hanging indefinitely, or just taking a long time, and if so 2. where is the freezing happening, is it actually in TaskLoader.__call__?

[MartinSJRogers]

Thanks @davidwilby I will send a private gist now. I haven't run this code with a debugger, I have just used my trusted print statements all over the place again. The print statements prove the code is hanging indefinitely/ taking a very long time when generating tasks, but I guess the debugger is needed to identify where in the actual DeepSensor codebase the code is hanging indefinitely?
In the code I have sent you, if you set patching to False the code runs fine.

[davidwilby]

In running your example @MartinSJRogers , I've tracked the hanging behaviour down to this section of deepsensor.data.TaskLoader.task_generation:

deepsensor/deepsensor/data/loader.py

Lines 1367 to 1395 in 23733df

	# Keep trying until we get a target set with at least one target point
	keep_searching = True
	while keep_searching:
	added_mask_date = rng.choice(self.context[context_idx].time)
	added_mask = (
	self.context[context_idx].sel(time=added_mask_date).isnull()
	)
	curr_mask = context_var.isnull()
	# Mask out added missing values
	context_var = context_var.where(~added_mask)
	# TEMP: Inefficient to convert all non-targets to NaNs and then remove NaNs
	# when we could just slice the target values here
	target_mask = added_mask & ~curr_mask
	if isinstance(target_var, xr.Dataset):
	keep_searching = np.all(target_mask.to_array().data == False)
	else:
	keep_searching = np.all(target_mask.data == False)
	if keep_searching:
	continue # No target points -- use a different `added_mask`
	target_var = target_var.where(
	target_mask
	) # Only keep target locations
	context_slices[context_idx] = context_var
	target_slices[target_idx] = target_var

In this section keep_searching is never being set to True for some or all of your patches I think, so this while loop just runs indefinitely.

target_mask = added_mask & ~curr_mask
if isinstance(target_var, xr.Dataset):
    keep_searching = np.all(target_mask.to_array().data == False)
else:
    keep_searching = np.all(target_mask.data == False)
if keep_searching:
    continue  # No target points -- use a different `added_mask`

[davidwilby]

Resolved by #11 - moving the spatial slicing operation after the gapfill operation.

Will also be addressed by #12 - but at a later date.