Rebuilding model during load gives error for multi GPU model

[suvojit-0x55aa]

PConv-Keras/libs/pconv_model.py

Line 264 in 7c8f2cb

self.model, inputs_mask = self.build_pconv_unet(train_bn)

Rebuilding the model without the init options gives error when training on multi GPU.
Error traceback:

Traceback (most recent call last):
  File "train_pconvunet.py", line 270, in <module>
    model.load(args.checkpoint)
  File "/home/ubuntu/image-inpainting/libs/pconv_model.py", line 307, in load
    self.model.load_weights(filepath)
  File "/home/ubuntu/image-inpainting/env/lib/python3.6/site-packages/keras/engine/topology.py", line 2667, in load_weights
    f, self.layers, reshape=reshape)
  File "/home/ubuntu/image-inpainting/env/lib/python3.6/site-packages/keras/engine/topology.py", line 3365, in load_weights_from_hdf5_group
    str(len(filtered_layers)) + ' layers.')
ValueError: You are trying to load a weight file containing 1 layers into a model with 31 layers.

Commenting the rebuilding and compilation statements in the load function works however.

[ghost]

This is because your saved model weights from multi GPU model, and push it into single GPU model.
There are some solutions.
1. Go to Pconv_model.py, and change "load model" function to build multi GPU model with the same number of GPU, which your model trained on.
2. Save your weights for single GPUs. Make sure your multi GPU model with model.summary() and model.layer[index]

In my private opinion,
I prefer the second, because sometimes the number of GPUs for training is different for ones for inference.