How to properly preprocess images?

[romanvelichkin]

I see, many models have preprocess layer:
add_pre_post_process(model, rescale_mode=rescale_mode)`

But examples says I need to call preprocess forcibly:
pred = mm([mm.preprocess_input(chelsea())])

So what's the correct way, assuming I have raw images with [0, 256] range?

[leondgarse]

The add_pre_post_process just adds 2 functions preprocess_input and decode_predictions as model properties, but not into model layers. So the correct way is manually calling them like pred = mm(mm.preprocess_input(chelsea())).

[lucashervier]

Hi,

I have a similar question as I try to find how to get the same Top-1 Accuracy as you mention in the README for different models.

Initially I guessed that you used the evaluation method defined in the imagenet/eval_func.py script. So most preprocessing happens when building the test_dataset with the help of the data.init_dataset. However, if I do that with ConvNeXtV2Huge I have a model with less than 1% accuracy...
On the other hand, if I use predicts = model(model.preprocess_input(img_batch)) my results are getting closer. However, it is hard to know what is happening in the loader and what is happening in this preprocess_input method. For instance, I wonder if I can't have exactly your results because the central crop happened two times.

In a nutshell, I wonder if you have the configurations that allow the same accuracies as mentioned in the README somewhere? (I suppose they are the Top-1 acc on the validation set of ImageNet right?)

PS: I recently discovered your repository and it is really impressive, thank you a lot :)

[romanvelichkin]

In section I mentioned before, there is also evaluation for EfficientNet:
#19

I guess, this could help you to understand how accuracy was calculated and with what parameters.

[lucashervier]

I don't think we should mix preprocess_input with eval_script. Preprocess only transforms the image values from the range [0, 255] to [0, 1] or [-1, 1] or some other ranges, plus without scaling or with normalization to whole dataset. While eval_script does the rest of the data preparation: resizing (with or without antialiasing, with or without cropping, etc.), batching and creating classes.

I think in eval_script there is the preprocess_input included through the parameter rescale_mode. So if one uses the init_dataset method and still calls the preprocess_input then the image is rescaled two times I think. In the minimal example, one can use the preprocess_input method because the image loaded as an example is not loaded with the loader build with the init_dataset method.

Am I right or am I missing something? Because in my understanding one should either use a loader built with the init_dataset method, OR apply the transformations manually (including the preprocess_input) but not the two in the same time.

[romanvelichkin]

Indeed, it looks like eval_script contains everything that you need for evaluation, so you don't need to use preprocess_input also.

I guess preprocess_input is for custom datasets, where you set the rest of parameters for input data by yourself.

Now I get, what did you mean in your post!

ConvNext in discription (https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/convnext) uses 'torch' preprocess_input, so you could try it. Models use original weights, so rescale option depends on how your model was trained, with what rescaling mode. Also you could use small fraction of imagenet images and try different options.

About other preprocess params like crop etc, you better try to read original paper.

I think most preprocess options like antialising or gentle crop with 0.95 value shouldn't significantly affect accuracy, since models are trained on huge datasets. If you get 1% accuracy, you probably selected wrong rescale option.

Maybe you should try to evalutate model with only rescaling imagenet dataset properly without using other preprocess options?

[leondgarse]

Thanks @lucashervier for your reply!

• preprocess_input is added for processing a single image, just a helper function for testing. The key parameter is the rescale_mode, functional as common_layers.py#L669-L737.
• For eval_script.py, the default rescale_mode value "auto" means getting it from model eval_func.py#L170-L172, which is set in common_layers.py#L734.
• The accuracy in the table is mostly from the original repo or the paper, even if my testing result not matching, like maxvit. Most ported models ouputs are verified using a single random input, like convnext#verification-with-pytorch-version.
• The 1% accuracy discrepancy myabe caused by rescale_mode value, may test it later.

[leondgarse]

Just ran an accuracy test using

python eval_script.py -m ConvNeXtV2Huge -d imagenet_val.json --central_crop 0.875
# >>>> Accuracy top1: 0.86306 top5: 0.97866

May find the central_crop info in timm results-imagenet.csv.