Urban Wind Field: URock v2023a - Missing Data in Raster File

[AlexandraStankulova]

I am performing the wind analysis using the "Urban Wind Field: URock v2023a". The area is quite large and complex, but after 3hours the analysis finishes with success, loading as always the created raster file with the wind speed.
The problem is that the raster resullts with no data, while the vector file has the vaues in it.
What could be the problem?

Here I upload the input data and the output data created by UMEP:

https://we.tl/t-TZrhbqXpdl

[biglimp]

I have the same issue, but only on my windows machine. @AlexandraStankulova , are you on windows also?

I am simulating on 4 meter resolution, both in horizontal and vertical dimensions, then I have a 2 meter raster grid that I want to use for raster output.

I have QGIS 3.40, UMEP 2.1.3.

@j3r3m1 , how do you interpolate?

Below. you find my data and settings:

processing.run("umep:Urban Wind Field: URock", {'BUILDINGS':'D:/GRUPAS/THERMAL_COMFORT_FIVE_CITIES/MALMO/UROCK/UROCKIN/build_urock44.shp','HEIGHT_FIELD_BUILD':'ROOF_H','VEGETATION':'D:/GRUPAS/THERMAL_COMFORT_FIVE_CITIES/MALMO/UROCK/UROCKIN/veg_urock44.shp','VEGETATION_CROWN_TOP_HEIGHT':'VEG_H','VEGETATION_CROWN_BASE_HEIGHT':'trunk','ATTENUATION_FIELD':'','INPUT_PROFILE_FILE':'','INPUT_PROFILE_TYPE':0,'INPUT_WIND_HEIGHT':10,'INPUT_WIND_SPEED':2,'INPUT_WIND_DIRECTION':244,'RASTER_OUTPUT':'D:/GRUPAS/THERMAL_COMFORT_FIVE_CITIES/MALMO/UROCK/UROCKIN/temp_ClippedDSM.tif','HORIZONTAL_RESOLUTION':2,'VERTICAL_RESOLUTION':2,'WIND_HEIGHT':'1.5','UROCK_OUTPUT':'C:\\Users\\xlinfr\\Desktop\\Rockle_zones','OUTPUT_FILENAME':'urock_output','SAVE_RASTER':True,'SAVE_VECTOR':False,'SAVE_NETCDF':False,'LOAD_OUTPUT':True})

ResolutionErrorURock.zip

[AlexandraStankulova]

Yes, I also work with Windows, I have Windows 10.

[j3r3m1]

The interpolation depends on the resolution of your calculation and the resolution of the raster used as output constraint. However, most of the time (and this is the case for your data) this function is used:
https://github.com/UMEP-dev/UMEP-processing/blob/65b105bb04326870f306a693e5414c30bcd6baf2/functions/URock/saveData.py#L482

And then at the end the no data values are filled using the few lines after this one:
https://github.com/UMEP-dev/UMEP-processing/blob/65b105bb04326870f306a693e5414c30bcd6baf2/functions/URock/saveData.py#L452

I have not faced any issue with Ubuntu for your case @biglimp and I will try the @AlexandraStankulova case. The issue probably comes from one or the other of the code provided above.

I now have a Windows OS at home. It may help to fix Windows issues but I need to find time for that.

[j3r3m1]

@AlexandraStankulova can you please share a smaller file ? Your is more than 1Gb.

[AlexandraStankulova]

Yes sure. I will upload now only the input data and the output raster windspeed file that actually results with no data.

Here the files:

InputData.zip

urock_outputWS.zip

[j3r3m1]

Interesting, I could reproduce the error. I will investigate what is the issue

[j3r3m1]

Well the issue was that I did not use the right DSM for interpolating the results at the end... Thus I cannot reproduce the error on Ubuntu. I have to debug on a Windows machine, not sure when.

[lgzdbf]

I am encountering the same error on my side regarding outputs without data when running the process on Ubuntu.

@j3r3m1 Here is a sample to reproduce the error.

urock_grenoble_test.zip

Let me know if you need any additional information.

Thanks in advance for your help!

[j3r3m1]

OK there was quite a big problem. When the vegetation was not intersected at all by building wake zones (which is often the case in rural area obviously...), there vegetation was removed from the analysis... It is now solved.

[j3r3m1]

UMEP-dev/UMEP-processing@af8b178

[j3r3m1]

@lgzdbf it should solve your issue.
@AlexandraStankulova or @biglimp is this fix also solves your issue ? I guess not since I do not think it is related to Windows issue but we never know...

[AlexandraStankulova]

Hello. I guess it will solve it definetely because I was doing some different tests, and I noticed that sometimes when I increase the buffer of the data to be inserted also around the area of analysis, sometimes this problem was not appearing (but I coulnd't explain nmyself why with e certain buffer are it works and with another buffer area it doesnt).

[j3r3m1]

OK the TIN interpolation does not work in some specific cases. I have reported the bug in the QGIS repo. We will see how they can help (qgis/QGIS#60416).

Two alternatives can be used:

• using the IDW interpolation seems to work but not sure it can solve the problem in all situations (at least it solves it for the dataset provided by @lgzdbf) and I have set an arbitrary distance coefficient (I have tried several and this one results in something I feel comfortable with).
• using scipy interpolation methods and writing the raster using rasterio results in slightly faster writing but also a sligth shifting of the result and this means adding a new dependency with rasterio which seems not to be a good idea in terms of long-term maintenance.

@biglimp what do you think ?

[j3r3m1]

@biglimp @lgzdbf @AlexandraStankulova you can already try the first proposition as it it now in the last UMEP version.

[lgzdbf]

Thanks for your information @j3r3m1. I ran it again for a sample and it works without a problem. I will run it on several tiles to see what happens and let you know.
Thanks for your help!

[biglimp]

IDW might smooth out the result, but since you have lots of data points it should not be an issue.
rasterio (1.4.3) is included in the osgeo repository for windows installation so it should also work. We can add it to the umep-reqs also.

[AlexandraStankulova]

@j3r3m1, thank you for the information. I will retry to run the tile that I was analysing with the updated version from the repository and I will let you know.

[agiudiceandrea]

Have you tried the workaround proposed in my comment qgis/QGIS#60416 (comment)?

[j3r3m1]

Have you tried the workaround proposed in my comment qgis/QGIS#60416 (comment)?

I confess that after my first reading I only thought that the comment was a confirmation of the error. But actually this is both a workaround that solves the problem but which also speeds up the TIN algorithm... Thanks a lot for your proposition. This is now solved in last UMEP version (2.1.10)

[lgzdbf]

In version 2.1.10, the simulations seem to run indefinitely (two runs have been running since yesterday at 9 PM). This was not the case in V2.1.9. No errors are reported, but the algorithm appears to run endlessly. The raster WS file is saved within a few minutes after launch, while the VWS and HWS files, as well as the rockle zones, are still not generated.

[j3r3m1]

Can you please share the settings of the simulation that does not end ?

[lgzdbf]

Here are the settings for a tile and a day-time that doesn't work. But I don't think you will find anything like this because it's variable. When I rerun the algorithm, some runs that previously didn't finish are processed in just a few minutes.
(Example with the data from c-1 that you already have)

`Inputs

BUILDINGS: /workdir/hmcis/lagelout/5_application/To_workdir/spatial_input_data/va/grenoble/c-1/vector_build_c-1.fgb|layername=vector_build_c-1
HEIGHT_FIELD_BUILD: ROOF_HEIGHT
HORIZONTAL_RESOLUTION: 3
INPUT_PROFILE_TYPE: 0
INPUT_WIND_DIRECTION: 257.110664
INPUT_WIND_HEIGHT: 10
INPUT_WIND_SPEED: 0.773589
LOAD_OUTPUT: false
OUTPUT_FILENAME: urock_output
RASTER_OUTPUT: /workdir/hmcis/lagelout/5_application/To_workdir/output_data/tmrt/grenoble/era5_data/canicule3_20230811-26/c-1/2023-223/Tmrt_average.tif
SAVE_NETCDF: false
SAVE_RASTER: true
SAVE_VECTOR: true
UROCK_OUTPUT: /workdir/hmcis/lagelout/5_application/To_workdir/output_data/va/grenoble/era5_data/canicule3_20230811-26/c-1/223-7/urock_output
VEGETATION: /workdir/hmcis/lagelout/5_application/To_workdir/spatial_input_data/va/grenoble/c-1/vector_veg_c-1.fgb|layername=vector_veg_c-1
VEGETATION_CROWN_BASE_HEIGHT:
VEGETATION_CROWN_TOP_HEIGHT: VEG_HEIGHT
VERTICAL_RESOLUTION: 3
WIND_HEIGHT: 1.5
area_units: m2
distance_units: meters
ellipsoid: EPSG:7019
`

[j3r3m1]

OK the problem is probably raised by the fact that the points are randomly sorted before the TIN interpolation. In some situations the random order does not work but works for other orders. I have been trying to implement a way to reapply a new random sorting if the interpolation lasts too long but without success.
@agiudiceandrea any idea to solve the problem ? I have tried other ways to sort the data (sorting by ascending order for the variable I want to interpolate but it results in some weird patterns in some areas).
Otherwise I will have to be back to the IDW interpolation (which is longer).

[agiudiceandrea]

Unfortunately, it looks like my workaround doesn't always workaround the issue...
A proper fix is needed in the "TIN interpolation" processing algorithm.

[j3r3m1]

Concerning version 2.1.10, the problem might come from my side. I have made a slight modification at this branch (https://github.com/j3r3m1/UMEP-processing/tree/TIN_test - it is named version 2.1.11).
@lgzdbf can you please try if this one still get into troubles for some cases ? It is definitely faster when it works...
Otherwise you can update to the last UMEP version 2.1.11 (the real one, not the branch located on my repo) which uses the IDW algo but which is slower.

Let us know.

[lgzdbf]

Thank you for your feedback. Still the same issue with the version 2.1.11 test :
787754 calcul c-1_224_ lagelout R 9:17:09 1 ige-calcul1 787774 calcul c-2_223_ lagelout R 9:15:50 1 ige-calcul1 787779 calcul c-2_223_ lagelout R 9:15:44 1 ige-calcul1 787802 calcul c-2_224_ lagelout R 9:13:26 1 ige-calcul1

I will therefore try with IDW to see the result.

[lgzdbf]

All runs with IDW completed without errors. It is indeed faster with TIN than with IDW, averaging 1 minute compared to 7-8 minutes. However, the result template is different, which is expected.

Would you like me to test on a larger sample, or would you prefer to wait and decide if TIN will be kept as the official version?

[agiudiceandrea]

Is the "Order by expression" output a GeoPackage layer in your processing algorithm? If yes, then features are not actually reordered. See qgis/QGIS#60432.

[j3r3m1]

All runs with IDW completed without errors. It is indeed faster with TIN than with IDW, averaging 1 minute compared to 7-8 minutes. However, the result template is different, which is expected.

What do you mean by the result template is different ? Can you share screenshots with same value ranges for two simulations using a different version of the plugin (test and main) ?

Thank you for your feedback. Still the same issue with the version 2.1.11 test

All tiles have the same behavior or is it a random effects with most of them going up to the end ?

[lgzdbf]

(1) test version :

(2) main version :

It is a random effect. Nn the same tile, s dome days-times may not work during a run but can work if I start a new one. For the moment, I have only tested 4 tiles, so I don't know if the others behave the same way.

[j3r3m1]

Is the "Order by expression" output a GeoPackage layer in your processing algorithm? If yes, then features are not actually reordered. See qgis/QGIS#60432.

Yes the output is a gpkg file. But it is resulting in the creation of the primary indexes and a randomization of the order of the point (see below, left is the output gpkg, right is the initial fgb file):

Note that if I output a fgb file, there is no primary index created and the rows are still in the same initial order.

[j3r3m1]

It is a random effect. Nn the same tile, s dome days-times may not work during a run but can work if I start a new one. For the moment, I have only tested 4 tiles, so I don't know if the others behave the same way.

OK and how often do you get the problem for this tile ? Like 1 / 10 ?

[lgzdbf]

OK and how often do you get the problem for this tile ? Like 1 / 10 ?

On a tile, over 48 runs, there are about 3-5 problematic runs on average, each time different.

[agiudiceandrea]

Yes the output is a gpkg file. But it is resulting in the creation of the primary indexes and a randomization of the order of the point (see below, left is the output gpkg, right is the initial fgb file):

OK. The "Order by expression" issue only occurs if both input and output layers are GeoPackages or if they have a fid/primary key.