Error with rmapshaper::ms_clip

[jldupouey]

I am trying to clip a (rather large) sf layer using another (small) sf layer using rmaspshaper::ms_clip:

ms_clip(MySfLayer,MySfClippingLayer,sys=TRUE,sys_mem=30)

Despite using the system mapshaper library (option sys=TRUE) and a large memory allocation (option sys_mem=30), the process is very long (hours...).

The latest version of mapshaper seems to be installed correctly:

check_sys_mapshaper()

mapshaper version 0.6.102 is installed and in your PATH
                                              mapshaper-xl 
"C:\\Users\\JLDUPO~1\\AppData\\Roaming\\npm\\MA5A58~1.CMD"

I have the latest versions of R and rmapshaper installed.

When I do the same clipping directly, using mapshaper installed on the same Windows system, the operation is fast. It seems that rmapshaper is not using the system mapshaper library. Or maybe there is something sf does in ms_clip that is long?

I tried converting the layers to GeoJSON first. It seems to go further, and to use the system library, but now I get an error:

MySfLayer <- 
ms_clip(MyGeoJLayer,MyGeoJClippingLayer,sys=TRUE,sys_mem=30)

Allocating 30 GB of heap memory
[o] Wrote C:\Users\JLDUPO~1\AppData\Local\Temp\RtmpScM8M9\filec6066891abc.geojson
Error in unlink(c(target_layer, overlay_layer)) :
file name conversion problem -- name too long?

I am not sure if this is a bug, but I do not understand the origin of the problem.

[ateucher]

Interesting, thanks for reporting @jldupouey. Are you able to share the files you are trying to clip?

[jldupouey]

The large one is over 25MB, the upper limit for my GitHub repository. Sending larger files seems to be painful.

[ateucher]

Could you put it in a dropbox/google drive folder?

[jldupouey]

Here it is:

https://filesender.renater.fr/?s=download&token=540cfb37-ebe1-46c8-936e-99b0f72383c5

and my script:

library(sf)
library(geojsonsf)
library(rmapshaper)

MySfLayer <- st_read("MySfLayer.gpkg")
MySfClippingLayer <- st_read("MySfClippingLayer.gpkg")

MySfLayerWGS <- st_transform(MySfLayer,crs=4326)
MySfClippingLayerWGS <- st_transform(MySfClippingLayer,crs=4326)

MyGeoJLayerWGS <- sf_geojson(MySfLayerWGS)
MyGeoJClippingLayerWGS <- sf_geojson(MySfClippingLayerWGS)

# less than 1', but error
# works well with system mapshaper (~50 s.) :
ClipGeoJ <- ms_clip(MyGeoJLayerWGS,MyGeoJClippingLayerWGS,sys=TRUE,sys_mem=30) 

Registered S3 method overwritten by 'jsonify':
  method     from    
  print.json jsonlite
Allocating 30 GB of heap memory
[o] Wrote C:\Users\JLDUPO~1\AppData\Local\Temp\RtmpOGTnjE\file74843f976c23.geojson
Error in unlink(c(target_layer, overlay_layer)) : 
  file name conversion problem -- name too long?

# no end...
ClipSf <- ms_clip(MySfLayerWGS,MySfClippingLayerWGS,sys=TRUE,sys_mem=30) 

[ateucher]

Hi @jldupouey this runs ok for me. How much memory does your machine have? I ran it allocating only 16GB and it completed in about 55s; I wonder if you might be overallocating memory if you don't have enough free (i.e., < 30GB, or another process is using a lot)?

library(sf)
#> Linking to GEOS 3.11.0, GDAL 3.5.3, PROJ 9.1.0; sf_use_s2() is TRUE
library(geojsonsf)
library(rmapshaper)

check_sys_mapshaper()
#> mapshaper version 0.6.102 is installed and on your PATH
#>                     mapshaper-xl 
#> "/opt/homebrew/bin/mapshaper-xl"

MySfLayer <- st_read("~/Downloads/MyLayers/MySfLayer.gpkg")
MySfClippingLayer <- st_read("~/Downloads/MyLayers/MySfClippingLayer.gpkg")

MySfLayerWGS <- st_transform(MySfLayer, crs = 4326)
MySfClippingLayerWGS <- st_transform(MySfClippingLayer, crs = 4326)

tictoc::tic()
ClipSf <- ms_clip(
  MySfLayerWGS,
  MySfClippingLayerWGS,
  sys = TRUE,
  sys_mem = 16
)
tictoc::toc()
#> 56.772 sec elapsed

^{Created on 2025-02-27 with reprex v2.1.1}

[jldupouey]

Thanks for the help. I have reinstalled everything (R, Node, mapshaper, R libraries ...). I still can't run the ms_clip command. I have 60 GB of memory. I have tested with different values of sys-mem, without any success. I have no other jobs running and there is plenty of free memory available while running ms_clip.

It seems that the problem comes from the R side (rmapshaper/R memory management/rmapshaper-sf interactions...?), as the following command works perfectly:

shell("mapshaper MyLayer.shp -clip MyClippingLayer.shp -o Clip.shp")

One solution, of course, is to run mapshaper's shell commands. But rmapshaper was a very nice interface, which among other things avoided the burden of reading and writing external files.

[ateucher]

I agree it doesn't seem like the problem is with the mapshaper node library, something is happening in rmapshaper. Unfortunately since I can't reproduce it I don't know what the issue is. I can see if I can reproduce it on a Windows machine over the weekend