GBFS: Performance/Memory Improvements + Vehicle Types

[pablohoch]

• Adds proper vehicle type support. The API now has a single RENTAL mode with additional filters for form factors (bike, cargo bike, scooter etc.), propulsion type (human, electric etc.) and providers.
  • Note that currently all vehicle types use the bike sharing osr profile, i.e. cars are not yet properly supported.
• Providers are now partitioned into segments based on vehicle form factors, propulsion types, return constraints, geofencing zones and vehicle type return restrictions for stations. osr routing is performed for each provider segment matching the query.
• Vehicles with roundtrip station return constraints (vehicle must be returned to the departure station) can now be used for direct queries (only). Note that the return leg is not part of the connection, but this can be detected using the rental.returnConstraint field in the response. This is often used for cargo bikes and car sharing.
• Faster updates:
  • The ttl field is now processed to only download files when necessary.
  • If a file is not updated (either the ttl is not expired or the contents are unchanged), the existing data is reused.
  • Only bitfields for providers in the cache (see below) where data was changed are calculated during the update.
• Memory usage improvements (GBFS memory usage #660):
  • The bitfields for osr are only calculated once a provider is first used in a routing query or during the GBFS update if the cache is not yet full. (Queries where GBFS data is not in the cache will be slower. Providers with large geofencing zones will be especially slow.)
  • Only a limited number of bitfields (new setting: gbfs.cache_size) are kept in memory. These bitfields are also compressed.
  • Only the necessary bitfields are decompressed when processing a routing query.

Most GBFS providers don't actually use multiple vehicle form factors. Here is an example config for Switzerland, where some providers in Zurich provide bikes and scooters:

gbfs:
  feeds:
    Link_Basel:
      url: https://mds.linkyour.city/gbfs/ch_basel/gbfs.json
    bird-basel:
      url: https://mds.bird.co/gbfs/v2/public/basel/gbfs.json
    bird-biel:
      url: https://mds.bird.co/gbfs/v2/public/biel/gbfs.json
    bird-bulle:
      url: https://mds.bird.co/gbfs/v2/public/bulle/gbfs.json
    bird-kloten:
      url: https://mds.bird.co/gbfs/v2/public/kloten/gbfs.json
    bird-uster:
      url: https://mds.bird.co/gbfs/v2/public/uster/gbfs.json
    bird-winterthur:
      url: https://mds.bird.co/gbfs/v2/public/winterthur/gbfs.json
    bird-zurich:
      url: https://mds.bird.co/gbfs/v2/public/zurich/gbfs.json
    bolt_basel:
      url: https://api.mobidata-bw.de/sharing/gbfs/bolt_basel/gbfs
    bolt_zurich:
      url: https://api.mobidata-bw.de/sharing/gbfs/bolt_zurich/gbfs
    carvelo2go_ch:
      url: https://api.mobidata-bw.de/sharing/gbfs/carvelo2go_ch/gbfs
    donkey_ge:
      url: https://stables.donkey.bike/api/public/gbfs/2/donkey_ge/gbfs
    donkey_kreuzlingen:
      url: https://stables.donkey.bike/api/public/gbfs/2/donkey_kreuzlingen/gbfs.json
    donkey_le_locle:
      url: https://stables.donkey.bike/api/public/gbfs/2/donkey_le_locle/gbfs
    donkey_neuchatel:
      url: https://stables.donkey.bike/api/public/gbfs/2/donkey_neuchatel/gbfs.json
    donkey_sion:
      url: https://stables.donkey.bike/api/public/gbfs/2/donkey_sion/gbfs.json
    donkey_thun:
      url: https://stables.donkey.bike/api/public/gbfs/2/donkey_thun/gbfs.json
    donkey_yverdon-les-bains:
      url: https://stables.donkey.bike/api/public/gbfs/2/donkey_yverdon-les-bains/gbfs.json
    edrivecarsharing_ch:
      url: https://api.mobidata-bw.de/sharing/gbfs/edrivecarsharing_ch/gbfs
    lime_basel:
      url: https://api.mobidata-bw.de/sharing/gbfs/lime_basel/gbfs
    lime_opfikon:
      url: https://data.lime.bike/api/partners/v2/gbfs/opfikon/gbfs.json
    lime_uster:
      url: https://api.mobidata-bw.de/sharing/gbfs/lime_uster/gbfs
    lime_zug:
      url: https://data.lime.bike/api/partners/v2/gbfs/zug/gbfs.json
    lime_zurich:
      url: https://api.mobidata-bw.de/sharing/gbfs/lime_zurich/gbfs
    nextbike_ch:
      url: https://gbfs.nextbike.net/maps/gbfs/v2/nextbike_ch/gbfs.json
    pickebike_aubonne:
      url: https://api.mobidata-bw.de/sharing/gbfs/pickebike_aubonne/gbfs
    pickebike_basel:
      url: https://api.mobidata-bw.de/sharing/gbfs/pickebike_basel/gbfs
    pickebike_fribourg:
      url: https://api.mobidata-bw.de/sharing/gbfs/pickebike_fribourg/gbfs
    publibike:
      url: https://api.publibike.ch/v1/gbfs/v2/gbfs.json
    share_birrer_ch:
      url: https://www.share-birrer.ch/gbfs/gbfs.json
    sharedmobility.ch:
      url: https://www.sharedmobility.ch/gbfs.json
    tier_basel:
      url: https://api.mobidata-bw.de/sharing/gbfs/tier_basel/gbfs
    tier_bern:
      url: https://api.mobidata-bw.de/sharing/gbfs/tier_bern/gbfs
    tier_stgallen:
      url: https://api.mobidata-bw.de/sharing/gbfs/tier_stgallen/gbfs
    tier_winterthur:
      url: https://api.mobidata-bw.de/sharing/gbfs/tier_winterthur/gbfs
    tier_zurich:
      url: https://api.mobidata-bw.de/sharing/gbfs/tier_zurich/gbfs
    velospot_ch:
      url: https://api.mobidata-bw.de/sharing/gbfs/velospot_ch/gbfs
    voi_ch:
      url: https://api.mobidata-bw.de/sharing/gbfs/voi_ch/gbfs
    zem_ch:
      url: https://api.mobidata-bw.de/sharing/gbfs/zem_ch/gbfs
  update_interval: 60
  http_timeout: 10
  cache_size: 50

[felixguendling]

Suggested change

	std::find(lru_order_.begin(), lru_order_.end(), key);
	std::find(lru_order_.begin(), lru_order_.end(), key);

utl::find

But in general looks expensive in case the cache becomes large?

[pablohoch]

lru_order_ only stores numbers and I expected the cache size to be <= 1000. Finding a number and moving memory around (in move_to_front) in <= 8 KB should be fast enough I hope. We could probably limit gbfs_provider_idx_t (the entries in this vector) to uint16_t instead of size_t to make it more compact.

What alternative did you have in mind? For example, I don't think an additional map key -> lru index is very helpful because the indices of all the entries change every time move_to_front is called (unless only the first entry is accessed), so keeping that map updated would be expensive.

[felixguendling]

how is this related to boost::interval_set or boost::interval_map?

[pablohoch]

Unless I'm missing something, I don't think they are related. The GBFS code doesn't use intervals. The vehicle types are just mapped to numbers 0..n (index into the list of vehicle types) for the partitioning.

partition.h implements Partition refinement, which I don't think Boost.Icl does (and I don't see how it would relate to intervals). The idea is that initially, all vehicle types are equal (= a single set containing all vehicle types), then refine is called for each combination of vehicle types that share some property but are different from all other vehicle types (this includes the form factor (bike, scooter etc.), vehicle types that can be returned at a station (vs. all the other ones that can't be returned there), and geofencing zone rules that are vehicle type specific). If not all vehicle types in a set share that property, the set is split into two sets.