POMDP States with different state variables types, definition and management with basicPOMCP online solver

[Piaktipik]

Hi, I wonder how to implement and deal with a POMDP State with different state variables types using the QuickPOMDP approach?

Currently, I am using the following states, actions, and observations:

states = [s_LANDED, s_HOVERING, s_H_EXPLORATION, s_V_INSPECTION, s_LANDING],
actions = [a_STAY_ON_GROUND, a_HOVER, a_H_EXPLORATION,a_V_INSPECTION,a_LAND],
observations = [s_LANDED, s_HOVERING, s_H_EXPLORATION, s_V_INSPECTION, s_LANDING],
initialstate = Deterministic(s_LANDED),
discount = discountFactor,

With the variables defined as int-type objects:

#  --------- States
s_LANDED = 0 
s_HOVERING = 1
s_H_EXPLORATION = 2 
s_V_INSPECTION = 3 
s_LANDING = 4
# --------- Actions
a_STAY_ON_GROUND = 0
a_HOVER = 1
a_H_EXPLORATION = 2
a_V_INSPECTION = 3
a_LAND = 4

I am trying to include multiple float-type state variables (BatteryLevel, SurfaceType, ExplorationPercentage) and observations, something like:

states = [s_LANDED, s_HOVERING, s_H_EXPLORATION, s_V_INSPECTION, s_LANDING, BatteryLevel, SurfaceType, ExplorationPercentage],
actions = [a_STAY_ON_GROUND, a_HOVER, a_H_EXPLORATION,a_V_INSPECTION,a_LAND],
observations = [s_LANDED, s_HOVERING, s_H_EXPLORATION, s_V_INSPECTION, s_LANDING, BatteryLevel, SurfaceType, ExplorationPercentage],
initialstate = Deterministic(s_LANDED, BatteryLevel=1, SurfaceType=st_SAFE, ExplorationPercentage=0),
discount = discountFactor,

1. Is it possible to implement this multi-Type State variable using the QuickPOMCP approach?
2. Is there a recommended way to implement these multi-type states? Perhaps using tuples, sets, dictionaries, Multidimensional Arrays, or custom types?
3. How to manage these states and observations across the function calls and beliefs updating using the basicPOMCP online solver?

Thanks in advance for your help!

[zsunberg]

One important conceptual thing to understand is that the word "states" is used in two ways in the literature and discussions. Sometimes people use "states" to refer to the state dimensions, i.e. what are the variables needed to describe the state. For example someone might say "the states of the pendulum system are theta and theta dot". The second meaning is a set of all of the possible values that the state can take, i.e. the state can be s_LANDED or s_HOVERING. In POMDPs.jl, we use the second meaning when a function or argument is called states.

In your example above, you seem to be mixing the two meanings, i.e. s_LANDED and s_HOVERING are values of one of the state variables, whereas BatteryLevel is a state dimension that may have values between 0.0 and 1.0.

Is it possible to implement this multi-Type State variable using the QuickPOMCP approach?
Is there a recommended way to implement these multi-type states? Perhaps using tuples, sets, dictionaries, Multidimensional Arrays, or custom types?

Yes, you can now use the product function from CommonRLSpaces.jl to accomplish what you want, i.e.

using CommonRLSpaces

statuses = [1, 2, 3, 4, 5]
battery_levels = 0..1 # this represents the continuous interval from 0 to 1

state_space = product(statuses, battery_levels)

m = QuickPOMDP(
    states = state_space,
    # Other arguments
)

To see what an element of your new space looks like, you can use rand(state_space), which will return something like (5, 0.2488), which indicates that the system is in status 5 with 24.88% battery.

Now, whenever you represent states, they will be tuples with an integer and float in them, so, for example, you might have:

initialstate = Deterministic((1, 1.0))

transition = function(s, a)
    ImplicitDistribution() do rng
        if a == 2 # say action 2 is hovering
            return (2, s[2] - 0.05 - 0.01*rand(rng)) # s[2] is the current battery. For hovering, one step costs 5% battery plus a small random amount
        else
            # other cases
        end
    end
end

observation = function(a, sp)
    ImplicitDistribution() do rng
        return (sp[1], sp[2] + 0.01*randn(rng))
    end
end

Note however, that POMCP will not perform well in continuous observation spaces as noted here: https://arxiv.org/abs/1709.06196. ARDESPOT is better, but still not optimal. POMCPOW requires an explicit observation function, so you will have to define your own distribution since we haven't implemented a product for distributions yet.

[Piaktipik]

Thank you so much @zsunberg! It seems what I am trying to do. Also, thanks for the note about the performance of POMCP with continuous spaces; I will try using discrete values ~ 0.1 or 0.05 as an initial approach.