Shareable RAW disk

[luboss79]

Please add the support for using the shared RAW disk between the VM instances. Just to create the virtual disk in UI, mark it as shared and then attach it to more than 1 VM instance. This the common requirements for installing the Oracle Database, where the OCFS2 or ACFS cluster filesystem is taking the responsibility for IO/locking and preserving the data corruption.
For example oVirt has that solution implemented,very good description of the feature is here https://www.ovirt.org/develop/release-management/features/storage/sharedrawdisk.html .
I know there will be added support for the SharedFilesystem in v 4.20 but this is not the case I need to cover. We discussed it little in CCC in Madrid few days ago.
Thank you
Lubomir

[wido]

Think this is a valid and easy use-case. It could also be used to perform application failover between VMs. I know some people do this to quickly start their SQL database on another server without needing to wait for the VM to reboot on a different location.

@luboss79 Do you see the oppertunity to work on this and submit a PR?

[luboss79]

@wido Thanks for the positive reaction.. I'm not a developer, but I can participate in testing :)

[weizhouapache]

good idea

[krokodilerian]

Hi,

To weigh in here, this is a very dangerous feature and very hard to implement correctly.

The use cases I've seen require specific SCSI support from the underlying storage for either "persistent reservations" (Microsoft's cluster shared volumes) or atomics (VMWare VMFS), or other guarantees.

Take, for example, the following use case: two VMs (v1 and v2) on two hosts (h1 and h2) have the same disk and use some kind of communication (which is not via the storage) on who does what. Then, h2 loses connectivity to the network and storage but has some operations in flight that are in the queue.

There's no mechanism that v2 can use to "cancel" these operations. There's also no way for v1 to know if there are such operations or where they write. So if v1 takes over and the connectivity of h2/v2 is restored, this pretty much guarantees data corruption because the queued old operations will be executed on top of the things v1 has already done.

This is solvable with storage or host fencing but would have to be exposed to VMs as an API, which has its own problems.

Looking at the oVirt documentation, there's the cop-out "it's the responsibility of the VM to take care of locking," but to reiterate, to be able to do this in a useful way, the guests will need access to low-level SCSI storage functions (or their emulation, like via qemu-pr-helper) or fencing.

[wido]

Hi,

To weigh in here, this is a very dangerous feature and very hard to implement correctly.

The use cases I've seen require specific SCSI support from the underlying storage for either "persistent reservations" (Microsoft's cluster shared volumes) or atomics (VMWare VMFS), or other guarantees.

Take, for example, the following use case: two VMs (v1 and v2) on two hosts (h1 and h2) have the same disk and use some kind of communication (which is not via the storage) on who does what. Then, h2 loses connectivity to the network and storage but has some operations in flight that are in the queue.

There's no mechanism that v2 can use to "cancel" these operations. There's also no way for v1 to know if there are such operations or where they write. So if v1 takes over and the connectivity of h2/v2 is restored, this pretty much guarantees data corruption because the queued old operations will be executed on top of the things v1 has already done.

This is solvable with storage or host fencing but would have to be exposed to VMs as an API, which has its own problems.

Looking at the oVirt documentation, there's the cop-out "it's the responsibility of the VM to take care of locking," but to reiterate, to be able to do this in a useful way, the guests will need access to low-level SCSI storage functions (or their emulation, like via qemu-pr-helper) or fencing.

I do not disagree that this is a dangerous feature, but having it can be beneficial. Implementing it from the CloudStack side isn't that difficult.

• Only shared storage would be supported: NFS, Ceph, StorPool
• Caching needs to be forced to 'none' on any shared disk
• Mark the disk as shareable in libvirt: https://libvirt.org/formatdomain.html#elementsDisks

If present, this indicates the device is expected to be shared between domains (assuming the hypervisor and OS support this), which means that caching should be deactivated for that device.

It's up to the user on how to use this, not CloudStack.

[bkrosnov]

Without something like SCSI reservations or storage fencing integrated the the in-VM cluster, data corruption will often occur when you disconnect/reconnect one of the hosts from the storage network. There is nothing the VM can do to prevent it because the storage stack in the host has operations in flight, which it holds and then submits when it can. The write operations in flight might be waiting not only in the block layers of the host, they might be stuck even in TCP buffers of the iSCSI session for example, just waiting for an opportunity be retransmitted, 10s of seconds later.

Disabling caching doesn't fix it. You can't have block storage without a queue (operations which have been requested by the VM, but are not complete yet). And as @krokodilerian said, there are a bunch of ways in which you can make useful clusters in VMs, just giving them a shared disk which can delay some operations by 10s of seconds is not a way to enable clusters in VMs. Even active-standby clusters are risky in such environment.

I had a quick look at OpenStack and oVirt. Both completely glance over this issue and leave it to the user to figure it out. This is bad.
If we do the same in CloudStack, users wouldn't understand the limitations. They'd shoot themselves in the foot. They'd blame CloudStack or the storage.

On the other hand VMWare supports SCSI persistent reservations for shared virtual disks, which is good.
To get a similar good and usable shared disks in CloudStack/KVM, we'd need much more than just permitting two simultaneous attachments. For example, we'd need SCSI PR emulation for the raw file on NFS scenario. For iSCSI and FC we'd need virtio-scsi and scsi-block (passthrough) devices, maybe qemu-pr-helper plus additional checks that the host configuration, multipath are correctly configured, etc.

If implemented we need a way for a storage plugin to signal if it supports it or not. The StorPool driver will opt out of the shared disk feature. Currently we have a different way of supporting the cluster-in-VMs use-case, which is not pretty, but a workaround exists for our users.

[wido]

Without something like SCSI reservations or storage fencing integrated the the in-VM cluster, data corruption will often occur when you disconnect/reconnect one of the hosts from the storage network. There is nothing the VM can do to prevent it because the storage stack in the host has operations in flight, which it holds and then submits when it can. The write operations in flight might be waiting not only in the block layers of the host, they might be stuck even in TCP buffers of the iSCSI session for example, just waiting for an opportunity be retransmitted, 10s of seconds later.

Disabling caching doesn't fix it. You can't have block storage without a queue (operations which have been requested by the VM, but are not complete yet). And as @krokodilerian said, there are a bunch of ways in which you can make useful clusters in VMs, just giving them a shared disk which can delay some operations by 10s of seconds is not a way to enable clusters in VMs. Even active-standby clusters are risky in such environment.

I had a quick look at OpenStack and oVirt. Both completely glance over this issue and leave it to the user to figure it out. This is bad. If we do the same in CloudStack, users wouldn't understand the limitations. They'd shoot themselves in the foot. They'd blame CloudStack or the storage.

On the other hand VMWare supports SCSI persistent reservations for shared virtual disks, which is good. To get a similar good and usable shared disks in CloudStack/KVM, we'd need much more than just permitting two simultaneous attachments. For example, we'd need SCSI PR emulation for the raw file on NFS scenario. For iSCSI and FC we'd need virtio-scsi and scsi-block (passthrough) devices, maybe qemu-pr-helper plus additional checks that the host configuration, multipath are correctly configured, etc.

If implemented we need a way for a storage plugin to signal if it supports it or not. The StorPool driver will opt out of the shared disk feature. Currently we have a different way of supporting the cluster-in-VMs use-case, which is not pretty, but a workaround exists for our users.

Great feedback!

virtio-scsi seems to support reservations, so we might only support it if virtio-scsi is being used. But then it depends if RBD or StorPool could support these reservations.

It does seem like RBD support reservations through RBD locking. Would it be a possibility that this is supported?

[btzq]

And Linstor too! @rp-

[luboss79]

@wido @bkrosnov we are using Oracle OLVM (Oracle customized oVirt) for the database cluster, I can confirm that virtio-scsi interface used for the shared disks. The shared disks are stored and managed by the oVirt Storage Domain which is physically a Fibre Channel Block Storage (attached to all the KVM hypervisors - 2 hosts)

Would be great to have such an option in Cloudstack!