It is advised not to remove devices that were present when the server started; only because the server for some operations will assume that those devices are accessible. It is absolutely not dangerous to remove the devices, it will only waste the server's time and processing power somewhat.