Merge pull request #2695 from shwethanidd/jira/3

Bug fixes and updates to vcfg rabbitmq federation and shovel commands

README.md

@@ -150,7 +150,7 @@ You can deactivate the environment at any time by running `deactivate`.
 ##### 5. Create RabbitMQ setup for VOLTTRON:
 
 ```sh
-vcfg --rabbitmq single [optional path to rabbitmq_config.yml]
+vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 ```
 
 Refer to [examples/configurations/rabbitmq/rabbitmq_config.yml](examples/configurations/rabbitmq/rabbitmq_config.yml)
@@ -180,9 +180,9 @@ be configured. The VOLTTRON instance name will be read from volttron_home/config
 if available, if not the user will be prompted for VOLTTRON instance name. To
 run the scripts without any prompts, save the VOLTTRON instance name in
 volttron_home/config file and pass the VOLTTRON home directory as a command line
-argument. For example: `vcfg --vhome /home/vdev/.new_vhome --rabbitmq single`
+argument. For example: `vcfg --vhome /home/vdev/.new_vhome rabbitmq single`
 
-The Following are the example inputs for `vcfg --rabbitmq single` command. Since no
+The Following are the example inputs for `vcfg rabbitmq single` command. Since no
 config file is passed the script prompts for necessary details.
 
 ```sh

docs/source/deploying-volttron/platform-configuration.rst

@@ -161,14 +161,31 @@ Optional Arguments
   - **--list-agents** - Display a list of configurable agents (Listener, Platform Driver, Platform Historian, VOLTTRON
     Central, VOLTTRON Central Platform)
   - **--agent AGENT [AGENT ...]** - Configure listed agents
-  - **--rabbitmq RABBITMQ [RABBITMQ ...]** - Configure rabbitmq for single instance, federation, or shovel either based
-    on configuration file in yml format or providing details when prompted.
+  - **--secure-agent-users** - Require that agents run as their own Unix users (this requires running
+    `scripts/secure_user_permissions.sh` as `sudo`)
+
+RabbitMQ Arguments
+------------------
+vcfg command to configure a single RabbitMQ instance of VOLTTRON.
 
         Usage:
 
         .. code-block:: bash
 
-            vcfg --rabbitmq single|federation|shovel [rabbitmq config file]``
+            vcfg rabbitmq single [--config Optional path to rabbitmq config file]``
 
-  - **--secure-agent-users** - Require that agents run as their own Unix users (this requires running
-    `scripts/secure_user_permissions.sh` as `sudo`)
+vcfg command to configure a federation instance of RabbitMQ VOLTTRON.
+
+        Usage:
+
+        .. code-block:: bash
+
+            vcfg rabbitmq federation [--config Optional path to rabbitmq federation config file] [--max-retries Optional maximum CSR retry attempt]``
+
+vcfg command to create shovel to send messages from one RabbitMQ instance of VOLTTRON to another.
+
+        Usage:
+
+        .. code-block:: bash
+
+            vcfg rabbitmq shovel [--config Optional path to shovel config file] [--max-retries Optional maximum CSR retry attempt]``

docs/source/introduction/platform-install.rst

@@ -293,7 +293,7 @@ Step 5 - Configure RabbitMQ setup for VOLTTRON
 
 .. code-block:: bash
 
-    vcfg --rabbitmq single [optional path to rabbitmq_config.yml]
+    vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 
 A sample configuration file can be found in the VOLTTRON repository in **examples/configurations/rabbitmq/rabbitmq_config.yml**.
 At a minimum you will need to provide the host name and a unique common-name (under certificate-data) in the configuration file.
@@ -332,7 +332,7 @@ exchange to capture unrouteable messages.
 
     The default behavior generates a certificate which is valid for a period of 1 year.
 
-The Following are the example inputs for `vcfg --rabbitmq single` command.  Since no config file is passed the script
+The Following are the example inputs for `vcfg rabbitmq single` command.  Since no config file is passed the script
 prompts for necessary details.
 
 .. code-block:: console

docs/source/platform-features/message-bus/rabbitmq/rabbitmq-ssl-auth.rst

@@ -90,7 +90,7 @@ The parameters of interest for SSL based configuration are
 
 We can then configure the VOLTTRON instance to use SSL based authentication with the below command:
 
-    vcfg --rabbitmq single <optional path to rabbitmq_config.yml>
+    vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 
 When one creates a single instance of RabbitMQ, the following is created / re-created in the VOLTTRON_HOME/certificates
 directory:

docs/source/platform-features/message-bus/rabbitmq/rabbitmq-volttron.rst

@@ -73,7 +73,7 @@ To configure the VOLTTRON instance to use RabbitMQ message bus, run the followin
 
 .. code-block:: bash
 
-    vcfg --rabbitmq single [optional path to rabbitmq_config.yml]
+    vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 
 At the end of the setup process, a RabbitMQ broker is setup to use the configuration provided.  A new topic exchange for
 the VOLTTRON instance is created within the configured virtual host.

docs/source/volttron-topics/troubleshooting/troubleshooting-rmq.rst

@@ -74,7 +74,7 @@ There are few things that are essential for SSL certificates to work right.
 
 a. Please use a unique common-name for CA certificate for each VOLTTRON instance.  This is configured under
    `certificate-data` in the `rabbitmq_config.yml` or if no yml file is used while configuring a VOLTTRON single
-   instance (using ``vcfg --rabbitmq single``).  Certificate generated for agent will automatically get agent's VIP
+   instance (using ``vcfg rabbitmq single``).  Certificate generated for agent will automatically get agent's VIP
    identity as the certificate's common-name
 
 b. The host name in the SSL certificate should match hostname used to access the server.  For example, if the fully

examples/configurations/rabbitmq/rabbitmq_federation_config.yml

@@ -2,18 +2,18 @@
 federation-upstream:
   volttron4: # hostname of upstream server
     port: '5671'
+    https-port: '8443'
     virtual-host: volttron4
     certificates:
-        csr: true
         private_key: "path to private cert" # For example, /home/volttron/vhome/test_fed/certificates/private/volttron1.federation.pem
         public_cert: "path to public cert" # For example, /home/volttron/vhome/test_fed/certificates/federation/volttron2.volttron1.federation.crt
         remote_ca: "path to CA cert" # For example, /home/volttron/vhome/test_fed/certificates/federation/volttron2_ca.crt
     federation-user: volttron4.federation #<local instance name>.federation
   volttron5: # hostname of upstream server
     port: '5671'
+    https-port: '8443'
     virtual-host: volttron5
     certificates:
-        csr: true
         private_key: "path to private cert"
         public_cert: "path to public cert"
         remote_ca: "path to CA cert"

examples/configurations/rabbitmq/rabbitmq_shovel_config.yml

@@ -1,22 +1,24 @@
 # Mandatory parameters for shovel setup
 shovel:
-  rabbit-2:
-    port: '5671'
-    virtual-host: volttron
-    certificates:
-        csr: true
-        private_cert: "path to private cert" # For example, /home/volttron/vhome/test_shovel/certificates/private/volttron1.shovelvolttron2.pem
-        public_cert: "path to public cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2.volttron1.shovelvolttron2.crt
-        remote_ca: "path to CA cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2_ca.crt
-    # Configuration to forward pubsub topics
-    pubsub:
-      # Identity of agent that is publishing the topic
-      platform.driver:
-        # Topic pattern to be forwarded
-        - devices
-    # Configuration to make remote RPC calls
-    rpc:
-      # Remote instance name
-      volttron2:
-        # List of pair of agent identities (local caller, remote callee)
-        - [scheduler, platform.actuator]
+    volttron2: # remote hostname
+        https-port: 8443
+        port: 5671
+        shovel-user: volttron1.shovelvolttron2 #<instance_name>.<unique name>
+        virtual-host: volttron
+        certificates:
+            private_cert: "path to private cert" # For example, /home/volttron/vhome/test_shovel/certificates/private/volttron1.shovelvolttron2.pem
+            public_cert: "path to public cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2.volttron1.shovelvolttron2.crt
+            remote_ca: "path to CA cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2_ca.crt
+        # Configuration to forward pubsub topics
+        pubsub:
+            # Identity of agent that is publishing the topic
+            platform.driver:
+                # Topic pattern to be forwarded
+                - devices
+
+        # Configuration to make remote RPC calls
+        rpc:
+          # Remote instance name
+          volttron2:
+            # List of pair of agent identities (local caller, remote callee)
+            - [scheduler, platform.actuator]

pytest.ini

@@ -62,4 +62,6 @@ markers =
         sqlitefuncts: level one integration tests for sqlitefuncts
         unit: Run all unit/level one integration tests
         influxdbutils: level one integration tests for influxdb
+        federation: Tests for rabbitmq federation communication
+        shovel: Tests for rabbitmq shovel communication
         contrib: tests for community-contributed agents

volttron/platform/control.py

@@ -289,6 +289,7 @@ def stop_platform(self):
 
     @RPC.export
     def list_agents(self):
+        _log.info("CONTROL RPC list_agents")
         tag = self._aip.agent_tag
         priority = self._aip.agent_priority
         return [{'name': name, 'uuid': uuid,
@@ -2215,6 +2216,64 @@ def list_shovel_parameters(opts):
         _stdout.write("Error in getting shovel parameters")
 
 
+def list_fed_links(opts):
+    links = None
+    try:
+        links = rmq_mgmt.get_federation_links()
+    except requests.exceptions.HTTPError as e:
+        _stdout.write("No Federation links Found \n")
+        return
+    except ConnectionError as e:
+        _stdout.write("Error making request to RabbitMQ Management interface.\n"
+                      "Check Connection Parameters: {} \n".format(e))
+        return
+    try:
+        if links:
+            name_width = max(5, max(len(lk['name']) for lk in links))
+            status_width = max(3, max(len(lk['status']) for lk in links))
+            fmt = '{:{}} {:{}}\n'
+            _stderr.write(
+                fmt.format('NAME', name_width, 'STATUS', status_width))
+            for link in links:
+                _stdout.write(fmt.format(link['name'], name_width,
+                                         link['status'], status_width))
+    except (AttributeError, KeyError) as ex:
+        _stdout.write("Error in federation links")
+
+
+def list_shovel_links(opts):
+    links = None
+    try:
+        links = rmq_mgmt.get_shovel_links()
+    except requests.exceptions.HTTPError as e:
+        _stdout.write("No Shovel links Found \n")
+        return
+    except ConnectionError as e:
+        _stdout.write("Error making request to RabbitMQ Management interface.\n"
+                      "Check Connection Parameters: {} \n".format(e))
+        return
+    try:
+        if links:
+            name_width = max(5, max(len(lk['name']) for lk in links))
+            status_width = max(3, max(len(lk['status']) for lk in links))
+            src_exchange_key_width = max(3, max(len(lk['src_exchange_key']) for lk in links))
+            src_uri_width = max(3, max(len(lk['src_uri']) for lk in links))
+            dest_uri_width = max(3, max(len(lk['dest_uri']) for lk in links))
+            fmt = '{:{}}  {:{}}  {:{}}  {:{}}  {:{}}\n'
+            _stderr.write(
+                fmt.format('NAME', name_width, 'STATUS', status_width, 'SRC_URI',
+                           src_uri_width, 'DEST_URI', dest_uri_width,
+                           'SRC_EXCHANGE_KEY', src_exchange_key_width))
+            for link in links:
+                _stdout.write(fmt.format(link['name'], name_width,
+                                         link['status'], status_width,
+                                         link['src_uri'], src_uri_width,
+                                         link['dest_uri'], dest_uri_width,
+                                         link['src_exchange_key'], src_exchange_key_width))
+    except (AttributeError, KeyError) as ex:
+        _stdout.write(f"Error in shovel links as {ex}")
+
+
 def list_bindings(opts):
     bindings = None
     try:
@@ -2317,7 +2376,8 @@ def remove_queues(opts):
 def remove_fed_parameters(opts):
     try:
         for param in opts.parameters:
-            rmq_mgmt.delete_multiplatform_parameter('federation-upstream', param)
+            delete_certs = _ask_yes_no(f'Do you wish to delete certificates as well for {param}?')
+            rmq_mgmt.delete_multiplatform_parameter('federation-upstream', param, delete_certs=delete_certs)
     except requests.exceptions.HTTPError as e:
         _stdout.write("No Federation Parameters Found {} \n".format(opts.parameters))
     except ConnectionError as e:
@@ -2328,7 +2388,8 @@ def remove_fed_parameters(opts):
 def remove_shovel_parameters(opts):
     try:
         for param in opts.parameters:
-            rmq_mgmt.delete_multiplatform_parameter('shovel', param)
+            delete_certs = _ask_yes_no('Do you wish to delete certificates as well?')
+            rmq_mgmt.delete_multiplatform_parameter('shovel', param, delete_certs=delete_certs)
     except requests.exceptions.HTTPError as e:
         _stdout.write("No Shovel Parameters Found {} \n".format(opts.parameters))
     except ConnectionError as e:
@@ -2957,6 +3018,14 @@ def add_parser(*args, **kwargs) -> argparse.ArgumentParser:
                                                   subparser=rabbitmq_subparsers)
         rabbitmq_list_fed_parameters.set_defaults(func=list_fed_parameters)
 
+        rabbitmq_list_fed_links = add_parser('list-federation-links', help='list all federation links',
+                                             subparser=rabbitmq_subparsers)
+        rabbitmq_list_fed_links.set_defaults(func=list_fed_links)
+
+        rabbitmq_list_shovel_links = add_parser('list-shovel-links', help='list all Shovel links',
+                                                subparser=rabbitmq_subparsers)
+        rabbitmq_list_shovel_links.set_defaults(func=list_shovel_links)
+
         rabbitmq_list_shovel_parameters = add_parser('list-shovel-parameters', help='list all shovel parameters',
                                                      subparser=rabbitmq_subparsers)
         rabbitmq_list_shovel_parameters.set_defaults(func=list_shovel_parameters)
@@ -2986,13 +3055,13 @@ def add_parser(*args, **kwargs) -> argparse.ArgumentParser:
         rabbitmq_remove_queues.add_argument('queues', nargs='+', help='Queue')
         rabbitmq_remove_queues.set_defaults(func=remove_queues)
 
-        rabbitmq_remove_fed_parameters = add_parser('remove-federation-parameters',
+        rabbitmq_remove_fed_parameters = add_parser('remove-federation-links',
                                                     help='Remove federation parameter',
                                                     subparser=rabbitmq_subparsers)
         rabbitmq_remove_fed_parameters.add_argument('parameters', nargs='+', help='parameter name/s')
         rabbitmq_remove_fed_parameters.set_defaults(func=remove_fed_parameters)
 
-        rabbitmq_remove_shovel_parameters = add_parser('remove-shovel-parameters',
+        rabbitmq_remove_shovel_parameters = add_parser('remove-shovel-links',
                                                        help='Remove shovel parameter',
                                                        subparser=rabbitmq_subparsers)
         rabbitmq_remove_shovel_parameters.add_argument('parameters', nargs='+', help='parameter name/s')