Merge remote-tracking branch 'upstream/develop' into update/gh-actions

.gitattributes

@@ -0,0 +1,2 @@
+# Handles CLRF/RF EOL issue
+* text=auto

README.md

@@ -151,7 +151,7 @@ You can deactivate the environment at any time by running `deactivate`.
 ##### 5. Create RabbitMQ setup for VOLTTRON:
 
 ```sh
-vcfg --rabbitmq single [optional path to rabbitmq_config.yml]
+vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 ```
 
 Refer to [examples/configurations/rabbitmq/rabbitmq_config.yml](examples/configurations/rabbitmq/rabbitmq_config.yml)
@@ -181,9 +181,9 @@ be configured. The VOLTTRON instance name will be read from volttron_home/config
 if available, if not the user will be prompted for VOLTTRON instance name. To
 run the scripts without any prompts, save the VOLTTRON instance name in
 volttron_home/config file and pass the VOLTTRON home directory as a command line
-argument. For example: `vcfg --vhome /home/vdev/.new_vhome --rabbitmq single`
+argument. For example: `vcfg --vhome /home/vdev/.new_vhome rabbitmq single`
 
-The Following are the example inputs for `vcfg --rabbitmq single` command. Since no
+The Following are the example inputs for `vcfg rabbitmq single` command. Since no
 config file is passed the script prompts for necessary details.
 
 ```sh

docs/source/deploying-volttron/platform-configuration.rst

@@ -161,14 +161,31 @@ Optional Arguments
   - **--list-agents** - Display a list of configurable agents (Listener, Platform Driver, Platform Historian, VOLTTRON
     Central, VOLTTRON Central Platform)
   - **--agent AGENT [AGENT ...]** - Configure listed agents
-  - **--rabbitmq RABBITMQ [RABBITMQ ...]** - Configure rabbitmq for single instance, federation, or shovel either based
-    on configuration file in yml format or providing details when prompted.
+  - **--secure-agent-users** - Require that agents run as their own Unix users (this requires running
+    `scripts/secure_user_permissions.sh` as `sudo`)
+
+RabbitMQ Arguments
+------------------
+vcfg command to configure a single RabbitMQ instance of VOLTTRON.
 
         Usage:
 
         .. code-block:: bash
 
-            vcfg --rabbitmq single|federation|shovel [rabbitmq config file]``
+            vcfg rabbitmq single [--config Optional path to rabbitmq config file]``
 
-  - **--secure-agent-users** - Require that agents run as their own Unix users (this requires running
-    `scripts/secure_user_permissions.sh` as `sudo`)
+vcfg command to configure a federation instance of RabbitMQ VOLTTRON.
+
+        Usage:
+
+        .. code-block:: bash
+
+            vcfg rabbitmq federation [--config Optional path to rabbitmq federation config file] [--max-retries Optional maximum CSR retry attempt]``
+
+vcfg command to create shovel to send messages from one RabbitMQ instance of VOLTTRON to another.
+
+        Usage:
+
+        .. code-block:: bash
+
+            vcfg rabbitmq shovel [--config Optional path to shovel config file] [--max-retries Optional maximum CSR retry attempt]``

docs/source/developing-volttron/community.rst

@@ -44,5 +44,10 @@ available through a Zoom meeting. To be invited to the link meeting, contact the
 
 Meetings are recorded and can be reviewed `here <https://volttron.org/office-hours>`__.
 
+Publications
+============
+
+VOLTTRON white papers and technical reports can be found at https://volttron.org/publications
+
 
 .. |VOLTTRON| unicode:: VOLTTRON U+2122

docs/source/introduction/platform-install.rst

@@ -293,7 +293,7 @@ Step 5 - Configure RabbitMQ setup for VOLTTRON
 
 .. code-block:: bash
 
-    vcfg --rabbitmq single [optional path to rabbitmq_config.yml]
+    vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 
 A sample configuration file can be found in the VOLTTRON repository in **examples/configurations/rabbitmq/rabbitmq_config.yml**.
 At a minimum you will need to provide the host name and a unique common-name (under certificate-data) in the configuration file.
@@ -332,7 +332,7 @@ exchange to capture unrouteable messages.
 
     The default behavior generates a certificate which is valid for a period of 1 year.
 
-The Following are the example inputs for `vcfg --rabbitmq single` command.  Since no config file is passed the script
+The Following are the example inputs for `vcfg rabbitmq single` command.  Since no config file is passed the script
 prompts for necessary details.
 
 .. code-block:: console

docs/source/platform-features/message-bus/rabbitmq/rabbitmq-ssl-auth.rst

@@ -90,7 +90,7 @@ The parameters of interest for SSL based configuration are
 
 We can then configure the VOLTTRON instance to use SSL based authentication with the below command:
 
-    vcfg --rabbitmq single <optional path to rabbitmq_config.yml>
+    vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 
 When one creates a single instance of RabbitMQ, the following is created / re-created in the VOLTTRON_HOME/certificates
 directory:

docs/source/platform-features/message-bus/rabbitmq/rabbitmq-volttron.rst

@@ -73,7 +73,7 @@ To configure the VOLTTRON instance to use RabbitMQ message bus, run the followin
 
 .. code-block:: bash
 
-    vcfg --rabbitmq single [optional path to rabbitmq_config.yml]
+    vcfg rabbitmq single [--config optional path to rabbitmq_config.yml]
 
 At the end of the setup process, a RabbitMQ broker is setup to use the configuration provided.  A new topic exchange for
 the VOLTTRON instance is created within the configured virtual host.

docs/source/volttron-topics/troubleshooting/troubleshooting-rmq.rst

@@ -74,7 +74,7 @@ There are few things that are essential for SSL certificates to work right.
 
 a. Please use a unique common-name for CA certificate for each VOLTTRON instance.  This is configured under
    `certificate-data` in the `rabbitmq_config.yml` or if no yml file is used while configuring a VOLTTRON single
-   instance (using ``vcfg --rabbitmq single``).  Certificate generated for agent will automatically get agent's VIP
+   instance (using ``vcfg rabbitmq single``).  Certificate generated for agent will automatically get agent's VIP
    identity as the certificate's common-name
 
 b. The host name in the SSL certificate should match hostname used to access the server.  For example, if the fully

examples/configurations/rabbitmq/rabbitmq_federation_config.yml

@@ -2,18 +2,18 @@
 federation-upstream:
   volttron4: # hostname of upstream server
     port: '5671'
+    https-port: '8443'
     virtual-host: volttron4
     certificates:
-        csr: true
         private_key: "path to private cert" # For example, /home/volttron/vhome/test_fed/certificates/private/volttron1.federation.pem
         public_cert: "path to public cert" # For example, /home/volttron/vhome/test_fed/certificates/federation/volttron2.volttron1.federation.crt
         remote_ca: "path to CA cert" # For example, /home/volttron/vhome/test_fed/certificates/federation/volttron2_ca.crt
     federation-user: volttron4.federation #<local instance name>.federation
   volttron5: # hostname of upstream server
     port: '5671'
+    https-port: '8443'
     virtual-host: volttron5
     certificates:
-        csr: true
         private_key: "path to private cert"
         public_cert: "path to public cert"
         remote_ca: "path to CA cert"

examples/configurations/rabbitmq/rabbitmq_shovel_config.yml

@@ -1,22 +1,24 @@
 # Mandatory parameters for shovel setup
 shovel:
-  rabbit-2:
-    port: '5671'
-    virtual-host: volttron
-    certificates:
-        csr: true
-        private_cert: "path to private cert" # For example, /home/volttron/vhome/test_shovel/certificates/private/volttron1.shovelvolttron2.pem
-        public_cert: "path to public cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2.volttron1.shovelvolttron2.crt
-        remote_ca: "path to CA cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2_ca.crt
-    # Configuration to forward pubsub topics
-    pubsub:
-      # Identity of agent that is publishing the topic
-      platform.driver:
-        # Topic pattern to be forwarded
-        - devices
-    # Configuration to make remote RPC calls
-    rpc:
-      # Remote instance name
-      volttron2:
-        # List of pair of agent identities (local caller, remote callee)
-        - [scheduler, platform.actuator]
+    volttron2: # remote hostname
+        https-port: 8443
+        port: 5671
+        shovel-user: volttron1.shovelvolttron2 #<instance_name>.<unique name>
+        virtual-host: volttron
+        certificates:
+            private_cert: "path to private cert" # For example, /home/volttron/vhome/test_shovel/certificates/private/volttron1.shovelvolttron2.pem
+            public_cert: "path to public cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2.volttron1.shovelvolttron2.crt
+            remote_ca: "path to CA cert" # For example, /home/volttron/vhome/test_shovel/certificates/shovels/volttron2_ca.crt
+        # Configuration to forward pubsub topics
+        pubsub:
+            # Identity of agent that is publishing the topic
+            platform.driver:
+                # Topic pattern to be forwarded
+                - devices
+
+        # Configuration to make remote RPC calls
+        rpc:
+          # Remote instance name
+          volttron2:
+            # List of pair of agent identities (local caller, remote callee)
+            - [scheduler, platform.actuator]

pytest.ini

@@ -62,4 +62,6 @@ markers =
         sqlitefuncts: level one integration tests for sqlitefuncts
         unit: Run all unit/level one integration tests
         influxdbutils: level one integration tests for influxdb
+        federation: Tests for rabbitmq federation communication
+        shovel: Tests for rabbitmq shovel communication
         contrib: tests for community-contributed agents

services/ops/MessageDebuggerAgent/tests/test_message_debugging.py → services/contrib/MessageDebuggerAgent/tests/test_message_debugging.py

@@ -82,7 +82,7 @@ class TestMessageDebugger:
         Regression tests for the MessageDebuggerAgent.
     """
 
-    # @pytest.mark.skip(reason="Dependency on SQLAlchemy library")
+    @pytest.mark.skip(reason="Community contributed agent")
     def test_rpc_calls(self, agent):
         """Test the full range of RPC calls to the MessageDebuggerAgent, except those related to streaming."""
 

services/ops/FileWatchPublisher/Tests/test_file_watcher.py

@@ -46,26 +46,12 @@
 from volttron.platform import get_ops, get_home
 from volttron.platform.vip.agent import Agent
 
-test_path = os.path.join(get_home(), "test.txt")
-
-test_config = {
-    "files": [
-        {
-            "file": test_path,
-            "topic": "platform/test_topic"
-        }
-    ]
-}
-
 
 @pytest.fixture(scope="module")
 def publish_agent(request, volttron_instance):
     # 1: Start a fake agent to publish to message bus
     agent = volttron_instance.build_agent(identity='test-agent')
 
-    with open(test_path, "w") as textfile:
-        textfile.write("test_data")
-
     agent.callback = MagicMock(name="callback")
     agent.callback.reset_mock()
 
@@ -75,7 +61,6 @@ def stop_agent():
         print("In teardown method of publish_agent")
         if isinstance(agent, Agent):
             agent.core.stop()
-        os.remove(test_path)
 
     request.addfinalizer(stop_agent)
     return agent
@@ -98,18 +83,33 @@ def test_default_config(volttron_instance, publish_agent):
     volttron_instance.remove_agent(watcher_uuid)
 
 
-# def test_file_watcher(volttron_instance, publish_agent):
-#     watcher_uuid = volttron_instance.install_agent(
-#         agent_dir=get_ops("FileWatchPublisher"),
-#         config_file=test_config,
-#         start=True,
-#         vip_identity="health_test")
-#
-#     with open(test_path, "w+") as textfile:
-#         textfile.write("more test_data")
-#
-#     gevent.sleep(1)
-#
-#     assert publish_agent.callback.call_count == 1
-#     print(publish_agent.callback.call_args)
-#     volttron_instance.remove_agent(watcher_uuid)
+def test_file_watcher(volttron_instance, publish_agent):
+    test_path = os.path.join(get_home(), "test.txt")
+
+    with open(test_path, "w") as textfile:
+        textfile.write("test_data")
+
+    test_config = {
+        "files": [
+            {
+                "file": test_path,
+                "topic": "platform/test_topic"
+            }
+        ]
+    }
+
+    watcher_uuid = volttron_instance.install_agent(
+        agent_dir=get_ops("FileWatchPublisher"),
+        config_file=test_config,
+        start=True,
+        vip_identity="health_test")
+
+    with open(test_path, "w+") as textfile:
+        textfile.write("more test_data")
+
+    gevent.sleep(1)
+
+    assert publish_agent.callback.call_count == 1
+    print(publish_agent.callback.call_args)
+    volttron_instance.remove_agent(watcher_uuid)
+    os.remove(test_path)

services/ops/LogStatisticsAgent/Tests/test_log_statistics.py

@@ -45,25 +45,29 @@
 from volttron.platform.vip.agent import Agent
 from volttron.platform import get_ops, get_home
 
-# TODO fix finding test logs
 test_config = {
-    "file_path": os.path.join(get_home(), "volttron.log"),
     "analysis_interval_sec": 2,
     "publish_topic": "platform/log_statistics",
     "historian_topic": "analysis/log_statistics"
 }
 
-
 @pytest.fixture(scope="module")
 def publish_agent(request, volttron_instance):
+    test_config = {
+        "file_path": os.path.join(volttron_instance.volttron_home, "volttron.log"),
+        "analysis_interval_sec": 2,
+        "publish_topic": "platform/log_statistics",
+        "historian_topic": "analysis/log_statistics"
+    }
     # 1: Start a fake agent to publish to message bus
     agent = volttron_instance.build_agent()
 
-    publish_agent.callback = MagicMock(name="callback")
-    publish_agent.callback.reset_mock()
+    agent.callback = MagicMock(name="callback")
+    agent.callback.reset_mock()
 
-    agent.vip.pubsub.subscribe(peer='pubsub', prefix=test_config.get("publish_topic"),
-                               callback=publish_agent.callback).get()
+    agent.vip.pubsub.subscribe(peer='pubsub',
+                               prefix=test_config.get("publish_topic"),
+                               callback=agent.callback).get()
 
     def stop_agent():
         print("In teardown method of publish_agent")
@@ -91,20 +95,24 @@ def test_default_config(volttron_instance, publish_agent):
     volttron_instance.remove_agent(stats_uuid)
 
 
-# def test_log_stats(volttron_instance, publish_agent):
-#     stats_uuid = volttron_instance.install_agent(
-#         agent_dir=get_ops("LogStatisticsAgent"),
-#         config_file=test_config,
-#         start=True,
-#         vip_identity="health_test")
-#
-#     gevent.sleep(1)
-#
-#     # building another agent should populate the logs
-#     volttron_instance.build_agent(identity="log_populate")
-#
-#     gevent.sleep(2)
-#
-#     # TODO do mock asserts
-#
-#     volttron_instance.remove_agent(stats_uuid)
+def test_log_stats(volttron_instance, publish_agent):
+    test_config["file_path"] = volttron_instance.log_path
+    print(f"File path: {test_config['file_path']}")
+
+    stats_uuid = volttron_instance.install_agent(
+        agent_dir=get_ops("LogStatisticsAgent"),
+        config_file=test_config,
+        start=True,
+        vip_identity="health_test")
+
+    import gevent
+    gevent.sleep(1)
+
+    # building another agent should populate the logs
+    volttron_instance.build_agent(identity="log_populate")
+
+    gevent.sleep(1)
+
+    assert publish_agent.callback.call_count >= 1
+
+    volttron_instance.remove_agent(stats_uuid)

services/ops/SysMonAgent/tests/test_sysmonagent.py

@@ -102,8 +102,8 @@ def add_topic(peer, sender, bus, topic, headers, messages):
 
     agent.vip.pubsub.subscribe('pubsub', base_topic, callback=add_topic)
 
-    max_wait = 1 + max(value for key, value in _test_config.items() if key.endswith('_interval'))
-
+    max_wait = 1 + max(value for key, value in _test_config.items() if key.endswith('_interval')) + 8
+    print(f"Max wait: {max_wait}, topics: {topics}, seen_topics: {seen_topics}")
     assert poll_gevent_sleep(max_wait, lambda: set(topics) <= seen_topics)
 
 
@@ -124,6 +124,7 @@ def test_reconfigure_then_listen(sysmon_tester_agent):
     listen(sysmon_tester_agent, new_config)
 
 
+@pytest.mark.dev
 def test_default_config(sysmon_tester_agent):
     """
     Test that the topic can be reconfigured

services/ops/TopicWatcher/tests/test_remote_topic_watcher.py

@@ -182,4 +182,5 @@ def onmessage(peer, sender, bus, topic, headers, message):
            u"Topic(s) not published within time limit: [('fakedevice2/all', 'point'), " \
            u"'fakedevice2/all', 'fakedevice']" in \
            alert_messages
+
     alert_messages.clear()