diff --git a/docs/source/agent-framework/agents-overview.rst b/docs/source/agent-framework/agents-overview.rst new file mode 100644 index 0000000000..b34dfdf171 --- /dev/null +++ b/docs/source/agent-framework/agents-overview.rst @@ -0,0 +1,43 @@ +.. _Agent-Framework: + +=============== +Agents Overview +=============== + +Agents in VOLTTRON can be loosely defined as software modules communicating on the platform which perform some function +on behalf of the user. Agents may perform a huge variety of tasks, but common use cases involve data collection, +control of ICS and IOT devices, and various platform management tasks. Agents implemented using the VOLTTRON agent +framework inherit a number of capabilities, including message bus connectivity and agent lifecycle. + +Agents deployed on VOLTTRON can perform one or more roles which can be broadly classified into the following groups: + +- Platform Agents: Agents which are part of the platform and provide a service to other agents. Examples are the + Actuator and Master Driver agents which serve as interfaces between control agents and drivers. +- Control Agents: These agents implement algorithms to control the devices of interest and interact with other + resources to achieve some goal. +- Service Agents: These agents perform various data collection or platform management services. Agents in this + category include weather service agents which collect weather data from remote sources or operations agents which + help users maintain situational awareness of their deployment. +- Cloud Agents: These agents represent a remote application which needs access to the messages and data on the + platform. This agent would subscribe to topics of interest to the remote application and would also allow it publish + data to the platform. + +The platform includes some valuable services which can be leveraged by agents: + +- Message Bus: All agents and services publish and subscribe to topics on the message bus. This provides a single + interface that abstracts the details of devices and agents from each other. Components in the platform basically + produce and consume events. +- Configuration Store: Using the configuration store, agent operations can be altered ad-hoc without significant + disruption or downtime. +- Historian Framework: Historian agents automatically collect data from a subset of topics on the message bus and store + them in a data store of choice. Currently SQL, MongoDB, CrateDB and other historians exist, and more can be + developed to fit the needs of a deployment by inheriting from the base historian. The base historian has been + developed to be fast and reliable, and to handle many common pitfalls of data collection over a network. +- Weather Information: These agents periodically retrieve data from the a remote weather API then format the + response and publish it to the platform message bus on a weather topic. +- Device interfaces: Drivers publish device data onto the message bus and send control signals issued from control + agents to the corresponding device. Drivers are capable of handling the locking of devices to prevent multiple + conflicting directives. +- Application Scheduling: This service allows the scheduling of agents’ access to devices in order to prevent conflicts. +- Logging service: Agents can publish arbitrary strings to a logging topic and this service will push them to a + historian for later analysis. diff --git a/docs/source/core_services/control/Agent-Execution-Environment.rst b/docs/source/agent-framework/aip.rst similarity index 64% rename from docs/source/core_services/control/Agent-Execution-Environment.rst rename to docs/source/agent-framework/aip.rst index 5ea200dc04..28b6fe0ca2 100644 --- a/docs/source/core_services/control/Agent-Execution-Environment.rst +++ b/docs/source/agent-framework/aip.rst @@ -1,3 +1,10 @@ +.. _Agent-Instantiation-and-Packaging: + +======================================= +AIP - Agent Instantiation and Packaging +======================================= + + .. _Agent-Execution-Environment: Used Environmental Variables @@ -6,3 +13,6 @@ Used Environmental Variables - **AGENT_VIP_IDENTITY** - The router address an agent will attempt to connect to. - **AGENT_CONFIG** - The path to a configuration file to use during agent launch. - **VOLTTRON_HOME** - The home directory where the volttron instances is located. + +Documentation coming soon! + diff --git a/docs/source/agent-framework/core-service-agents/dnp3_and_mesa/dnp3-agent.rst b/docs/source/agent-framework/core-service-agents/dnp3_and_mesa/dnp3-agent.rst new file mode 100644 index 0000000000..1fcfdddd57 --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/dnp3_and_mesa/dnp3-agent.rst @@ -0,0 +1,197 @@ +.. _DNP3-Agent: + +========== +DNP3 Agent +========== + +`DNP3 `_ (Distributed Network Protocol) is a set of communications protocols that +are widely used by utilities such as electric power companies, primarily for +`SCADA `_ purposes. It was adopted in 2010 as +`IEEE Std 1815-2010 `_, +later updated to `1815-2012 `_. + +VOLTTRON's DNP3 Agent is an implementation of a DNP3 Outstation as specified in IEEE Std 1815-2012. It engages in +bidirectional network communications with a DNP3 Master, which might be located at a power utility. + +Like some other VOLTTRON protocol agents (e.g. IEEE2030_5Agent), the DNP3 Agent can optionally be front-ended by a DNP3 +device driver running under VOLTTRON's MasterDriverAgent. This allows a DNP3 Master to be treated like any other device +in VOLTTRON's ecosystem. + +The VOLTTRON DNP3 Agent implementation of an Outstation is built on PyDNP3, an open-source library from Kisensum +containing Python language bindings for Automatak's C++ `opendnp3 `_ library, the +de facto reference implementation of DNP3. + +The DNP3 Agent exposes DNP3 application-layer functionality, creating an extensible base from which specific custom +behavior can be designed and supported. By default, the DNP3 Agent acts as a simple transfer agent, publishing data +received from the Master on the VOLTTRON Message Bus, and responding to RPCs from other VOLTTRON agents by sending data +to the Master. + + +Requirements +============ + +PyDNP3 can be installed in an activated environment with: + +.. code-block:: bash + + pip install pydnp3 + + +RPC Calls +--------- + +The DNP3 Agent exposes the following VOLTTRON RPC calls: + +.. code-block:: python + + def get_point(self, point_name): + """ + Look up the most-recently-received value for a given output point. + + @param point_name: The point name of a DNP3 PointDefinition. + @return: The (unwrapped) value of a received point. + """ + + def get_point_by_index(self, group, index): + """ + Look up the most-recently-received value for a given point. + + @param group: The group number of a DNP3 point. + @param index: The index of a DNP3 point. + @return: The (unwrapped) value of a received point. + """ + + def get_points(self): + """ + Look up the most-recently-received value of each configured output point. + + @return: A dictionary of point values, indexed by their VOLTTRON point names. + """ + + def set_point(self, point_name, value): + """ + Set the value of a given input point. + + @param point_name: The point name of a DNP3 PointDefinition. + @param value: The value to set. The value's data type must match the one in the DNP3 PointDefinition. + """ + + def set_points(self, point_list): + """ + Set point values for a dictionary of points. + + @param point_list: A dictionary of {point_name: value} for a list of DNP3 points to set. + """ + + def config_points(self, point_map): + """ + For each of the agent's points, map its VOLTTRON point name to its DNP3 group and index. + + @param point_map: A dictionary that maps a point's VOLTTRON point name to its DNP3 group and index. + """ + + def get_point_definitions(self, point_name_list): + """ + For each DNP3 point name in point_name_list, return a dictionary with each of the point definitions. + + The returned dictionary looks like this: + + { + "point_name1": { + "property1": "property1_value", + "property2": "property2_value", + ... + }, + "point_name2": { + "property1": "property1_value", + "property2": "property2_value", + ... + } + } + + If a definition cannot be found for a point name, it is omitted from the returned dictionary. + + :param point_name_list: A list of point names. + :return: A dictionary of point definitions. + """ + + +Pub/Sub Calls +------------- + +The DNP3 Agent uses two topics when publishing data to the VOLTTRON message bus: + + * **Point Values (default topic: `dnp3/point`)**: As the DNP3 Agent communicates with the Master, + it publishes received point values on the VOLTTRON message bus. + + * **Outstation status (default topic: dnp3/status)**: If the status of the DNP3 Agent outstation + changes, for example if it is restarted, it publishes its new status on the VOLTTRON message bus. + + +Data Dictionary of Point Definitions +------------------------------------ + +The DNP3 Agent loads and uses a data dictionary of point definitions, which are maintained by agreement between the +(DNP3 Agent) Outstation and the DNP3 Master. The data dictionary is stored in the agent's registry. + + +Current Point Values +-------------------- + +The DNP3 Agent tracks the most-recently-received value for each point definition in its data dictionary, regardless of +whether the point value's source is a VOLTTRON RPC call or a message from the DNP3 Master. + + +Agent Configuration +------------------- + +The DNP3Agent configuration file specifies the following fields: + + - **local_ip**: (string) Outstation's host address (DNS resolved). Default: ``0.0.0.0``. + - **port**: (integer) Outstation's port number - the port that the remote endpoint (Master) is listening on. Default: + 20000. + - **point_topic**: (string) VOLTTRON message bus topic to use when publishing DNP3 point values. Default: + ``dnp3/point``. + - **outstation_status_topic**: (string) Message bus topic to use when publishing outstation status. Default: + ``dnp3/outstation_status``. + - **outstation_config**: (dictionary) Outstation configuration parameters. All are optional. Parameters include: + + - **database_sizes**: (integer) Size of each outstation database buffer. Default: 10. + - **event_buffers**: (integer) Size of the database event buffers. Default: 10. + - **allow_unsolicited**: (boolean) Whether to allow unsolicited requests. Default: ``True``. + - **link_local_addr**: (integer) Link layer local address. Default: 10. + - **link_remote_addr**: (integer) Link layer remote address. Default: 1. + - **log_levels**: (list) List of bit field names (`OR'd` together) that filter what gets logged by DNP3. Default: + ``NORMAL``. Possible values: ``ALL``, ``ALL_APP_COMMS``, ``ALL_COMMS``, ``NORMAL``, ``NOTHING``. + - **threads_to_allocate**: (integer) Threads to allocate in the manager's thread pool. Default: 1. + +A sample DNP3 Agent configuration file is available in `services/core/DNP3Agent/dnp3agent.config`. + + +VOLTTRON DNP3 Device Driver +=========================== + +VOLTTRON's DNP3 device driver exposes `get_point`/`set_point` RPC calls and scrapes for DNP3 points. + +The driver periodically issues DNP3Agent RPC calls to refresh its cached representation of DNP3 data. It issues RPC +calls to the DNP3 Agent as needed when responding to `get_point`, `set_point` and `scrape_all` calls. + +For information about the DNP3 driver, see :ref:`DNP3 Driver Configuration `. + + +Installing the DNP3 Agent +========================= + +To install DNP3Agent, please consult the installation advice in `services/core/DNP3Agent/README.md`. `README.md` +specifies a default agent configuration, which can be overridden as needed. + +An agent installation script is available: + +.. code-block:: shell + + $ export VOLTTRON_ROOT= + $ cd $VOLTTRON_ROOT + $ source services/core/DNP3Agent/install_dnp3_agent.sh + +When installing the Mesa Agent, please note that the agent's point definitions must be loaded into the agent's config +store. See `install_dnp3_agent.sh` for an example of how to load them. diff --git a/docs/source/agent-framework/core-service-agents/dnp3_and_mesa/mesa-agent.rst b/docs/source/agent-framework/core-service-agents/dnp3_and_mesa/mesa-agent.rst new file mode 100644 index 0000000000..199c0903eb --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/dnp3_and_mesa/mesa-agent.rst @@ -0,0 +1,226 @@ +.. _MESA: + +========== +Mesa Agent +========== + +The Mesa Agent is a VOLTTRON agent that handles MESA-ESS DNP3 outstation communications. It subclasses and extends the +functionality of VOLTTRON's DNP3 Agent. Like the DNP3 Agent, the Mesa Agent models a DNP3 outstation, communicating +with a DNP3 master. + +For a description of DNP3 and the VOLTTRON DNP3 agent, please refer to the :ref:`DNP3 Agent documentation `. + +VOLTTRON's Mesa Agent and DNP3 Agent are implementations of a DNP3 Outstation as specified in IEEE Std 1815-2012. They +engage in bidirectional network communications with a DNP3 Master, which might be located at a power utility. + +MESA-ESS is an extension and enhancement to DNP3. It builds on the basic DNP3 communications protocol, adding support +for more complex structures, including functions, arrays, curves and schedules. The draft specification for MESA-ESS, +as well as a spreadsheet of point definitions, can be found at http://mesastandards.org/mesa-standards/. + +VOLTTRON's DNP3 Agent and Mesa Agent implementations of an Outstation are built on pydnp3, an open-source library from +Kisensum containing Python language bindings for Automatak's C++ `opendnp3 `_ +library, the de facto reference implementation of DNP3. + +MesaAgent exposes DNP3 application-layer functionality, creating an extensible base from which specific custom behavior +can be designed and supported, including support for MESA functions, arrays and selector blocks. By default, the Mesa +Agent acts as a simple transfer agent, publishing data received from the Master on the VOLTTRON Message Bus, and +responding to RPCs from other VOLTTRON agents by sending data to the Master. Properties of the point and function +definitions also enable the use of more complex controls for point data capture and publication. + +The Mesa Agent was developed by Kisensum for use by 8minutenergy, which provided generous financial support for the +open-source contribution to the VOLTTRON platform, along with valuable feedback based on experience with the agent in a +production context. + + +RPC Calls +========= + +MesaAgent exposes the following VOLTTRON RPC calls: + +.. code-block:: python + + def get_point(self, point_name): + """ + Look up the most-recently-received value for a given output point. + + @param point_name: The point name of a DNP3 PointDefinition. + @return: The (unwrapped) value of a received point. + """ + + def get_point_by_index(self, data_type, index): + """ + Look up the most-recently-received value for a given point. + + @param data_type: The data_type of a DNP3 point. + @param index: The index of a DNP3 point. + @return: The (unwrapped) value of a received point. + """ + + def get_points(self): + """ + Look up the most-recently-received value of each configured output point. + + @return: A dictionary of point values, indexed by their point names. + """ + + def get_configured_points(self): + """ + Look up the most-recently-received value of each configured point. + + @return: A dictionary of point values, indexed by their point names. + """ + + def set_point(self, point_name, value): + """ + Set the value of a given input point. + + @param point_name: The point name of a DNP3 PointDefinition. + @param value: The value to set. The value's data type must match the one in the DNP3 PointDefinition. + """ + + def set_points(self, point_dict): + """ + Set point values for a dictionary of points. + + @param point_dict: A dictionary of {point_name: value} for a list of DNP3 points to set. + """ + + def config_points(self, point_map): + """ + For each of the agent's points, map its VOLTTRON point name to its DNP3 group and index. + + @param point_map: A dictionary that maps a point's VOLTTRON point name to its DNP3 group and index. + """ + + def get_point_definitions(self, point_name_list): + """ + For each DNP3 point name in point_name_list, return a dictionary with each of the point definitions. + + The returned dictionary looks like this: + + { + "point_name1": { + "property1": "property1_value", + "property2": "property2_value", + ... + }, + "point_name2": { + "property1": "property1_value", + "property2": "property2_value", + ... + } + } + + If a definition cannot be found for a point name, it is omitted from the returned dictionary. + + :param point_name_list: A list of point names. + :return: A dictionary of point definitions. + """ + + def get_selector_block(self, point_name, edit_selector): + """ + Return a dictionary of point values for a given selector block. + + :param point_name: Name of the first point in the selector block. + :param edit_selector: The index (edit selector) of the block. + :return: A dictionary of point values. + """ + + def reset(self): + """ + Reset the agent's internal state, emptying point value caches. Used during iterative testing. + """ + + +Pub/Sub Calls +============= + +MesaAgent uses three topics when publishing data to the VOLTTRON message bus: + + * **Point Values (default topic: dnp3/point)**: As MesaAgent communicates with the Master, it publishes received point + values on the VOLTTRON message bus. + + * **Functions (default topic: mesa/function)**: When MesaAgent receives a function step with a "publish" action value, + it publishes the current state of the function (all steps received to date) on the VOLTTRON message bus. + + * **Outstation status (default topic: mesa/status)**: If the status of the MesaAgent outstation + changes, for example if it is restarted, it publishes its new status on the VOLTTRON message bus. + + +Data Dictionaries of Point and Function Definitions +--------------------------------------------------- + +The Mesa Agent loads and uses data dictionaries of point and function definitions, which are maintained by agreement +between the (Mesa Agent) Outstation and the DNP3 Master. The data dictionaries are stored in the agent's registry. + + +Current Point Values +-------------------- + +MesaAgent tracks the most-recently-received value for each point definition in its data dictionary, regardless of +whether the point value's source is a VOLTTRON RPC call or a message from the DNP3 Master. + + +Agent Configuration +------------------- + +The MesaAgent configuration specifies the following fields: + + - **local_ip**: (string) Outstation's host address (DNS resolved). Default: ``0.0.0.0``. + - **port**: (integer) Outstation's port number - the port that the remote endpoint (Master) is listening on. Default: + 20000. + - **point_topic**: (string) VOLTTRON message bus topic to use when publishing DNP3 point values. Default: + ``dnp3/point``. + - **function_topic**: (string) Message bus topic to use when publishing MESA-ESS functions. Default: + ``mesa/function``. + - **outstation_status_topic**: (string) Message bus topic to use when publishing outstation status. Default: + ``mesa/outstation_status``. + - **all_functions_supported_by_default**: (boolean) When deciding whether to reject points for unsupported functions, + ignore the values of their 'supported' points: simply treat all functions as supported. Used primarily during + testing. Default: ``False``. + - **function_validation**: (boolean) When deciding whether to support sending single points to the Mesa Agent. If + ``function_validation`` is ``True``, the Mesa Agent will raise an exception when receiving any invalid point in + current function. If function_validation is ``False``, Mesa Agent will reset current function to None instead of + raising the exception. Default: ``False``. + - **outstation_config**: (dictionary) Outstation configuration parameters. All are optional. Parameters include: + + - **database_sizes**: (integer) Size of each outstation database buffer. Default: 10. + - **event_buffers**: (integer) Size of the database event buffers. Default: 10. + - **allow_unsolicited**: (boolean) Whether to allow unsolicited requests. Default: ``True``. + - **link_local_addr**: (integer) Link layer local address. Default: 10. + - **link_remote_addr**: (integer) Link layer remote address. Default: 1. + - **log_levels**: (list) List of bit field names (OR'd together) that filter what gets logged by DNP3. + Default: [NORMAL]. Possible values: ``ALL``, ``ALL_APP_COMMS``, ``ALL_COMMS``, ``NORMAL``, ``NOTHING``. + - **threads_to_allocate**: (integer) Threads to allocate in the manager's thread pool. Default: 1. + +A sample Mesa Agent configuration file is available in ``services/core/DNP3Agent/mesaagent.config``. + + +Installing MesaAgent +==================== + +To install the Mesa Agent, please consult the installation advice in ``services/core/DNP3Agent/README.md``, +which includes advice on installing ``pydnp3``, a library upon which the DNP3 Agent depends. + +After installing libraries as described in README.md, the agent can be installed from a command-line shell as follows: + +.. code-block:: shell + + $ export VOLTTRON_ROOT= + $ cd $VOLTTRON_ROOT + $ source services/core/DNP3Agent/install_mesa_agent.sh + +README.md specifies a default agent configuration, which can be overridden as needed. + +Here are some things to note when installing MesaAgent: + + - MesaAgent source code resides in, and is installed from, a dnp3 subdirectory, thus allowing it to be implemented as a + subclass of the base DNP3 agent class. When installing the Mesa Agent, inform the install script that it should + build from the mesa subdirectory by exporting the following environment variable: + + .. code-block:: shell + + $ export AGENT_MODULE=dnp3.mesa.agent + + - The agent's point and function definitions must be loaded into the agent's config store. See the + ``install_mesa_agent.sh`` script for an example of how to load them. diff --git a/docs/source/core_services/service_agents/externaldata/externaldata.rst b/docs/source/agent-framework/core-service-agents/external-data/external-data.rst similarity index 100% rename from docs/source/core_services/service_agents/externaldata/externaldata.rst rename to docs/source/agent-framework/core-service-agents/external-data/external-data.rst diff --git a/docs/source/specifications/files/volttron_ieee2030_5.jpg b/docs/source/agent-framework/core-service-agents/ieee2030_5-agent/files/volttron_ieee2030_5.jpg similarity index 100% rename from docs/source/specifications/files/volttron_ieee2030_5.jpg rename to docs/source/agent-framework/core-service-agents/ieee2030_5-agent/files/volttron_ieee2030_5.jpg diff --git a/docs/source/agent-framework/core-service-agents/ieee2030_5-agent/ieee2030_5-agent.rst b/docs/source/agent-framework/core-service-agents/ieee2030_5-agent/ieee2030_5-agent.rst new file mode 100644 index 0000000000..3b07e68d4d --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/ieee2030_5-agent/ieee2030_5-agent.rst @@ -0,0 +1,220 @@ +.. _IEEE2030_5-Agent: + +===================== +IEEE 2030.5 DER Agent +===================== + +Version 1.0 + +Smart Energy Profile 2.0 (SEP2, IEEE 2030.5) specifies a REST architecture built around the core HTTP verbs: `GET`, +`HEAD`, `PUT`, `POST` and `DELETE`. A specification for the IEEE 2030.5 protocol can be found +`here `_. + +IEEE 2030.5 EndDevices (clients) `POST` XML resources representing their state, and `GET` XML resources containing +command and control information from the server. The server never reaches out to the client unless a "subscription" is +registered and supported for a particular resource type. This implementation does not use IEEE 2030.5 registered +subscriptions. + +The IEEE 2030.5 specification requires HTTP headers, and it explicitly requires RESTful response codes, for example: + + - 201 - "Created" + - 204 - "No Content" + - 301 - "Moved Permanently" + - etc. + +IEEE 2030.5 message encoding may be either XML or EXI. Only XML is supported in this implementation. + +IEEE 2030.5 requires HTTPS/TLS version 1.2 along with support for the cipher suite `TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8`. +Production installation requires a certificate issued by a IEEE 2030.5 CA. The encryption requirement can be met by +using a web server such as Apache to proxy the HTTPs traffic. + +IEEE 2030.5 discovery, if supported, must be implemented by an xmDNS server. Avahi can be modified to perform this +function. + + +Function Sets +============= + +IEEE 2030.5 groups XML resources into "Function Sets." Some of these function sets provide a core set of functionality +used across higher-level function sets. This implementation implements resources from the following function sets: + + - Time + - Device Information + - Device Capabilities + - End Device + - Function Set Assignments + - Power Status + - Distributed Energy Resources + + +Distributed Energy Resources +============================ + +Distributed energy resources (DERs) are devices that generate energy, e.g., solar inverters, or store energy, e.g., +battery storage systems, electric vehicle supply equipment (EVSEs). These devices are managed by a IEEE 2030.5 DER +server using DERPrograms which are described by the IEEE 2030.5 specification as follows: + + Servers host one or more DERPrograms, which in turn expose DERControl events to DER clients. DERControl instances + contain attributes that allow DER clients to respond to events that are targeted to their device type. A DERControl + instance also includes scheduling attributes that allow DER clients to store and process future events. These + attributes include start time and duration, as well an indication of the need for randomization of the start and / + or duration of the event. The IEEE 2030.5 DER client model is based on the SunSpec Alliance Inverter Control Model + [SunSpec] which is derived from IEC 61850-90-7 [61850] and [EPRI]. + +EndDevices post multiple IEEE 2030.5 resources describing their status. The following is an example of a Power Status +resource that might be posted by an EVSE (vehicle charging station): + +.. code-block:: xml + + + 4 + 1487812095 + 1 + 9300 + + + 3 + -5 + + + 3 + 22 + + + 3 + 7 + + 11280 + 10000 + 9223372036854775807 + 1487812095 + + + + +Design Details +-------------- + +.. image:: files/volttron_ieee2030_5.jpg + +VOLTTRON's IEEE 2030.5 implementation includes a `IEEE2030_5` Agent and a IEEE 2030.5 device driver, as described below. + + +VOLTTRON IEEE2030_5 Agent +========================= + +The IEEE2030_5 Agent implements a IEEE 2030.5 server that receives HTTP `POST`/`PUT` requests from IEEE 2030.5 devices. +The requests are routed to IEEE2030_5 Agent over the VOLTTRON message bus by VOLTTRON's Master Web Service. The +IEEE2030_5 Agent returns an appropriate HTTP response. In some cases (e.g., DERControl requests), this response +includes a data payload. + +The IEEE2030_5 Agent maps IEEE 2030.5 resource data to a VOLTTRON IEEE 2030.5 data model based on SunSpec, using block +numbers and point names as defined in the SunSpec Information Model, which in turn is harmonized with 61850. The data +model is given in detail below. + +Each device's data is stored by the IEEE2030_5 Agent in an `EndDevice` memory structure. This structure is not +persisted to a database. Each `EndDevice` retains only the most recently received value for each field. + +The IEEE2030_5 Agent exposes RPC calls for getting and setting EndDevice data. + + +VOLTTRON IEEE 2030.5 Device Driver +---------------------------------- + +The :ref:`IEEE 2030.5 device driver ` is a new addition to VOLTTRON Master Driver Agent's +family of standard device drivers. It exposes ``get_point``/``set_point calls`` for IEEE 2030.5 EndDevice fields. + +The IEEE 2030.5 device driver periodically issues IEEE2030_5 Agent RPC calls to refresh its cached representation of +EndDevice data. It issues RPC calls to IEEE2030_5Agent as needed when responding to ``get_point``, ``set_point`` and +``scrape_all`` calls. + + +Field Definitions +^^^^^^^^^^^^^^^^^ + +These field IDs correspond to the ones in the IEEE 2030.5 device driver's configuration file, ``ieee2030_5.csv``. +They have been used in that file's "Volttron Point Name" column and also in its "Point Name" column. + +================= ============================= ==================================================== ======= ====== +Field ID IEEE 2030.5 Resource/Property Description Units Type +================= ============================= ==================================================== ======= ====== +b1_Md device_information Model (32 char lim). string + mfModel +b1_Opt device_information Long-form device identifier (32 char lim). string + lfdi +b1_SN abstract_device Short-form device identifier (32 char lim). string + sfdi +b1_Vr device_information Version (16 char lim). string + mfHwVer +b113_A mirror_meter_reading AC current. A float + PhaseCurrentAvg +b113_DCA mirror_meter_reading DC current. A float + InstantPackCurrent +b113_DCV mirror_meter_reading DC voltage. V float + LineVoltageAvg +b113_DCW mirror_meter_reading DC power. W float + PhasePowerAvg +b113_PF mirror_meter_reading AC power factor. % float + PhasePFA +b113_WH mirror_meter_reading AC energy. Wh float + EnergyIMP +b120_AhrRtg der_capability Usable capacity of the battery. Ah float + rtgAh Maximum charge minus minimum charge. +b120_ARtg der_capability Maximum RMS AC current level capability of the A float + rtgA inverter. +b120_MaxChaRte der_capability Maximum rate of energy transfer into the device. W float + rtgMaxChargeRate +b120_MaxDisChaRte der_capability Maximum rate of energy transfer out of the device. W float + rtgMaxDischargeRate +b120_WHRtg der_capability Nominal energy rating of the storage device. Wh float + rtgWh +b120_WRtg der_capability Continuous power output capability of the inverter. W float + rtgW +b121_WMax der_settings Maximum power output. Default to WRtg. W float + setMaxChargeRate +b122_ActWh mirror_meter_reading AC lifetime active (real) energy output. Wh float + EnergyEXP +b122_StorConn der_status CONNECTED=0, AVAILABLE=1, OPERATING=2, TEST=3. enum + storConnectStatus +b124_WChaMax der_control Setpoint for maximum charge. This is the only W float + opModFixedFlow field that is writable with a set_point call. +b403_Tmp mirror_meter_reading Pack temperature. C float + InstantPackTemp +b404_DCW PEVInfo Power flow in or out of the inverter. W float + chargingPowerNow +b404_DCWh der_availability Output energy (absolute SOC). Wh float + availabilityDuration Calculated as (availabilityDuration / 3600) * WMax. +b802_LocRemCtl der_status Control Mode: REMOTE=0, LOCAL=1. enum + localControlModeStatus +b802_SoC der_status State of Charge %. % WHRtg float + stateOfChargeStatus +b802_State der_status DISCONNECTED=1, INITIALIZING=2, CONNECTED=3, enum + inverterStatus STANDBY=4, SOC PROTECTION=5, FAULT=99. +================= ============================= ==================================================== ======= ====== + + +Revising and Expanding the Field Definitions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The IEEE 2030.5-to-SunSpec field mappings in this implementation are a relatively thin subset of all possible +field definitions. Developers are encouraged to expand the definitions. + +The procedure for expanding the field mappings requires you to make changes in two places: + +1. Update the driver's point definitions in ``services/core/MasterDriverAgent/master_driver/ieee2030_5.csv`` +2. Update the IEEE 2030.5-to-SunSpec field mappings in ``services/core/IEEE2030_5Agent/ieee2030_5/end_device.py`` and + ``__init__.py`` + +When updating VOLTTRON's IEEE 2030.5 data model, please use field IDs that conform to the SunSpec +block-number-and-field-name model outlined in the SunSpec Information Model Reference (see the link below). + + +For Further Information +----------------------- + +SunSpec References: + + - Information model specification: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Information-Models-12041.pdf + - Information model reference spreadsheet: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Information-Model-Reference.xlsx + - Inverter models: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Inverter-Models-12020.pdf + - Energy storage models: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Energy-Storage-Models-12032.pdf diff --git a/docs/source/agent-framework/core-service-agents/index.rst b/docs/source/agent-framework/core-service-agents/index.rst new file mode 100644 index 0000000000..f2b8323f68 --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/index.rst @@ -0,0 +1,22 @@ +.. _Core-Service-Agents: + +============= +Core Services +============= + +Agents in the `services/core` directory support the most common use cases of the platform. For details on each, please +refer to the corresponding documents. + +.. toctree:: + :maxdepth: 1 + + master-driver/master-driver-agent + market-service/market-service-agent + volttron-central/volttron-central + dnp3_and_mesa/dnp3-agent + dnp3_and_mesa/mesa-agent + external-data/external-data + ieee2030_5-agent/ieee2030_5-agent + obix/obix-history-agent + openadr-ven/ven-agent + diff --git a/docs/source/core_services/service_agents/market_service/MarketServiceAgent.rst b/docs/source/agent-framework/core-service-agents/market-service/market-service-agent.rst similarity index 64% rename from docs/source/core_services/service_agents/market_service/MarketServiceAgent.rst rename to docs/source/agent-framework/core-service-agents/market-service/market-service-agent.rst index 6e93a44d9e..e13edeaef4 100644 --- a/docs/source/core_services/service_agents/market_service/MarketServiceAgent.rst +++ b/docs/source/agent-framework/core-service-agents/market-service/market-service-agent.rst @@ -7,34 +7,37 @@ Market Service Agent Introduction ============ -The MarketServiceAgent implements a variation of a double-blind auction, in which each market participant bids +The Market Service Agent implements a variation of a double-blind auction, in which each market participant bids to buy or sell a commodity for a given price. -In contrast to other common implementations, participants do not bid single price-quantity pairs. -Instead, they bid a price-quantity curve, or “flexibility curve” into their respective markets. -Market participants may be both buyers in one market and sellers in another. +In contrast to other common implementations, participants do not bid single price-quantity pairs. Instead, they bid a +price-quantity curve, or “flexibility curve” into their respective markets. Market participants may be both buyers in +one market and sellers in another. + Settling of the market is a “single shot” process that begins with bidding that progresses from the bottom up -and concludes with a clearing of the markets from the top down. This is termed “single shot” because there is no +and concludes with a clearing of the markets from the top down. This is termed “single shot” because there is no iteration required to find the clearing price or quantity at any level of the market structure. -Once the market has cleared, the process begins again for the next market interval, and -new bids are submitted based on the updated states of the agents. + +Once the market has cleared, the process begins again for the next market interval, and new bids are submitted based on +the updated states of the agents. + Requirements ------------ -The Market Service Agent requires the Transitions (version 0.6.9) and NumPy (version 1.15.4) packages. These +The Market Service Agent requires the `Transitions` (version 0.6.9) and `NumPy` (version 1.15.4) packages. These packages can be installed in an activated environment with: -:: +.. code-block:: bash pip install transitions==0.6.9 pip install numpy==1.15.4 + Market Timing -------------- +============= -The MarketServiceAgent is driven by the Director. The Director -drives the MarketServiceAgent through a timed loop. The Director has just a few parameters -that are configured by default with adequate values. They are: +The MarketServiceAgent is driven by the Director. The Director drives the MarketServiceAgent through a timed loop. The +Director has just a few parameters that are configured by default with adequate values. They are: 1. The market_period with a default value of 5 minutes 2. The reservation_delay with a default value of 0 minutes @@ -51,13 +54,13 @@ The timing loop works as follows: * Error messages are published when discovered and usually occur at the end of one of the delays. * The cycle repeats. + How to Use the MarketServiceAgent ================================= A given agent participates in one or more markets by inheriting from the -:ref:`base MarketAgent`. -The base MarketAgent handles all of the communication between the agent and the MarketServiceAgent. -The agent only needs to join each market with the +:ref:`base MarketAgent`. The base MarketAgent handles all of the communication between the +agent and the MarketServiceAgent. The agent only needs to join each market with the :py:meth:`join_market ` method and then respond to the appropriate callback methods. The callback methods are described at the :ref:`base MarketAgent`. diff --git a/docs/source/agent-framework/core-service-agents/master-driver/master-driver-agent.rst b/docs/source/agent-framework/core-service-agents/master-driver/master-driver-agent.rst new file mode 100644 index 0000000000..3ea7957137 --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/master-driver/master-driver-agent.rst @@ -0,0 +1,471 @@ +.. _Master-Driver-Configuration: + +=================== +Master Driver Agent +=================== + +The Master Driver Agent manages all device communication. To communicate with devices you must setup and deploy the +Master Driver Agent. For more information on the Master Driver Agent's operations, read about the +:ref:`Master Driver ` in the driver framework docs. + + +.. _Master-Driver-Config: + +Configuring the Master Driver +============================= + +Configuration for each device consists of 3 parts: + +* Master Driver Agent configuration file - lists all driver configuration files to load +* Driver configuration file - contains the general driver configuration and device settings +* Device Register configuration file - contains the settings for each individual data point on the device + +For each device, you must create a driver configuration file, device register configuration file, and an entry in the +Master Driver Agent configuration file. + +Once configured, the Master Driver Agent is configured and deployed in a manner similar to any other agent: + +.. code-block:: bash + + python scripts/install-agent.py -s services/core/MasterDriverAgent -c + + +Requirements +------------ + +VOLTTRON drivers operated by the master driver may have additional requirements for installation. +Required libraries: + +:: + + BACnet driver - bacpypes + Modbus driver - pymodbus + Modbus_TK driver - modbus-tk + DNP3 and IEEE 2030.5 drivers - pydnp3 + +The easiest way to install the requirements for drivers included in the VOLTTRON repository is to use ``bootstrap.py`` +(see :ref:`platform installation for more detail `) + + +Master Driver Agent Configuration +--------------------------------- + +The Master Driver Agent configuration consists of general settings for all devices. The default values of the Master +Driver should be sufficient for most users. The user may optionally change the interval between device scrapes with the +driver_scrape_interval. + +The following example sets the driver_scrape_interval to 0.05 seconds or 20 devices per second: + +.. code-block:: json + + { + "driver_scrape_interval": 0.05, + "publish_breadth_first_all": false, + "publish_depth_first": false, + "publish_breadth_first": false, + "publish_depth_first_all": true, + "group_offset_interval": 0.0 + } + +* **driver_scrape_interval** - Sets the interval between devices scrapes. Defaults to 0.02 or 50 devices per second. + Useful for when the platform scrapes too many devices at once resulting in failed scrapes. +* **group_offset_interval** - Sets the interval between when groups of devices are scraped. Has no effect if all devices + are in the same group. + +In order to improve the scalability of the platform unneeded device state publishes for all devices can be turned off. +All of the following setting are optional and default to `True`. + +* **publish_depth_first_all** - Enable "depth first" publish of all points to a single topic for all devices. +* **publish_breadth_first_all** - Enable "breadth first" publish of all points to a single topic for all devices. +* **publish_depth_first** - Enable "depth first" device state publishes for each register on the device for all devices. +* **publish_breadth_first** - Enable "breadth first" device state publishes for each register on the device for all + devices. + +An example master driver configuration file can be found in the VOLTTRON repository in +`services/core/MasterDriverAgent/master-driver.agent`. + + +.. _driver-configuration-file: + +Driver Configuration File +------------------------- + +.. note:: + + The terms `register` and `point` are used interchangeably in the documentation and in the configuration setting + names. They have the same meaning in the context of VOLTTRON drivers. + +Each device configuration has the following form: + +.. code-block:: json + + { + "driver_config": {"device_address": "10.1.1.5", + "device_id": 500}, + "driver_type": "bacnet", + "registry_config":"config://registry_configs/vav.csv", + "interval": 60, + "heart_beat_point": "heartbeat", + "group": 0 + } + +The following settings are required for all device configurations: + + - **driver_config** - Driver specific setting go here. See below for driver specific settings. + - **driver_type** - Type of driver to use for this device: bacnet, modbus, fake, etc. + - **registry_config** - Reference to a configuration file in the configuration store for registers + on the device. See the `Registry-Configuration-File`_ + and `Adding Device Configurations to the Configuration Store`_ sections below. + +These settings are optional: + + - **interval** - Period which to scrape the device and publish the results in seconds. Defaults to 60 seconds. + - **heart_beat_point** - A Point which to toggle to indicate a heartbeat to the device. A point with this ``Volttron + Point Name`` must exist in the registry. If this setting is missing the driver will not send a heart beat signal + to the device. Heart beats are triggered by the :ref:`Actuator Agent ` which must be running to + use this feature. + - **group** - Group this device belongs to. Defaults to 0 + +These settings are used to create the topic that this device will be referenced by following the VOLTTRON convention of +``{campus}/{building}/{unit}``. This will also be the topic published on, when the device is periodically scraped for +it's current state. + +The topic used to reference the device is derived from the name of the device configuration in the store. See the +`Adding Device Configurations to the Configuration Store`_ section. + + +Device Grouping +^^^^^^^^^^^^^^^ + +Devices may be placed into groups to separate them logically when they are scraped. This is done by setting the `group` +in the device configuration. `group` is a number greater than or equal to 0. Only number of devices in the same group +and the `group_offset_interval` are considered when determining when to scrape a device. + +This is useful in two cases: + +* If you need to ensure that certain devices are scraped in close proximity to each other you can put them in their own + group. If this causes devices to be scraped too quickly the groups can be separated out time wise using the + `group_offset_interval` setting. +* You may scrape devices on different networks in parallel for performance. For instance BACnet devices behind a single + MSTP router need to be scraped slowly and serially, but devices behind different routers may be scraped in parallel. + Grouping devices by router will do this automatically. + +The `group_offset_interval` is applied by multiplying it by the `group` number. If you intend to use +`group_offset_interval` only use consecutive `group` values that start with 0. + + +.. _Registry-Configuration-File: + +Registry Configuration File +--------------------------- +Registry configuration files setup each individual point on a device. Typically this file will be in CSV format, but the +exact format is driver specific. See the section for a particular driver for the registry configuration format. + +The following is a simple example of a Modbus registry configuration file: + +.. csv-table:: Catalyst 371 + :header: Reference Point Name,Volttron Point Name,Units,Units Details,Modbus Register,Writable,Point Address,Default Value,Notes + + CO2Sensor,ReturnAirCO2,PPM,0.00-2000.00,>f,FALSE,1001,,CO2 Reading 0.00-2000.0 ppm + CO2Stpt,ReturnAirCO2Stpt,PPM,1000.00 (default),>f,TRUE,1011,1000,Setpoint to enable demand control ventilation + HeatCall2,HeatCall2,On / Off,on/off,BOOL,FALSE,1114,,Status indicator of heating stage 2 need + + +.. _Adding-Devices-To-Config-Store: + +Adding Device Configurations to the Configuration Store +======================================================= + +Configurations are added to the Configuration Store using the command line: + +.. code-block:: bash + + volttron-ctl config store platform.driver + +* **name** - The name used to refer to the file from the store. +* **file name** - A file containing the contents of the configuration. +* **file type** - ``--raw``, ``--json``, or ``--csv``. Indicates the type of the file. Defaults to ``--json``. + +The main configuration must have the name ``config`` + +Device configuration but **not** registry configurations must have a name prefixed with ``devices/``. Scripts that +automate the process will prefix registry configurations with ``registry_configs/``, but that is not a requirement for +registry files. + +The name of the device's configuration in the store is used to create the topic used to reference the device. For +instance, a configuration named `devices/PNNL/ISB1/vav1` will publish scrape results to `devices/PNNL/ISB1/vav1` and +is accessible with the Actuator Agent via `PNNL/ISB1/vav1`. + +The name of a registry configuration must match the name used to refer to it in the driver configuration. The reference +is not case sensitive. + +If the Master Driver Agent is running any changes to the configuration store will immediately affect the running devices +according to the changes. + +Consider the following three configuration files: + +A master driver configuration called `master-driver.agent`: + +.. code-block:: json + + { + "driver_scrape_interval": 0.05 + } + +A Modbus device configuration file called `modbus1.config`: + +.. code-block:: json + + { + "driver_config": {"device_address": "10.1.1.2", + "port": 502, + "slave_id": 5}, + "driver_type": "modbus", + "registry_config":"config://registry_configs/hvac.csv", + "interval": 60, + "timezone": "UTC", + "heart_beat_point": "heartbeat" + } + +A Modbus registry configuration file called `catalyst371.csv`: + +.. csv-table:: catalyst371.csv + :header: Reference Point Name,Volttron Point Name,Units,Units Details,Modbus Register,Writable,Point Address,Default Value,Notes + + CO2Sensor,ReturnAirCO2,PPM,0.00-2000.00,>f,FALSE,1001,,CO2 Reading 0.00-2000.0 ppm + CO2Stpt,ReturnAirCO2Stpt,PPM,1000.00 (default),>f,TRUE,1011,1000,Setpoint to enable demand control ventilation + HeatCall2,HeatCall2,On / Off,on/off,BOOL,FALSE,1114,,Status indicator of heating stage 2 need + +To store the master driver configuration run the command: + +.. code-block:: bash + + volttron-ctl config store platform.driver config master-driver.agent + +To store the registry configuration run the command (note the ``--csv`` option): + +.. code-block:: bash + + volttron-ctl config store platform.driver registry_configs/hvac.csv catalyst371.csv --csv + +.. Note:: + + The name `registry_configs/hvac.csv` matches the configuration reference in the file `modbus1.config` - this + is required. + +To store the driver configuration run the command: + +.. code-block:: bash + + volttron-ctl config store platform.driver devices/my_campus/my_building/hvac1 modbus1.config + + +Converting Old Style Configuration +---------------------------------- + +The new Master Driver no longer supports the old style of device configuration. The old `device_list` setting is +ignored. + +To simplify updating to the new format `scripts/update_master_driver_config.py` is provide to automatically update to +the new configuration format. + +With the platform running run: + +.. code-block:: bash + + python scripts/update_master_driver_config.py + +old_configuration`` is the main configuration file in the old format. The script automatically modifies the driver +files to create references to CSV files and adds the CSV files with the appropriate name. + +`output` is the target output directory. + +If the ``--keep-old`` switch is used the old configurations in the output directory (if any) will not be deleted before +new configurations are created. Matching names will still be overwritten. + +The output from `scripts/update_master_driver_config.py` can be automatically added to the configuration store +for the Master Driver agent with `scripts/install_master_driver_configs.py`. + +Creating and naming configuration files in the form needed by `scripts/install_master_driver_configs.py` can speed up +the process of changing and updating a large number of configurations. See the ``--help`` message for +`scripts/install_master_driver_configs.py` for more details. + + +Device State Publishes +---------------------- + +By default, the value of each register on a device is published 4 different ways when the device state is published. +Consider the following settings in a driver configuration stored under the name ``devices/pnnl/isb1/vav1``: + +.. code-block:: json + + { + "driver_config": {"device_address": "10.1.1.5", + "device_id": 500}, + + "driver_type": "bacnet", + "registry_config":"config://registry_configs/vav.csv", + } + +In the `vav.csv` file is a register with the name `temperature`. For these examples the current value of the +register on the device happens to be 75.2 and the meta data is + +.. code-block:: json + + {"units": "F"} + +When the driver publishes the device state the following 2 things will be published for this register: + + A "depth first" publish to the topic `devices/pnnl/isb1/vav1/temperature` with the following message: + + .. code-block:: python + + [75.2, {"units": "F"}] + + A "breadth first" publish to the topic `devices/temperature/vav1/isb1/pnnl` with the following message: + + .. code-block:: python + + [75.2, {"units": "F"}] + + These publishes can be turned off by setting `publish_depth_first` and `publish_breadth_first` to `false` + respectively. + +Also these two publishes happen once for all registers: + + A "depth first" publish to the topic `devices/pnnl/isb1/vav1/all` with the following message: + + .. code-block:: python + + [{"temperature": 75.2, ...}, {"temperature":{"units": "F"}, ...}] + + A "breadth first" publish to the topic `devices/all/vav1/isb1/pnnl` with the following message: + + .. code-block:: python + + [{"temperature": 75.2, ...}, {"temperature":{"units": "F"}, ...}] + + These publishes can be turned off by setting `publish_depth_first_all` and `publish_breadth_first_all` to + ``false`` respectively. + + +Device Scalability Settings +--------------------------- + +In order to improve the scalability of the platform unneeded device state publishes for a device can be turned off. +All of the following setting are optional and will override the value set in the main master driver configuration. + + - **publish_depth_first_all** - Enable "depth first" publish of all points to a single topic. + - **publish_breadth_first_all** - Enable "breadth first" publish of all points to a single topic. + - **publish_depth_first** - Enable "depth first" device state publishes for each register on the device. + - **publish_breadth_first** - Enable "breadth first" device state publishes for each register on the device. + +It is common practice to set `publish_breadth_first_all`, `publish_depth_first`, and +`publish_breadth_first` to `False` unless they are specifically needed by an agent running on +the platform. + + +.. note:: + + All Historian Agents require `publish_depth_first_all` to be set to `True` in order to capture data. + + +.. _Master_Driver_Override: + +Master Driver Override +====================== + +By default, every user is allowed write access to the devices by the master driver. The override feature will allow the +user (for example, building administrator) to override this default behavior and enable the user to lock the write +access on the devices for a specified duration of time or indefinitely. + + +Set Override On +--------------- + +The Master Driver's ``set_override_on`` RPC method can be used to set the override condition for all drivers with topic +matching the provided pattern. This can be specific devices, groups of devices, or even all configured devices. The +pattern matching is based on bash style filename matching semantics. + +Parameters: + + - pattern: Override pattern to be applied. For example, + * If the pattern is ``campus/building1/*`` the override condition is applied for all the devices under + `campus/building1/`. + * If the pattern is ``campus/building1/ahu1`` the override condition is applied for only the + `campus/building1/ahu1` device. The pattern matching is based on bash style filename matching semantics. + - duration: Time duration for the override in seconds. If duration <= 0.0, it implies an indefinite duration. + - failsafe_revert: Flag to indicate if all the devices falling under the override condition has to be set to its + default state/value immediately. + - staggered_revert: If this flag is set, reverting of devices will be staggered. + +Example ``set_override_on`` RPC call: + +.. code-block:: python + + self.vip.rpc.call(PLATFORM_DRIVER, "set_override_on", , ) + + +Set Override Off +---------------- + +The override condition can also be toggled off based on a provided pattern using the Master Driver's +``set_override_off`` RPC call. + +Parameters: + + - pattern: Override pattern to be applied. For example, + * If the pattern is ``campus/building1/*`` the override condition is removed for all the devices under + `campus/building1/`. + * If the pattern is ``campus/building1/ahu1`` the override condition is removed for only for the + `campus/building1/ahu1` device. The pattern matching is based on bash style filename matching semantics. + +Example ``set_override_off`` RPC call: + +.. code-block:: python + + self.vip.rpc.call(PLATFORM_DRIVER, "set_override_off", ) + + +Get Override Devices +-------------------- + +A list of all overridden devices can be obtained with the Master Driver's ``get_override_devices`` RPC call. + +This method call has no additional parameters. + +Example ``get_override_devices`` RPC call: + +.. code-block:: python + + self.vip.rpc.call(PLATFORM_DRIVER, "get_override_devices") + + +Get Override Patterns +--------------------- + +A list of all patterns which have been requested for override can be obtained with the Master Driver's +``get_override_patterns`` RPC call. + +This method call has no additional parameters + +Example "get_override_patterns" RPC call: + +.. code-block:: python + + self.vip.rpc.call(PLATFORM_DRIVER, "get_override_patterns") + + +Clear Overrides +--------------- + +All overrides set by RPC calls described above can be toggled off at using a single ``clear_overrides`` RPC call. + +This method call has no additional parameters + +Example "clear_overrides" RPC call: + +.. code-block:: python + + self.vip.rpc.call(PLATFORM_DRIVER, "clear_overrides") + diff --git a/docs/source/core_services/drivers/Obix-History-Agent.rst b/docs/source/agent-framework/core-service-agents/obix/obix-history-agent.rst similarity index 65% rename from docs/source/core_services/drivers/Obix-History-Agent.rst rename to docs/source/agent-framework/core-service-agents/obix/obix-history-agent.rst index b34eeb6868..b3511be4eb 100644 --- a/docs/source/core_services/drivers/Obix-History-Agent.rst +++ b/docs/source/agent-framework/core-service-agents/obix/obix-history-agent.rst @@ -1,22 +1,23 @@ -.. _Obix-history: +.. _Obix-History: +================== Obix History Agent ------------------- +================== -The Obix History Agent captures data history data from an Obix RESTful interface and publishes -it to the message bus like a driver for capture by agents and historians. The Agent will setup -its queries to ensure that data is only publishes once. For points queried for the first time -it will go back in time and publish old data as configured. +The Obix History Agent captures data history data from an Obix RESTful interface and publishes it to the message bus +like a driver for capture by agents and historians. The Agent will setup its queries to ensure that data is only +publishes once. For points queried for the first time it will go back in time and publish old data as configured. -The data will be colated into device all publishs automatically and will use a timestamp in the -header based on the timestamps reported by the Obix interface. The publishes will be made in chronological order. +The data will be collated into device `all` publishes automatically and will use a timestamp in the header based on the +timestamps reported by the Obix interface. The publishes will be made in chronological order. Units data is automatically read from the device. For sending commands to devices see :ref:`Obix-config`. + Agent Configuration -******************* +=================== There are three arguments for the **driver_config** section of the device configuration file: @@ -30,7 +31,7 @@ There are three arguments for the **driver_config** section of the device config Here is an example device configuration file: -.. code-block:: json +:: { "url": "http://example.com/obix/histories/EXAMPLE/", @@ -48,22 +49,26 @@ Here is an example device configuration file: A sample Obix configuration file can be found in the VOLTTRON repository in ``services/core/ObixHistoryPublish/config`` + Registry Configuration File -*************************** +=========================== Similar to a driver the Obix History Agent requires a registry file to select the points to publish. -The registry configuration file is a `CSV `_ file. Each row configures a point on the device. +The registry configuration file is a `CSV `_ file. Each row +configures a point on the device. The following columns are required for each row: - **Device Name** - Name of the device to associate with this point. - - **Volttron Point Name** - The Volttron Point name to use when publishing this value. - - **Obix Name** - Name of the point on the obix interface. Escaping of spaces and dashes for use with the interface is handled internaly. + - **Volttron Point Name** - The VOLTTRON Point name to use when publishing this value. + - **Obix Name** - Name of the point on the Obix interface. Escaping of spaces and dashes for use with the interface + is handled internally. -Any additional columns will be ignored. It is common practice to include a **Notes** or **Unit Details** for additional information about a point. +Any additional columns will be ignored. It is common practice to include a `Notes` or `Unit Details` for additional +information about a point. -The following is an example of a Obix History Agent registry confugration file: +The following is an example of a Obix History Agent registry configuration file: .. csv-table:: Obix :header: Device Name,Volttron Point Name,Obix Name @@ -74,25 +79,30 @@ The following is an example of a Obix History Agent registry confugration file: device2,Boiler Plant Hourly Gas Usage,Boiler Plant Hourly Gas Usage device3,CG-1 Water Flow H-1,CG-1 Water Flow H-1 -A sample Obix History Agent configuration can be found in the VOLTTRON repository in ``services/core/ObixHistoryPublish/registry_config.csv`` +A sample Obix History Agent configuration can be found in the VOLTTRON repository in +`services/core/ObixHistoryPublish/registry_config.csv` + .. _Obix-History-AutoConfiguration: Automatic Obix Configuration File Creation -****************************************** -A script that will automatically create both a device and register -configuration file for a site is located in the repository at ``scripts/obix/get_obix_history_config.py``. +========================================== + +A script that will automatically create both a device and register configuration file for a site is located in the +repository at `scripts/obix/get_obix_history_config.py`. The utility is invoked with the command: - ``python get_obix_history_config.py -u -p -d `` +.. code-block:: bash + + python get_obix_history_config.py -u -p -d If either the registry_file or driver_file is omitted the script will output those files to stdout. -If either the username or password options are left out the script will ask for them on the command line before proceeding. +If either the username or password options are left out the script will ask for them on the command line before +proceeding. The device name option specifies a default device for every point in the configuration. -The registry file produced by this script assumes that the `Volttron Point Name` and the `Obix Name` have the same value. -Also, it is assumed that all points should be read only. Users are expected to fix this as appropriate. - +The registry file produced by this script assumes that the `Volttron Point Name` and the `Obix Name` have the same +value. Also, it is assumed that all points should be read only. Users are expected to fix this as appropriate. diff --git a/docs/source/agent-framework/core-service-agents/openadr-ven/ven-agent.rst b/docs/source/agent-framework/core-service-agents/openadr-ven/ven-agent.rst new file mode 100644 index 0000000000..215d2a0c0a --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/openadr-ven/ven-agent.rst @@ -0,0 +1,443 @@ +.. _OpenADR-VEN-Agent: + +====================== +OpenADR 2.0b VEN Agent +====================== + +OpenADR (Automated Demand Response) is a standard for alerting and responding to the need to adjust electric power +consumption in response to fluctuations in grid demand. + +OpenADR communications are conducted between Virtual Top Nodes (VTNs) and Virtual End Nodes (VENs). In this +implementation a VOLTTRON agent, the VEN agent, acts as a VEN, communicating with its VTN by means of EIEvent and +EIReport services in conformance with a subset of the OpenADR 2.0b specification. This document's +`VOLTTRON Interface `_ section defines how the VEN agent relays information to, +and receives data from, other VOLTTRON agents. + +The OpenADR 2.0b specification (http://www.openadr.org/specification) is available from the OpenADR Alliance. This +implementation also generally follows the DR program characteristics of the Capacity Program described in Section 9.2 +of the OpenADR Program Guide (http://www.openadr.org/assets/openadr_drprogramguide_v1.0.pdf). + + +DR Capacity Bidding and Events +============================== + +The OpenADR Capacity Bidding program relies on a pre-committed agreement about the VEN’s load shed capacity. This +agreement is reached in a bidding process transacted outside of the OpenADR interaction, typically with a long-term +scope, perhaps a month or longer. The VTN can “call an event,” indicating that a load-shed event should occur +in conformance with this agreement. The VTN indicates the level of load shedding desired, when the event should occur, +and for how long. The VEN responds with an `optIn` acknowledgment. (It can also `optOut`, but since it has been +pre-committed, an `optOut` may incur penalties.) + + +Reporting +--------- + +The VEN agent reports device status and usage telemetry to the VTN, relying on information received periodically from +other VOLTTRON agents. + + +General Approach +================ + +Events: + +- The VEN agent maintains a persistent record of DR events. +- Event updates (including creation) trigger publication of event JSON on the VOLTTRON message bus. +- Other VOLTTRON agents can also call a get_events() RPC to retrieve the current status of + particular events, or of all active events. + +Reporting: + +- The VEN agent configuration defines telemetry values (data points) that can be reported to the VTN. +- The VEN agent maintains a persistent record of telemetry values over time. +- Other VOLTTRON agents are expected to call report_telemetry() to supply the VEN agent + with a regular stream of telemetry values for reporting. +- Other VOLTTRON agents can receive notification of changes in telemetry reporting + requirements by subscribing to publication of telemetry parameters. + + +VEN Agent VOLTTRON Interface +============================ + +The VEN agent implements the following VOLTTRON PubSub and RPC calls. + +PubSub: event update + +.. code-block:: python + + def publish_event(self, an_event): + """ + Publish an event. + + When an event is created/updated, it is published to the VOLTTRON bus + with a topic that includes 'openadr/event_update'. + + Event JSON structure: + { + "event_id" : String, + "creation_time" : DateTime, + "start_time" : DateTime, + "end_time" : DateTime or None, + "signals" : String, # Values: json string describing one or more signals. + "status" : String, # Values: unresponded, far, near, active, + # completed, canceled. + "opt_type" : String # Values: optIn, optOut, none. + } + + If an event status is 'unresponded', the VEN agent is awaiting a decision on + whether to optIn or optOut. The downstream agent that subscribes to this PubSub + message should communicate that choice to the VEN agent by calling respond_to_event() + (see below). The VEN agent then relays the choice to the VTN. + + @param an_event: an EiEvent. + """ + +PubSub: telemetry parameters update + +.. code-block:: python + + def publish_telemetry_parameters_for_report(self, report): + """ + Publish telemetry parameters. + + When the VEN agent telemetry reporting parameters have been updated (by the VTN), + they are published with a topic that includes 'openadr/telemetry_parameters'. + If a particular report has been updated, the reported parameters are for that report. + + Telemetry parameters JSON example: + { + "telemetry": { + "baseline_power_kw": { + "r_id": "baseline_power", + "frequency": "30", + "report_type": "baseline", + "reading_type": "Mean", + "method_name": "get_baseline_power" + } + "current_power_kw": { + "r_id": "actual_power", + "frequency": "30", + "report_type": "reading", + "reading_type": "Mean", + "method_name": "get_current_power" + } + "manual_override": "False", + "report_status": "active", + "online": "False", + } + } + + The above example indicates that, for reporting purposes, telemetry values + for baseline_power and actual_power should be updated -- via report_telemetry() -- at + least once every 30 seconds. + + Telemetry value definitions such as baseline_power and actual_power come from the + agent configuration. + + @param report: (EiReport) The report whose parameters should be published. + """ + +RPC calls: + +.. code-block:: python + + @RPC.export + def respond_to_event(self, event_id, opt_in_choice=None): + """ + Respond to an event, opting in or opting out. + + If an event's status=unresponded, it is awaiting this call. + When this RPC is received, the VENAgent sends an eventResponse to + the VTN, indicating whether optIn or optOut has been chosen. + If an event remains unresponded for a set period of time, + it times out and automatically optsIn to the event. + + Since this call causes a change in the event's status, it triggers + a PubSub call for the event update, as described above. + + @param event_id: (String) ID of an event. + @param opt_in_choice: (String) 'OptIn' to opt into the event, anything else is treated as 'OptOut'. + """ + +.. code-block:: python + + @RPC.export + def get_events(self, event_id=None, in_progress_only=True, started_after=None, end_time_before=None): + """ + Return a list of events as a JSON string. + + Sample request: + self.get_events(started_after=utils.get_aware_utc_now() - timedelta(hours=1), + end_time_before=utils.get_aware_utc_now()) + + Return a list of events. + + By default, return only event requests with status=active or status=unresponded. + + If an event's status=active, a DR event is currently in progress. + + @param event_id: (String) Default None. + @param in_progress_only: (Boolean) Default True. + @param started_after: (DateTime) Default None. + @param end_time_before: (DateTime) Default None. + @return: (JSON) A list of events -- see 'PubSub: event update'. + """ + +.. code-block:: python + + @RPC.export + def get_telemetry_parameters(self): + """ + Return the VEN agent's current set of telemetry parameters. + + @return: (JSON) Current telemetry parameters -- see 'PubSub: telemetry parameters update'. + """ + +.. code-block:: python + + @RPC.export + def set_telemetry_status(self, online, manual_override): + """ + Update the VEN agent's reporting status. + + Set these properties to either 'TRUE' or 'FALSE'. + + @param online: (Boolean) Whether the VEN agent's resource is online. + @param manual_override: (Boolean) Whether resource control has been overridden. + """ + +.. code-block:: python + + @RPC.export + def report_telemetry(self, telemetry): + """ + Receive an update of the VENAgent's report metrics, and store them in the agent's database. + + Examples of telemetry are: + { + 'baseline_power_kw': '15.2', + 'current_power_kw': '371.1', + 'start_time': '2017-11-21T23:41:46.051405', + 'end_time': '2017-11-21T23:42:45.951405' + } + + @param telemetry_values: (JSON) Current value of each report metric, with reporting-interval start/end. + """ + + +PubSub: Event Update +-------------------- + +When an event is created/updated, the event is published with a topic that includes `openadr/event/{ven_id}`. + +Event JSON structure: + +:: + + { + "event_id" : String, + "creation_time" : DateTime - UTC, + "start_time" : DateTime - UTC, + "end_time" : DateTime - UTC, + "priority" : Integer, # Values: 0, 1, 2, 3. Usually expected to be 1. + "signals" : String, # Values: json string describing one or more signals. + "status" : String, # Values: unresponded, far, near, active, completed, canceled. + "opt_type" : String # Values: optIn, optOut, none. + } + +If an event status is 'unresponded', the VEN is awaiting a decision on whether to `optIn` or `optOut`. The downstream +agent that subscribes to this `PubSub` message should communicate that choice to the VEN by calling respond_to_event() +(see below). The VEN then relays the choice to the VTN. + + +PubSub: Telemetry Parameters Update +----------------------------------- + +When the VEN telemetry reporting parameters have been updated (by the VTN), they are published with a topic that +includes `openadr/status/{ven_id}`. + +These parameters include state information about the current report. + +Telemetry parameters structure: + +:: + + { + 'telemetry': '{ + "baseline_power_kw": { + "r_id" : "baseline_power", # ID of the reporting metric + "report_type" : "baseline", # Type of reporting metric, e.g. baseline or reading + "reading_type" : "Direct Read", # (per OpenADR telemetry_usage report requirements) + "units" : "powerReal", # (per OpenADR telemetry_usage reoprt requirements) + "method_name" : "get_baseline_power", # Name of the VEN agent method that gets the metric + "min_frequency" : (Integer), # Data capture frequency in seconds (minimum) + "max_frequency" : (Integer) # Data capture frequency in seconds (maximum) + }, + "current_power_kw": { + "r_id" : "actual_power", # ID of the reporting metric + "report_type" : "reading", # Type of reporting metric, e.g. baseline or reading + "reading_type" : "Direct Read", # (per OpenADR telemetry_usage report requirements) + "units" : "powerReal", # (per OpenADR telemetry_usage report requirements) + "method_name" : "get_current_power", # Name of the VEN agent method that gets the metric + "min_frequency" : (Integer), # Data capture frequency in seconds (minimum) + "max_frequency" : (Integer) # Data capture frequency in seconds (maximum) + } + }' + 'report parameters': '{ + "status" : (String), # active, inactive, completed, or cancelled + "report_specifier_id" : "telemetry", # ID of the report definition + "report_request_id" : (String), # ID of the report request; supplied by the VTN + "request_id" : (String), # Request ID of the most recent VTN report modification + "interval_secs" : (Integer), # How often a report update is sent to the VTN + "granularity_secs" : (Integer), # How often a report update is sent to the VTN + "start_time" : (DateTime - UTC), # When the report started + "end_time" : (DateTime - UTC), # When the report is scheduled to end + "last_report" : (DateTime - UTC), # When a report update was last sent + "created_on" : (DateTime - UTC) # When this set of information was recorded in the VEN db + }', + 'manual_override' : (Boolean) # VEN manual override status, as supplied by Control Agent + 'online' : (Boolean) # VEN online status, as supplied by Control Agent + } + +Telemetry value definitions such as `baseline_power_kw` and `current_power_kw` come from the VEN agent config. + + +.. _OpenADR-VEN-Agent-Config: + +OpenADR VEN Agent: Installation and Configuration +================================================= + +The VEN agent can be configured, built and launched using the VOLTTRON agent installation process described in +http://volttron.readthedocs.io/en/develop/devguides/agent_development/Agent-Development.html#agent-development. + +The VEN agent depends on some third-party libraries that are not in the standard VOLTTRON installation. They should be +installed in the VOLTTRON virtual environment prior to building the agent: + +.. code-block:: bash + + (volttron) $ cd $VOLTTRON_ROOT/services/core/OpenADRVenAgent + (volttron) $ pip install -r requirements.txt + +where ``$VOLTTRON_ROOT`` is the base directory of the cloned VOLTTRON code repository. + +The VEN agent is designed to work in tandem with a “control agent,” another VOLTTRON agent that uses VOLTTRON RPC calls +to manage events and supply report data. A sample control agent has been provided in the `test/ControlAgentSim` +subdirectory under OpenADRVenAgent. + +The VEN agent maintains a persistent store of event and report data in ``$VOLTTRON_HOME/data/openadr.sqlite``. Some +care should be taken in managing the disk consumption of this data store. If no events or reports are active, it is +safe to take down the VEN agent and delete the file; the persistent store will be reinitialized automatically on agent +startup. + + +Configuration Parameters +------------------------ + +The VEN agent’s configuration file contains JSON that includes several parameters for configuring VTN server +communications and other behavior. A sample configuration file, `openadrven.config`, has been provided in the agent +directory. + +The VEN agent supports the following configuration parameters: + +========================= ======================== ==================================================== +Parameter Example Description +========================= ======================== ==================================================== +db_path “$VOLTTRON_HOME/data/ Pathname of the agent's sqlite database. Shell + openadr.sqlite” variables will be expanded if they are present + in the pathname. +ven_id “0” The OpenADR ID of this virtual end node. Identifies + this VEN to the VTN. If automated VEN registration + is used, the ID is assigned by the VTN at that + time. If the VEN is registered manually with the + VTN (i.e., via configuration file settings), then + a common VEN ID should be entered in this config + file and in the VTN's site definition. +ven_name "ven01" Name of this virtual end node. This name is used + during automated registration only, identiying + the VEN before its VEN ID is known. +vtn_id “vtn01” OpenADR ID of the VTN with which this VEN + communicates. +vtn_address “http://openadr-vtn. URL and port number of the VTN. + ki-evi.com:8000” +send_registration “False” (“True” or ”False”) If “True”, the VEN sends + a one-time automated registration request to + the VTN to obtain the VEN ID. If automated + registration will be used, the VEN should be run + in this mode initially, then shut down and run + with this parameter set to “False” thereafter. +security_level “standard” If 'high', the VTN and VEN use a third-party + signing authority to sign and authenticate each + request. The default setting is “standard”: the + XML payloads do not contain Signature elements. +poll_interval_secs 30 (integer) How often the VEN should send an OadrPoll + request to the VTN. The poll interval cannot be + more frequent than the VEN’s 5-second process + loop frequency. +log_xml “False” (“True” or “False”) Whether to write each + inbound/outbound request’s XML data to the + agent's log. +opt_in_timeout_secs 1800 (integer) How long to wait before making a + default optIn/optOut decision. +opt_in_default_decision “optOut” (“True” or “False”) Which optIn/optOut choice + to make by default. +request_events_on_startup "False" ("True" or "False") Whether to ask the VTN for a + list of current events during VEN startup. +report_parameters (see below) A dictionary of definitions of reporting/telemetry + parameters. +========================= ======================== ==================================================== + + +Reporting Configuration +----------------------- + +The VEN’s reporting configuration, specified as a dictionary in the agent configuration, defines each telemetry element +(metric) that the VEN can report to the VTN, if requested. By default, it defines reports named “telemetry” and +"telemetry_status", with a report configuration dictionary containing the following parameters: + +======================================================= =========================== ==================================================== +"telemetry" report: parameters Example Description +======================================================= =========================== ==================================================== +report_name "TELEMETRY_USAGE" Friendly name of the report. +report_name_metadata "METADATA_TELEMETRY_USAGE" Friendly name of the report’s metadata, when sent + by the VEN’s oadrRegisterReport request. +report_specifier_id "telemetry" Uniquely identifies the report’s data set. +report_interval_secs_default "300" How often to send a reporting update to the VTN. +telemetry_parameters (baseline_power_kw): r_id "baseline_power" (baseline_power) Unique ID of the metric. +telemetry_parameters (baseline_power_kw): report_type "baseline" (baseline_power) The type of metric being reported. +telemetry_parameters (baseline_power_kw): reading_type "Direct Read" (baseline_power) How the metric was calculated. +telemetry_parameters (baseline_power_kw): units "powerReal" (baseline_power) The reading's data type. +telemetry_parameters (baseline_power_kw): method_name "get_baseline_power" (baseline_power) The VEN method to use when + extracting the data for reporting. +telemetry_parameters (baseline_power_kw): min_frequency 30 (baseline_power) The metric’s minimum sampling + frequency. +telemetry_parameters (baseline_power_kw): max_frequency 60 (baseline_power) The metric’s maximum sampling + frequency. +telemetry_parameters (current_power_kw): r_id "actual_power" (current_power) Unique ID of the metric. +telemetry_parameters (current_power_kw): report_type "reading" (current_power) The type of metric being reported. +telemetry_parameters (current_power_kw): reading_type "Direct Read" (current_power) How the metric was calculated. +telemetry_parameters (current_power_kw): units "powerReal" (baseline_power) The reading's data type. +telemetry_parameters (current_power_kw): method_name "get_current_power" (current_power) The VEN method to use when + extracting the data for reporting. +telemetry_parameters (current_power_kw): min_frequency 30 (current_power) The metric’s minimum sampling + frequency. +telemetry_parameters (current_power_kw): max_frequency 60 (current_power) The metric’s maximum sampling + frequency. +======================================================= =========================== ==================================================== + +======================================================= =========================== ==================================================== +"telemetry_status" report: parameters Example Description +======================================================= =========================== ==================================================== +report_name "TELEMETRY_STATUS" Friendly name of the report. +report_name_metadata "METADATA_TELEMETRY_STATUS" Friendly name of the report’s metadata, when sent + by the VEN’s oadrRegisterReport request. +report_specifier_id "telemetry_status" Uniquely identifies the report’s data set. +report_interval_secs_default "300" How often to send a reporting update to the VTN. +telemetry_parameters (Status): r_id "Status" Unique ID of the metric. +telemetry_parameters (Status): report_type "x-resourceStatus" The type of metric being reported. +telemetry_parameters (Status): reading_type "x-notApplicable" How the metric was calculated. +telemetry_parameters (Status): units "" The reading's data type. +telemetry_parameters (Status): method_name "" The VEN method to use when extracting the data + for reporting. +telemetry_parameters (Status): min_frequency 60 The metric’s minimum sampling frequency. +telemetry_parameters (Status): max_frequency 120 The metric’s maximum sampling frequency. +======================================================= =========================== ==================================================== diff --git a/docs/source/devguides/walkthroughs/files/01-add-devices.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/01-add-devices.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/01-add-devices.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/01-add-devices.png diff --git a/docs/source/devguides/walkthroughs/files/02-install-devices.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/02-install-devices.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/02-install-devices.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/02-install-devices.png diff --git a/docs/source/devguides/walkthroughs/files/03-start-scan.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/03-start-scan.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/03-start-scan.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/03-start-scan.png diff --git a/docs/source/devguides/walkthroughs/files/04-devices-found.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/04-devices-found.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/04-devices-found.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/04-devices-found.png diff --git a/docs/source/devguides/walkthroughs/files/05-get-device-points.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/05-get-device-points.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/05-get-device-points.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/05-get-device-points.png diff --git a/docs/source/devguides/walkthroughs/files/07-edit-points.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/07-edit-points.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/07-edit-points.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/07-edit-points.png diff --git a/docs/source/devguides/walkthroughs/files/07-select-point-a.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/07-select-point-a.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/07-select-point-a.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/07-select-point-a.png diff --git a/docs/source/devguides/walkthroughs/files/07-select-point-b.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/07-select-point-b.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/07-select-point-b.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/07-select-point-b.png diff --git a/docs/source/devguides/walkthroughs/files/07-select-point-c.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/07-select-point-c.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/07-select-point-c.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/07-select-point-c.png diff --git a/docs/source/devguides/walkthroughs/files/08-filter-points-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/08-filter-points-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/08-filter-points-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/08-filter-points-button.png diff --git a/docs/source/devguides/walkthroughs/files/09-filter-set.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/09-filter-set.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/09-filter-set.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/09-filter-set.png diff --git a/docs/source/devguides/walkthroughs/files/10-clear-filter.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/10-clear-filter.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/10-clear-filter.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/10-clear-filter.png diff --git a/docs/source/devguides/walkthroughs/files/11-add-new-point.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/11-add-new-point.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/11-add-new-point.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/11-add-new-point.png diff --git a/docs/source/devguides/walkthroughs/files/12-add-point-dialog.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/12-add-point-dialog.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/12-add-point-dialog.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/12-add-point-dialog.png diff --git a/docs/source/devguides/walkthroughs/files/13-remove-points-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/13-remove-points-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/13-remove-points-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/13-remove-points-button.png diff --git a/docs/source/devguides/walkthroughs/files/14-confirm-remove-points.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/14-confirm-remove-points.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/14-confirm-remove-points.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/14-confirm-remove-points.png diff --git a/docs/source/devguides/walkthroughs/files/15-edit-column-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/15-edit-column-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/15-edit-column-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/15-edit-column-button.png diff --git a/docs/source/devguides/walkthroughs/files/16-edit-column-menu.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/16-edit-column-menu.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/16-edit-column-menu.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/16-edit-column-menu.png diff --git a/docs/source/devguides/walkthroughs/files/17-name-column.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/17-name-column.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/17-name-column.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/17-name-column.png diff --git a/docs/source/devguides/walkthroughs/files/18-duplicated-column.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/18-duplicated-column.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/18-duplicated-column.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/18-duplicated-column.png diff --git a/docs/source/devguides/walkthroughs/files/19-find-in-column-b.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/19-find-in-column-b.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/19-find-in-column-b.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/19-find-in-column-b.png diff --git a/docs/source/devguides/walkthroughs/files/19-find-in-column.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/19-find-in-column.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/19-find-in-column.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/19-find-in-column.png diff --git a/docs/source/devguides/walkthroughs/files/20-replace-in-column.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/20-replace-in-column.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/20-replace-in-column.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/20-replace-in-column.png diff --git a/docs/source/devguides/walkthroughs/files/21-edit-point-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/21-edit-point-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/21-edit-point-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/21-edit-point-button.png diff --git a/docs/source/devguides/walkthroughs/files/22-edit-point-dialog.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/22-edit-point-dialog.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/22-edit-point-dialog.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/22-edit-point-dialog.png diff --git a/docs/source/devguides/walkthroughs/files/23-start-keyboard-commands.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/23-start-keyboard-commands.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/23-start-keyboard-commands.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/23-start-keyboard-commands.png diff --git a/docs/source/devguides/walkthroughs/files/24-keyboard-highlight.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/24-keyboard-highlight.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/24-keyboard-highlight.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/24-keyboard-highlight.png diff --git a/docs/source/devguides/walkthroughs/files/25-keyboard-select.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/25-keyboard-select.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/25-keyboard-select.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/25-keyboard-select.png diff --git a/docs/source/devguides/walkthroughs/files/26-keyboard-shortcuts-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/26-keyboard-shortcuts-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/26-keyboard-shortcuts-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/26-keyboard-shortcuts-button.png diff --git a/docs/source/devguides/walkthroughs/files/27-keyboard-shortcuts.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/27-keyboard-shortcuts.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/27-keyboard-shortcuts.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/27-keyboard-shortcuts.png diff --git a/docs/source/devguides/walkthroughs/files/28-save-registry-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/28-save-registry-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/28-save-registry-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/28-save-registry-button.png diff --git a/docs/source/devguides/walkthroughs/files/29-registry-preview-table.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/29-registry-preview-table.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/29-registry-preview-table.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/29-registry-preview-table.png diff --git a/docs/source/devguides/walkthroughs/files/30-preview-registry-csv.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/30-preview-registry-csv.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/30-preview-registry-csv.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/30-preview-registry-csv.png diff --git a/docs/source/devguides/walkthroughs/files/31-name-registry-file.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/31-name-registry-file.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/31-name-registry-file.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/31-name-registry-file.png diff --git a/docs/source/devguides/walkthroughs/files/32-registry-saved.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/32-registry-saved.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/32-registry-saved.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/32-registry-saved.png diff --git a/docs/source/devguides/walkthroughs/files/33-configure-device-dialog.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/33-configure-device-dialog.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/33-configure-device-dialog.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/33-configure-device-dialog.png diff --git a/docs/source/devguides/walkthroughs/files/34-save-device-config.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/34-save-device-config.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/34-save-device-config.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/34-save-device-config.png diff --git a/docs/source/devguides/walkthroughs/files/35-subdevice-path.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/35-subdevice-path.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/35-subdevice-path.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/35-subdevice-path.png diff --git a/docs/source/devguides/walkthroughs/files/36-subdevice2.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/36-subdevice2.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/36-subdevice2.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/36-subdevice2.png diff --git a/docs/source/devguides/walkthroughs/files/37-device-added-b.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/37-device-added-b.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/37-device-added-b.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/37-device-added-b.png diff --git a/docs/source/devguides/walkthroughs/files/37-device-added.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/37-device-added.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/37-device-added.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/37-device-added.png diff --git a/docs/source/devguides/walkthroughs/files/38-select-saved-registry-file.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/38-select-saved-registry-file.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/38-select-saved-registry-file.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/38-select-saved-registry-file.png diff --git a/docs/source/devguides/walkthroughs/files/39-saved-registry-selector.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/39-saved-registry-selector.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/39-saved-registry-selector.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/39-saved-registry-selector.png diff --git a/docs/source/devguides/walkthroughs/files/40-file-import-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/40-file-import-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/40-file-import-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/40-file-import-button.png diff --git a/docs/source/devguides/walkthroughs/files/41-reload-points-from-device.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/41-reload-points-from-device.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/41-reload-points-from-device.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/41-reload-points-from-device.png diff --git a/docs/source/devguides/walkthroughs/files/43-reconfigure-device-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/43-reconfigure-device-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/43-reconfigure-device-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/43-reconfigure-device-button.png diff --git a/docs/source/devguides/walkthroughs/files/44-reconfiguring-device.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/44-reconfiguring-device.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/44-reconfiguring-device.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/44-reconfiguring-device.png diff --git a/docs/source/devguides/walkthroughs/files/45-reconfigure-option-selector.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/45-reconfigure-option-selector.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/45-reconfigure-option-selector.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/45-reconfigure-option-selector.png diff --git a/docs/source/devguides/walkthroughs/files/46-reconfigure-device-config.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/46-reconfigure-device-config.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/46-reconfigure-device-config.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/46-reconfigure-device-config.png diff --git a/docs/source/devguides/walkthroughs/files/47-file-export-button.png b/docs/source/agent-framework/core-service-agents/volttron-central/files/47-file-export-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/47-file-export-button.png rename to docs/source/agent-framework/core-service-agents/volttron-central/files/47-file-export-button.png diff --git a/docs/source/agent-framework/core-service-agents/volttron-central/vc-device-configuration-demo.rst b/docs/source/agent-framework/core-service-agents/volttron-central/vc-device-configuration-demo.rst new file mode 100644 index 0000000000..7a6758c497 --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/volttron-central/vc-device-configuration-demo.rst @@ -0,0 +1,402 @@ +.. _Device-Configuration-in-VOLTTRON-Central: + +======================================== +Device Configuration in VOLTTRON Central +======================================== + +Devices in your network can be detected and configured through the VOLTTRON Central UI. The current version of VOLTTRON +enables device detection and configuration for BACnet devices. The following sections describe the processes involved +with performing scans to detect physical devices and get their points, and configuring them as virtual devices installed +on VOLTTRON instances. + +- `Launching Device Configuration `__ +- `Scanning for Devices `__ +- `Scanning for Points `__ +- `Registry Configuration File `__ +- `Additional Attributes `__ +- `Quick Edit Features `__ +- `Keyboard Commands `__ +- `Registry Preview `__ +- `Registry Configuration Options `__ +- `Reloading Device Points `__ +- `Device Configuration Form `__ +- `Configuring Sub-devices `__ +- `Reconfiguring Devices `__ +- `Exporting Registry Configuration Files `__ + + +Launching Device Configuration +------------------------------ + +To begin device configuration in VOLTTRON Central, extend the side panel on the left and find the cogs button next to +the platform instance you want to add a device to. Click the cogs button to launch the device configuration feature. + +|Add Devices| + +|Install Devices| + +Currently the only method of adding devices is to conduct a scan to detect BACnet devices. A BACnet Proxy Agent must be +running in order to do the scan. If more than one BACnet Proxy is installed on the platform, choose the one that will +be used for the scan. + +The scan can be conducted using default settings that will search for all physical devices on the network. However, +optional settings can be used to focus on specific devices or change the duration of the scan. Entering a range of +device IDs will limit the scan to return only devices with IDs in that range. Advanced options include the ability to +specify the IP address of a device to detect as well as the ability to change the duration of the scan from the default +of five seconds. + + +Scanning for Devices +-------------------- + +To start the scan, click the large cog button to the right of the scan settings. + +|Start Scan| + +Devices that are detected will appear in the space below the scan settings. Scanning can be repeated at any time by +clicking the large cog button again. + +|Devices Found| + + +Scanning for Points +------------------- + +Another scan can be performed on each physical device to retrieve its available points. This scan is initiated by +clicking the triangle next to the device in the list. The first time the arrow is clicked, it initiates the scan. +After the points are retrieved, the arrow becomes a hide-and-show toggle button and won't re-initiate scanning the +device. + +|Get Device Points| + +After the points have been retrieved once, the only way to scan the same device for points again is to relaunch the +device configuration process from the start by clicking on the small cogs button next to the platform instance in the +panel tree. + + +Registry Configuration File +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The registry configuration determines which points on the physical device will be associated with the virtual device +that uses that particular registry configuration. The registry configuration determines which points' data will be +published to the message bus and recorded by the historian, and it determines how the data will be presented. + +When all the points on the device have been retrieved, the points are loaded into the registry configuration editor. +There, the points can be modified and selected to go into the registry configuration file for a device. + +Each row in the registry configuration editor represents a point, and each cell in the row represents an attribute of +the point. + +Only points that have been selected will be included in the registry configuration file. To select a point, check the +box next to the point in the editor. + +|Select Point Before| + +|Select Point During| + +|Select Point After| + +Type directly in a cell to change an attribute value for a point. + +|Edit Points| + + +Additional Attributes +--------------------- + +The editor's default view shows the attributes that are most likely to be changed during configuration: the VOLTTRON +point name, the writable setting, and the units. Other attributes are present but not shown in the default view. To +see the entire set of attributes for a point, click the `Edit Point` button (the three dots) at the end of the point +row. + +|Edit Point Button| + +In the window that opens, point attributes can be changed by typing in the fields and clicking the Apply button. + +|Edit Point Dialog| + +Checking or unchecking the `Show in Table` box for an attribute will add or remove it as a column in the registry +configuration editor. + + +Quick Edit Features +------------------- + +Several quick-edit features are available in the registry configuration editor. + +The list of points can be filtered based on values in the first column by clicking the filter button in the first +column's header and entering a filter term. + +|Filter Points Button| + +|Filter Set| + +The filter feature allows points to be edited, selected, or deselected more quickly by narrowing down potentially large +lists of points. However, the filter doesn't select points, and if the registry configuration is saved while a filter +is applied, any selected points not included in the filter will still be included in the registry file. + +To clear the filter, click on the `Clear Filter` button in the filter popup. + +|Clear Filter| + +To add a new point to the points listed in the registry configuration editor, click on the `Add Point` button in the +header of the first column. + +|Add New Point| + +|Add Point Dialog| + +Provide attribute values, and click the `Apply` button to add the new point, which will be appended to the bottom of the +list. + +To remove points from the list, select the points and click the `Remove Points` button in the header of the first +column. + +|Remove Points| + +|Confirm Remove Points| + +Each column has an `Edit Column` button in its header. + +|Edit Columns| + +Click on the button to display a popup menu of operations to perform on the column. The options include inserting a +blank new column, duplicating an existing column, removing a column, or searching for a value within a column. + +|Edit Column Menu| + +A duplicate or new column has to be given a unique name. + +|Name Column| + +|Duplicated Column| + +To search for values in a column, choose the `Find and Replace` option in the popup menu. + +|Find in Column| + +Type the term to search for, and click the `Find Next` button to highlight all the matched fields in the column. + +|Find Next| + +Click the `Find Next` button again to advance the focus down the list of matched terms. + +To quickly replace the matched term in the cell with focus, type a replacement term, and click on the `Replace` button. + +|Replace in Column| + +To replace all the matched terms in the column, click on the `Replace All` button. Click the `Clear Search` button to +end the search. + + +Keyboard Commands +----------------- + +Some keyboard commands are available to expedite the selection or de-selection of points. To initiate use of the +keyboard commands, strike the `Control` key on the keyboard. For keyboard commands to be activated, the registry +configuration editor has to have focus, which comes from interacting with it. But the commands won't be activated if +the cursor is in a type-able field. + +If the keyboard commands have been successfully activated, a faint highlight will appear over the first row in the +registry configuration editor. + +|Start Keyboard Commands| + +Keyboard commands are deactivated when the mouse cursor moves over the configuration editor. If unintentional +deactivation occurs, strike the `Control` key again to reactivate the commands. + +With keyboard commands activated, the highlighted row can be advanced up or down by striking the `up` or `down arrow` on +the keyboard. A group of rows can be highlighted by striking the up or down arrow while holding down the `Shift` key. + +|Keyboard Highlight| + +To select the highlighted rows, strike the `Enter` key. + +|Keyboard Select| + +Striking the `Enter` key with rows highlighted will also deselect any rows that were already selected. + +Click on the `Keyboard Shortcuts` button to show a popup list of the available keyboard commands. + +|Keyboard Shortcuts Button| + +|Keyboard Shortcuts| + + +Registry Preview +---------------- + +To save the registry configuration, click the `Save` button at the bottom of the registry configuration editor. + +|Save Registry Button| + +A preview will appear to let you confirm that the configuration is what you intended. + +|Registry Preview Table| + +The configuration also can be inspected in the comma-separated format of the actual registry configuration file. + +|Registry Preview CSV| + +Provide a name for the registry configuration file, and click the `Save` button to save the file. + +|Name Registry File| + +|Registry Saved| + + +Registry Configuration Options +------------------------------ + +Different subsets of configured points can be saved from the same physical device and used to create separate registry +files for multiple virtual devices and sub-devices. Likewise, a single registry file can be reused by multiple virtual +devices and sub-devices. + +To reuse a previously saved registry file, click on the `Select Registry File (CSV)` button at the end of the physical +device's listing. + +|Select Saved Registry File| + +The `Previously Configured Registry Files` window will appear, and a file can be selected to load it into the registry +configuration editor. + +|Saved Registry Selector| + +Another option is to import a registry configuration file from the computer running the VOLTTRON Central web +application, if one has been saved to local storage connected to the computer. To import a registry configuration file +from local storage, click on the `Import Registry File (CSV)` button at the end of the physical device's listing, and +use the file selector window to locate and load the file. + +|File Import Button| + + +Reloading Device Points +----------------------- + +Once a physical device has been scanned, the original points from the scan can be reloaded at any point during device +configuration by clicking on the `Reload Points From Device` button at the end of the device's listing. + +|Reload Points| + + +Device Configuration Form +^^^^^^^^^^^^^^^^^^^^^^^^^ + +After the registry configuration file has been saved, the device configuration form appears. Creating the device +configuration results in the virtual device being installed in the platform and determines the device's position in the +side panel tree. It also contains some settings that determine how data is collected from the device. + +|Configure Device Dialog| + +After the device configuration settings have been entered, click the `Save` button to save the configuration and add the +device to the platform. + +|Save Device Config| + +|Device Added| + + +Configuring Sub-devices +----------------------- + +After a device has been configured, sub-devices can be configured by pointing to their position in the ``Path`` +attribute of the device configuration form. But a sub-device can't be configured until its parent device has been +configured first. + +|Sub-device Path| + +|Sub-device 2| + +As devices are configured, they're inserted into position in the side panel tree, along with their configured points. + +|Device Added to Tree| + + +Reconfiguring Devices +^^^^^^^^^^^^^^^^^^^^^ + +A device that's been added to a VOLTTRON instance can be reconfigured by changing its registry configuration or its +device configuration. To launch reconfiguration, click on the wrench button next to the device in the side panel tree. + +|Reconfigure Device Button| + +Reconfiguration reloads the registry configuration editor and the device configuration form for the virtual device. The +editor and the form work the same way in reconfiguration as during initial device configuration. + +|Reconfiguring Device| + +The reconfiguration view shows the name, address, and ID of the physical device that the virtual device was configured +from. It also shows the name of the registry configuration file associated with the virtual device as well as its +configured path. + +A different registry configuration file can be associated with the device by clicking on the `Select Registry File +(CSV)` button or the `Import Registry File (CSV)` button. + +The registry configuration can be edited by making changes to the configuration in the editor and clicking the `Save` +button. + +To make changes to the device configuration form, click on the `File to Edit` selector and choose `Device Config`. + +|Reconfigure Option Selector| + +|Reconfigure Device Config| + + +Exporting Registry Configuration Files +-------------------------------------- + +The registry configuration file associated with a virtual device can be exported from the web browser to the computer's +local storage by clicking on the `File Export` Button in the device reconfiguration view. + +|File Export Button| + +.. |Add Devices| image:: files/01-add-devices.png +.. |Install Devices| image:: files/02-install-devices.png +.. |Start Scan| image:: files/03-start-scan.png +.. |Devices Found| image:: files/04-devices-found.png +.. |Get Device Points| image:: files/05-get-device-points.png +.. |Select Point Before| image:: files/07-select-point-a.png +.. |Select Point During| image:: files/07-select-point-b.png +.. |Select Point After| image:: files/07-select-point-c.png +.. |Edit Points| image:: files/07-edit-points.png +.. |Edit Point Button| image:: files/21-edit-point-button.png +.. |Edit Point Dialog| image:: files/22-edit-point-dialog.png +.. |Filter Points Button| image:: files/08-filter-points-button.png +.. |Filter Set| image:: files/09-filter-set.png +.. |Clear Filter| image:: files/10-clear-filter.png +.. |Add New Point| image:: files/11-add-new-point.png +.. |Add Point Dialog| image:: files/12-add-point-dialog.png +.. |Remove Points| image:: files/13-remove-points-button.png +.. |Confirm Remove Points| image:: files/14-confirm-remove-points.png +.. |Edit Columns| image:: files/15-edit-column-button.png +.. |Edit Column Menu| image:: files/16-edit-column-menu.png +.. |Name Column| image:: files/17-name-column.png +.. |Duplicated Column| image:: files/18-duplicated-column.png +.. |Find in Column| image:: files/19-find-in-column.png +.. |Find Next| image:: files/19-find-in-column-b.png +.. |Replace in Column| image:: files/20-replace-in-column.png +.. |Start Keyboard Commands| image:: files/23-start-keyboard-commands.png +.. |Keyboard Highlight| image:: files/24-keyboard-highlight.png +.. |Keyboard Select| image:: files/25-keyboard-select.png +.. |Keyboard Shortcuts Button| image:: files/26-keyboard-shortcuts-button.png +.. |Keyboard Shortcuts| image:: files/27-keyboard-shortcuts.png +.. |Save Registry Button| image:: files/28-save-registry-button.png +.. |Registry Preview Table| image:: files/29-registry-preview-table.png +.. |Registry Preview CSV| image:: files/30-preview-registry-csv.png +.. |Name Registry File| image:: files/31-name-registry-file.png +.. |Registry Saved| image:: files/32-registry-saved.png +.. |Select Saved Registry File| image:: files/38-select-saved-registry-file.png +.. |Saved Registry Selector| image:: files/39-saved-registry-selector.png +.. |File Import Button| image:: files/40-file-import-button.png +.. |Reload Points| image:: files/41-reload-points-from-device.png +.. |Configure Device Dialog| image:: files/33-configure-device-dialog.png +.. |Save Device Config| image:: files/34-save-device-config.png +.. |Device Added| image:: files/37-device-added.png +.. |Sub-device Path| image:: files/35-subdevice-path.png +.. |Sub-device 2| image:: files/36-subdevice2.png +.. |Device Added to Tree| image:: files/37-device-added-b.png +.. |Reconfigure Device Button| image:: files/43-reconfigure-device-button.png +.. |Reconfiguring Device| image:: files/44-reconfiguring-device.png +.. |Reconfigure Option Selector| image:: files/45-reconfigure-option-selector.png +.. |Reconfigure Device Config| image:: files/46-reconfigure-device-config.png +.. |File Export Button| image:: files/47-file-export-button.png diff --git a/docs/source/agent-framework/core-service-agents/volttron-central/volttron-central-platform.rst b/docs/source/agent-framework/core-service-agents/volttron-central/volttron-central-platform.rst new file mode 100644 index 0000000000..c28d37cfcf --- /dev/null +++ b/docs/source/agent-framework/core-service-agents/volttron-central/volttron-central-platform.rst @@ -0,0 +1,23 @@ +.. _VOLTTRON-Central-Platform-Agent: + +=============================== +VOLTTRON Central Platform Agent +=============================== + +The Platform Agent allows communication from a VOLTTRON Central instance. Each VOLTTRON instance that is to be +controlled through the VOLTTRON Central agent should have one and only one Platform Agent. The Platform Agent must have +the VIP identity of `platform.agent` which is specified by default by VOLTTRON +:ref:`known identities `. + + +Configuration +------------- + +The minimal configuration (and most likely the only used) for a Platform Agent is as follows: + +:: + + { + # Agent id is used in the display on volttron central. + "agentid": "Platform 1", + } diff --git a/docs/source/core_services/service_agents/central_management/VOLTTRON-Central.rst b/docs/source/agent-framework/core-service-agents/volttron-central/volttron-central.rst similarity index 66% rename from docs/source/core_services/service_agents/central_management/VOLTTRON-Central.rst rename to docs/source/agent-framework/core-service-agents/volttron-central/volttron-central.rst index aff2727a1b..5080a44c86 100644 --- a/docs/source/core_services/service_agents/central_management/VOLTTRON-Central.rst +++ b/docs/source/agent-framework/core-service-agents/volttron-central/volttron-central.rst @@ -1,17 +1,18 @@ .. _VOLTTRON-Central: +================================= VOLTTRON Central Management Agent -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +================================= + Agent Introduction ================== -The VOLTTRON Central Agent (VCM) is responsible for controlling multiple -VOLTTRON instances through a single interfaces. The VOLTTRON instances -can be either local or remote. VCM leverages an internal VOLTTRON web server -providing a interface to our JSON-RPC based web api. Both the web api and -the interface are served through the VCM agent. There is a :ref:`VOLTTRON Central Demo ` that will allow you to quickly setup and see the current offerings of the interface. -VOLTTRON Central will allow you to +The VOLTTRON Central Agent (VCM) is responsible for controlling multiple VOLTTRON instances through a single interfaces. +The VOLTTRON instances can be either local or remote. VCM leverages an internal VOLTTRON web server providing a +interface to our JSON-RPC based web API. Both the web api and the interface are served through the VCM agent. +There is a :ref:`VOLTTRON Central Demo ` that will allow you to quickly setup and see the current +offerings of the interface. VOLTTRON Central will allow you to: - See a list of platforms being managed. - Add and remove platforms. @@ -21,15 +22,15 @@ VOLTTRON Central will allow you to .. note:: - see :ref:`VCM json-rpc web api ` for how the web interface - works. + see :ref:`VCM JSON-RPC web API ` for how the web interface works. + Instance Configuration ====================== -In order for any web agent to be enabled, there must be a port configured to -serve the content. The easiest way to do this is to create a config file in -the root of your VOLTTRON_HOME directory. ( to do this automatically see :ref:`VOLTTRON Config ` ) +In order for any web agent to be enabled, there must be a port configured to serve the content. The easiest way to do +this is to create a config file in the root of your VOLTTRON_HOME directory ( to do this automatically see +:ref:`VOLTTRON Config `.) The following is an example of the configuration file @@ -39,22 +40,20 @@ The following is an example of the configuration file vip-addres=tcp://127.0.0.1:22916 bind-web-address=http://127.0.0.1:8080/vc/ -** Note the above configuration will open a discoverable port for the volttron - instance. In addition, the opening of this web address allows you to serve - both static as well as dynamic pages. +.. Note: -Verify that the instance is serving properly by pointing your web browser to + The above configuration will open a discoverable port for the volttron instance. In addition, the opening of this + web address allows you to serve both static as well as dynamic pages. -:: +Verify that the instance is serving properly by pointing your web browser to ``http://127.0.0.1:8080/discovery/`` - http://127.0.0.1:8080/discovery/ +This is the required information for a VolttronCentralPlatform to be able to be managed. -This is the required information for a VolttronCentralPlatform to be able to -be managed. VOLTTRON Central Manager Configuration ====================================== -The following is the default configuration file for VOLTTRON Central + +The following is the default configuration file for VOLTTRON Central: :: @@ -98,18 +97,25 @@ The following is the default configuration file for VOLTTRON Central } } + Agent Execution =============== -To start VOLTTRON Central first make sure the -`VOLTTRON instance is running <../../../devguides/eclipse/Eclipse-Dev-Environment.html#execute-volttron-through-shell>`__ -Next create/choose the config file to use. Finally from an activated -shell in the root of the VOLTTRON repository execute +To start VOLTTRON Central first make sure the :ref:`VOLTTRON instance is running `. Next +create/choose the config file to use. Finally from an activated shell in the root of the VOLTTRON repository execute: -:: +.. code-block:: bash # Arguments are package to execute, config file to use, tag to use as reference ./scripts/core/pack_install.sh services/core/VolttronCentral services/core/VolttronCentral/config vc # Start the agent vctl start --tag vc + + +.. toctree:: + + vc-device-configuration-demo + volttron-central-platform + webservice-api + diff --git a/docs/source/core_services/service_agents/central_management/Webservice-API.rst b/docs/source/agent-framework/core-service-agents/volttron-central/webservice-api.rst similarity index 94% rename from docs/source/core_services/service_agents/central_management/Webservice-API.rst rename to docs/source/agent-framework/core-service-agents/volttron-central/webservice-api.rst index 2959b6010e..a6a8dd80ce 100644 --- a/docs/source/core_services/service_agents/central_management/Webservice-API.rst +++ b/docs/source/agent-framework/core-service-agents/volttron-central/webservice-api.rst @@ -1,30 +1,28 @@ -.. _VCM-Webservice-API: +.. _VCM-Web-Service-API: =============================================== VOLTTRON Central Web Services Api Documentation =============================================== -VOLTTRON Central (VC) is meant to be the hub of communication within a cluster of -VOLTTRON instances. VC exposes a -`JSON-RPC 2.0 `_ based api that allows -a user to control multple instances of VOLTTRON. +VOLTTRON Central (VC) is meant to be the hub of communication within a cluster of VOLTTRON instances. VC exposes a +`JSON-RPC 2.0 `_ based API that allows a user to control multiple instances of +VOLTTRON. Why JSON-RPC ============ -SOAP messaging is unfriendly to many developers, especially those wanting to -make calls in a browser from AJAX environment. We have therefore have -implemented a JSON-RPC API capability to VC, as a more JSON/JavaScript -friendly mechanism. +SOAP messaging is unfriendly to many developers, especially those wanting to make calls in a browser from AJAX +environment. We have therefore have implemented a JSON-RPC API capability to VC, as a more JSON/JavaScript friendly +mechanism. How the API is Implemented ========================== * All calls are made through a POST to `/vc/jsonrpc` -* All calls (not including the call to authenticate) will - include an authorization token (a json-rpc extension). +* All calls (not including the call to authenticate) will include an authorization token (a json-rpc extension). + JSON-RPC Request Payload ------------------------ @@ -59,15 +57,15 @@ As an alternative, the params can be an array as illustrated by the following: "authorization": "server_authorization_token" } -For full documentation of the Request object please see section 4 of the +For full documentation of the `Request` object please see section 4 of the `JSON-RPC 2.0 `_ specification. + JSON-RPC Response Payload ------------------------- -All responses shall have either an either an error response or a result -response. The result key shown below can be a single instance of a json -type, an array or a JSON object. +All responses shall have either an either an error response or a result response. The result key shown below can be a +single instance of a JSON type, an array or a JSON object. A result response will have the following format: @@ -92,9 +90,10 @@ An error response will have the following format: "id": "sent_in_unique_message_id_or_null" } -For full documenation of the Response object please see section 5 of the +For full documentation of the Response object please see section 5 of the `JSON-RPC 2.0 `_ specification. + JSON-RPC Data Objects ===================== @@ -210,10 +209,13 @@ Retrieve Authorization Token } Failure - HTTP Status Code 401 + + :: + + HTTP Status Code 401 -Register A Volttron Platform Instance (Using Discovery) +Register a VOLTTRON Platform Instance (Using Discovery) .. code-block:: Python # POST /vc/jsonrpc @@ -324,7 +326,7 @@ Retrieve Managed Instances "id": # } -TODO: change repsonse Retrieve Installed Agents From platform1 +TODO: change response Retrieve Installed Agents From platform1 .. code-block:: Python # POST /vc/jsonrpc diff --git a/docs/source/agent-framework/example-agents/c-agent.rst b/docs/source/agent-framework/example-agents/c-agent.rst new file mode 100644 index 0000000000..c2108be229 --- /dev/null +++ b/docs/source/agent-framework/example-agents/c-agent.rst @@ -0,0 +1,48 @@ +.. _C-Agent: + +======= +C Agent +======= + +The C Agent uses the `ctypes` module to load a shared object into memory so its functions can be called from Python. + +There are two versions of the C Agent: + +* A standard agent that can be installed with the agent installation process +* A driver which can can be controlled using the Master Driver Agent + + +Building the Shared Object +-------------------------- + +The shared object library must be built before installing C Agent examples. Running ``make`` in the C Agent source +directory will compile the provided C code using the position independent flag, a requirement for creating shared +objects. + +Files created by make can be removed by running + +.. code-block:: bash + + make clean + + +Agent Installation +------------------ + +After building the shared object library the standard agent can be installed with the ``scripts/install-agent.py`` +script: + +.. code-block:: bash + + python scripts/install-agent.py -s examples/CAgent + +The other is a driver interface for the Master Driver. To use the C driver, the driver code file must be moved into +the Master Driver's `interfaces` directory: + + :: + + examples/CAgent/c_agent/driver/cdriver -> services/core/MasterDriverAgent/master_driver/interfaces + + +The C Driver configuration tells the interface where to find the shared object. An example is available in the C +Agent's `driver` directory. diff --git a/docs/source/agent-framework/example-agents/config-actuation.rst b/docs/source/agent-framework/example-agents/config-actuation.rst new file mode 100644 index 0000000000..b8e2fd8cdc --- /dev/null +++ b/docs/source/agent-framework/example-agents/config-actuation.rst @@ -0,0 +1,22 @@ +.. _Config-Actuation: + +======================== +Config Actuation Example +======================== + +The Config Actuation example attempts to set points on a device when files are added or updated in its +:ref:`configuration store `. + + +Configuration +------------- + +The name of a configuration file must match the name of the device to be actuated. The configuration file is a JSON +dictionary of point name and value pairs. Any number of points on the device can be listed in the config. + +.. code-block:: python + + { + "point0": value, + "point1": value + } diff --git a/docs/source/devguides/supporting/examples/CSVHistorianAgent.rst b/docs/source/agent-framework/example-agents/csv-historian.rst similarity index 58% rename from docs/source/devguides/supporting/examples/CSVHistorianAgent.rst rename to docs/source/agent-framework/example-agents/csv-historian.rst index cbf2a74f32..7aa02b9d29 100644 --- a/docs/source/devguides/supporting/examples/CSVHistorianAgent.rst +++ b/docs/source/agent-framework/example-agents/csv-historian.rst @@ -1,18 +1,17 @@ -.. _CSVHistorian: +.. _CSV-Historian: -CSVHistorian -============ +============= +CSV Historian +============= -The CSV Historian Agent is an example historian agent that writes device data -to the CSV file specified in the configuration file. +The CSV Historian Agent is an example historian agent that writes device data to the CSV file specified in the +configuration file. -This is the code created during Kyle Monson's presentation on VOLTTRON Historians -at the 2017 VOLTTRON Technical Meeting. -Explanation of CSVHistorian ---------------------------- +Explanation of CSV Historian +============================ -Setup logging for later. +The Utils module of the VOLTTRON platform includes functions for setting up global logging for the platform: .. code-block:: python @@ -20,9 +19,9 @@ Setup logging for later. _log = logging.getLogger(__name__) -The `historian` method is called by `utils.vip_main` when the agents is started (see below). `utils.vip_main` -expects a callable object that returns an instance of an Agent. This method -of dealing with a configuration file and instantiating an Agent is common practice. +The ``historian`` method is called by ``utils.vip_main`` when the agents is started (see below). ``utils.vip_main`` +expects a callable object that returns an instance of an Agent. This method of dealing with a configuration file and +instantiating an Agent is common practice. .. code-block:: python @@ -36,18 +35,18 @@ of dealing with a configuration file and instantiating an Agent is common practi return CSVHistorian(output_path = output_path, **kwargs) -All historians must inherit from `BaseHistorian`. The `BaseHistorian` class handles the capturing -and caching of all device, logging, analysis, and record data published to the message bus. +All historians must inherit from `BaseHistorian`. The `BaseHistorian` class handles the capturing and caching of all +device, logging, analysis, and record data published to the message bus. .. code-block:: python class CSVHistorian(BaseHistorian): -The Base Historian creates a separate thread to handle publishing data to the data store. In this thread -the Base Historian calls two methods on the created historian, `historian_setup` and `publish_to_historian`. +The Base Historian creates a separate thread to handle publishing data to the data store. In this thread the Base +Historian calls two methods on the created historian, ``historian_setup`` and ``publish_to_historian``. -The Base Historian created the new thread in it's `__init__` method. This means that any instance variables -must assigned in `__init__` before calling the Base Historian's `__init__` method. +The Base Historian created the new thread in it's ``__init__`` method. This means that any instance variables +must assigned in ``__init__`` before calling the Base Historian's ``__init__`` method. .. code-block:: python @@ -56,12 +55,11 @@ must assigned in `__init__` before calling the Base Historian's `__init__` metho self.csv_dict = None super(CSVHistorian, self).__init__(**kwargs) -Historian setup is called shortly after the new thread starts. This is where a Historian sets up a connect -the first time. In our example we create the Dictwriter object that we will use to create and add lines to the -CSV file. +Historian setup is called shortly after the new thread starts. This is where a Historian sets up a connect the first +time. In our example we create the `Dictwriter` object that we will use to create and add lines to the CSV file. -We keep a reference to the file object so that we may flush its contents to disk after writing the header -and after we have written new data to the file. +We keep a reference to the file object so that we may flush its contents to disk after writing the header and after we +have written new data to the file. The CSV file we create will have 4 columns: `timestamp`, `source`, `topic`, and `value`. @@ -73,8 +71,8 @@ The CSV file we create will have 4 columns: `timestamp`, `source`, `topic`, and self.csv_dict.writeheader() self.f.flush() -`publish_to_historian` is called when data is ready to be published. It is passed a list of dictionaries. -Each dictionary contains a record of a single value that was published to the message bus. +``publish_to_historian`` is called when data is ready to be published. It is passed a list of dictionaries. Each +dictionary contains a record of a single value that was published to the message bus. The dictionary takes the form: @@ -89,9 +87,9 @@ The dictionary takes the form: 'meta': {"units": "F", "tz": "UTC", "type": "float"} #Meta data published with the topic } -Once the data is written to the historian we call `self.report_all_handled()` to inform the `BaseHistorian` -that all data we received was successfully published and can be removed from the cache. Then we can flush the -file to ensure that the data is written to disk. +Once the data is written to the historian we call ``self.report_all_handled()`` to inform the `BaseHistorian` that all +data we received was successfully published and can be removed from the cache. Then we can flush the file to ensure +that the data is written to disk. .. code-block:: python @@ -111,11 +109,13 @@ file to ensure that the data is written to disk. This agent does not support the Historian Query interface. + Agent Testing ------------- The CSV Historian can be tested by running the included `launch_my_historian.sh` script. + Agent Installation ------------------ diff --git a/docs/source/devguides/supporting/utilities/DataPublisher.rst b/docs/source/agent-framework/example-agents/data-publisher.rst similarity index 85% rename from docs/source/devguides/supporting/utilities/DataPublisher.rst rename to docs/source/agent-framework/example-agents/data-publisher.rst index 9a81cdd890..e98fcbd0e7 100644 --- a/docs/source/devguides/supporting/utilities/DataPublisher.rst +++ b/docs/source/agent-framework/example-agents/data-publisher.rst @@ -1,19 +1,25 @@ -.. _DataPublisher: +.. _Data-Publisher: -============= -DataPublisher -============= +============== +Data Publisher +============== + +This is a simple agent that plays back data either from the config store or a CSV to the configured topic. It can also +provide basic emulation of the Actuator Agent for testing agents that expect to be able to set points on a device in +response to device publishes. -This is a simple agent that plays back data either from the config -store or a CSV to the configured topic. It can also provide basic -emulation of the actuator agent for testing agents that expect to -be able to set points on a device in response to device publishes. Installation notes ------------------ -In order to simulate the actuator you must install the agent -with the VIP identity of `platform.actuator`. +In order to simulate the actuator you must install the agent with the VIP identity of `platform.actuator`. If an +an actuator is already installed on the platform, this will cause VIP identity conflicts. To install the agent, the +agent install script can be used: + +.. code-block:: bash + + python scripts/install-agent.py -s examples/DataPublisher -c + Configuration ------------- @@ -74,19 +80,18 @@ Configuration "replay_data": false } + CSV File Format --------------- -The CSV file must have a single header line. The column names are appended to the -`basepath` setting in the configuration file and the resulting topic is normalized -to remove extra `/` characters. The values are all treated as floating -point values and converted accordingly. +The CSV file must have a single header line. The column names are appended to the `basepath` setting in the +configuration file and the resulting topic is normalized to remove extra` ``/`` characters. The values are all treated +as floating point values and converted accordingly. -The corresponding device for each point is determined and the values are combined -together to create an `all` topic publish for each device. +The corresponding device for each point is determined and the values are combined together to create an `all` topic +publish for each device. -If a `Timestamp` column is in the input it may be used to set the timestamp in the -header of the published data. +If a `Timestamp` column is in the input it may be used to set the timestamp in the header of the published data. .. csv-table:: Publisher Data :header: Timestamp,centrifugal_chiller/OutsideAirTemperature,centrifugal_chiller/DischargeAirTemperatureSetPoint,fuel_cell/DischargeAirTemperature,fuel_cell/CompressorStatus,absorption_chiller/SupplyFanSpeed,absorption_chiller/SupplyFanStatus,boiler/DuctStaticPressureSetPoint,boiler/DuctStaticPressure diff --git a/docs/source/supporting/examples/DDSAgent.rst b/docs/source/agent-framework/example-agents/dds-agent.rst similarity index 50% rename from docs/source/supporting/examples/DDSAgent.rst rename to docs/source/agent-framework/example-agents/dds-agent.rst index 50a4f3a83c..c25acfa7f9 100644 --- a/docs/source/supporting/examples/DDSAgent.rst +++ b/docs/source/agent-framework/example-agents/dds-agent.rst @@ -1,46 +1,41 @@ -.. _DDSAgent: +.. _DDS-Agent: ========= DDS Agent ========= -The DDS example agent demonstrates VOLTTRON's capacity to be extended -with tools and libraries not used in the core codebase. DDS is a messaging -platform that implements a publish-subscribe system for well defined data -types. +The DDS example agent demonstrates VOLTTRON's capacity to be extended with tools and libraries not used in the core +codebase. DDS is a messaging platform that implements a publish-subscribe system for well defined data types. -This agent example is meant to be run the command line, as opposed to -installing it like other agents. From the `examples/DDSAgent` directory, -the command to start it is: +This agent example is meant to be run the command line, as opposed to installing it like other agents. From the +`examples/DDSAgent` directory, the command to start it is: .. code-block:: shell $ AGENT_CONFIG=config python -m ddsagent.agent -The rticonnextdds-connector library needs to be installed for this example -to function properly. We'll retrieve it from GitHub since it is not available -through pip. Download the source with +The `rticonnextdds-connector` library needs to be installed for this example to function properly. We'll retrieve it +from GitHub since it is not available through Pip. Download the source with: .. code-block:: shell $ wget https://github.com/rticommunity/rticonnextdds-connector/archive/master.zip -and unpack it in `examples/DDSAgent/ddsagent` with +and unpack it in `examples/DDSAgent/ddsagent` with: .. code-block:: shell $ unzip master.zip -The `demo_publish()` output can be viewed with the rtishapesdemo available -from rti. +The ``demo_publish()`` output can be viewed with the `rtishapesdemo` available from RTI. + Configuration ------------- -Each data type that this agent will have access to needs to have an xml document -defining its structure. The xml will include a participant name, publisher name, -and a subscriber name. These are recorded in the configuration with the location -on disk of the xml file. +Each data type that this agent will have access to needs to have an XML document defining its structure. The XML will +include a participant name, publisher name, and a subscriber name. These are recorded in the configuration with the +location on disk of the XML file. .. code-block:: json diff --git a/docs/source/devguides/supporting/examples/files/cmd-image.png b/docs/source/agent-framework/example-agents/files/cmd-image.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/cmd-image.png rename to docs/source/agent-framework/example-agents/files/cmd-image.png diff --git a/docs/source/devguides/supporting/examples/files/cmd-image_2.png b/docs/source/agent-framework/example-agents/files/cmd-image_2.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/cmd-image_2.png rename to docs/source/agent-framework/example-agents/files/cmd-image_2.png diff --git a/docs/source/devguides/supporting/examples/files/env-vars-image_1.png b/docs/source/agent-framework/example-agents/files/env-vars-image_1.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/env-vars-image_1.png rename to docs/source/agent-framework/example-agents/files/env-vars-image_1.png diff --git a/docs/source/devguides/supporting/examples/files/env-vars-image_2.png b/docs/source/agent-framework/example-agents/files/env-vars-image_2.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/env-vars-image_2.png rename to docs/source/agent-framework/example-agents/files/env-vars-image_2.png diff --git a/docs/source/devguides/supporting/examples/files/extract-image_1.png b/docs/source/agent-framework/example-agents/files/extract-image_1.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/extract-image_1.png rename to docs/source/agent-framework/example-agents/files/extract-image_1.png diff --git a/docs/source/devguides/supporting/examples/files/extract-image_2.png b/docs/source/agent-framework/example-agents/files/extract-image_2.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/extract-image_2.png rename to docs/source/agent-framework/example-agents/files/extract-image_2.png diff --git a/docs/source/devguides/supporting/examples/files/github-image.png b/docs/source/agent-framework/example-agents/files/github-image.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/github-image.png rename to docs/source/agent-framework/example-agents/files/github-image.png diff --git a/docs/source/devguides/supporting/examples/files/github-zip-image.png b/docs/source/agent-framework/example-agents/files/github-zip-image.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/github-zip-image.png rename to docs/source/agent-framework/example-agents/files/github-zip-image.png diff --git a/docs/source/devguides/supporting/examples/files/matlab-agent-diagram.png b/docs/source/agent-framework/example-agents/files/matlab-agent-diagram.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/matlab-agent-diagram.png rename to docs/source/agent-framework/example-agents/files/matlab-agent-diagram.png diff --git a/docs/source/devguides/supporting/examples/files/node-red-flow.png b/docs/source/agent-framework/example-agents/files/node-red-flow.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/node-red-flow.png rename to docs/source/agent-framework/example-agents/files/node-red-flow.png diff --git a/docs/source/devguides/supporting/examples/files/node-red.png b/docs/source/agent-framework/example-agents/files/node-red.png similarity index 100% rename from docs/source/devguides/supporting/examples/files/node-red.png rename to docs/source/agent-framework/example-agents/files/node-red.png diff --git a/docs/source/agent-framework/example-agents/index.rst b/docs/source/agent-framework/example-agents/index.rst new file mode 100644 index 0000000000..f071612c7f --- /dev/null +++ b/docs/source/agent-framework/example-agents/index.rst @@ -0,0 +1,27 @@ +.. _Example-Agents: + +============== +Example Agents +============== + +Some example agents are included with the platform to help explore its features. These agents represent concrete +implementations of important agent sub-types such as Historians or Weather Agents, or demonstrate a development pattern +for accomplishing common tasks. + +More complex agents contributed by other researchers can also be found in the examples directory. It is recommended +that developers new to VOLTTRON understand the example agents first before diving into the other agents. + + +.. toctree:: + :maxdepth: 1 + + c-agent + config-actuation + csv-historian + data-publisher + dds-agent + listener-agent + matlab-agent + node-red + scheduler-example-agent + simple-web-agent-walk-through diff --git a/docs/source/devguides/supporting/examples/ListenerAgent.rst b/docs/source/agent-framework/example-agents/listener-agent.rst similarity index 63% rename from docs/source/devguides/supporting/examples/ListenerAgent.rst rename to docs/source/agent-framework/example-agents/listener-agent.rst index 0f921f6008..3b7673179b 100644 --- a/docs/source/devguides/supporting/examples/ListenerAgent.rst +++ b/docs/source/agent-framework/example-agents/listener-agent.rst @@ -1,15 +1,15 @@ -.. _ListenerAgent: +.. _Listener-Agent: -ListenerAgent -------------- +============== +Listener Agent +============== -The ListenerAgent subscribes to all topics and is useful for testing -that agents being developed are publishing correctly. It also provides a -template for building other agents as it expresses the requirements of a -platform agent. +The ListenerAgent subscribes to all topics and is useful for testing that agents being developed are publishing +correctly. It also provides a template for building other agents as it expresses the requirements of a platform agent. -Explanation of ListenerAgent -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Explanation of Listener Agent Code +================================== Use :code:`utils` to setup logging, which we’ll use later. @@ -19,20 +19,19 @@ Use :code:`utils` to setup logging, which we’ll use later. _log = logging.getLogger(__name__) -The Listener agent extends (inherits from) the Agent class for its -default functionality such as responding to platform commands: +The Listener agent extends (inherits from) the Agent class for its default functionality such as responding to platform +commands: .. code-block:: python class ListenerAgent(Agent): - '''Listens to everything and publishes a heartbeat according to the + ''' + Listens to everything and publishes a heartbeat according to the heartbeat period specified in the settings module. ''' -After the class definition, the Listener agent reads the configuration -file, extracts the configuration parameters, and initializes any -Listener agent instance variable. This is done through the agent's :code:`__init__` -method: +After the class definition, the Listener agent reads the configuration file, extracts the configuration parameters, and +initializes any Listener agent instance variable. This is done through the agent's ``__init__`` method: .. code-block:: python @@ -50,10 +49,9 @@ method: else: self._logfn = _log.info -Next, the Listener agent will run its setup method. This method is -tagged to run after the agent is initialized by the decorator -``@Core.receiver('onsetup')``. This method accesses the configuration -parameters, logs a message to the platform log, and sets the agent ID. +Next, the Listener agent will run its setup method. This method is tagged to run after the agent is initialized by the +decorator ``@Core.receiver('onsetup')``. This method accesses the configuration parameters, logs a message to the +platform log, and sets the agent ID. .. code-block:: python @@ -63,19 +61,11 @@ parameters, logs a message to the platform log, and sets the agent ID. _log.info(self.config.get('message', DEFAULT_MESSAGE)) self._agent_id = self.config.get('agentid') -The Listener agent subscribes to all topics published on the message -bus. Publish and subscribe interactions with the message bus are handled by -the PubSub module located at: - - ``~/volttron/volttron/platform/vip/agent/subsystems/pubsub.py`` +The Listener agent subscribes to all topics published on the message bus. Publish and subscribe interactions with the +message bus are handled by the `PubSub` module located at `~/volttron/volttron/platform/vip/agent/subsystems/pubsub.py`. -The Listener agent uses an empty string to subscribe to all messages -published. This is done in a -`decorator `__ -for simplifying subscriptions. - -It also checks for the sender being ``pubsub.compat`` in case there are -any VOLTTRON 2.0 agents running on the platform. +The Listener agent uses an empty string to subscribe to all messages published. This is done in a +`decorator `_ for simplifying subscriptions. .. code-block:: python @@ -87,5 +77,3 @@ any VOLTTRON 2.0 agents running on the platform. self._logfn( "Peer: %r, Sender: %r:, Bus: %r, Topic: %r, Headers: %r, " "Message: %r", peer, sender, bus, topic, headers, message) - - diff --git a/docs/source/agent-framework/example-agents/matlab-agent.rst b/docs/source/agent-framework/example-agents/matlab-agent.rst new file mode 100644 index 0000000000..88ed164fab --- /dev/null +++ b/docs/source/agent-framework/example-agents/matlab-agent.rst @@ -0,0 +1,606 @@ +.. _Matlab-Agent: + +============ +MatLab Agent +============ + +The MatLab agent and Matlab Standalone Agent together are example agents that allow for MatLab scripts to be run in a +Windows environment and interact with the VOLTTRON platform running in a Linux environment. + +The MatLab agent takes advantage of the config store to dynamically send scripts and commandline arguments across the +message bus to one or more Standalone Agents in Windows. The Standalone Agent then executes the requested script and +arguments, and sends back the results to the MatLab agent. + + +Overview of Matlab Agents +========================= + +There are multiple components that are used for the MatLab agent. This diagram is to represent the components that are +connected to the MatLab Agents. In this example, the scripts involved are based on the default settings in the MatLab +Agent. + +|matlab-agent-diagram| + + +MatLabAgentV2 +------------- + +MatLabAgentV2 publishes the name of a python script along with any command line arguments that are needed for the script +to the appropriate topic. The agent then listens on another topic, and whenever anything is published on this topic, it +stores the message in the log file chosen when the VOLTTRON instance is started. If there are multiple standalone +agents, the agent can send a a script to each of them, along with their own set of command line arguments. In this +case, each script name and set of command line arguments should be sent to separate subtopics. This is done so that no +matter how many standalone agents are in use, MatLabAgentV2 will record all of their responses. + +.. code:: + + class MatlabAgentV2(Agent): + + def __init__(self,script_names=[], script_args=[], topics_to_matlab=[], + topics_to_volttron=None,**kwargs): + + super(MatlabAgentV2, self).__init__(**kwargs) + _log.debug("vip_identity: " + self.core.identity) + + self.script_names = script_names + self.script_args = script_args + self.topics_to_matlab = topics_to_matlab + self.topics_to_volttron = topics_to_volttron + self.default_config = {"script_names": script_names, + "script_args": script_args, + "topics_to_matlab": topics_to_matlab, + "topics_to_volttron": topics_to_volttron} + + + #Set a default configuration to ensure that self.configure is called immediately to setup + #the agent. + self.vip.config.set_default("config", self.default_config) + #Hook self.configure up to changes to the configuration file "config". + self.vip.config.subscribe(self.configure, actions=["NEW", "UPDATE"], pattern="config") + + def configure(self, config_name, action, contents): + """ + Called after the Agent has connected to the message bus. + If a configuration exists at startup this will be + called before onstart. + Is called every time the configuration in the store changes. + """ + config = self.default_config.copy() + config.update(contents) + + _log.debug("Configuring Agent") + + try: + script_names = config["script_names"] + script_args = config["script_args"] + topics_to_matlab = config["topics_to_matlab"] + topics_to_volttron = config["topics_to_volttron"] + + except ValueError as e: + _log.error("ERROR PROCESSING CONFIGURATION: {}".format(e)) + return + + self.script_names = script_names + self.script_args = script_args + self.topics_to_matlab = topics_to_matlab + self.topics_to_volttron = topics_to_volttron + self._create_subscriptions(self.topics_to_volttron) + + for script in range(len(self.script_names)): + cmd_args = "" + for x in range(len(self.script_args[script])): + cmd_args += ",{}".format(self.script_args[script][x]) + _log.debug("Publishing on: {}".format(self.topics_to_matlab[script])) + self.vip.pubsub.publish('pubsub', topic=self.topics_to_matlab[script], + message="{}{}".format(self.script_names[script],cmd_args)) + _log.debug("Sending message: {}{}".format(self.script_names[script],cmd_args)) + + _log.debug("Agent Configured!") + +For this example, the agent is publishing to the `matlab/to_matlab/1` topic, and is listening to the +`matlab/to_volttron` topic. It is sending the script name `testScript.py` with the argument 20. These are the default +values found in the agent, if no configuration is loaded. + +.. code:: + + script_names = config.get('script_names', ["testScript.py"]) + script_args = config.get('script_args', [["20"]]) + topics_to_matlab = config.get('topics_to_matlab', ["matlab/to_matlab/1"]) + topics_to_volttron = config.get('topics_to_volttron', "matlab/to_volttron/") + + +StandAloneMatLab.py +------------------- + +The `StandAloneMatLab.py` script is a standalone agent designed to be able to run in a Windows environment. Its purpose +is to listen to a topic, and when something is published to this topic, it takes the message, and sends it to the +``script_runner`` function in `scriptwrapper.py`. This function processes the inputs, and then the output is published +to another topic. + +.. code:: + + class StandAloneMatLab(Agent): + '''The standalone version of the MatLab Agent''' + + @PubSub.subscribe('pubsub', _topics['volttron_to_matlab']) + def print_message(self, peer, sender, bus, topic, headers, message): + print('The Message is: ' + str(message)) + messageOut = script_runner(message) + self.vip.pubsub.publish('pubsub', _topics['matlab_to_volttron'], message=messageOut) + + +settings.py +----------- + +The topic to listen to and the topic to publish to are defined in `settings.py`, along with the information needed to +connect the Standalone Agent to the primary VOLTTRON instance. These should be the same topics that the MatLabAgentV2 +is publishing and listening to, so that the communication can be successful. To connect the Standalone Agent to the +primary VOLTTRON instance, the IP address and port of the instance are needed, along with the server key. + +.. code:: + + _topics = { + 'volttron_to_matlab': 'matlab/to_matlab/1', + 'matlab_to_volttron': 'matlab/to_volttron/1' + } + + # The parameters dictionary is used to populate the agent's + # remote vip address. + _params = { + # The root of the address. + # Note: + # 1. volttron instance should be configured to use tcp. use command vcfg + # to configure + 'vip_address': 'tcp://192.168.56.101', + 'port': 22916, + + # public and secret key for the standalone_matlab agent. + # These can be created using the command: volttron-ctl auth keypair + # public key should also be added to the volttron instance auth + # configuration to enable standalone agent access to volttron instance. Use + # command 'vctl auth add' Provide this agent's public key when prompted + # for credential. + + 'agent_public': 'dpu13XKPvGB3XJNVUusCNn2U0kIWcuyDIP5J8mAgBQ0', + 'agent_secret': 'Hlya-6BvfUot5USdeDHZ8eksDkWgEEHABs1SELmQhMs', + + # Public server key from the remote platform. This can be + # obtained using the command: + # volttron-ctl auth serverkey + 'server_key': 'QTIzrRGQ0-b-37AbEYDuMA0l2ETrythM2V1ac0v9CTA' + + } + + def remote_url(): + return "{vip_address}:{port}?serverkey={server_key}" \ + "&publickey={agent_public}&" \ + "secretkey={agent_secret}".format(**_params) + +The primary VOLTTRON instance will then need to add the public key from the Standalone Agent. In this example, the +topic that the Standalone Agent is listening to is `matlab/to_matlab/1`, and the topic it is publishing to is +`matlab/to_volttron/1`. + + +scriptwrapper.py +---------------- + +`Scriptwrapper.py` contains the script_runner function. The purpose of this function is to take in a string that +contains a Python script and command line arguments separated by commas. This string is parsed and passed to the system +arguments, which allows the script sent to the function to use the command line arguments. The function then redirects +standard output to a `StringIO` file object, and then attempts to execute the script. If there are any errors with the +script, the error that is generated is returned to the standalone agent. Otherwise, the file object stores the output +from the script, is converted to a string, and is sent to the standalone agent. In this example, the script that is to +be run is `testScript.py`. + +.. code:: + + #Script to take in a string, run the program, + #and output the results of the command as a string. + + import time + import sys + from io import StringIO + + + def script_runner(message): + original = sys.stdout + # print(message) + # print(sys.argv) + sys.argv = message.split(',') + # print(sys.argv) + + try: + out = StringIO() + sys.stdout = out + exec(open(sys.argv[0]).read()) + sys.stdout = original + return out.getvalue() + except Exception as ex: + out = str(ex) + sys.stdout = original + return out + +.. note:: + + The script that is to be run needs to be in the same folder as the agent and the `scriptwrapper.py` script. The + `script_runner` function needs to be edited if it is going to call a script at a different location. + + +testScript.py +------------- + +This is a very simple test script designed to demonstrate the calling of a MatLab function from within Python. First it +initializes the MatLab engine for Python. It then takes in a single command line argument, and passes it to the MatLab +function `testPy.m`. If no arguments are sent, it will send 0 to the `testPy.m` function. It then prints the result of +the `testPy.m` function. In this case, since standard output is being redirected to a file object, this is how the +result is passed from this function to the Standalone Agent. + +.. code-block:: python + + import matlab.engine + import sys + + + eng = matlab.engine.start_matlab() + + if len(sys.argv) == 2: + result = eng.testPy(float(sys.argv[1])) + else: + result = eng.testPy(0.0) + + print(result) + + +testPy.m +-------- + +This MatLab function is a very simple example, designed to show a function that takes an argument, and produces an array +as the output. The input argument is added to each element in the array, and the entire array is then returned. + +.. code:: + + function out = testPy(z) + x = 1:100 + out = x + z + end + + +Setup on Linux +-------------- + +1. Setup and run VOLTTRON from develop branch using instructions :ref:`here `. + +2. Configure volttron instance using the ``vcfg`` command. When prompted for the vip address use + ``tcp://``. This is necessary to enable volttron communication with external + processes. + + .. note:: + + If you are running VOLTTRON from within VirtualBox, jit would be good to set one of your adapters as a + `Host-only` adapter. This can be done within the VM's settings, under the `Network` section. Once this is + done, use this IP for the VIP address. + + +.. _Matlab-Agent-Config: + +3. Update the configuration for MatLabAgent_v2 at `/example/MatLabAgent_v2/config`. + + The configuration file for the MatLab agent has four variables. + + 1. script_names + + 2. script_args + + 3. topics_to_matlab + + 4. topics_to_volttron + + An example config file included with the folder. + + .. code:: + + { + # VOLTTRON config files are JSON with support for python style comments. + "script_names": ["testScript.py"], + "script_args": [["20"]], + "topics_to_matlab": ["matlab/to_matlab/1"], + "topics_to_volttron": "matlab/to_volttron/" + } + + To edit the configuration, the format should be as follows: + + .. code-block:: json + + { + "script_names": ["script1.py", "script2.py", "..."], + "script_args": [["arg1","arg2"], ["arg1"], ["..."]], + "topics_to_matlab": ["matlab/to_matlab/1", "matlab/to_matlab/2", "..."], + "topics_to_volttron": "matlab/to_volttron/" + } + + The config requires that each script name lines up with a set of commandline arguments and a topic. A + commandline argument must be included, even if it is not used. The placement of brackets are important, even when + only communicating with one standalone agent. + + For example, if only one standalone agent is used, and no command line arguments are in place, the config file may + look like this. + + .. code-block:: json + + { + "script_names": ["testScript.py"], + "script_args": [["0"]], + "topics_to_matlab": ["matlab/to_matlab/1"], + "topics_to_volttron": "matlab/to_volttron/" + } + + +4. Install MatLabAgent_v2 and start agent (from volttron root directory) + + .. code-block:: bash + + python ./scripts/install-agent.py -s examples/MatLabAgent_v2 --start + + .. note:: + + The MatLabAgent_v2 publishes the command to be run to the message bus only on start or on a configuration + update. Once we configure the `standalone_matlab` agent on the Windows machine, we will send a configuration + update to the running MatLabAgent_v2. The configuration would contain the topics to which the Standalone Agent + is listening to and will be publishing result to. + + .. seealso:: + + The MatLab agent uses the configuration store to dynamically change inputs. More information on the config + store and how it used can be found here. + + * :ref:`VOLTTRON Configuration Store ` + + * :ref:`Agent Configuration Store ` + + * :ref:`Agent Configuration Store Interface ` + +5. Run the below command and make a note of the server key. This is required for configuring the stand alone agent + on Windows. (This is run on the linux machine) + + .. code-block:: bash + + vctl auth serverkey + + +Setup on Windows +---------------- + +Install pre-requisites +^^^^^^^^^^^^^^^^^^^^^^ + +1. Install Python3.6 64-bit from the `Python website `_. + +2. Install the MatLab engine from + `MathWorks `_. + + .. warning:: + + The MatLab engine for Python only supports certain version of Python depending on the version of MatLab used. + Please check `here `__ to see if the current + version of MatLab supports your version of Python. + + +.. note:: + + At this time, you may want to verify that you are able to communicate with your Linux machine across your network. + The simplest method would be to open up the command terminal and use ``ping ``, and ``telnet + ``. Please make sure that the port is + opened for outside access. + + +Install Standalone MatLab Agent +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The standalone MatLab agent is designed to be usable in a Windows environment. + +.. warning:: + + VOLTTRON is not designed to run in a Windows environment. Outside of cases where it is stated to be usable in a + Windows environment, it should be assumed that it will **NOT** function as expected. + +#. Download VOLTTRON + + Download the VOLTTRON develop repository from Github. Download the zip from + `GitHub `_. + + |github-image| + + |github-zip-image| + + Once the zipped file has been downloaded, go to your `Downloads` folder, right-click on the file, and select + `Extract All...` + + |extract-image_1| + + Choose a location for the extracted folder, and select "Extract" + + |extract-image_2| + +#. Setup the `PYTHONPATH` + + Open the Windows explorer, and navigate to `Edit environment variables for your account`. + + |cmd-image| + + Select "New" + + |env-vars-image_1| + + For "Variable name" enter: ``PYTHONPATH`` + For "Variable value" either browse to your VOLTTRON installation, or enter in the path to your VOLTTRON + installation. + + |env-vars-image_2| + + Select `OK` twice. + +#. Set Python version in MatLab + + Open your MatLab application. Run the command: + + .. code-block:: bash + + pyversion + + This should print the path to Python2.7. If you have multiple versions of python on your machine and `pyversion` + points to a different version of Python, use: + + .. code-block:: bash + + pyversion /path/to/python.exe + + to set the appropriate version of python for your system. + + For example, to use python 3.6 with MatLab: + + .. code-block:: console + + pyversion C:\Python36\python.exe + +#. Set up the environment. + + Open up the command prompt + + |cmd-image_2| + + Navigate to your VOLTTRON installation + + ``cd \Your\directory\path\to\volttron-develop`` + + Use pip to install and setup dependencies. + + ``pip install -r examples\StandAloneMatLab\requirements.txt`` + + ``pip install -e .`` + + .. note:: + + If you get the error doing the second step because of an already installed volttron from a different directory, + manually delete the `volttron-egg.` link file from your `\\Lib\\site-packages` directory (for + example: + + .. code-block:: bash + + del C:\\Python27\\lib\\site-packages\\volttron-egg.link + + and re-run the second command + +#. Configure the agent + + The configuration settings for the standalone agent are in setting.py (located in + `volttron-develop\\examples\\StandAloneMatLab\\`) + + **settings.py** + + * `volttron_to_matlab` needs to be set to the topic that will send your script and command line arguments to your + stand alone agent. This was defined in the :ref:`config. ` + + * `matlab_to_volttron` needs to be set to the topic that will send your script's + output back to your volttron platform. This was defined in :ref:`config. ` + + * `vip_address` needs to be set to the address of your volttron instance + + * `port` needs to be set to the port of your volttron instance + + * `server_key` needs to be set to the public server key of your primary volttron platform. This can be obtained + from the primary volttron platform using ``vctl auth serverkey`` (VOLTTRON must be running to use this command.) + + It is possible to have multiple standalone agents running. In this case, copy the `StandAloneMatLab` folder, and + make the necessary changes to the new `settings.py` file. Unless it is connecting to a separate VOLTTRON instance, + you should only need to change the `volttron_to_matlab` setting. + + .. note:: + + It is recommended that you generate a new "agent_public" and "agent_private" key for your standalone agent. + This can be done using the ``vctl auth keypair`` command on your primary VOLTTRON platform on Linux. If you + plan to use multiple standalone agents, they will each need their own keypair. + +6. Add standalone agent key to VOLTTRON platform + + * Copy the public key from `settings.py` in the StandAloneMatLab folder. + + * While the primary VOLTTRON platform is running on the linux machine, add the agent public key using the ``vctl + auth`` command on the Linux machine. This will make VOLTTRON platform allow connections from the standalone agent + + .. code-block:: bash + + vctl auth add --credentials + +7. Run standalone agent + + + At this point, the agent is ready to run. To use the agent, navigate to the example folder and use python to start + the agent. The agent will then wait for a message to be published to the selected topic by the MatLab agent. + + .. code-block:: bash + + cd examples\StandAloneMatLab\ + + python standalone_matlab.py + + The output should be similar to this: + + .. code-block:: console + + 2019-08-01 10:42:47,592 volttron.platform.vip.agent.core DEBUG: identity: standalone_matlab + 2019-08-01 10:42:47,592 volttron.platform.vip.agent.core DEBUG: agent_uuid: None + 2019-08-01 10:42:47,594 volttron.platform.vip.agent.core DEBUG: serverkey: None + 2019-08-01 10:42:47,596 volttron.platform.vip.agent.core DEBUG: AGENT RUNNING on ZMQ Core standalone_matlab + 2019-08-01 10:42:47,598 volttron.platform.vip.zmq_connection DEBUG: ZMQ connection standalone_matlab + 2019-08-01 10:42:47,634 volttron.platform.vip.agent.core INFO: Connected to platform: router: ebae9efa-5e8f-49e3-95a0-2020ddff9e8a version: 1.0 identity: standalone_matlab + 2019-08-01 10:42:47,634 volttron.platform.vip.agent.core DEBUG: Running onstart methods. + + + .. note:: + + If you have Python3 as your default Python run the command ``python -2 standalone_matlab.py`` + +8. On the Linux machine configure the Matlab Agent to publish commands to the topic standalone agent is listening to. +To load a new configuration or to change the current configuration enter + + .. code-block:: bash + + vctl config store config + + Whenever there is a change in the configuration in the config store, or whenever the agent starts, the MatLab Agent + sends the configured command to the topic configured. As long as the standalone agent has been started and is + listening to the appropriate topic, the output in the log should look similar to this: + + .. code:: + + 2019-08-01 10:43:18,925 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Configuring Agent + 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Publishing on: matlab/to_matlab/1 + 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Sending message: testScript2.py,20 + 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Agent Configured! + 2019-08-01 10:43:18,979 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent INFO: Agent: matlab/to_volttron/1 + Message: + '20' + + Once the matlab agent publishes the message (in the above case, "testScript2.py,20") on the windows command prompt + running the standalone agent, you should see the message that was received by the standalone agent. + + .. code:: + + 2019-08-01 10:42:47,671 volttron.platform.vip.agent.subsystems.configstore DEBUG: Processing callbacks for affected files: {} + The Message is: testScript2.py,20 + + .. note:: + + If MatLabAgent_v2 has been installed and started, and you have not started the `standalone_matlab agent`, you + will need to either restart the matlab_agentV2, or make a change to the configuration in the config store to + send command to the topic standalone agent is actively listening to. + +.. |github-image| image:: files/github-image.png +.. |cmd-image| image:: files/cmd-image.png +.. |env-vars-image_1| image:: files/env-vars-image_1.png +.. |env-vars-image_2| image:: files/env-vars-image_2.png +.. |cmd-image_2| image:: files/cmd-image_2.png +.. |github-zip-image| image:: files/github-zip-image.png +.. |extract-image_1| image:: files/extract-image_1.png +.. |extract-image_2| image:: files/extract-image_2.png +.. |matlab-agent-diagram| image:: files/matlab-agent-diagram.png diff --git a/docs/source/devguides/supporting/examples/NodeRed.rst b/docs/source/agent-framework/example-agents/node-red.rst similarity index 67% rename from docs/source/devguides/supporting/examples/NodeRed.rst rename to docs/source/agent-framework/example-agents/node-red.rst index 132e7eee32..523c5718f5 100644 --- a/docs/source/devguides/supporting/examples/NodeRed.rst +++ b/docs/source/agent-framework/example-agents/node-red.rst @@ -1,45 +1,44 @@ -.. _NodeRed: +.. _Node-Red: +================ Node Red Example ================ -Node Red is a visual programming wherein users connect small units of -functionality "nodes" to create "flows". +Node Red is a visual programming language wherein users connect small units of functionality "nodes" to create "flows". + +There are two example nodes that allow communication between Node-Red and VOLTTRON. One node reads subscribes to +messages on the VOLTTRON message bus and the other publishes to it. -There are two example nodes that allow communication between Node Red and -VOLTTRON. One node reads subscribes to messages on the VOLTTRON message bus -and the other publishes to it. Dependencies ------------ -The example nodes depend on `python-shell` to be installed and available to -the Node Red environment. +The example nodes depend on `python-shell` to be installed and available to the Node Red environment. + Installation ------------ -Copy all files from `volttron/examples/NodeRed` to your `~/.node-red/nodes` -directory. `~/.node-red` is the default directory for Node Red files. If you -have set a different directory use that instead. +Copy all files from `volttron/examples/NodeRed` to your `~/.node-red/nodes` directory. `~/.node-red` is the default +directory for Node Red files. If you have set a different directory use that instead. -Set the variables at the beginning of the `volttron.js` file to be a valid -VOLTTRON environment, VOLTTRON home, and python path. +Set the variables at the beginning of the `volttron.js` file to be a valid VOLTTRON environment, VOLTTRON home, and +Python PATH. -Valid CURVE keys need to be added to the `settings.py` file. If they are -generated with the `vctl auth keypair` command then the public key -should be added to VOLTTRON's authorization file with the following: +Valid CURVE keys need to be added to the `settings.py` file. If they are generated with the `vctl auth keypair` command +then the public key should be added to VOLTTRON's authorization file with the following: .. code-block:: console $ vctl auth add -The serverkey can be found with +The serverkey can be found with: .. code-block:: console $ vctl auth serverkey + Usage ----- @@ -67,8 +66,8 @@ Start VOLTTRON and Node Red. 11 Jan 15:26:49 - [info] Starting flows 11 Jan 15:26:49 - [info] Started flows -The output from the Node Red command indicates the address of its web -interface. Nodes available for use are in the left sidebar. +The output from the Node Red command indicates the address of its web interface. Nodes available for use are in the +left sidebar. |Node Red| diff --git a/docs/source/agent-framework/example-agents/scheduler-example-agent.rst b/docs/source/agent-framework/example-agents/scheduler-example-agent.rst new file mode 100644 index 0000000000..83817ae8e8 --- /dev/null +++ b/docs/source/agent-framework/example-agents/scheduler-example-agent.rst @@ -0,0 +1,57 @@ +.. _Scheduler-Example-Agent: + +======================= +Scheduler Example Agent +======================= + +The Scheduler Example Agent demonstrates how to use the scheduling feature of the :ref`Actuator Agent ` +as well as how to send a command. This agent publishes a request for a reservation on a (fake) device then takes an +action when it's scheduled time appears. The ActuatorAgent must be running to exercise this example. + +.. Note:: + + Since there is no actual device, an error is produced when the agent attempts to take its action. + +.. code-block:: python + + def publish_schedule(self): + '''Periodically publish a schedule request''' + headers = { + 'AgentID': agent_id, + 'type': 'NEW_SCHEDULE', + 'requesterID': agent_id, #The name of the requesting agent. + 'taskID': agent_id + "-ExampleTask", #The desired task ID for this task. It must be unique among all other scheduled tasks. + 'priority': 'LOW', #The desired task priority, must be 'HIGH', 'LOW', or 'LOW_PREEMPT' + } + + start = str(datetime.datetime.now()) + end = str(datetime.datetime.now() + datetime.timedelta(minutes=1)) + + + msg = [ + ['campus/building/unit',start,end] + ] + self.vip.pubsub.publish( + 'pubsub', topics.ACTUATOR_SCHEDULE_REQUEST, headers, msg) + +The agent listens to schedule announcements from the actuator and then issues a command: + +.. code-block:: python + + @PubSub.subscribe('pubsub', topics.ACTUATOR_SCHEDULE_ANNOUNCE(campus='campus', + building='building',unit='unit')) + def actuate(self, peer, sender, bus, topic, headers, message): + print ("response:",topic,headers,message) + if headers[headers_mod.REQUESTER_ID] != agent_id: + return + '''Match the announce for our fake device with our ID + Then take an action. Note, this command will fail since there is no + actual device''' + headers = { + 'requesterID': agent_id, + } + self.vip.pubsub.publish( + 'pubsub', topics.ACTUATOR_SET(campus='campus', + building='building',unit='unit', + point='point'), + headers, 0.0) diff --git a/docs/source/agent-framework/example-agents/simple-web-agent-walk-through.rst b/docs/source/agent-framework/example-agents/simple-web-agent-walk-through.rst new file mode 100644 index 0000000000..adcf642274 --- /dev/null +++ b/docs/source/agent-framework/example-agents/simple-web-agent-walk-through.rst @@ -0,0 +1,71 @@ +.. _Simple-Web-Agent-Walk-through: + +============================= +Simple Web Agent Walk-through +============================= + +A simple web enabled agent that will hook up with a VOLTTRON message bus and allow interaction between it via HTTP. +This example agent shows a simple file serving agent, a JSON-RPC based call, and a websocket based connection mechanism. + + +Starting VOLTTRON Platform +-------------------------- + +.. note:: + + Activate the environment first :ref:`active the environment ` + +In order to start the simple web agent, we need to bind the VOLTTRON instance to the a web server. We need to specify +the address and the port for the web server. For example, if we want to bind the `localhost:8080` as the web server +we start the VOLTTRON platform as follows: + +.. code-block:: bash + + ./start-volttron --bind-web-address http://127.0.0.1:8080 + +Once the platform is started, we are ready to run the Simple Web Agent. + + +Running Simple Web Agent +------------------------ + +.. note:: + + The following assumes the shell is located at the :ref:`VOLTTRON_ROOT`. + +Copy the following into your shell (save it to a file for executing it again later): + +.. code-block:: console + + python scripts/install-agent.py \ + --agent-source examples/SimpleWebAgent \ + --tag simpleWebAgent \ + --vip-identity webagent \ + --force \ + --start + +This will create a web server on ``http://localhost:8080``. The `index.html` file under `simpleweb/webroot/simpleweb/` +can be any HTML page which binds to the VOLTTRON message bus .This provides a simple example of providing a web endpoint +in VOLTTRON. + + +Path based registration examples +-------------------------------- + +- Files will need to be in `webroot/simpleweb` in order for them to be browsed from + ``http://localhost:8080/simpleweb/index.html`` + +- Filename is required as we don't currently auto-redirect to any default pages as shown in + ``self.vip.web.register_path("/simpleweb", os.path.join(WEBROOT))`` + +The following two examples show the way to call either a JSON-RPC (default) endpoint and one that returns a different +content-type. With the JSON-RPC example from volttron central we only allow post requests, however this is not +required. + +- Endpoint will be available at `http://localhost:8080/simple/text` + ``self.vip.web.register_endpoint("/simple/text", self.text)`` + +- Endpoint will be available at `http://localhost:8080/simple/jsonrpc` + ``self.vip.web.register_endpoint("/simpleweb/jsonrpc", self.rpcendpoint)`` +- ``text/html`` content type specified so the browser can act appropriately like ``[("Content-Type", "text/html")]`` +- The default response is ``application/json so our`` endpoint returns appropriately with a JSON based response. diff --git a/docs/source/agent-framework/historian-agents/crate/crate-historian.rst b/docs/source/agent-framework/historian-agents/crate/crate-historian.rst new file mode 100644 index 0000000000..bb12a915ae --- /dev/null +++ b/docs/source/agent-framework/historian-agents/crate/crate-historian.rst @@ -0,0 +1,79 @@ +.. _Crate-Historian: + +=============== +Crate Historian +=============== + +Crate is an open source SQL database designed on top of a No-SQL design. It allows automatic data replication and +self-healing clusters for high availability, automatic sharding, and fast joins, aggregations and sub-selects. + +Find out more about crate from ``_. + + +Prerequisites +============= + +1. Crate Database +----------------- + +For Arch Linux, Debian, RedHat Enterprise Linux and Ubuntu distributions there is a simple installer to get Crate up and +running on your system. + +.. code-block:: bash + + sudo bash -c "$(curl -L https://try.crate.io)" + +This command will download and install all of the requirements for running Crate, create a Crate user and install a +Crate service. After the installation the service will be available for viewing at ``http://localhost:4200`` by +default. + +.. note:: + + There is no authentication support within crate. + + +2. Crate Driver +--------------- + +There is a Python library for crate that must be installed in the VOLTTRON Python virtual environment in order to access +Crate. From an activated environment, in the root of the volttron folder, execute the following command: + +.. code-block:: bash + + python bootstrap.py --crate + +or + +.. code-block:: bash + + python bootstrap.py --databases + + +or + +.. code-block:: bash + + pip install crate + + +Configuration +============= + +Because there is no authorization to access a crate database the configuration for the Crate Historian is very easy. + +.. code-block:: python + + { + "connection": { + "type": "crate", + # Optional table prefix defaults to historian + "schema": "testing", + "params": { + "host": "localhost:4200" + } + } + } + +Finally package, install and start the Crate Historian agent. + +.. seealso:: :ref:`Agent Development Walk-through ` diff --git a/docs/source/agent-framework/historian-agents/data-mover/data-mover-historian.rst b/docs/source/agent-framework/historian-agents/data-mover/data-mover-historian.rst new file mode 100644 index 0000000000..ad0d283015 --- /dev/null +++ b/docs/source/agent-framework/historian-agents/data-mover/data-mover-historian.rst @@ -0,0 +1,55 @@ +.. _Data-Mover-Historian: + +==================== +Data Mover Historian +==================== + +The Data Mover sends data from its platform to a remote platform in cases where there are not sufficient resources to +store data locally. It shares this functionality with the :ref:`Forward Historian `, however the +Data Mover does not have the goal of data appearing "live" on the remote platform. This allows DataMover to be more +efficient by both batching data and by sending an RPC call to a remote historian instead of publishing data on the +remote message bus. This allows allows the Data Mover to be more robust by ensuring that the receiving historian is +running. If the target is unreachable, the Data Mover will cache data until it is available. + + +Configuration +============= + +The default configuration file is `services/core/DataMover/config`. Change the `destination-vip` value to +point towards the foreign Volttron instance. + +The following is an example configuration: + +:: + + { + "destination-vip": "ipc://@/home/volttron/.volttron/run/vip.socket", + "destination-serverkey": null, + "required_target_agents": [], + "custom_topic_list": [], + "services_topic_list": [ + "devices", "analysis", "record", "datalogger", "actuators" + ], + "topic_replace_list": [ + #{"from": "FromString", "to": "ToString"} + ] + } + + +The `services_topic_list` allows you to specify which of the main data topics to forward. If there is no entry, the +historian defaults to sending all. + +`topic_replace_list` allows you to replace portions of topics if needed. This could be used to correct or standardize +topics or to replace building/device names with an anonymous version. The receiving platform will only see the +replaced values. + +Adding the configuration option below will limit the backup cache to `n` gigabytes. This will keep a hard drive from +filling up if the agent is disconnected from its target for a long time. + +:: + + "backup_storage_limit_gb": n + +.. seealso:: + + :ref:`Historian Framework ` diff --git a/docs/source/agent-framework/historian-agents/forwarder/forward-historian.rst b/docs/source/agent-framework/historian-agents/forwarder/forward-historian.rst new file mode 100644 index 0000000000..4d66784b4e --- /dev/null +++ b/docs/source/agent-framework/historian-agents/forwarder/forward-historian.rst @@ -0,0 +1,61 @@ +.. _Forward-Historian: + +================= +Forward Historian +================= + +The primary use case for the Forward Historian is to send data to another instance of VOLTTRON as if the data were live. +This allows agents running on a more secure and/or more powerful machine to run analysis on data being collected on a +potentially less secure/powerful board. + +Given this use case, it is not optimized for batching large amounts of data when "live-ness" is not needed. For this +use case, please see the :ref:`Data Mover Historian `. + +The Forward Historian can be found in the `services/core directory`. + + +Configuration +============= + +The default configuration file is `services/core/ForwardHistorian/config`. Change the `destination-vip` value to +point towards the foreign VOLTTRON instance. + +.. code-block:: json + + { + "agentid": "forwarder", + "destination-vip": "ipc://@/home/volttron/.volttron/run/vip.socket" + } + +In order to send to a remote platform, you will need its VIP address and server key. The server key can be found by +running: + +.. code-block:: bash + + vctl auth serverkey + +Put the result into the following example: + +.. note:: + + The example shown uses the local IP address, the IP address for your configuration should match the intended target + +.. code-block:: json + + { + "agentid": "forwarder", + "destination-vip": "tcp://127.0.0.1:22916", + "destination-serverkey": "" + } + + +Adding the configuration option below will limit the backup cache to `n` gigabytes. This will help keep a hard drive +from filling up if the agent is disconnected from its target for a long time. + +:: + + "backup_storage_limit_gb": n + +.. seealso:: + + :ref:`Historian Framework ` diff --git a/docs/source/core_services/historians/index.rst b/docs/source/agent-framework/historian-agents/historian-framework.rst similarity index 63% rename from docs/source/core_services/historians/index.rst rename to docs/source/agent-framework/historian-agents/historian-framework.rst index 85d18d95da..e714cd61ab 100644 --- a/docs/source/core_services/historians/index.rst +++ b/docs/source/agent-framework/historian-agents/historian-framework.rst @@ -1,30 +1,40 @@ -.. _Historian Index: +.. _Historian-Framework: -============================ -VOLTTRON Historian Framework -============================ +=================== +Historian Framework +=================== -Historian Agents are the way by which device, actuator, datalogger, and -analysis are captured and stored in some sort of data store. Historians exist for the following storage options: +Historian Agents are the way by which `device`, `actuator`, `datalogger`, and `analysis` topics are automatically +captured and stored in some sort of data store. Historians exist for the following storage options: - A general :ref:`SQL Historian ` implemented for MySQL, SQLite, PostgreSQL, and Amazon Redshift - :ref:`MongoDB Historian ` - :ref:`Crate Historian ` - :ref:`Forward Historian ` for sending data to another VOLTTRON instance -- :ref:`OpenEIS Historian ` +- :ref:`OpenEIS Historian ` - :ref:`MQTT Historian ` Forwards data to an MQTT broker - :ref:`InfluxDB Historian ` -Other implementations of historians can be created by following the -:ref:`developing historian agents ` section of -the wiki. +Other implementations of Historians can be created by following the +:ref:`Developing Historian Agents ` guide. -Historians are all built upon the BaseHistorian which provides general -functionality the specific implementations is built upon. -In most cases the default settings are fine for all deployments. +Base Historian +============== -All historians support the following settings: +Historians are all built upon the `BaseHistorian` which provides general functionality the specific implementations are +built upon. + +This base Historian will cache all received messages to a local database before publishing it to the Historian. This +allows recovery from unexpected happenings before the successful writing of data to the Historian. + + +Configuration +============= + +In most cases the default configuration settings are fine for all deployments. + +All Historians support the following settings: .. code-block:: python @@ -118,38 +128,63 @@ All historians support the following settings: } } -By default the base historian will listen to 4 separate root topics -`datalogger/*`, `record/*`, `analysis/*`, and `devices/*`. -Each root -topic has a :ref:`specific message syntax ` that -it is expecting for incoming data. +Topics +====== + +By default the base historian will listen to 4 separate root topics: + +* `datalogger/*` +* `record/*` +* `analysis/*` +* `devices/*` + +Each root topic has a :ref:`specific message syntax ` that it is expecting for incoming data. -Messages published to `datalogger` -will be assumed to be timepoint data that is composed of units and -specific types with the assumption that they have the ability to be -graphed easily. +Messages published to `datalogger` will be assumed to be `timepoint` data that is composed of units and specific types +with the assumption that they have the ability to be plotted easily. -Messages published to `devices` are data that comes -directly from drivers. +Messages published to `devices` are data that comes directly from drivers. -Messages published to `analysis` are analysis data published by agents -in the form of key value pairs. +Messages published to `analysis` are analysis data published by agents in the form of key value pairs. -Finally, messages that are published to `record` -will be handled as string data and can be customized to the user +Finally, messages that are published to `record` will be handled as string data and can be customized to the user specific situation. -Please consult the :ref:`Historian Topic -Syntax ` page for a specific syntax. -This base historian will cache all received messages to a local -database before publishing it to the historian. This allows recovery from -unexpected happenings before the successful writing of data to the historian. +.. _Platform-Historian: + +Platform Historian +================== + +A platform historian is a :ref:`"friendly named" ` historian on a VOLTTRON instance. It always has +the identity of `platform.historian`. A platform historian is made available to a VOLTTRON Central agent for monitoring +of the VOLTTRON instances health and plotting topics from the platform historian. In order for one of the historians to +be turned into a platform historian the `identity` keyword must be added to it's configuration with the value of +`platform.historian`. The following configuration file shows a SQLite based platform historian configuration: + +.. code-block:: json + + { + "agentid": "sqlhistorian-sqlite", + "identity": "platform.historian", + "connection": { + "type": "sqlite", + "params": { + "database": "~/.volttron/data/platform.historian.sqlite" + } + } + } .. toctree:: - :glob: - :maxdepth: 2 - * + historian-topic-syntax + crate/crate-historian + influxdb/influxdb-historian + mongodb/mongo-historian + mqtt/mqtt-historian + openeis/openeis-historian + sql-historian/sql-historian + data-mover/data-mover-historian + forwarder/forward-historian diff --git a/docs/source/agent-framework/historian-agents/historian-topic-syntax.rst b/docs/source/agent-framework/historian-agents/historian-topic-syntax.rst new file mode 100644 index 0000000000..f01d55ed92 --- /dev/null +++ b/docs/source/agent-framework/historian-agents/historian-topic-syntax.rst @@ -0,0 +1,116 @@ +.. _Historian-Topic-Syntax: + +====================== +Historian Topic Syntax +====================== + +Each historian will subscribe to the following message bus topics: + +* `datalogger/*` +* `anaylsis/*` +* `record/\*` +* `devices/\*` + +For each of these topics there is a different message syntax that must be adhered to in order for the correct +interpretation of the data being specified. + + +record/\* +--------- +The record topic is the most flexible of all of the topics. This topic allows any serializable message to be published +to any topic under the root topic `record/`. + +.. Note:: + + This topic is not recommended to plot, as the structure of the messages are not necessarily numeric + +:: + + # Example messages that can be published + + # Dictionary data + {'foo': 'world'} + + # Numerical data + 52 + + # Time data (note: not a `datetime` object) + '2015-12-02T11:06:32.252626' + + +devices/\* +---------- + +The `devices` topic is meant to be data structured from a scraping of a Modbus or BACnet device. Currently drivers for +both of these protocols write data to the message bus in the proper format. VOLTTRON drivers also publish an +aggregation of points in an `all` topic. + +**Only the `all` topic messages are read and published to a historian.** + +Both the all topic and point topic have the same header information, but the message body for each is slightly +different. For a complete working example of these messages please see +:py:mod:`examples.ExampleSubscriber.subscriber.subscriber_agent` + +The format of the header and message for device topics (i.e. messages published to topics with pattern "devices/\*/all") +follows the following pattern: + +:: + + # Header contains the data associated with the message. + { + # python code to get this is + # from datetime import datetime + # from volttron.platform.messaging import headers as header_mod + # from volttron.platform.agent import utils + # now = utils.format_timestamp( datetime.utcnow()) + # { + # headers_mod.DATE: now, + # headers_mod.TIMESTAMP: now + # } + "Date": "2015-11-17 21:24:10.189393+00:00", + "TimeStamp": "2015-11-17 21:24:10.189393+00:00" + } + + # Message Format: + + # WITH METADATA + # Messages contains a two element list. The first element contains a + # dictionary of all points under a specific parent. While the second + # element contains a dictionary of meta data for each of the specified + # points. For example devices/pnnl/building/OutsideAirTemperature and + # devices/pnnl/building/MixedAirTemperature ALL message would be created as: + [ + {"OutsideAirTemperature ": 52.5, "MixedAirTemperature ": 58.5}, + { + "OutsideAirTemperature ": {'units': 'F', 'tz': 'UTC', 'type': 'float'}, + "MixedAirTemperature ": {'units': 'F', 'tz': 'UTC', 'type': 'float'} + } + ] + + #WITHOUT METADATA + # Message contains a dictionary of all points under a specific parent + {"OutsideAirTemperature ": 52.5, "MixedAirTemperature ": 58.5} + + +analysis/\* +----------- + +Data sent to `analysis/*` topics is result of analysis done by applications. The format of data sent to `analysis/*` +topics is similar to data sent to `devices/\*/all` topics. + + +datalogger/\* +------------- +Messages published to `datalogger/\*` will be assumed to be time point data that is composed of units and specific types +with the assumption that they have the ability to be graphed easily. + +:: + + {"MixedAirTemperature": {"Readings": ["2015-12-02T00:00:00", + `_. + + +Prerequisites +============= + +InfluxDB Installation +--------------------- + +To install InfluxDB on an Ubuntu or Debian operating system, run the script: + + :: + + services/core/InfluxdbHistorian/scripts/install-influx.sh + +For installation on other operating systems, +see ``_. + +Authentication in InfluxDB +-------------------------- + +By default, the InfluxDB *Authentication* option is disabled, and no user authentication is required to access any +InfluxDB database. You can enable authentication by updating the InfluxDB configuration file. For detailed information +on enabling authentication, see: +``_. + +If *Authentication* is enabled, authorization privileges are enforced. There must be at least one defined admin user +with access to administrative queries as outlined in the linked document above. Additionally, you must pre-create the +``user`` and ``database`` that are specified in the configuration file (the default configuration file for InfluxDB +is `services/core/InfluxdbHistorian/config`). If your ``user`` is a non-admin user, they must be granted a full set of +privileges on the desired ``database``. + + +InfluxDB Driver +--------------- + +In order to connect to an InfluxDb client, the Python library for InfluxDB must be installed in VOLTTRON's virtual +environment. From the command line, after enabling the virtual environment, install the InfluxDB library as follows: + +.. code-block:: bash + + python bootstrap.py --influxdb + +or + +.. code-block:: bash + + python bootstrap.py --databases + +or + +.. code-block:: bash + + pip install influxdb + + +Configuration +============= + +The default configuration file for VOLTTRON's InfluxDB Historian agent should be in the format: + +.. code-block:: python + + { + "connection": { + "params": { + "host": "localhost", + "port": 8086, # Don't change this unless default bind port + # in influxdb config is changed + "database": "historian", + "user": "historian", # user is optional if authentication is turned off + "passwd": "historian" # passwd is optional if authentication is turned off + } + }, + "aggregations": { + "use_calendar_time_periods": true + } + } + + +The InfluxDB Historian agent can be packaged, installed and started according to the standard VOLTTRON agent creation +procedure. A sample VOLTTRON configuration file has been provided: `services/core/InfluxdbHistorian/config`. + +.. seealso:: + + :ref:`Agent Development Walk-through ` + + +Connection +---------- + +The ``host``, ``database``, ``user`` and ``passwd`` values in the VOLTTRON configuration file +can be modified. ``user`` and ``passwd`` are optional if InfluxDB *Authentication* is disabled. + +.. note:: + + Be sure to initialize or pre-create the ``database`` and ``user`` defined in the configuration file, and if ``user`` + is a non-admin user, be make sure to grant privileges for the user on the specified ``database``. For more + information, see `Authentication in InfluxDB`_. + + +Aggregations +------------ + +In order to use aggregations, the VOLTTRON configuration file must also specify a value, either ``true`` or ``false``, +for ``use_calendar_time_periods``, indicating whether the aggregation period should align to calendar time periods. If +this value is omitted from the configuration file, aggregations cannot be used. + +For more information on historian aggregations, see: +:ref:`Aggregate Historian Agent Specification `. + +Supported Influxdb aggregation functions: + + * Aggregations: COUNT(), DISTINCT(), INTEGRAL(), MEAN(), MEDIAN(), MODE(), SPREAD(), STDDEV(), SUM() + + * Selectors: FIRST(), LAST(), MAX(), MIN() + + * Transformations: CEILING(),CUMULATIVE_SUM(), DERIVATIVE(), DIFFERENCE(), ELAPSED(), NON_NEGATIVE_DERIVATIVE(), + NON_NEGATIVE_DIFFERENCE() + +More information how to use those functions: ``_ + +.. note:: + + Historian aggregations in InfluxDB are different from aggregations employed by other historian agents in VOLTTRON. + InfluxDB doesn't have a separate agent for aggregations. Instead, aggregation is supported through the + ``query_historian`` function. Other agents can execute an aggregation query directly in InfluxDB by calling the + `RPC.export` method ``query``. For an example, see + :ref:`Aggregate Historian Agent Specification ` + + +Database Schema +=============== + +Each InfluxDB database has a `meta` table as well as other tables for different measurements, e.g. one table for +"power_kw", one table for "energy", one table for "voltage", etc. (An InfluxDB `measurement` is similar to a +relational table, so for easier understanding, InfluxDB measurements will be referred to below as tables.) + + +Measurement Table +----------------- + +Example: If a topic name is `CampusA/Building1/Device1/Power_KW`, the `power_kw` table might look as follows: + ++-------------------------------+-----------+---------+----------+-------+------+ +|time |building |campus |device |source |value | ++-------------------------------+-----------+---------+----------+-------+------+ +|2017-12-28T20:41:00.004260096Z |building1 |campusa |device1 |scrape |123.4 | ++-------------------------------+-----------+---------+----------+-------+------+ +|2017-12-30T01:05:00.004435616Z |building1 |campusa |device1 |scrape |567.8 | ++-------------------------------+-----------+---------+----------+-------+------+ +|2018-01-15T18:08:00.126345Z |building1 |campusa |device1 |scrape |10 | ++-------------------------------+-----------+---------+----------+-------+------+ + +``building``, ``campus``, ``device``, and ``source`` are InfluxDB *tags*. ``value`` is an InfluxDB *field*. + +.. note:: + + The topic is converted to all lowercase before being stored in the table. In other words, a set of *tag* names, as + well as a table name, are created by splitting `topic_id` into substrings (see `meta table`_ below). + + +In this example, where the typical format of a topic name is `///`, `campus`, +`building` and `device` are each stored as tags in the database. + +A topic name might not confirm to that convention: + + #. The topic name might contain additional substrings, e.g. `CampusA/Building1/LAB/Device/OutsideAirTemperature`. + In this case, `campus` will be ``campusa/building``, `building` will be ``lab``, and `device` will be ``device``. + + #. The topic name might contain fewer substrings, e.g. `LAB/Device/OutsideAirTemperature`. In this case, the + `campus` tag will be empty, `building` will be ``lab``, and `device` will be ``device``. + + +Meta Table +========== + +The meta table will be structured as in the following example: + ++---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ +|time |last_updated |meta_dict |topic |topic_id | ++---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ +|1970-01-01T00:00:00Z |2017-12-28T20:47:00.003051+00:00 |{u'units': u'kw', u'tz': u'US/Pacific', u'type': u'float'} |CampusA/Building1/Device1/Power_KW |campusa/building1/device1/power_kw | ++---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ +|1970-01-01T00:00:00Z |2017-12-28T20:47:00.003051+00:00 |{u'units': u'kwh', u'tz': u'US/Pacific', u'type': u'float'} |CampusA/Building1/Device1/Energy_KWH |campusa/building1/device1/energy_kwh | ++---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ + +In the InfluxDB, `last_updated`, `meta_dict` and `topic` are *fields* and `topic_id` is a *tag*. + +Since InfluxDB is a time series database, the ``time`` column is required, and a dummy value (``time=0``, which is +``1970-01-01T00:00:00Z`` based on epoch unix time) is assigned to all topics for easier metadata updating. Hence, if the +contents of `meta_dict` change for a specific topic, both `last_updated` and `meta_dict` values for that topic will be +replaced in the table. diff --git a/docs/source/agent-framework/historian-agents/mongodb/mongo-historian.rst b/docs/source/agent-framework/historian-agents/mongodb/mongo-historian.rst new file mode 100644 index 0000000000..9fd13532d8 --- /dev/null +++ b/docs/source/agent-framework/historian-agents/mongodb/mongo-historian.rst @@ -0,0 +1,110 @@ +.. _Mongo-Historian: + +=============== +Mongo Historian +=============== + +MongoDB is a NoSQL document database, which allows for great performance for transactional data. Because MongoDB +documents do not have a schema, it is easy to store and query data which changes over time. MongoDB also scales +horizontally using sharding. + +For more information about MongoDB, read the `MongoDB documentation `_ + + +Prerequisites +============= + + +1. Mongodb +---------- + +Setup mongodb based on using one of the three installation scripts for the corresponding environment: + +1. Install as root on Redhat or Cent OS + + .. code-block:: bash + + sudo scripts/historian-scripts/root_install_mongo_rhel.sh + + The above script will prompt user for os version, db user name, password and database name. Once installed you can + start and stop the service using the command: + + .. code-block:: bash + + **sudo service mongod [start|stop|service]** + +2. Install as root on Ubuntu + + .. code-block:: bash + + sudo scripts/historian-scripts/root_install_mongo_ubuntu.sh + + The above script will prompt user for os version, db user name, password and database name. Once installed you can + start and stop the service using the command: + + .. code-block:: bash + + **sudo service mongod [start|stop|service]** + +3. Install as non root user on any Linux machine + + .. code-block:: bash + + scripts/historian-scripts/install_mongodb.sh + + Usage: + + .. code-block:: bash + + install_mongodb.sh [-h] [-d download_url] [-i install_dir] [-c config_file] [-s] + + Optional arguments: + + -s setup admin user and test collection after install and startup + + -d download url. defaults to https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.4.tgz + + -i install_dir. defaults to current_dir/mongo_install + + -c config file to be used for mongodb startup. Defaults to default_mongodb.conf in the same directory as this + script. Any data path mentioned in the config file should already exist and should have write access to the + current user + + -h print the help message + + +2. Mongodb connector +-------------------- +This historian requires a mongodb connector installed in your activated VOLTTRON virtual environment to talk to MongoDB. +Please execute the following from an activated shell in order to install it: + +.. code-block:: bash + + python bootstrap.py --mongo + + +or + +.. code-block:: bash + + python bootstrap.py --databases + + +or + +.. code-block:: bash + + pip install pymongo + + +3. Configuration Options +------------------------ + +The historian configuration file can specify + +:: + + "history_limit_days": + +which will remove entries from the data and rollup collections older than `n` days. Timestamps passed to the +``manage_db_size`` method are truncated to the day. diff --git a/docs/source/core_services/historians/MQTT-Historian.rst b/docs/source/agent-framework/historian-agents/mqtt/mqtt-historian.rst similarity index 53% rename from docs/source/core_services/historians/MQTT-Historian.rst rename to docs/source/agent-framework/historian-agents/mqtt/mqtt-historian.rst index 1874240fa3..2ed25a0b54 100644 --- a/docs/source/core_services/historians/MQTT-Historian.rst +++ b/docs/source/agent-framework/historian-agents/mqtt/mqtt-historian.rst @@ -1,27 +1,27 @@ .. _MQTT-Historian: +============== MQTT Historian ============== Overview --------- -The MQTT Historian agent publishes data to an MQTT broker. +======== + +The MQTT Historian agent publishes data to an MQTT broker. The ``mqttlistener.py`` script will connect to the broker +and print all messages. -The mqttlistener.py script will connect to the broker and print -all messages. Dependencies ------------- -The Paho MQTT library from Eclipse is needed for the agent and can -be installed with: +============ +The Paho MQTT library from Eclipse is needed for the agent and can be installed with: -:: +.. code-block:: bash pip install paho-mqtt The Mosquitto MQTT broker may be useful for testing and can be installed with -:: +.. code-block:: bash apt-get install mosquitto diff --git a/docs/source/core_services/historians/OpenEIS-Historian.rst b/docs/source/agent-framework/historian-agents/openeis/openeis-historian.rst similarity index 80% rename from docs/source/core_services/historians/OpenEIS-Historian.rst rename to docs/source/agent-framework/historian-agents/openeis/openeis-historian.rst index 97b4a3724c..7909e3c2fb 100644 --- a/docs/source/core_services/historians/OpenEIS-Historian.rst +++ b/docs/source/agent-framework/historian-agents/openeis/openeis-historian.rst @@ -1,17 +1,23 @@ .. _OpenEIS-Historian: +================= OpenEIS Historian -=================== +================= -An OpenEIS Historian has been developed to integrate real -time data ingestion into the OpenEIS platform. In order for the OpenEIS -historian to be able to communicate with an OpenEIS server a datasource -must be created on the OpenEIS server. The process of creating a dataset -is documented in the `OpenEIS User's -Guide `__ -under *Creating a Dataset* heading. Once a dataset is created you will -be able to add datasets through the configuration file. An example -configuration for the historian is as follows: +An OpenEIS Historian has been developed to integrate real time data ingestion into the OpenEIS platform. In order for +the OpenEIS Historian to be able to communicate with an OpenEIS server a datasource must be created on the OpenEIS +server. + +The process of creating a dataset is documented in the +`OpenEIS User's Guide `__ +under `Creating a Dataset` heading. + + +Configuration +============= + +Once a dataset is created you will be able to add datasets through the configuration file. An example configuration for +the historian is as follows: :: @@ -84,4 +90,3 @@ configuration for the historian is as follows: # } } } - diff --git a/docs/source/core_services/historians/SQL-Historian.rst b/docs/source/agent-framework/historian-agents/sql-historian/sql-historian.rst similarity index 57% rename from docs/source/core_services/historians/SQL-Historian.rst rename to docs/source/agent-framework/historian-agents/sql-historian/sql-historian.rst index f4566d8478..cdd7763a84 100644 --- a/docs/source/core_services/historians/SQL-Historian.rst +++ b/docs/source/agent-framework/historian-agents/sql-historian/sql-historian.rst @@ -1,19 +1,24 @@ .. _SQL-Historian: +============= SQL Historian ============= -An SQL Historian is available as a core service. The sql historian has -been programmed to allow for inconsistent network connectivity -(automatic re-connection to tcp based databases). All additions to the -historian are batched and wrapped within a transaction with commit and -rollback functions properly implemented. This allows the maximum -throughput of data with the most protection. The following example -configurations show the different options available for configuring the -SQL Historian Agent. +An SQL Historian is available as a core service (`services/core/SQLHistorian` in the VOLTTRON repository). + +The SQL Historian has been programmed to handle for inconsistent network connectivity (automatic re-connection to tcp +based databases). All additions to the historian are batched and wrapped within a transaction with commit and rollback +functions. This allows the maximum throughput of data with the most protection. + + +Configuration +============= + +The following example configurations show the different options available for configuring the SQL Historian Agent: + MySQL Specifics -~~~~~~~~~~~~~~~ +--------------- MySQL requires a third party driver (mysql-connector) to be installed in order for it to work. Please execute the following from an activated @@ -62,8 +67,9 @@ or } } + Sqlite3 Specifics -~~~~~~~~~~~~~~~~~ +----------------- An Sqlite Historian provides a convenient solution for under powered systems. The database is parameter is a location on the file system. By @@ -84,59 +90,58 @@ will respect a rooted or relative path to the database. PostgreSQL and Redshift -~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- Installation notes ------------------- +^^^^^^^^^^^^^^^^^^ -1. The PostgreSQL database driver supports recent PostgreSQL versions. - It was tested on 10.x, but should work with 9.x and 11.x. +1. The PostgreSQL database driver supports recent PostgreSQL versions. It has been tested on 10.x, but should work with + 9.x and 11.x. -2. The user must have SELECT, INSERT, and UPDATE privileges on historian - tables. +2. The user must have SELECT, INSERT, and UPDATE privileges on historian tables. -3. The tables in the database are created as part of the execution of - the SQLHistorianAgent, but this will fail if the database user does not - have CREATE privileges. +3. The tables in the database are created as part of the execution of the SQL Historian Agent, but this will fail if the + database user does not have CREATE privileges. -4. Care must be exercised when using multiple historians with the same - database. This configuration may be used only if there is no overlap in - the topics handled by each instance. Otherwise, duplicate topic IDs - may be created, producing strange results. +4. Care must be exercised when using multiple historians with the same database. This configuration may be used only if + there is no overlap in the topics handled by each instance. Otherwise, duplicate topic IDs may be created, producing + strange results. + +5. Redshift databases do not support unique constraints. Therefore, it is possible that tables may contain some + duplicate data. The Redshift driver handles this by using distinct queries. It does not remove duplicates from the + tables. -5. Redshift databases do not support unique constraints. Therefore, it is - possible that tables may contain some duplicate data. The Redshift driver - handles this by using distinct queries. It does not remove duplicates - from the tables. Dependencies ------------- +^^^^^^^^^^^^ -The PostgreSQL and Redshift database drivers require the **psycopg2** Python package. +The PostgreSQL and Redshift database drivers require the `psycopg2` Python package. From an activated shell execute: - :: + .. code-block:: bash pip install psycopg2-binary -Configuration -------------- -The following are minimal configuration files for using a psycopg2-based -historian. Other options are available and are documented -http://initd.org/psycopg/docs/module.html -**Not all parameters have been tested, use at your own risk**. +PostgreSQL and Redshift Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following are minimal configuration files for using a psycopg2-based historian. Other options are available and are +`documented `_. + +.. warning:: + + Not all parameters have been tested, use at your own risk. + Local PostgreSQL Database -+++++++++++++++++++++++++ +""""""""""""""""""""""""" -The following snippet demonstrates how to configure the -SQLHistorianAgent to use a PostgreSQL database on the local system -that is configured to use Unix domain sockets. The user executing -volttron must have appropriate privileges. +The following snippet demonstrates how to configure the SQL Historian Agent to use a PostgreSQL database on the local +system that is configured to use Unix domain sockets. The user executing VOLTTRON must have appropriate privileges. -:: +.. code-block:: json { "connection": { @@ -147,13 +152,13 @@ volttron must have appropriate privileges. } } + Remote PostgreSQL Database -++++++++++++++++++++++++++ +"""""""""""""""""""""""""" -The following snippet demonstrates how to configure the -SQLHistorianAgent to use a remote PostgreSQL database. +The following snippet demonstrates how to configure the SQL Historian Agent to use a remote PostgreSQL database. -:: +.. code-block:: json { "connection": { @@ -168,19 +173,17 @@ SQLHistorianAgent to use a remote PostgreSQL database. } } + TimescaleDB Support -+++++++++++++++++++ +""""""""""""""""""" -Both of the above PostgreSQL connection types can make -use of TimescaleDB's high performance Hypertable backend -for the primary timeseries table. The agent assumes you -have completed the TimescaleDB installation and setup -the database by following the instructions here: -https://docs.timescale.com/latest/getting-started/setup -To use, simply add 'timescale_dialect: true' to the -connection params in the Agent Config as below +Both of the above PostgreSQL connection types can make use of TimescaleDB's high performance Hypertable backend for the +primary time-series table. The agent assumes you have completed the TimescaleDB installation and setup +the database by following the instructions `here `_. -:: +To use, simply add ``timescale_dialect: true`` to the connection params in the Agent Config as below: + +.. code-block:: json { "connection": { @@ -196,13 +199,13 @@ connection params in the Agent Config as below } } + Redshift Database -+++++++++++++++++ +""""""""""""""""" -The following snippet demonstrates how to configure the -SQLHistorianAgent to use a Redshift database. +The following snippet demonstrates how to configure the SQL Historian Agent to use a Redshift database. -:: +.. code-block:: json { "connection": { diff --git a/docs/source/core_services/service_agents/emailer/EmailerAgent.rst b/docs/source/agent-framework/operations-agents/emailer/emailer-agent.rst similarity index 90% rename from docs/source/core_services/service_agents/emailer/EmailerAgent.rst rename to docs/source/agent-framework/operations-agents/emailer/emailer-agent.rst index 0df6c9ddbc..4056c79afb 100644 --- a/docs/source/core_services/service_agents/emailer/EmailerAgent.rst +++ b/docs/source/agent-framework/operations-agents/emailer/emailer-agent.rst @@ -1,8 +1,9 @@ -.. _EmailerAgent: +.. _Emailer-Agent: ============= Emailer Agent ============= + Emailer agent is responsible for sending emails for an instance. It has been written so that any agent on the instance can send emails through it via the "send_email" method or through the pubsub message bus using the topic "platform/send_email". @@ -10,8 +11,10 @@ can send emails through it via the "send_email" method or through the pubsub mes By default any alerts will be sent through this agent. In addition all emails will be published to the "record/sent_email" topic for a historian to be able to capture that data. + Configuration -~~~~~~~~~~~~~ +============= + A typical configuration for this agent is as follows. We need to specify the SMTP server address, email address of the sender, email addresses of all the recipients and minimum time for duplicate emails based upon the key. @@ -25,4 +28,5 @@ sender, email addresses of all the recipients and minimum time for duplicate ema "allow-frequency-minutes": 10 } -Finally package, install and start the agent. For more details, see :ref:`Agent Creation Walkthrough ` +Finally package, install and start the agent. For more details, see +:ref:`Agent Creation Walk-through ` diff --git a/docs/source/core_services/service_agents/failover/Failover.rst b/docs/source/agent-framework/operations-agents/failover/failover.rst similarity index 100% rename from docs/source/core_services/service_agents/failover/Failover.rst rename to docs/source/agent-framework/operations-agents/failover/failover.rst diff --git a/docs/source/core_services/service_agents/file_watch_publisher/FileWatchPublisher-Agent.rst b/docs/source/agent-framework/operations-agents/file-watch-publisher/file-watch-publisher-agent.rst similarity index 100% rename from docs/source/core_services/service_agents/file_watch_publisher/FileWatchPublisher-Agent.rst rename to docs/source/agent-framework/operations-agents/file-watch-publisher/file-watch-publisher-agent.rst diff --git a/docs/source/agent-framework/operations-agents/index.rst b/docs/source/agent-framework/operations-agents/index.rst new file mode 100644 index 0000000000..b0b3b50260 --- /dev/null +++ b/docs/source/agent-framework/operations-agents/index.rst @@ -0,0 +1,19 @@ +.. _Operations-Agents: + +========== +Operations +========== + +Operations agents assist with the operations of the platform systems and provide alerts for various platform and +environmental conditions. For details on each, please refer to the corresponding documents. + +.. toctree:: + :maxdepth: 1 + + emailer/emailer-agent + failover/failover + file-watch-publisher/file-watch-publisher-agent + message-debugger/message-debugging + sysmon/sysmon + threshold/threshold-agent + topic-watcher/topic-watcher-agent diff --git a/docs/source/devguides/agent_development/files/40-message-debugger.jpg b/docs/source/agent-framework/operations-agents/message-debugger/files/40-message-debugger.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/40-message-debugger.jpg rename to docs/source/agent-framework/operations-agents/message-debugger/files/40-message-debugger.jpg diff --git a/docs/source/devguides/agent_development/Message-Debugging.rst b/docs/source/agent-framework/operations-agents/message-debugger/message-debugging.rst similarity index 99% rename from docs/source/devguides/agent_development/Message-Debugging.rst rename to docs/source/agent-framework/operations-agents/message-debugger/message-debugging.rst index f5dcce121c..acba842aa2 100644 --- a/docs/source/devguides/agent_development/Message-Debugging.rst +++ b/docs/source/agent-framework/operations-agents/message-debugger/message-debugging.rst @@ -1,5 +1,6 @@ .. _Message-Debugging: +================= Message Debugging ================= @@ -66,7 +67,7 @@ fashion as other agents, for example: e1 vcplatformagent-3.5.4 platform.agent vcp 47 volttroncentralagent-3.5.5 volttron.central vc -See :ref:`Agent Creation Walkthrough ` for further details on +See :ref:`Agent Creation Walk-through ` for further details on installing and starting agents from vctl. Once the Message Debugger Agent is running, it begins capturing message data and diff --git a/docs/source/core_services/service_agents/sysmon/sysmon.rst b/docs/source/agent-framework/operations-agents/sysmon/sysmon.rst similarity index 100% rename from docs/source/core_services/service_agents/sysmon/sysmon.rst rename to docs/source/agent-framework/operations-agents/sysmon/sysmon.rst diff --git a/docs/source/core_services/service_agents/threshold/ThresholdAgent.rst b/docs/source/agent-framework/operations-agents/threshold/threshold-agent.rst similarity index 86% rename from docs/source/core_services/service_agents/threshold/ThresholdAgent.rst rename to docs/source/agent-framework/operations-agents/threshold/threshold-agent.rst index b9abbadbc9..8f0ac3d4c3 100644 --- a/docs/source/core_services/service_agents/threshold/ThresholdAgent.rst +++ b/docs/source/agent-framework/operations-agents/threshold/threshold-agent.rst @@ -1,5 +1,6 @@ -.. _ThresholdAgent: +.. _Threshold-Agent: +========================= Threshold Detection Agent ========================= @@ -8,9 +9,9 @@ topic exceeds or falls below a configured value. The agent can be configured to watch topics are associated with a single value or to watch devices' all topics. Configuration -------------- +============= -The ThresholdDetectionAgent supports the :ref:`configstore ` +The Threshold Detection Agent supports the :ref:`config store ` and can be configured with a file named "config". The file must be in the following format: diff --git a/docs/source/core_services/service_agents/topic_watcher/TopicWatcherAgent.rst b/docs/source/agent-framework/operations-agents/topic-watcher/topic-watcher-agent.rst similarity index 100% rename from docs/source/core_services/service_agents/topic_watcher/TopicWatcherAgent.rst rename to docs/source/agent-framework/operations-agents/topic-watcher/topic-watcher-agent.rst diff --git a/docs/source/core_services/service_agents/Platform-Service-Standardization.rst b/docs/source/agent-framework/platform-service-standardization.rst similarity index 100% rename from docs/source/core_services/service_agents/Platform-Service-Standardization.rst rename to docs/source/agent-framework/platform-service-standardization.rst diff --git a/docs/source/setup/Third-Party-Agents.rst b/docs/source/agent-framework/third-party-agents.rst similarity index 100% rename from docs/source/setup/Third-Party-Agents.rst rename to docs/source/agent-framework/third-party-agents.rst diff --git a/docs/source/specifications/webframework.rst b/docs/source/agent-framework/web-framework.rst similarity index 96% rename from docs/source/specifications/webframework.rst rename to docs/source/agent-framework/web-framework.rst index d96d27322e..84ca564eeb 100644 --- a/docs/source/specifications/webframework.rst +++ b/docs/source/agent-framework/web-framework.rst @@ -1,14 +1,15 @@ -.. _WebFramework: +.. _Web-Framework: -VOLTTRON Web Framwwork -====================== +============= +Web Framework +============= -This document describes the interaction between web enabled agents and the MasterWebService agent. +This document describes the interaction between web enabled agents and the Master Web Service agent. The web framework enables agent developers to expose JSON, static, and websocket endpoints. Web SubSystem -+++++++++++++ +============= Enabling -------- @@ -78,5 +79,3 @@ Websocket endpoints allow bi-directional communication between the client and th @Core.receiver('onstart') def onstart(self, sender, **kwargs): self.vip.web.register_websocket(r'/vc/ws', self.open_authenticate_ws_endpoint, self._ws_closed, self._ws_received) - - diff --git a/docs/source/community_resources/documentation.rst b/docs/source/community_resources/documentation.rst deleted file mode 100644 index 401eda7fa7..0000000000 --- a/docs/source/community_resources/documentation.rst +++ /dev/null @@ -1,40 +0,0 @@ -.. _documentation: - -Contributing Documentation -============================= - -The Community is encouraged to contribute documentation back to the project as they work through use cases the -developers may not have considered or documented. By contributing documentation back, the community can -learn from each other and build up a much more extensive knowledge base. - -|VOLTTRON| documentation utilizes ReadTheDocs: http://volttron.readthedocs.io/en/develop/ and is built -using the `Sphinx `_ Python library with static content in -`Restructured Text `_. - -Building the Documentation ---------------------------- - -Static documentation can be found in the `docs/source` directory. Edit or create new .rst files to add new content -using the `Restructured Text `_ format. To see the results -of your changes. the documentation can be built locally through the command line using the following instructions. - -If you've already :ref:`bootstrapped ` |VOLTTRON|, do the following while activated. If not, -this will also pull down the necessary |VOLTTRON| libraries. - -.. code-block:: bash - - python bootstrap.py --documentation - cd docs - make html - -Then, open your browser to the created local files: - -.. code-block:: bash - - file:///home//git/volttron/docs/build/html/overview/index.html - - -When complete, changes can be contributed back using the same process as code :ref:`contributions ` by -creating a pull request. When the changes are accepted and merged, they will be reflected in the ReadTheDocs site. - -.. |VOLTTRON| unicode:: VOLTTRON U+2122 diff --git a/docs/source/core_services/config_store/index.rst b/docs/source/core_services/config_store/index.rst deleted file mode 100644 index 74b2d358b1..0000000000 --- a/docs/source/core_services/config_store/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. _VOLTTRON-Configuration-Store: - -============================ -VOLTTRON Configuration Store -============================ - -The configuration store provides storage for agent configurations and an agent interface to facilitate dynamic agent configuration. - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/control/AgentAutostart.rst b/docs/source/core_services/control/AgentAutostart.rst deleted file mode 100644 index 6308fb081d..0000000000 --- a/docs/source/core_services/control/AgentAutostart.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. _Agent-Autostart: - -Agent Autostart -=============== - -An agent can be setup to start when the platform is started with the -"enable" command. This command also allows a priority to be set (0-100, -default 50) so that agents can be started after any dependencies. This -command can also be used with the --tag or --name options. - -``vctl enable <--priority PRIORITY>`` diff --git a/docs/source/core_services/control/AgentManagement.rst b/docs/source/core_services/control/AgentManagement.rst deleted file mode 100644 index df8ef1c6b4..0000000000 --- a/docs/source/core_services/control/AgentManagement.rst +++ /dev/null @@ -1,113 +0,0 @@ -.. _Agent-Lifecycle-Management: - -Agent Lifecyle Management -~~~~~~~~~~~~~~~~~~~~~~~~~ - -The VOLTTRON platform has several commands for controlling the lifecycle -of agents. This page discusses how to use them, for details of operation -please see :ref:`PlatformConfiguration ` - -**These examples assume the VOLTTRON environment has been activated (. -env/bin/activate). If not, add "bin/" to all commands.** - -Agent Packaging -=============== - -The "vpkg" command is used for packaging and configuring agents. -It is not necessary to have the platform running to use this command. -The platform uses `Python Wheel `__ -for its packaging and follows the Wheel naming -`convention `__. - -To create an agent package, call ``vpkg ``. - -For instance: ``vpkg package examples/ListenerAgent`` - -The ``package`` command uses the setup.py in the agent directory to -create the package. The name and version number portion of the Wheel -filename come from this. The resulting wheels are created at -"~/.volttron/packaged". - -For example: -``~/.volttron/packaged/listeneragent-3.0-py2-none-any.whl``. - -Agent Configuration -=================== - -Agent packages are configured with the -``vpkg configure `` command. It is -suggested that this file use json formatting but the agent can be -written to interpret any format it requires. The configuration of a -particular agent is opaque to the VOLTTRON platform. The location of the -agent config file is passed as an environmental variable "AGENT\_CONFIG" -which the provided utilities read in and pass to the agent. - -An example config file passing in some parameters: - -:: - - { - - "agentid": "listener1", - "message": "hello" - } - -Agent Installation and Removal -============================== - -| Agents are installed into the platform using: - -``vctl install ``. -| When agents are installed onto a platform, it creates a uuid for that -instance of an agent. This allows multiple instances of the same agent -package to be installed on the platform. - -Agents can also be installed with a :ref:`tag ` by using: - -``vctl install =`` - -This allows the user to refer to the agent with "--tag " instead of the -uuid when issuing commands. This tag can also distinguish instances of -an agent from each other. - -A stopped agent can be removed with: - -- ``vctl remove `` -- ``vctl remove --tag `` -- ``vctl remove --name `` - -Removal by tag and name potentially allows multiple agents to be removed -at once and should be used with caution. A "-f" option is required to -delete more than one agent at a time. - -Agent Control -============= - -Starting and Stopping an Agent ------------------------------- - -Agent that are installed in the platform can be launched with the -"start" command. By default this operates off the agent's UUID but can -be used with "--tag" or "--name" to launch agents by those attributes. -This can allow multiple agents to be started at once. For instance: -``vctl start --name myagent-0.1`` would start all instances of -that agent regardless of their uuid, tag, or configuration information. -After an agent is started, it will show up in -:ref:`AgentStatus ` as "running" with a process id. - -Similarly, ``volttron-ctl stop `` can also operate off the tag and -name of agent(s). After an agent is stopped, it will show an exit code -of 0 in :ref:`AgentStatus ` - -Running an agent ----------------- - -For testing purposes, an agent package not installed in the platform can -be run by using: ``vctl run ``. - -Agent Status -============ -olttron- -| ``vctl list`` lists the agents installed on the platform and their priority -| The ``vctl status`` shows the list of installed agents and whether they are running or have exited. -| See :ref:`AgentStatus ` for more details. diff --git a/docs/source/core_services/control/AgentStatus.rst b/docs/source/core_services/control/AgentStatus.rst deleted file mode 100644 index f744741f5a..0000000000 --- a/docs/source/core_services/control/AgentStatus.rst +++ /dev/null @@ -1,56 +0,0 @@ -.. _Agent-Status: - -Agent List Display -~~~~~~~~~~~~~~~~~~ - -:: - - AGENT IDENTITY TAG PRI - - d listeneragent-3.0 listeneragent-3.0_1 30 - 2 testeragent-0.1 testeragent-0.1_1 - -``vctl list`` shows the agents which have been installed on the -platform along with their uuid, associated `tag `__, and -`priority `__. - -- uuid is the first column of the display and is displayed as the - shorted unique portion. Using this portion, agents can be started, - stopped, removed, etc. -- AGENT is the "name" of this agent based on the name of the wheel file - which was installed. Agents can be controlled with this using "--name - ". Note, if multiple instances of a wheel are installed they will all - have the same name and can be controlled as a group. -- `TAG `__ is a user-provided tag which makes it simpler to - track and refer to agents. Agents can be controlled by using "--tag". -- PRI is the priority for agents which have been "enabled" using the - ``vctl enable`` command. When enabled, agents will be - automatically started in priority order along with the platform. - -Agent Status Display -==================== - -:: - - AGENT TAG STATUS - - d listeneragent-3.0 listener running [3813] - 2 testeragent-0.1 0 - -``vctl status`` shows a list of all agents installed on the -platform and their current status. - -- uuid is the first column of the display and is displayed as the - shorted unique portion. Using this portion, agents can be started, - stopped, removed, etc. -- AGENT is the "name" of this agent based on the name of the wheel file - which was installed. Agents can be controlled with this using "--name - ". Note, if multiple instances of a wheel are installed they will all - have the same name and can be controlled as a group. -- `TAG `__ is a user provided tag which makes it simpler to - track and refer to agents. Using "--tag " agents can be controlled - using this -- STATUS is the current condition of the agent. If the agent is - currently executing, it has "running" and the process id of the - agent. If the agent is not running, the exit code is shown. - diff --git a/docs/source/core_services/control/AgentTag.rst b/docs/source/core_services/control/AgentTag.rst deleted file mode 100644 index 8f3894b42e..0000000000 --- a/docs/source/core_services/control/AgentTag.rst +++ /dev/null @@ -1,24 +0,0 @@ -.. _AgentTag: - -Tagging Agents -============== - -Agents can be tagged as they are installed with: - -``vctl install =`` - -Agents can be tagged after installation with: - -``vctl tag `` - -Agents can be "tagged" to provide a meaningful user defined way to -reference the agent instead of the uuid or the name. This allows users -to differentiate between instances of agents which use the same codebase -but are configured differently. For instance, the AFDDAgent can be -configured to work against a single HVAC unit and can have any number of -instances running on one platform. A tagging scheme for this could be by -unit: afdd-rtu1, afdd-rtu2, etc. - -Commands which operate off an agent's UUID can optionally operate off -the tag by using "--tag ". This can use wildcards to catch multiple -agents at once. diff --git a/docs/source/core_services/control/AuthenticationCommands.rst b/docs/source/core_services/control/AuthenticationCommands.rst deleted file mode 100644 index 9a672de567..0000000000 --- a/docs/source/core_services/control/AuthenticationCommands.rst +++ /dev/null @@ -1,71 +0,0 @@ -.. _AuthenticationCommands: - -Authentication Commands -======================= - -All authentication sub-commands can be viewed by entering following command. - -.. code-block:: console - - vctl auth --help - -.. code-block:: console - - optional arguments: - -h, --help show this help message and exit - -c FILE, --config FILE - read configuration from FILE - --debug show tracbacks for errors rather than a brief message - -t SECS, --timeout SECS - timeout in seconds for remote calls (default: 30) - --vip-address ZMQADDR - ZeroMQ URL to bind for VIP connections - --keystore-file FILE use keystore from FILE - --known-hosts-file FILE - get known-host server keys from FILE - - subcommands: - add add new authentication record - add-group associate a group name with a set of roles - add-known-host add server public key to known-hosts file - add-role associate a role name with a set of capabilities - keypair generate CurveMQ keys for encrypting VIP connections - list list authentication records - list-groups show list of group names and their sets of roles - list-known-hosts list entries from known-hosts file - list-roles show list of role names and their sets of capabilities - publickey show public key for each agent - remove removes one or more authentication records by indices - remove-group disassociate a group name from a set of roles - remove-known-host remove entry from known-hosts file - remove-role disassociate a role name from a set of capabilities - serverkey show the serverkey for the instance - update updates one authentication record by index - update-group update group to include (or remove) given roles - update-role update role to include (or remove) given capabilities - -Authentication record ---------------------- - -An authentication record consist of following parameters - -.. code-block:: console - - domain []: - address []: Either a single agent identity or an array of agents identities - user_id []: Arbitrary string to identify the agent - capabilities (delimit multiple entries with comma) []: Array of strings referring to authorized capabilities defined by exported RPC methods - roles (delimit multiple entries with comma) []: - groups (delimit multiple entries with comma) []: - mechanism [CURVE]: - credentials []: Public key string for the agent - comments []: - enabled [True]: - -For more details on how to create authentication record, please see section :ref:`Agent Authentication` - - - - - - diff --git a/docs/source/core_services/control/PlatformConfigFile.rst b/docs/source/core_services/control/PlatformConfigFile.rst deleted file mode 100644 index 8045a93ee2..0000000000 --- a/docs/source/core_services/control/PlatformConfigFile.rst +++ /dev/null @@ -1,45 +0,0 @@ -.. _PlatformConfigFile: - -VOLTTRON Config File -==================== - -The VOLTTRON platform config file can contain any of the command line -arguments for starting the platform... - -.. code-block:: console - - -c FILE, --config FILE - read configuration from FILE - -l FILE, --log FILE send log output to FILE instead of stderr - -L FILE, --log-config FILE - read logging configuration from FILE - -q, --quiet decrease logger verboseness; may be used multiple - times - -v, --verbose increase logger verboseness; may be used multiple - times - --verboseness LEVEL set logger verboseness - --help show this help message and exit - --version show program's version number and exit - -agent options: - -.. code-block:: console - - --autostart automatically start enabled agents and services - --publish-address ZMQADDR - ZeroMQ URL for used for agent publishing - --subscribe-address ZMQADDR - ZeroMQ URL for used for agent subscriptions - -control options: - -.. code-block:: console - - --control-socket FILE - path to socket used for control messages - --allow-root allow root to connect to control socket - --allow-users LIST users allowed to connect to control socket - --allow-groups LIST user groups allowed to connect to control socket - -| Boolean options, which take no argument, may be inversed by prefixing the -| option with no- (e.g. --autostart may be inversed using --no-autostart). diff --git a/docs/source/core_services/control/PlatformConfiguration.rst b/docs/source/core_services/control/PlatformConfiguration.rst deleted file mode 100644 index 9889803eaa..0000000000 --- a/docs/source/core_services/control/PlatformConfiguration.rst +++ /dev/null @@ -1,22 +0,0 @@ -volttron-.. _PlatformConfiguration: - -VOLTTRON Environment -==================== - -By default, the VOLTTRON projects bases its files out of VOLTTRON\_HOME -which defaults to "~/.volttron". - -- ``$VOLTTRON_HOME/agents`` contains the agents installed on the - platform -- ``$VOLTTRON_HOME/certificates`` contains the certificates for use - with the Licensed VOLTTRON code. -- ``$VOLTTRON_HOME/run`` contains files create by the platform during - execution. The main ones are the 0MQ files created for publish and - subcribe. -- ``$VOLTTRON_HOME/ssh`` keys used by agent mobility in the Licensed - VOLTTRON code -- ``$VOLTTRON_HOME/config`` Default location to place a config file to - override any platform settings. -- ``$VOLTTRON_HOME/packaged`` is where agent packages created with - \`volttron-pkg package are created - diff --git a/docs/source/core_services/control/index.rst b/docs/source/core_services/control/index.rst deleted file mode 100644 index 3d9b8b5223..0000000000 --- a/docs/source/core_services/control/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. _Control: - -=========================== -Base Platform Functionality -=========================== - -The base platform functionality focuses on the agent lifecycle, management of the platform itself, and security. - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/drivers/BACnet-Auto-Configuration.rst b/docs/source/core_services/drivers/BACnet-Auto-Configuration.rst deleted file mode 100644 index 5f0b75cf4e..0000000000 --- a/docs/source/core_services/drivers/BACnet-Auto-Configuration.rst +++ /dev/null @@ -1,279 +0,0 @@ -.. _BACnet-Auto-Configuration: - -=================================================== -Automatically Generating BACnet Configuration Files -=================================================== - -Included with the platform are two scripts for finding and configuring BACnet devices. -These scripts are located in ``scripts/bacnet``. ``bacnet_scan.py`` will scan -the network for devices. ``grab_bacnet_config.py`` creates a CSV file for -the BACnet driver that can be used as a starting point for creating -your own register configuration. - -Both scripts are configured with the file ``BACpypes.ini``. - -Configuring the Utilities -------------------------- - -While running both scripts create a temporary virtual BACnet device -using the ``bacpypes`` library. The virtual -device must be configured properly in order to work. This -configuration is stored in ``scripts/bacnet/BACpypes.ini`` and will be -read automatically when the utility is run. - -The only value that (usually) needs to be changed is the **address** field. -**This is the address bound to the port on the machine you are running the script from, NOT -A TARGET DEVICE!** This value should be set to the IP address of the -network interface used to communicate with the remote device. If there -is more than one network interface you must use the address of the interface -connected to the network that can reach the device. - -In Linux you can usually get the addresses bound to all interfaces by running -``ifconfig`` from the command line. - -If a different outgoing port other than the default 47808 must be used, -it can be specified as part of the address in the form - - ``
:`` - -In some cases, the netmask of the network will be needed for proper configuration. -This can be done following this format - - ``
/:`` - -where ```` is the netmask length. The most common value is 24. See http://www.computerhope.com/jargon/n/netmask.htm - -In some cases, you may also need to specify a different device ID by -changing the value of **objectIdentifier** so the virtual BACnet device does -not conflict with any devices on the network. **objectIdentifier** -defaults to 599. - -Sample BACpypes.ini -******************* - -:: - - [BACpypes] - objectName: Betelgeuse - address: 10.0.2.15/24 - objectIdentifier: 599 - maxApduLengthAccepted: 1024 - segmentationSupported: segmentedBoth - vendorIdentifier: 15 - -Scanning for BACnet Devices ---------------------------- - -If the addresses for BACnet devices are unknown they can be discovered -using the ``bacnet_scan.py`` utility. - -To run the utility simply execute the following command: - - ``python bacnet_scan.py`` - -and expect output similar to this: - -:: - - Device Address =
- Device Id = 699 - maxAPDULengthAccepted = 1024 - segmentationSupported = segmentedBoth - vendorID = 15 - - Device Address = - Device Id = 540011 - maxAPDULengthAccepted = 480 - segmentationSupported = segmentedBoth - vendorID = 5 - -Reading Output -************** - -The address where the device can be reached is listed on the **Device Address** line. -The BACnet device ID is listed on the **Device Id** line. -The remaining lines are informational and not needed to configure the BACnet driver. - -For the first example, the IP address ``192.168.1.42`` can be used to reach -the device. The second device is behind a BACnet router and can be -reached at ``1002:11``. See RouterAddressing Remote Station addressing. - -BACNet Scan Options -******************* - - - ``--address ADDRESS`` Send the WhoIs request only to a specific address. Useful as a way to ping devices on a network that blocks broadcast traffic. - - ``--range LOW HIGH`` Specify the device ID range for the results. Useful for filtering. - - ``--timeout SECONDS`` Specify how long to wait for responses to the original broadcast. This defaults to 5 which should be sufficient for most networks. - - ``--csv-out CSV_OUT`` Write the discovered devices to a CSV file. This can be used as inout for ``grab_multiple_configs.py``. See `Scraping Multiple Devices`_. - -Automatically Generating a BACnet Registry Configuration File -------------------------------------------------------------- - -A CSV registry configuration file for the BACnet driver can be generated with the -``grab_bacnet_config.py`` script. **This configuration will need to be edited -before it can be used.** - -The utility is invoked with the command: - - ``python grab_bacnet_config.py `` - -This will query the device with the matching device ID for configuration -information and print the resulting CSV file to the console. - -In order to save the configuration to a file use the ``--out-file`` option to specify the -output file name. - -Optionally the ``--address`` option can be used to specify the address of the target. In some cases, this is needed to help -establish a route to the device. - -Output and Assumptions -********************** - -Attempts at determining if a point is writable proved too unreliable. -Therefore all points are considered to be read-only in the output. - -The only property for which a point is setup for an object is -**presentValue**. - -By default, the **Volttron Point Name** is set to the value of the **name** -property of the BACnet object on the device. In most cases this name is vague. -No attempt is made at choosing a better name. A -duplicate of "Volttron Point Name" column called "Reference Point Name" is created to so that -once "Volttron Point Name" is changed a reference remains to the actual -BACnet device object name. - -Meta data from the objects on the device is used to attempt to put -useful info in the **Units** **Unit Details**, and **Notes** columns. -Information such as the range of valid values, defaults, the resolution -or sensor input, and enumeration or state names are scraped from the -device. - -With a few exceptions "Units" is pulled from the object's "units" -property and given the name used by the bacpypes library to describe it. -If a value in the **Units** column takes the form - - ``UNKNOWN UNIT ENUM VALUE: `` - -then the device is using a nonstandard value for the units on that -object. - -Scraping Multiple Devices -------------------------- - -The ``grab_multiple_configs.py`` script will use the CSV output of bacnet_scan.py to automatically run -``grab_bacnet_config.py`` on every device listed in the CSV file. - -The output is put in two directories. ``devices/`` contains basic driver configurations for the scrapped devices. -``registry_configs/`` contains the registry file generated by grab_bacnet_config.py. - -``grab_multiple_configs.py`` makes no assumptions about device names or topics, however the output is appropriate for the -``install_master_driver_configs.py`` script. - -Grab Multiple Configs Options -***************************** - - - ``--out-directory OUT_DIRECTORY`` Specify the output directory. - - ``--use-proxy`` Use ``proxy_grab_bacnet_config.py`` to gather configuration data. - - -BACnet Proxy Alternative Scripts --------------------------------- - -Both ``grab_bacnet_config.py`` and ``bacnet_scan.py`` have alternative versions called -``proxy_grab_bacnet_config.py`` and ``proxy_bacnet_scan.py`` repectively. These versions require that the -VOLTTRON platform is running and BACnet Proxy agent is running. Both of these agents use the same command line -arguments as their independent counterparts. - -.. warning:: - - These versions of the BACnet scripts are intended as a proof of concept and have not been optimized for performance. - ``proxy_grab_bacnet_config.py`` takes about 10 times longer to grab a configuration than ``grab_bacnet_config.py`` - - - -Problems and Debugging ----------------------- - -Both ``grab_bacnet_config.py`` and ``bacnet_scan.py`` creates a virtual device that open up a port for communication with devices. -If BACnet Proxy is running on the VOLTTRON platform it will cause both of these scripts to fail at startup. -Stopping the BACnet Proxy will resolve the problem. - -Typically the utility should run quickly and finish in 30 seconds or -less. In our testing, we have never seen a successful scrape take more -than 15 seconds on a very slow device with many points. Many devices -will scrape in less that 3 seconds. - -If the utility has not finished after about 60 seconds it -is probably having trouble communicating with the device and should be -stopped. Rerunning with debug output can help diagnose the problem. - -To output debug messages to the console add the ``--debug`` switch to -the **end** of the command line arguments. - - ``python grab_bacnet_config.py --out-file test.csv --debug`` - -On a successful run you will see output similar to this: - -:: - - DEBUG:main:initialization - DEBUG:main: - args: Namespace(address='10.0.2.20', buggers=False, debug=[], ini=, max_range_report=1e+20, out_file=) - DEBUG:main.SynchronousApplication:init (, '10.0.2.15') - DEBUG:main:starting build - DEBUG:main:pduSource =
- DEBUG:main:iAmDeviceIdentifier = ('device', 500) - DEBUG:main:maxAPDULengthAccepted = 1024 - DEBUG:main:segmentationSupported = segmentedBoth - DEBUG:main:vendorID = 5 - DEBUG:main:device_name = MS-NCE2560-0 - DEBUG:main:description = - DEBUG:main:objectCount = 32 - DEBUG:main:object name = Building/FCB.Local Application.Room Real Temp 2 - DEBUG:main: object type = analogInput - DEBUG:main: object index = 3000274 - DEBUG:main: object units = degreesFahrenheit - DEBUG:main: object units details = -50.00 to 250.00 - DEBUG:main: object notes = Resolution: 0.1 - DEBUG:main:object name = Building/FCB.Local Application.Room Real Temp 1 - DEBUG:main: object type = analogInput - DEBUG:main: object index = 3000275 - DEBUG:main: object units = degreesFahrenheit - DEBUG:main: object units details = -50.00 to 250.00 - DEBUG:main: object notes = Resolution: 0.1 - DEBUG:main:object name = Building/FCB.Local Application.OSA - DEBUG:main: object type = analogInput - DEBUG:main: object index = 3000276 - DEBUG:main: object units = degreesFahrenheit - DEBUG:main: object units details = -50.00 to 250.00 - DEBUG:main: object notes = Resolution: 0.1 - ... - -and will finish something like this: - -:: - - ... - DEBUG:main:object name = Building/FCB.Local Application.MOTOR1-C - DEBUG:main: object type = binaryOutput - DEBUG:main: object index = 3000263 - DEBUG:main: object units = Enum - DEBUG:main: object units details = 0-1 (default 0) - DEBUG:main: object notes = BinaryPV: 0=inactive, 1=active - DEBUG:main:finally - -Typically if the BACnet device is unreachable for any reason (wrong IP, -network down/unreachable, wrong interface specified, device failure, -etc) the scraper will stall at this message: - -:: - - DEBUG:main:starting build - -If you have not specified a valid interface in BACpypes.ini you will see -the following error with a stack trace: - -:: - - ERROR:main:an error has occurred: [Errno 99] Cannot assign requested address - - diff --git a/docs/source/core_services/drivers/BACnet-Proxy-Agent.rst b/docs/source/core_services/drivers/BACnet-Proxy-Agent.rst deleted file mode 100644 index c62b384eea..0000000000 --- a/docs/source/core_services/drivers/BACnet-Proxy-Agent.rst +++ /dev/null @@ -1,220 +0,0 @@ -.. _BACnet-Proxy-Agent: - -================== -BACnet Proxy Agent -================== - -Introduction ------------- - -Communication with BACnet device on a network happens via a single -virtual BACnet device. In VOLTTRON driver framework, we use a separate -agent specifically for communicating with BACnet devices and managing -the virtual BACnet device. - -Requirements ------------- -The BACnet Proxy agent requires the BACPypes package. This package can -be installed in an activated environment with: - -:: - - pip install bacpypes - -Configuration -------------- - -The agent configuration sets up the virtual BACnet device. - -.. code-block:: json - - { - "device_address": "10.0.2.15", - "max_apdu_length": 1024, - "object_id": 599, - "object_name": "Volttron BACnet driver", - "vendor_id": 15, - "segmentation_supported": "segmentedBoth" - } - -BACnet device settings -********************** - -- **device_address** - Address bound to the network port over which - BACnet communication will happen on the computer running VOLTTRON. - This is **NOT** the address of any target device. See `Device Addressing`_. -- **object_id** - ID of the Device object of the virtual BACnet - device. Defaults to 599. Only needs to be changed if there is - a conflicting BACnet device ID on your network. - -These settings determine the capabilities of the virtual BACnet device. -BACnet communication happens at the lowest common denominator between -two devices. For instance, if the BACnet proxy supports segmentation and -the target device does not communication will happen without -segmentation support and will be subject to those limitations. -Consequently, there is little reason to change the default settings -outside of the **max_apdu_length** (the default is not the largest -possible value). - -- **max_apdu_length** - (From bacpypes documentation) BACnet works on - lots of different types of networks, from high-speed Ethernet to - “slower” and “cheaper” ARCNET or MS/TP (a serial bus protocol used - for a field bus defined by BACnet). For devices to exchange messages - they have to know the maximum size message the device can handle. - (End BACpypes docs) - - This setting determines the largest APDU accepted by the BACnet - virtual device. Valid options are 50, 128, 206, 480, 1024, and 1476. - Defaults to 1024.(Optional) - - -- **object_name** - Name of the object. Defaults to "Volttron BACnet - driver". (Optional) -- **vendor_id** - Vendor ID of the virtual BACnet device. Defaults to - 15. (Optional) -- **segmentation_supported** - (From bacpypes documentation) A vast - majority of BACnet communications traffic fits into one message, but - there can be times when larger messages are convenient and more - efficient. Segmentation allows larger messages to be broken up into - segments and spliced back together. It is not unusual for “low power” - field equipment to not support segmentation. (End BACpypes docs) - - Possible setting are "segmentedBoth" (default), "segmentedTransmit", - "segmentedReceive", or "noSegmentation" (Optional) - -Device Addressing ------------------ - -In some cases, it will be needed to specify the subnet mask of the -virtual device or a different port number to listen on. The full format -of the BACnet device address is - - ``
/:`` - -where ```` is the port to use and ```` is the netmask length. -The most common value is 24. See http://www.computerhope.com/jargon/n/netmask.htm - -For instance, if you need to specify a subnet mask of 255.255.255.0 -and the IP address bound to the network port is 192.168.1.2 you -would use the address - -:: - - 192.168.1.2/24 - -If your BACnet network is on a different port (47809) besides the -default (47808) you would use the address - -:: - - 192.168.1.2:47809 - -If you need to do both - -:: - - 192.168.1.2/24:47809 - -.. _bacnet-proxy-multiple-networks: - -Communicating With Multiple BACnet Networks -------------------------------------------- - -If two BACnet devices are connected to different ports they are -considered to be on different BACnet networks. In order to communicate -with both devices, you will need to run one BACnet Proxy Agent per -network. - -Each proxy will need to be bound to different ports appropriate for -each BACnet network and will need a different VIP identity specified. -When configuring drivers you will need to specify which proxy to use by -:ref:`specifying the VIP identity `. - -TODO: Add link to docs showing how to specify the VIP IDENTITY when installing an agent. - -For example, a proxy connected to the default BACnet network - -.. code-block:: json - - { - "device_address": "192.168.1.2/24" - } - -and another on port 47809 - -.. code-block:: json - - { - "device_address": "192.168.1.2/24:47809" - } - -a device on the first network - -.. code-block:: json - - { - "driver_config": {"device_address": "1002:12", - "proxy_address": "platform.bacnet_proxy_47808", - "timeout": 10}, - "driver_type": "bacnet", - "registry_config":"config://registry_configs/bacnet.csv", - "interval": 60, - "timezone": "UTC", - "heart_beat_point": "Heartbeat" - } - -and a device on the second network - -.. code-block:: json - - { - "driver_config": {"device_address": "12000:5", - "proxy_address": "platform.bacnet_proxy_47809", - "timeout": 10}, - "driver_type": "bacnet", - "registry_config":"config://registry_configs/bacnet.csv", - "interval": 60, - "timezone": "UTC", - "heart_beat_point": "Heartbeat" - } - -Notice that both configs use the same registry configuration -(config://registry_configs/bacnet.csv). This is perfectly fine as long as the -registry configuration is appropriate for both devices. -For scraping large numbers of points from a single BACnet device, -there is an optional timeout parameter provided, to prevent the master driver -timing out while the BACnet Proxy Agent is collecting points. - - -BACnet Change of Value Services -------------------------------- - -|BACnet Change of Value Communications| - -Change of Value Services added in version 0.5 of the BACnet Proxy and version -3.2 of the Master Driver. - -There are a variety of scenarios in which a user may desire data from some -BACnet device point values to be published independently of the regular -scrape interval. Bacpypes provides a "ChangeOfValueServices" (hereby -referred to as 'COV') module, which enables a device to push updates to the -platform. - -The BACnet COV requires that points on the device be properly configured for -COV. A point on the BACnet device can be configured with the 'covIncrement' -property, which determines the threshold for a COV notification (note: this -property must be configured by the device operator - VOLTTRON does not -provide the ability to set or modify this property). - -Based on configuration options for BACnet drivers, the driver will instruct the -BACnet Proxy to establish a COV subscription with the device. The subscription -will last for an amount of time specified in the driver configuration, and will -auto-renew the subscription. If the proxy loses communication with the device or -the device driver is stopped the subscription will be removed when the lifetime -expires. While the subscription exists, the device will send (confirmed) -notifications to which will be published, with the topic based on the -driver's configured publish topics. - -https://bacpypes.readthedocs.io/en/latest/modules/service/cov.html - -.. |BACnet Change of Value Communications| image:: files/bacnet_cov.png diff --git a/docs/source/core_services/drivers/BACnet-Router-Addressing.rst b/docs/source/core_services/drivers/BACnet-Router-Addressing.rst deleted file mode 100644 index aee348d75a..0000000000 --- a/docs/source/core_services/drivers/BACnet-Router-Addressing.rst +++ /dev/null @@ -1,38 +0,0 @@ -.. _BACnet-Router-Addressing: - -======================== -BACnet Router Addressing -======================== - -The underlying library that Volttron uses for BACnet supports IP to -MS/TP routers. Devices behind the router use a Remote Station address in -the form - -:: - - :
- -where **** is the configured network ID of the router and **
** -is the address of the device behind the router. - -For example to access the device at **
** 12 for a router configured -for **** 1002 can be accessed with this address: - -:: - - 1002:12 - -**** must be number from 0 to 65534 and **
** must be a number -from 0 to 255. - -This type of address can be used anywhere an address is required in -configuration of the Volttron BACnet driver. - -Caveats -------- - -VOLTTRON uses a UDP broadcast mechanism to establish the route to the device. -If the route cannot be established it will fall back to a UDP broadcast for -all communication with the device. -If the IP network where the router is connected blocks UDP -broadcast traffic then these addresses will not work. diff --git a/docs/source/core_services/drivers/Using-Third-Party-Drivers.rst b/docs/source/core_services/drivers/Using-Third-Party-Drivers.rst deleted file mode 100644 index 304176fc01..0000000000 --- a/docs/source/core_services/drivers/Using-Third-Party-Drivers.rst +++ /dev/null @@ -1,23 +0,0 @@ -.. _Using-Third-Party-Drivers: - -========================= -Using Third Party Drivers -========================= - -In some cases you will need to use a driver provided by a third-party to interact with a device. -While the interface file can be copied into ``services/core/MasterDriverAgent/master_driver/interfaces`` -this does not work well with third-party code that is under source control. - -The recommended method is to create a symbolic link to the interface file in -``services/core/MasterDriverAgent/master_driver/interfaces``. This will work in both -a development environment and in production. When packing the agent for installation -a copy of the linked file will be put in the resulting wheel file. - -:: - - #A copy of the interface file lives in ~/my_driver/my_driver.py - #Create the link - ln -s ~/my_driver/my_driver.py services/core/MasterDriverAgent/master_driver/interfaces/my_driver.py - - #remove the link - rm services/core/MasterDriverAgent/master_driver/interfaces/my_driver.py diff --git a/docs/source/core_services/drivers/driver_configuration/_Driver-Configuration.rst b/docs/source/core_services/drivers/driver_configuration/_Driver-Configuration.rst deleted file mode 100644 index c588015419..0000000000 --- a/docs/source/core_services/drivers/driver_configuration/_Driver-Configuration.rst +++ /dev/null @@ -1,306 +0,0 @@ -.. _Driver-Configuration: - -==================== -Driver Configuration -==================== -The Master Driver Agent manages all device communication. To communicate with devices you must setup and deploy the Master Driver Agent. - -Configuration for each device consists of 3 parts: - -* Master Driver Agent configuration file - lists all driver configuration files to load -* Driver configuration file - contains the general driver configuration and device settings -* Device Register configuration file - contains the settings for each individual data point on the device - -For each device, you must create a driver configuration file, device register configuration file, and an entry in the Master Driver Agent configuration file. - -Once configured, the Master Driver Agent is :ref:`configured and deployed -` in a manner similar to any other agent. - -.. _MasterDriverConfig: - -Requirements ------------- - -VOLTTRON drivers operated by the master driver may have additional requirements for installation. -Required libraries: - -:: - - BACnet driver - bacpypes - Modbus driver - pymodbus - Modbus_TK driver - modbus-tk - DNP3 and IEEE 2030.5 drivers - pydnp3 - -Master Driver Agent Configuration ---------------------------------- -The Master Driver Agent configuration consists of general settings for all devices. The default values of the master driver should be sufficient for most users. -The user may optionally change the interval between device scrapes with the driver_scrape_interval. - -The following example sets the driver_scrape_interval to 0.05 seconds or 20 devices per second: - -.. code-block:: json - - { - "driver_scrape_interval": 0.05, - "publish_breadth_first_all": false, - "publish_depth_first": false, - "publish_breadth_first": false, - "publish_depth_first_all": true, - "group_offset_interval": 0.0 - } - -* **driver_scrape_interval** - Sets the interval between devices scrapes. Defaults to 0.02 or 50 devices per second. Useful for when the platform scrapes too many devices at once resulting in failed scrapes. -* **group_offset_interval** - Sets the interval between when groups of devices are scraped. Has no effect if all devices are in the same group. - -In order to improve the scalability of the platform unneeded device state publishes for all devices can be turned off. -All of the following setting are optional and default to `True`. - -* **publish_depth_first_all** - Enable "depth first" publish of all points to a single topic for all devices. -* **publish_breadth_first_all** - Enable "breadth first" publish of all points to a single topic for all devices. -* **publish_depth_first** - Enable "depth first" device state publishes for each register on the device for all devices. -* **publish_breadth_first** - Enable "breadth first" device state publishes for each register on the device for all devices. - -An example master driver configuration file can be found in the VOLTTRON repository in ``services/core/MasterDriverAgent/master-driver.agent``. - -.. _driver-configuration-file: - -Driver Configuration File -------------------------- - -.. note:: - - The terms `register` and `point` are used interchangeably in the documentation and - in the configuration setting names. They have the same meaning. - -Each device configuration has the following form: - -.. code-block:: json - - { - "driver_config": {"device_address": "10.1.1.5", - "device_id": 500}, - "driver_type": "bacnet", - "registry_config":"config://registry_configs/vav.csv", - "interval": 60, - "heart_beat_point": "heartbeat", - "group": 0 - } - -The following settings are required for all device configurations: - - - **driver_config** - Driver specific setting go here. See below for driver specific settings. - - **driver_type** - Type of driver to use for this device: bacnet, modbus, fake, etc. - - **registry_config** - Reference to a configuration file in the configuration store for registers - on the device. See the `Registry Configuration File`_ - and `Adding Device Configurations to the Configuration Store`_ sections below. - -These settings are optional: - - - **interval** - Period which to scrape the device and publish the results in seconds. Defaults to 60 seconds. - - **heart_beat_point** - A Point which to toggle to indicate a heartbeat to the device. A point with this Volttron Point Name must exist in the registry. If this setting is missing the driver will not send a heart beat signal to the device. Heart beats are triggered by the Actuator Agent which must be running to use this feature. - - **group** - Group this device belongs to. Defaults to 0 - -These settings are used to create the topic that this device will be referenced by following the VOLTTRON convention of {campus}/{building}/{unit}. This will also be the topic published on, when the device is periodically scraped for it's current state. - -The topic used to reference the device is derived from the name of the device configuration in the store. See the `Adding Device Configurations to the Configuration Store`_ section. - -Device Grouping -............... - -Devices may be placed into groups to separate them logically when they are scraped. This is done by setting the `group` in the device configuration. `group` is a number greater than or equal to 0. -Only number of devices in the same group and the `group_offset_interval` are considered when determining when to scrape a device. - -This is useful in two cases. First, if you need to ensure that certain devices are scraped in close proximity to each other you can put them in their own group. -If this causes devices to be scraped too quickly the groups can be separated out time wise using the `group_offset_interval` setting. -Second, you may scrape devices on different networks in parallel for performance. For instance BACnet devices behind a single MSTP router need to be scraped slowly and serially, but devices behind different routers may be scraped in parallel. Grouping devices by router will do this automatically. - -The `group_offset_interval` is applied by multiplying it by the `group` number. If you intent to use `group_offset_interval` only use consecutive `group` values that start with 0. - - -Registry Configuration File ---------------------------- -Registry configuration files setup each individual point on a device. Typically this file will be in CSV format, but the exact format is driver specific. See the section for a particular driver for the registry configuration format. - -The following is a simple example of a MODBUS registry configuration file: - -.. csv-table:: Catalyst 371 - :header: Reference Point Name,Volttron Point Name,Units,Units Details,Modbus Register,Writable,Point Address,Default Value,Notes - - CO2Sensor,ReturnAirCO2,PPM,0.00-2000.00,>f,FALSE,1001,,CO2 Reading 0.00-2000.0 ppm - CO2Stpt,ReturnAirCO2Stpt,PPM,1000.00 (default),>f,TRUE,1011,1000,Setpoint to enable demand control ventilation - HeatCall2,HeatCall2,On / Off,on/off,BOOL,FALSE,1114,,Status indicator of heating stage 2 need - -.. _config-store: - -======================================================= -Adding Device Configurations to the Configuration Store -======================================================= - -Configurations are added to the Configuration Store using the command line `volttron-ctl config store platform.driver `. - -* **name** - The name used to refer to the file from the store. -* **file name** - A file containing the contents of the configuration. -* **file type** - `--raw`, `--json`, or `--csv`. Indicates the type of the file. Defaults to `--json`. - -The main configuration must have the name `config` - -Device configuration but **not** registry configurations must have a name prefixed with `devices/`. Scripts that automate the process will prefix registry configurations with `registry_configs/`, but that is not a requirement for registry files. - -The name of the device's configuration in the store is used to create the topic used to reference the device. For instance, a configuration named ``devices/PNNL/ISB1/vav1`` will publish scrape results to ``devices/PNNL/ISB1/vav1`` and is accessible with the Actuator Agent via ``PNNL/ISB1/vav1``. - -The name of a registry configuration must match the name used to refer to it in the driver configuration. The reference is not case sensitive. - -If the Master Driver Agent is running any changes to the configuration store will immediately affect the running devices according to the changes. - -Consider the following three configuration files: - -A master driver configuration called `master-driver.agent`: - -.. code-block:: json - - { - "driver_scrape_interval": 0.05 - } - -A MODBUS device configuration file called `modbus1.config`: - -.. code-block:: json - - { - "driver_config": {"device_address": "10.1.1.2", - "port": 502, - "slave_id": 5}, - "driver_type": "modbus", - "registry_config":"config://registry_configs/hvac.csv", - "interval": 60, - "timezone": "UTC", - "heart_beat_point": "heartbeat" - } - -A MODBUS registry configuration file called `catalyst371.csv`: - -.. csv-table:: catalyst371.csv - :header: Reference Point Name,Volttron Point Name,Units,Units Details,Modbus Register,Writable,Point Address,Default Value,Notes - - CO2Sensor,ReturnAirCO2,PPM,0.00-2000.00,>f,FALSE,1001,,CO2 Reading 0.00-2000.0 ppm - CO2Stpt,ReturnAirCO2Stpt,PPM,1000.00 (default),>f,TRUE,1011,1000,Setpoint to enable demand control ventilation - HeatCall2,HeatCall2,On / Off,on/off,BOOL,FALSE,1114,,Status indicator of heating stage 2 need - -To store the master driver configuration run the command - -``volttron-ctl config store platform.driver config master-driver.agent`` - -To store the registry configuration run the command (note the --csv option) - -``volttron-ctl config store platform.driver registry_configs/hvac.csv catalyst371.csv --csv`` - -Note the name ``registry_configs/hvac.csv`` matches the configuration reference in the file ``modbus1.config``. - -To store the driver configuration run the command - -``volttron-ctl config store platform.driver devices/my_campus/my_building/hvac1 modbus1.config`` - - -Converting Old Style Configuration ----------------------------------- - -The new Master Driver no longer supports the old style of device configuration. The old ``device_list`` setting is ignored. - -To simplify updating to the new format ``scripts/update_master_driver_config.py`` is provide to automatically update to the new configuration format. - -With the platform running run: - -``python scripts/update_master_driver_config.py `` - -**old_configuration** is the main configuration file in the old format. The script automatically modifies the driver files to create references to CSV files and adds the CSV files with the appropriate name. - -**output** is the target output directory. - -If the ``--keep-old`` switch is used the old configurations in the output directory (if any) will not be deleted before new configurations are created. Matching names will still be overwritten. - -The output from ``scripts/update_master_driver_config.py`` can be automatically added to the configuration store -for the Master Driver agent with ``scripts/install_master_driver_configs.py``. - -Creating and naming configuration files in the form needed by ``scripts/install_master_driver_configs.py`` -can speed up the process of changing and updating a large number of configurations. See the ``--help`` -message for ``scripts/install_master_driver_configs.py`` for more details. - -Device State Publishes ----------------------- - -By default, the value of each register on a device is published 4 different ways when the device state is published. -Consider the following settings in a driver configuration stored under the name ``devices/pnnl/isb1/vav1``: - -.. code-block:: json - - { - "driver_config": {"device_address": "10.1.1.5", - "device_id": 500}, - - "driver_type": "bacnet", - "registry_config":"config://registry_configs/vav.csv", - } - -In the ``vav.csv`` file is a register with the name ``temperature``. For these examples -the current value of the register on the device happens to be 75.2 and the meta data -is - -.. code-block:: python - - {"units": "F"} - -When the driver publishes the device state the following 2 things will be published for this register: - - A "depth first" publish to the topic ``devices/pnnl/isb1/vav1/temperature`` - with the following message: - - .. code-block:: python - - [75.2, {"units": "F"}] - - A "breadth first" publish to the topic ``devices/temperature/vav1/isb1/pnnl`` - with the following message: - - .. code-block:: python - - [75.2, {"units": "F"}] - - These publishes can be turned off by setting `publish_depth_first` and `publish_breadth_first` to `false` respectively. - -Also these two publishes happen once for all registers: - - A "depth first" publish to the topic ``devices/pnnl/isb1/vav1/all`` - with the following message: - - .. code-block:: python - - [{"temperature": 75.2, ...}, {"temperature":{"units": "F"}, ...}] - - A "breadth first" publish to the topic ``devices/all/vav1/isb1/pnnl`` - with the following message: - - .. code-block:: python - - [{"temperature": 75.2, ...}, {"temperature":{"units": "F"}, ...}] - - These publishes can be turned off by setting `publish_depth_first_all` and `publish_breadth_first_all` to `false` respectively. - -Device Scalability Settings ---------------------------- - -In order to improve the scalability of the platform unneeded device state publishes for a device can be turned off. -All of the following setting are optional and will override the value set in the main master driver configuration. - - - **publish_depth_first_all** - Enable "depth first" publish of all points to a single topic. - - **publish_breadth_first_all** - Enable "breadth first" publish of all points to a single topic. - - **publish_depth_first** - Enable "depth first" device state publishes for each register on the device. - - **publish_breadth_first** - Enable "breadth first" device state publishes for each register on the device. - -It is common practice to set **publish_breadth_first_all**, **publish_depth_first**, and -**publish_breadth_first** to `False` unless they are specifically needed by an agent running on -the platform. - - -.. note:: - - All Historian Agents require **publish_depth_first_all** to be set to `True` in order to capture data. diff --git a/docs/source/core_services/drivers/driver_configuration/bacnet-driver.rst b/docs/source/core_services/drivers/driver_configuration/bacnet-driver.rst deleted file mode 100644 index 426c0e6da5..0000000000 --- a/docs/source/core_services/drivers/driver_configuration/bacnet-driver.rst +++ /dev/null @@ -1,100 +0,0 @@ -.. _BACnet-Driver: - -BACnet Driver Configuration ---------------------------- -Communicating with BACnet devices requires that the BACnet Proxy Agent is configured and running. All device communication happens through this agent. - -Requirements ------------- -The BACnet driver requires the Pint package. This package can be installed in an -activated environment with: - -:: - - pip install bacpypes - -driver_config -************* - -There are nine arguments for the "driver_config" section of the device configuration file: - - - **device_address** - Address of the device. If the target device is behind an IP to MS/TP router then Remote Station addressing will probably be needed for the driver to find the device. - - **device_id** - BACnet ID of the device. Used to establish a route to the device at startup. - - **min_priority** - (Optional) Minimum priority value allowed for this device whether specifying the priority manually or via the registry config. Violating this parameter either in the configuration or when writing to the point will result in an error. Defaults to 8. - - **max_per_request** - (Optional) Configure driver to manually segment read requests. The driver will only grab up to the number of objects specified in this setting at most per request. This setting is primarily for scraping many points off of low resource devices that do not support segmentation. Defaults to 10000. - - **proxy_address** - (Optional) VIP address of the BACnet proxy. Defaults to "platform.bacnet_proxy". See :ref:`bacnet-proxy-multiple-networks` for details. Unless your BACnet network has special needs you should not change this value. - - **ping_retry_interval** - (Optional) The driver will ping the device to establish a route at startup. If the BACnet proxy is not available the driver will retry the ping at this interval until it succeeds. Defaults to 5. - - **use_read_multiple** - (Optional) During a scrape the driver will tell the proxy to use a ReadPropertyMultipleRequest to get data from the device. Otherwise the proxy will use multiple ReadPropertyRequest calls. If the BACnet proxy is reporting a device is rejecting requests try changing this to false for that device. Be aware that setting this to false will cause scrapes for that device to take much longer. Only change if needed. Defaults to true. - - **cov_lifetime** - (Optional) When a device establishes a change of value subscription for a point, this argument will be used to determine the lifetime and renewal period for the subscription, in seconds. Defaults to 180. (Added to Master Driver version 3.2) - -Here is an example device configuration file: - -.. code-block:: json - - { - "driver_config": {"device_address": "10.1.1.3", - "device_id": 500, - "min_priority": 10, - "max_per_request": 24 - }, - "driver_type": "bacnet", - "registry_config":"config://registry_configs/vav.csv", - "interval": 5, - "timezone": "UTC", - "heart_beat_point": "heartbeat" - } - -A sample BACnet configuration file can be found `here `_ or -in the VOLTTRON repository in ``examples/configurations/drivers/bacnet1.config`` - -.. _BACnet-Registry-Configuration-File: - -BACnet Registry Configuration File -********************************** - -The registry configuration file is a `CSV `_ file. Each row configures a point on the device. - -Most of the configuration file can be generated with the ``grab_bacnet_config.py`` utility in ``scripts/bacnet``. See :doc:`BACnet-Auto-Configuration`. - -Currently, the driver provides no method to access array type properties even if the members of the array are of a supported type. - -The following columns are required for each row: - - - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this point. For instance, if the Volttron Point Name is HeatCall1 (and using the example device configuration above) then an agent would use "pnnl/isb2/hvac1/HeatCall1" to refer to the point when using the RPC interface of the actuator agent. - - **Units** - Used for meta data when creating point information on the historian. - - **BACnet Object Type** - A string representing what kind of BACnet standard object the point belongs to. Examples include: - - * analogInput - * analogOutput - * analogValue - * binaryInput - * binaryOutput - * binaryValue - * multiStateValue - - - **Property** - A string representing the name of the property belonging to the object. Usually, this will be "presentValue". - - **Writable** - Either "TRUE" or "FALSE". Determines if the point can be written to. Only points labeled TRUE can be written to through the ActuatorAgent. Points labeled "TRUE" incorrectly will cause an error to be returned when an agent attempts to write to the point. - - **Index** - Object ID of the BACnet object. - -The following columns are optional: - - - **Write Priority** - BACnet priority for writing to this point. Valid values are 1-16. Missing this column or leaving the column blank will use the default priority of 16. - - **COV Flag** - Either "True" or False". Determines if a BACnet Change of Value subscription should be established for this point. Missing this column or leaving the column blank will result in no change of value subscriptions being established. (Added to Master Driver version 3.2) - -Any additional columns will be ignored. It is common practice to include a **Point Name** or **Reference Point Name** to include the device documentation's name for the point and **Notes** and **Unit Details**" for additional information about a point. - -.. csv-table:: BACnet - :header: Point Name,Volttron Point Name,Units,Unit Details,BACnet Object Type,Property,Writable,Index,Notes - - Building/FCB.Local Application.PH-T,PreheatTemperature,degreesFahrenheit,-50.00 to 250.00,analogInput,presentValue,FALSE,3000119,Resolution: 0.1 - Building/FCB.Local Application.RA-T,ReturnAirTemperature,degreesFahrenheit,-50.00 to 250.00,analogInput,presentValue,FALSE,3000120,Resolution: 0.1 - Building/FCB.Local Application.RA-H,ReturnAirHumidity,percentRelativeHumidity,0.00 to 100.00,analogInput,presentValue,FALSE,3000124,Resolution: 0.1 - Building/FCB.Local Application.CLG-O,CoolingValveOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000107,Resolution: 0.1 - Building/FCB.Local Application.MAD-O,MixedAirDamperOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000110,Resolution: 0.1 - Building/FCB.Local Application.PH-O,PreheatValveOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000111,Resolution: 0.1 - Building/FCB.Local Application.RH-O,ReheatValveOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000112,Resolution: 0.1 - Building/FCB.Local Application.SF-O,SupplyFanSpeedOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000113,Resolution: 0.1 - - -A sample BACnet registry file can be found `here `_ or -in the VOLTTRON repository in ``examples/configurations/drivers/bacnet.csv`` diff --git a/docs/source/core_services/drivers/driver_configuration/fake-driver.rst b/docs/source/core_services/drivers/driver_configuration/fake-driver.rst deleted file mode 100644 index 2c3a6ba907..0000000000 --- a/docs/source/core_services/drivers/driver_configuration/fake-driver.rst +++ /dev/null @@ -1,64 +0,0 @@ -.. _Fake-Driver: - -Fake Device Driver Configuration --------------------------------- -This driver does not connect to any actual device and instead produces random and or pre-configured values. - -driver_config -************* - -There are no arguments for the "driver_config" section of the device configuration file. The driver_config entry must still be present and should be left blank - -Here is an example device configuration file: - -.. code-block:: json - - { - "driver_config": {}, - "driver_type": "bacnet", - "registry_config":"config://registry_configs/vav.csv", - "interval": 5, - "timezone": "UTC", - "heart_beat_point": "heartbeat" - } - -A sample fake device configuration file can be found in the VOLTTRON repository in ``examples/configurations/drivers/fake.config`` - -Fake Device Registry Configuration File -*************************************** - -The registry configuration file is a `CSV `_ file. Each row configures a point on the device. - -The following columns are required for each row: - - - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this point. For instance, if the Volttron Point Name is HeatCall1 (and using the example device configuration above) then an agent would use *pnnl/isb2/hvac1/HeatCall1* to refer to the point when using the RPC interface of the actuator agent. - - **Units** - Used for meta data when creating point information on the historian. - - **Writable** - Either "TRUE" or "FALSE". Determines if the point can be written to. Only points labeled TRUE can be written to through the ActuatorAgent. Points labeled "TRUE" incorrectly will cause an error to be returned when an agent attempts to write to the point. - - -The following columns are optional: - - - **Starting Value** - Initial value for the point. If the point is reverted it will change back to this value. By default, points will start with a random value (1-100). - - **Type** - Value type for the point. Defaults to "string". Valid types are: - - * string - * integer - * float - * boolean - -Any additional columns will be ignored. It is common practice to include a **Point Name** or **Reference Point Name** to include the device documentation's name for the point and **Notes** and **Unit Details** for additional information about a point. Please note that there is nothing in the driver that will enforce anything specified in the **Unit Details** column. - -.. csv-table:: BACnet - :header: Volttron Point Name,Units,Units Details,Writable,Starting Value,Type,Notes - - Heartbeat,On/Off,On/Off,TRUE,0,boolean,Point for heartbeat toggle - OutsideAirTemperature1,F,-100 to 300,FALSE,50,float,CO2 Reading 0.00-2000.0 ppm - SampleWritableFloat1,PPM,10.00 (default),TRUE,10,float,Setpoint to enable demand control ventilation - SampleLong1,Enumeration,1 through 13,FALSE,50,int,Status indicator of service switch - SampleWritableShort1,%,0.00 to 100.00 (20 default),TRUE,20,int,Minimum damper position during the standard mode - SampleBool1,On / Off,on/off,FALSE,TRUE,boolean,Status indicator of cooling stage 1 - SampleWritableBool1,On / Off,on/off,TRUE,TRUE,boolean,Status indicator - - -A sample fake registry configuration file can be found `here `_ or -in the VOLTTRON repository in ``examples/configurations/drivers/fake.csv`` diff --git a/docs/source/core_services/drivers/driver_configuration/index.rst b/docs/source/core_services/drivers/driver_configuration/index.rst deleted file mode 100644 index 3a7cda0110..0000000000 --- a/docs/source/core_services/drivers/driver_configuration/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. _VOLTTRON-Drivers: - -================ -VOLTTRON Drivers -================ - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/drivers/driver_configuration/modbus-driver.rst b/docs/source/core_services/drivers/driver_configuration/modbus-driver.rst deleted file mode 100644 index 4d9932b290..0000000000 --- a/docs/source/core_services/drivers/driver_configuration/modbus-driver.rst +++ /dev/null @@ -1,92 +0,0 @@ -.. _MODBUS-config: - -Modbus Driver Configuration ---------------------------- -VOLTTRON's modbus driver supports the Modbus over TCP/IP protocol only. For Modbus RTU support, -see VOLTTRON's modbus-tk driver. - -Requirements ------------- -The Modbus driver requires the pymodbus package. This package can be installed in an -activated environment with: - -:: - - pip install pymodbus - -driver_config -************* - -There are three arguments for the **driver_config** section of the device configuration file: - - - **device_address** - IP Address of the device. - - **port** - Port the device is listening on. Defaults to 502 which is the standard port for MODBUS devices. - - **slave_id** - Slave ID of the device. Defaults to 0. Use 0 for no slave. - -Here is an example device configuration file: - -.. code-block:: json - - { - "driver_config": {"device_address": "10.1.1.2", - "port": 502, - "slave_id": 5}, - "driver_type": "modbus", - "registry_config":"config://registry_configs/hvac.csv", - "interval": 60, - "timezone": "UTC", - "heart_beat_point": "heartbeat" - } - -A sample MODBUS configuration file can be found in the VOLTTRON repository in ``examples/configurations/drivers/modbus.config`` - - -.. _MODBUS-Driver: - -Modbus Registry Configuration File -********************************** - -The registry configuration file is a `CSV `_ file. Each row configures a point on the device. - -The following columns are required for each row: - - - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this point. For instance, if the Volttron Point Name is HeatCall1 (and using the example device configuration above) then an agent would use ``pnnl/isb2/hvac1/HeatCall1`` to refer to the point when using the RPC interface of the actuator agent. - - **Units** - Used for meta data when creating point information on the historian. - - **Modbus Register** - A string representing how to interpret the data register and how to read it from the device. The string takes two forms: - - + "BOOL" for coils and discrete inputs. - + A format string for the Python struct module. See http://docs.python.org/2/library/struct.html for full documentation. The supplied format string must only represent one value. See the documentation of your device to determine how to interpret the registers. Some Examples: - - * ">f" - A big endian 32-bit floating point number. - * "l" - A big endian 32-bit integer. - - - **Writable** - Either "TRUE" or "FALSE". Determines if the point can be written to. Only points labeled TRUE can be written to through the ActuatorAgent. - - **Point Address** - Modbus address of the point. Cannot include any offset value, it must be the exact value of the address. - - **Mixed Endian** - (Optional) Either "TRUE" or "FALSE". For mixed endian values. This will reverse the order of the MODBUS registers that make up this point before parsing the value or writing it out to the device. Has no effect on bit values. - -The following column is optional: - - - **Default Value** - The default value for the point. When the point is reverted by an agent it will change back to this value. If this value is missing it will revert to the last known value not set by an agent. - -Any additional columns will be ignored. It is common practice to include a **Point Name** or **Reference Point Name** to include the device documentation's name for the point and **Notes** and **Unit Details** for additional information about a point. - -The following is an example of a MODBUS registry confugration file: - -.. csv-table:: Catalyst 371 - :header: Reference Point Name,Volttron Point Name,Units,Units Details,Modbus Register,Writable,Point Address,Default Value,Notes - - CO2Sensor,ReturnAirCO2,PPM,0.00-2000.00,>f,FALSE,1001,,CO2 Reading 0.00-2000.0 ppm - CO2Stpt,ReturnAirCO2Stpt,PPM,1000.00 (default),>f,TRUE,1011,1000,Setpoint to enable demand control ventilation - Cool1Spd,CoolSupplyFanSpeed1,%,0.00 to 100.00 (75 default),>f,TRUE,1005,75,Fan speed on cool 1 call - Cool2Spd,CoolSupplyFanSpeed2,%,0.00 to 100.00 (90 default),>f,TRUE,1007,90,Fan speed on Cool2 Call - Damper,DamperSignal,%,0.00 - 100.00,>f,FALSE,1023,,Output to the economizer damper - DaTemp,DischargeAirTemperature,F,(-)39.99 to 248.00,>f,FALSE,1009,,Discharge air reading - ESMEconMin,ESMDamperMinPosition,%,0.00 to 100.00 (5 default),>f,TRUE,1013,5,Minimum damper position during the energy savings mode - FanPower,SupplyFanPower, kW,0.00 to 100.00,>f,FALSE,1015,,Fan power from drive - FanSpeed,SupplyFanSpeed,%,0.00 to 100.00,>f,FALSE,1003,,Fan speed from drive - HeatCall1,HeatCall1,On / Off,on/off,BOOL,FALSE,1113,,Status indicator of heating stage 1 need - HeartBeat,heartbeat,On / Off,on/off,BOOL,FALSE,1114,,Status indicator of heating stage 2 need - -A sample MODBUS registry file can be found `here `_ or -in the VOLTTRON repository in ``examples/configurations/drivers/catalyst371.csv`` diff --git a/docs/source/core_services/drivers/driver_configuration/rainforest-driver.rst b/docs/source/core_services/drivers/driver_configuration/rainforest-driver.rst deleted file mode 100644 index 8d8cce3a73..0000000000 --- a/docs/source/core_services/drivers/driver_configuration/rainforest-driver.rst +++ /dev/null @@ -1,49 +0,0 @@ -.. _Rainforest-Driver: - -Rainforest Emu2 Driver Configuration ------------------------------------- - -The Emu2 is a device for connecting to and reading data from smart power meters. -We have an experimental driver to talk to this device. It requires cloning the -Rainforest Automation library which can be found -`here `_. - -.. note:: - - The Emu Serial Api library has its own dependencies which should be installed - with pip while the VOLTTRON environment is activated. - - :: - - pip install -r requirements.txt - -The Emu2 device interface is configured as follows. Set `emu_library_path` -to the location of the cloned library. `tty` should be set to the name of the -Emu2's character special file. One way to find this is to run `dmesg` before -and after plugging in the Emu2, and checking the new output. - -.. code-block:: json - - { - "driver_config": { - "tty": "ttyACM0", - "emu_library_path": "/home/volttron/Emu-Serial-Api" - }, - "driver_type": "rainforestemu2", - "interval": 30, - "registry_config": "config://emu2.json", - "timezone": "UTC" - } - -The registry config file referred to in the first configuration must be an array -of strings. This tells the interface which data points should be retrieved from -the device every interval. If the NetworkInfo point is omitted it will be -included automatically. - -.. code-block:: json - - [ - "NetworkInfo", - "InstantaneousDemand", - "PriceCluster" - ] diff --git a/docs/source/core_services/drivers/index.rst b/docs/source/core_services/drivers/index.rst deleted file mode 100644 index 8425b97d03..0000000000 --- a/docs/source/core_services/drivers/index.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _VOLTTRON-Driver-Framework: - -========================= -VOLTTRON Driver Framework -========================= - -All Voltton drivers are implemented through the Master Driver Agent and are technically sub-agents -running in the same process as the Master Driver Agent. Each of these driver sub-agents is responsible -for creating an interface to a single device. Creating that interface is facilitated by an instance of -an interface class. There are a variety of interface classes included. The most commonly used interfaces -are BACnet and Modbus. - -.. toctree:: - :glob: - :maxdepth: 2 - - * - - driver_configuration/index diff --git a/docs/source/core_services/drivers/master_driver_override.rst b/docs/source/core_services/drivers/master_driver_override.rst deleted file mode 100644 index 04f16975d4..0000000000 --- a/docs/source/core_services/drivers/master_driver_override.rst +++ /dev/null @@ -1,106 +0,0 @@ -.. _Master_Driver_Override: - -====================== -Master Driver Override -====================== - -By default, every user is allowed write access to the devices by the master driver. The override -feature will allow the user (for example, building administrator) to override this default -behavior and enable the user to lock the write access on the devices for a specified duration of -time or indefinitely. - -Set Override On ---------------- - -The Master Driver's "set_override_on" RPC method can be used to set the override condition for -all drivers with topic matching the provided pattern. This can be specific devices, groups of -devices, or even all configured devices. The pattern matching is based on bash style filename -matching semantics. - -Parameters: - - pattern: - Override pattern to be applied. For example, - If pattern is campus/building1/* - Override condition is applied for all the - devices under campus/building1/. - If pattern is campus/building1/ahu1 - Override condition is applied for only - campus/building1/ahu1. The pattern matching is based on bash style filename - matching semantics. - duration: - Time duration for the override in seconds. If duration <= 0.0, it implies as - indefinite duration. - failsafe_revert: - Flag to indicate if all the devices falling under the override condition has to be set - to its default state/value immediately. - staggered_revert: - If this flag is set, reverting of devices will be staggered. - -Example "set_override_on" RPC call: - -:: - - self.vip.rpc.call(PLATFORM_DRIVER, "set_override_on", , ) - -Set Override Off ----------------- - -The override condition can also be toggled off based on a provided pattern using the Master -Driver's "set_override_off" RPC call. - -Parameters: - - pattern: - Override pattern to be applied. For example, - If pattern is campus/building1/* - Override condition is applied for all the - devices under campus/building1/. - If pattern is campus/building1/ahu1 - Override condition is applied for only - campus/building1/ahu1. The pattern matching is based on bash style filename - matching semantics. - -Example "set_override_off" RPC call: - -:: - - self.vip.rpc.call(PLATFORM_DRIVER, "set_override_off", ) - -Get Override Devices --------------------- - -A list of all overridden devices can be obtained with the Master Driver's "get_override_devices" -RPC call. - -This method call has no additional parameters - -Example "get_override_devices" RPC call: - -:: - - self.vip.rpc.call(PLATFORM_DRIVER, "get_override_devices") - -Get Override Patterns ---------------------- - -A list of all patterns which have been requested for override can be obtained with the Master -Driver's "get_override_patterns" RPC call. - -This method call has no additional parameters - -Example "get_override_patterns" RPC call: - -:: - - self.vip.rpc.call(PLATFORM_DRIVER, "get_override_patterns") - -Clear Overrides ---------------- - -All overrides set by RPC calls described above can be toggled off at using a single -"clear_overrides" RPC call. - -This method call has no additional parameters - -Example "clear_overrides" RPC call: - -:: - - self.vip.rpc.call(PLATFORM_DRIVER, "clear_overrides") diff --git a/docs/source/core_services/historians/Crate-Historian.rst b/docs/source/core_services/historians/Crate-Historian.rst deleted file mode 100644 index 8ab0fa439b..0000000000 --- a/docs/source/core_services/historians/Crate-Historian.rst +++ /dev/null @@ -1,81 +0,0 @@ -.. _Crate-Historian: - -=============== -Crate Historian -=============== - -Crate is an open source SQL database designed on top of a No-SQL design. It -allows automatic data replication and self-healing clusters for high -availability, automatic sharding, and fast joins, aggregations and sub-selects. - -Find out more about crate from ``_. - - -Prerequisites -~~~~~~~~~~~~~ - -1. Crate Database ------------------ - -For Arch Linux, Debian, RedHat Enterprise Linux and Ubuntu distributions there -is a dead simple installer to get crate up and running on your system. - - :: - - sudo bash -c "$(curl -L https://try.crate.io)" - -This command will download and install all of the requirements for running -crate, create a crate user and install a crate service. After the installation -the service will be available for viewing at http://localhost:4200 by default. - -.. note:: There is no authentication support within crate. - - -2. Crate Driver ---------------- - -There is a python library for crate that must be installed in the volttron -python environment in order to access crate. From an activated environment, -in the root of the volttron folder, execute the following command: - - :: - - python bootstrap.py --crate - - -or - - :: - - python bootstrap.py --crate - - -or - - :: - - pip install crate - - -Configuration -~~~~~~~~~~~~~ - -Because there is no authorization to access a crate database the configuration -for the CrateHistorian is very easy. - -.. code-block:: python - - { - "connection": { - "type": "crate", - # Optional table prefix defaults to historian - "schema": "testing", - "params": { - "host": "localhost:4200" - } - } - } - -Finally package, install and start the CrateHistorian agent. - -.. seealso:: :ref:`Agent Development Walkthrough ` diff --git a/docs/source/core_services/historians/DataMover-Historian.rst b/docs/source/core_services/historians/DataMover-Historian.rst deleted file mode 100644 index f418b5aa9c..0000000000 --- a/docs/source/core_services/historians/DataMover-Historian.rst +++ /dev/null @@ -1,60 +0,0 @@ -.. _DataMover: - -DataMover Historian -=================== - -DataMover sends data from its platform to a remote platform in cases where -there is not sufficient resources to store locally. It shares this -functionality with the :ref:`Forward Historian `, -However DataMover does not have a goal of data appearing "live" on the -remote platform. This allows DataMover to be more efficient by both batching -data and by sending an RPC call to a remote historian instead of publishing -data on the remote message bus. This allows allows DataMover to be more -robust by ensuring that the receiving historian is running. If the target is -unreachable, DataMover will cache data until it is available. - -Configuration -------------- - -The default configuration file is -*services/core/DataMover/config* . Change **destination-vip** to -point towards the foreign Volttron instance. - -:: - - { - "destination-vip": "ipc://@/home/volttron/.volttron/run/vip.socket", - "destination-serverkey": null, - "required_target_agents": [], - "custom_topic_list": [], - "services_topic_list": [ - "devices", "analysis", "record", "datalogger", "actuators" - ], - "topic_replace_list": [ - #{"from": "FromString", "to": "ToString"} - ] - } - - -The **services_topic_list** allows you to specify which of the main data topics -to forward. If there is no entry, the historian defaults to sending all. - -**topic_replace_list** allows you to replace portions of topics if needed. This -could be used to correct or standardize topics or to replace building/device -names with an anonymized version. The receiving platform will only see the -replaced values. - - - -Adding the configuration option below will limit the backup cache -to *n* gigabytes. This will keep your hard drive from filling up if -the agent is disconnected from its target for a long time. - -:: - - "backup_storage_limit_gb": n - -See Also -~~~~~~~~ - -:ref:`Historians ` diff --git a/docs/source/core_services/historians/Forward-Historian.rst b/docs/source/core_services/historians/Forward-Historian.rst deleted file mode 100644 index a9a42bd7b4..0000000000 --- a/docs/source/core_services/historians/Forward-Historian.rst +++ /dev/null @@ -1,61 +0,0 @@ -.. _Forward-Historian: - -Forward Historian -================= - -The primary use case for the ForwardHistorian is to send data to another -instance of VOLTTRON as if the data were live. This allows agents running on a -more secure and/or more powerful machine to run analysis on data being -collected on a potentially less secure/powerful board. - -Given this use case, it is not optimized for batching large amounts of data -when liveness is not needed. For this use case, please see the -:ref:`DataMover Historian `. - -The forward historian can be found in the services/core directory. - -Configuration -------------- - -The default configuration file is -*services/core/ForwardHistorian/config* . Change **destination-vip** to -point towards the foreign Volttron instance. - -:: - - { - "agentid": "forwarder", - "destination-vip": "ipc://@/home/volttron/.volttron/run/vip.socket" - } - -In order to send to a remote platform, you will need its VIP address and server -key. The server key can be found by running - -:: - - vctl auth serverkey - -Put the result into the following example -(Note the example uses a local IP address) - -:: - - { - "agentid": "forwarder", - "destination-vip": "tcp://127.0.0.1:22916", - "destination-serverkey": "" - } - - -Adding the configuration option below will limit the backup cache -to *n* gigabytes. This will keep your hard drive from filling up if -the agent is disconnected from its target for a long time. - -:: - - "backup_storage_limit_gb": n - -See Also -~~~~~~~~ - -:ref:`Historians ` diff --git a/docs/source/core_services/historians/Historian-Topic-Syntax.rst b/docs/source/core_services/historians/Historian-Topic-Syntax.rst deleted file mode 100644 index 1f4467ae66..0000000000 --- a/docs/source/core_services/historians/Historian-Topic-Syntax.rst +++ /dev/null @@ -1,110 +0,0 @@ -.. _Historian-Topic-Syntax: - -Historian Topic Syntax -====================== - -Each historian will subscribe to the following message bus topics -(datalogger/*, anaylsis/*, record/\* and devices/\*). For each of these -topics there is a different message syntax that must be adhered to in -order for the correct interpretation of the data being specified. - -record/\* ---------- -The record topic is the most flexible of all of the topics. This topic allows -any serializable message to be published to any topic under the root topic -'/record'. - -**Note: this topic is not recommended to plot, as the structure of the -messages are not necessarily numeric.** - -:: - - # Example messages that can be published - - # Dictionary data - {'foo': 'world'} - - # Numerical data - 52 - - # Time data (note not a datatime object) - '2015-12-02T11:06:32.252626' - -devices/\* ----------- - -The devices topic is meant to be data structured from a scraping of a -ModBus or BacNet device. Currently drivers for both of these -protocols write data to the message bus in the proper format. VOLTTRON -drivers also publish an aggregation of points in an "all" topic. **Only the -"all" topic messages are read and published to a historian.** -Both the all topic and point topic have the same header information, b -ut the message body for each is slightly different. -For a complete working example of these messages please see - -- :py:mod:`examples.ExampleSubscriber.subscriber.subscriber_agent` - -Format of header and message for device topics (i.e. messages published to -topics with pattern "devices/\*/all"): - -:: - - # Header contains the data associated with the message. - { - # python code to get this is - # from datetime import datetime - # from volttron.platform.messaging import headers as header_mod - # from volttron.platform.agent import utils - # now = utils.format_timestamp( datetime.utcnow()) - # { - # headers_mod.DATE: now, - # headers_mod.TIMESTAMP: now - # } - "Date": "2015-11-17 21:24:10.189393+00:00", - "TimeStamp": "2015-11-17 21:24:10.189393+00:00" - } - - # Message Format: - - # WITH METADATA - # Messages contains a two element list. The first element contains a - # dictionary of all points under a specific parent. While the second - # element contains a dictionary of meta data for each of the specified - # points. For example devices/pnnl/building/OutsideAirTemperature and - # devices/pnnl/building/MixedAirTemperature ALL message would be created as: - [ - {"OutsideAirTemperature ": 52.5, "MixedAirTemperature ": 58.5}, - { - "OutsideAirTemperature ": {'units': 'F', 'tz': 'UTC', 'type': 'float'}, - "MixedAirTemperature ": {'units': 'F', 'tz': 'UTC', 'type': 'float'} - } - ] - - #WITHOUT METADATA - # Message contains a dictionary of all points under a specific parent - {"OutsideAirTemperature ": 52.5, "MixedAirTemperature ": 58.5} - -analysis/\* ------------ - -Data sent to analysis/* topics is result of analysis done by applications. -The format of data sent to analysis/* topics is similar to data sent to -devices/\*/all topics. - -datalogger/\* -------------- -Messages published to datalogger will be assumed to be time point data that -is composed of units and specific types with the assumption that they have -the ability to be graphed easily. - -:: - - {'MixedAirTemperature': {'Readings': ['2015-12-02T00:00:00', - mixed_reading], - 'Units': 'F', - 'tz': 'UTC', - 'data_type': 'float'}} - -If no datetime value is specified as a part of the reading, current time is -used. Message can be published without any header. In the above message -'Readings' and 'Units' are mandatory diff --git a/docs/source/core_services/historians/Influxdb-Historian.rst b/docs/source/core_services/historians/Influxdb-Historian.rst deleted file mode 100644 index 4161f5d1f2..0000000000 --- a/docs/source/core_services/historians/Influxdb-Historian.rst +++ /dev/null @@ -1,199 +0,0 @@ -.. _Influxdb-Historian: - -################## -Influxdb Historian -################## - -InfluxDB is an open source time series database with a fast, scalable engine and high availability. -It's often used to build DevOps Monitoring (Infrastructure Monitoring, Application Monitoring, -Cloud Monitoring), IoT Monitoring, and Real-Time Analytics solutions. - -More information about InfluxDB is available from ``_. - - -Prerequisites -############# - -InfluxDB Installation -===================== - -To install InfluxDB on an Ubuntu or Debian operating system, run the script: - - :: - - services/core/InfluxdbHistorian/scripts/install-influx.sh - -For installation on other operating systems, -see ``_. - -Authentication in InfluxDB -========================== - -By default, the InfluxDB *Authentication* option is disabled, and no user authentication is -required to access any InfluxDB database. You can enable authentication by updating the -InfluxDB configuration file. For detailed information on enabling authentication, see: -``_. - -If *Authentication* is enabled, authorization privileges are enforced. There must be at least -one defined admin user with access to administrative queries as outlined in the linked document -above. Additionally, you must pre-create the ``user`` and ``database`` that are specified in the -configuration file (the default configuration file for InfluxDB -is ``services/core/InfluxdbHistorian/config``). -If your ``user`` is a non-admin user, they must be granted a full set of privileges on the -desired ``database``. - -InfluxDB Driver -=============== - -In order to connect to an InfluxDb client, the Python library for InfluxDB must be installed -in VOLTTRON's virtual environment. From the command line, after enabling the virtual environment, -install the InfluxDB library as follows: - - :: - - python bootstrap.py --influxdb - - -or - - :: - - python bootstrap.py --databases - - -or - - :: - - pip install influxdb - -Configuration -############# - -The default configuration file for VOLTTRON's InfluxDBHistorian agent should be in the format: - -.. code-block:: python - - { - "connection": { - "params": { - "host": "localhost", - "port": 8086, # Don't change this unless default bind port - # in influxdb config is changed - "database": "historian", - "user": "historian", # user is optional if authentication is turned off - "passwd": "historian" # passwd is optional if authentication is turned off - } - }, - "aggregations": { - "use_calendar_time_periods": true - } - } - - -The InfluxDBHistorian agent can be packaged, installed and started according to the standard -VOLTTRON agent creation procedure. A sample VOLTTRON configuration file has been -provided: ``services/core/InfluxdbHistorian/config``. - -.. seealso:: :ref:`Agent Development Walkthrough ` - -Connection -========== - -The ``host``, ``database``, ``user`` and ``passwd`` values in the VOLTTRON configuration file -can be modified. ``user`` and ``passwd`` are optional if InfluxDB *Authentication* is disabled. - -.. note:: Be sure to initialize or pre-create the ``database`` and ``user`` that you defined in - the configuration file, and if ``user`` is a non-admin user, be make sure to grant - privileges for the user on the specified ``database``. - For more information, see `Authentication in InfluxDB`_. - -Aggregations -============ - -In order to use aggregations, the VOLTTRON configuration file must also specify a value, -either ``true`` or ``false``, for ``use_calendar_time_periods``, indicating whether the -aggregation period should align to calendar time periods. If this value is omitted from the -configuration file, aggregations cannot be used. - -For more information on historian aggregations, -see: :ref:`Aggregate Historian Agent Specification `. - -Supported Influxdb aggregation functions: - - Aggregations: COUNT(), DISTINCT(), INTEGRAL(), MEAN(), MEDIAN(), MODE(), SPREAD(), STDDEV(), SUM() - - Selectors: FIRST(), LAST(), MAX(), MIN() - - Transformations: CEILING(),CUMULATIVE_SUM(), DERIVATIVE(), DIFFERENCE(), ELAPSED(), NON_NEGATIVE_DERIVATIVE(), NON_NEGATIVE_DIFFERENCE() - -More information how to use those functions: ``_ - -.. note:: Historian aggregations in InfluxDB are different from aggregations employed - by other historian agents in VOLTTRON. InfluxDB doesn't have a separate agent for aggregations. - Instead, aggregation is supported through the ``query_historian`` function. Other agents can - execute an aggregation query directly in InfluxDB by calling the *RPC.export* method ``query``. - For an example, see :ref:`Aggregate Historian Agent Specification ` - -Database Schema -############### - -Each InfluxDB database has a ``meta`` table as well as other tables for different measurements, -e.g. one table for "power_kw", one table for "energy", one table for "voltage", etc. -(An InfluxDB ``measurement`` is similar to a relational table, so for easier understanding, InfluxDB -measurements will be referred to below as tables.) - -Measurement Table -================= - -Example: If a topic name is *"CampusA/Building1/Device1/Power_KW"*, the ``power_kw`` table might look as follows: - -+-------------------------------+-----------+---------+----------+-------+------+ -|time |building |campus |device |source |value | -+-------------------------------+-----------+---------+----------+-------+------+ -|2017-12-28T20:41:00.004260096Z |building1 |campusa |device1 |scrape |123.4 | -+-------------------------------+-----------+---------+----------+-------+------+ -|2017-12-30T01:05:00.004435616Z |building1 |campusa |device1 |scrape |567.8 | -+-------------------------------+-----------+---------+----------+-------+------+ -|2018-01-15T18:08:00.126345Z |building1 |campusa |device1 |scrape |10 | -+-------------------------------+-----------+---------+----------+-------+------+ - -``building``, ``campus``, ``device``, and ``source`` are InfluxDB *tags*. ``value`` is an InfluxDB *field*. - -.. note:: The topic is converted to all lowercase before being stored in the table. - In other words, a set of *tag* names, as well as a table name, are created by - splitting ``topic_id`` into substrings (see `meta table`_ below). - - -So in this example, where the typical format of a topic name is ``///``, -``campus``, ``building`` and ``device`` are each stored as tags in the database. - -A topic name might not confirm to that convention: - - #. The topic name might contain additional substrings, e.g. - *CampusA/Building1/LAB/Device/OutsideAirTemperature*. In this case, - ``campus`` will be *campusa/building*, ``building`` will be *lab*, and ``device`` will be *device*. - - #. The topic name might contain fewer substrings, e.g. *LAB/Device/OutsideAirTemperature*. - In this case, the ``campus`` tag will be empty, ``building`` will be *lab*, - and ``device`` will be *device*. - -Meta Table -========== - -The meta table will be structured as in the following example: - -+---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ -|time |last_updated |meta_dict |topic |topic_id | -+---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ -|1970-01-01T00:00:00Z |2017-12-28T20:47:00.003051+00:00 |{u'units': u'kw', u'tz': u'US/Pacific', u'type': u'float'} |CampusA/Building1/Device1/Power_KW |campusa/building1/device1/power_kw | -+---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ -|1970-01-01T00:00:00Z |2017-12-28T20:47:00.003051+00:00 |{u'units': u'kwh', u'tz': u'US/Pacific', u'type': u'float'} |CampusA/Building1/Device1/Energy_KWH |campusa/building1/device1/energy_kwh | -+---------------------+---------------------------------+------------------------------------------------------------------+-------------------------------------+--------------------------------------+ - -In the InfluxDB, ``last_updated``, ``meta_dict`` and ``topic`` are *fields* and ``topic_id`` is a *tag*. - -Since InfluxDB is a time series database, the ``time`` column is required, and a dummy value (``time=0``, -which is 1970-01-01T00:00:00Z based on epoch unix time) is assigned to all topics for easier -metadata updating. Hence, if the contents of ``meta_dict`` change for a specific topic, both ``last_updated`` -and ``meta_dict`` values for that topic will be replaced in the table. diff --git a/docs/source/core_services/historians/Mongo-Historian.rst b/docs/source/core_services/historians/Mongo-Historian.rst deleted file mode 100644 index cc96faa329..0000000000 --- a/docs/source/core_services/historians/Mongo-Historian.rst +++ /dev/null @@ -1,93 +0,0 @@ -.. _Mongo-Historian: - -=============== -Mongo Historian -=============== - -Prerequisites -~~~~~~~~~~~~~ - -1. Mongodb ----------- - -Setup mongodb based on using one of the three below scripts. - -1. Install as root on Redhat or Cent OS - - :: - - sudo scripts/historian-scripts/root_install_mongo_rhel.sh - - The above script will prompt user for os version, db user name, password and database name - Once installed you can start and stop the service using the command: - - **sudo service mongod [start|stop|service]** - -2. Install as root on Ubuntu - - :: - - sudo scripts/historian-scripts/root_install_mongo_ubuntu.sh - - The above script will prompt user for os version, db user name, password and database name - Once installed you can start and stop the service using the command: - - **sudo service mongod [start|stop|service]** - -3. Install as non root user on any Linux machine - - :: - - scripts/historian-scripts/install_mongodb.sh - - Usage: - install_mongodb.sh [-h] [-d download_url] [-i install_dir] [-c config_file] [-s] - Optional arguments: - -s setup admin user and test collection after install and startup - - -d download url. defaults to https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.4.tgz - - -i install_dir. defaults to current_dir/mongo_install - - -c config file to be used for mongodb startup. Defaults to - default_mongodb.conf in the same directory as this script.Any datapath - mentioned in the config file should already exist and should have write - access to the current user - - -h print this help message - -2. Mongodb connector --------------------- -This historian requires a mongodb connector installed in your activated -volttron environment to talk to mongodb. Please execute the following -from an activated shell in order to install it. - - :: - - python bootstrap.py --mongo - - -or - - :: - - python bootstrap.py --databases - - -or - - :: - - pip install pymongo - - -3. Configuration Options ------------------------- -The historian configuration file can specify - -:: - - "history_limit_days": - -which will remove entries from the data and rollup collections older than n -days. Timestamps passed to the manage_db_size method are truncated to the day. diff --git a/docs/source/core_services/historians/Platform-Historian.rst b/docs/source/core_services/historians/Platform-Historian.rst deleted file mode 100644 index ff5003e978..0000000000 --- a/docs/source/core_services/historians/Platform-Historian.rst +++ /dev/null @@ -1,30 +0,0 @@ -.. _Platform-Historian: - -Platform Historian -================== - -A platform historian is a `"friendly named" `__ -historian on a VOLTTRON instance. It always has the identity (see -`vip `__) of platform.historian. A platform -historian is made available to a volttron central agent for monitoring -of the VOLTTRON instances health and plotting topics from the platform -historian. In order for one of the (historians)[Historians] to be turned -into a platform historian the identity keyword must be added to it's -configuration with the value of platform.historian. The following -configuration file shows a sqlite based platform historian -configuration. - -:: - - { - "agentid": "sqlhistorian-sqlite", - "identity": "platform.historian", - "connection": { - "type": "sqlite", - "params": { - "database": "~/.volttron/data/platform.historian.sqlite" - } - } - } - - diff --git a/docs/source/core_services/index.rst b/docs/source/core_services/index.rst deleted file mode 100644 index e45d074a83..0000000000 --- a/docs/source/core_services/index.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. _core-services: - -============= -Core Services -============= - -Platform services provide underlying functionality used by applications to perform their tasks. - -.. toctree:: - :glob: - :maxdepth: 2 - - service_agents/index - control/index - historians/index - drivers/index - messagebus/index - security/index - restricted/index - config_store/index - multiplatform/index - messagebus_refactor/index - openadr/index - - * - - diff --git a/docs/source/core_services/messagebus/RPC.rst b/docs/source/core_services/messagebus/RPC.rst deleted file mode 100644 index a34b74e80f..0000000000 --- a/docs/source/core_services/messagebus/RPC.rst +++ /dev/null @@ -1,121 +0,0 @@ -.. _Remote-Procedure-Calls: - -Remote Procedure Calls -====================== - -Remote procedure calls (RPC) is a new feature added with VOLTTRON 3.0. -The new VOLTTRON Interconnect Protocol `VIP `__ introduced the -ability to create new point-to-point protocols, called subsystems, -enabling the implementation of `JSON-RPC -2.0 `__. This provides a simple -method for agent authors to write methods and expose or export them to -other agents, making request-reply or notify communications patterns as -simple as writing and calling methods. - -Exporting Methods ------------------ - -The *export()* method, defined on the RPC subsystem class, is used to -mark a method as remotely accessible. This *export()* method has a dual -use. The class method can be used as a decorator to statically mark -methods when the agent class is defined. The instance method dynamically -exports methods, and can be used with methods not defined on the agent -class. Each take an optional export name argument, which defaults to the -method name. Here are the two export method signatures: - -Instance method: - -.. code:: python - - RPC.export(method, name=None) - -Class method: - -:: - - RPC.export(name=None) - -And here is an example agent definition using both methods: - -.. code:: python - - from volttron.platform.vip import Agent, Core, RPC - - def add(a, b): - '''Add two numbers and return the result''' - return a + b - - - class ExampleAgent(Agent): - @RPC.export - def say_hello(self, name): - '''Build and return a hello string''' - return 'Hello, %s!' % (name,) - - @RPC.export('say_bye') - def bye(self, name): - '''Build and return a goodbye string''' - return 'Goodbye, %s.' % (name,) - - @Core.receiver('setup') - def onsetup(self, sender, **kwargs): - self.vip.rpc.export('add') - -Calling exported methods ------------------------- - -The RPC subsystem provides three methods for calling exported RPC -methods. - -.. code:: python - - RPC.call(peer, method, *args, **kwargs) - -Call the remote *method* exported by *peer* with the given arguments. -Returns a gevent *AsyncResult* object. - -.. code:: python - - RPC.batch(peer, requests) - -Batch call remote methods exported by *peer*. *requests* must be an -iterable of 4-tuples *(notify, method, args, kwargs)*, where *notify* is -a boolean indicating whether this is a notification or standard call, -*method* is the method name, *args* is a list and *kwargs* is a -dictionary. Returns a list of *AsyncResult* objects for any standard -calls. Returns *None* if all requests were notifications. - -.. code:: python - - RPC.notify(peer, method, *args, **kwargs) - -Send a one-way notification message to *peer* by calling *method* -without returning a result. - -Here are some examples: - -.. code:: python - - self.vip.rpc.call(peer, 'say_hello', 'Bob').get() - results = self.vip.rpc.batch(peer, [(False, 'say_bye', 'Alice', {}), (True, 'later', [], {})]) - self.vip.rpc.notify(peer, 'ready') - -Inspection ----------- - -A list of methods is available by calling the *inspect* method. -Additional information can be returned for any method by appending -'.inspect' to the method name. Here are a couple examples: - -.. code:: python - - self.vip.rpc.call(peer, 'inspect') # Returns a list of exported methods - self.vip.rpc.call(peer, 'say_hello.inspect') # Return metadata on say_hello method - -Implementation --------------- - -See the -`RPC module `_ for implementation details. - -Also see :ref:`Multi-Platform RPC Communication ` and :ref:`RPC in RabbitMQ ` for additional resources. diff --git a/docs/source/core_services/messagebus/VIP/VIP-Enhancements.rst b/docs/source/core_services/messagebus/VIP/VIP-Enhancements.rst deleted file mode 100644 index cd3a918ca3..0000000000 --- a/docs/source/core_services/messagebus/VIP/VIP-Enhancements.rst +++ /dev/null @@ -1,111 +0,0 @@ -.. _VIP-Enhancements: - -VIP Enhancements -================ - -Outline a vision of how VOLTTRON Message Bus should work - -When creating VIP for VOLTTRON 3.0 we wanted to address two security -concerns and one user request: - -- Security Concern 1: Agents can spoof each other on the VOLTTRON - message bus and fake messages. -- Security Concern 2: Agents can subscribe to topics that they are not - authorized to subscribe to. -- User Request 1: Several users requested means to transfer large - amounts of data between agents without using the message bus. - -VOLTTRON Interconnect Protocol (VIP) was created to address these issues -but unfortunately, it broke the easy to use pub-sub messaging model of -VOLTTRON. Additionally to use the security features of VOLTTRON in 3.0 -code has become an ordeal especially when multiple platforms are -concerned. Finally, VIP has introduced the requirement for knowledge of -specific other platforms to agents written by users in order to be able -to communicate. The rest of this memo focuses on defining the way -VOLTTRON message bus will work going forward indefinitely and should be -used as the guiding principles for any future work on VIP and VOLTTRON. -  - -VOLTTRON Message Bus Guiding Principles: - -#. | All communications between two or more different VOLTTRON platforms - MUST go through the VIP Router. Said another way, a user agent - (application) should have NO capability to reach out to an agent on a - different VOLTTRON platform directly. - | All communications between two or more VOLTTRON platforms must be - in the form of topics on the message bus. Agents MUST not use a - distinct platform address or name to communicate via a direct - connection between two platforms. - -#. VOLTTRON will use two TCP ports. One port is used to extend VIP - across platforms. A second port is used for the VOLTTRON discovery - protocol (more on this to come on a different document). VIP will - establish bi-directional communication via a single TCP port. - -#. In order to solve the bootstrapping problem that CurveMQ has punted - on, we will modify VIP to operate similar (behaviorally) to SSH. - -A. On a single VOLTTRON platform, the platform’s public key will be made -available via an API so that all agents will be able to communicate with -the platform. Additionally, the behavior of the platform will be changed -so that agents on the same platform will automatically be added to -auth.json file. No more need for user to add the agents manually to the -file. The desired behavior is similar to how SSH handles known\_hosts. -Note that this behavior still addresses the security request 1 & 2. - -B. When connecting VOLTTRON platforms, VOLTTRON Discovery Protocol (VDP) -will be used to discover the other platforms public key to establish the -router to router connection. Note that since we BANNED agent to agent -communication between two platforms, we have prevented an O(N^2) -communication pattern and key bootstrapping problem. - -#. Authorization determines what agents are allowed to access what - topics. Authorization MUST be managed by the VOLTTRON Central - platform on a per organization basis. It is not recommended to have - different authorization profiles on different VOLTTRON instances - belonging to the same organization. - -#. VOLTTRON message bus uses topics such as and will adopt an - information model agreed upon by the VOLTTRON community going - forward. Our initial information model is based on the OpenEIS schema - going forward. A different document will describe the information - model we have adopted going forward. All agents are free to create - their own topics but the VOLTTRON team (going forward) will support - the common VOLTTRON information model and all agents developed by - PNNL will be converted to use the new information model. - -#. Two connected VOLTTRON systems will exchange a list of available - topics via the message router. This will allow each VIP router to - know what topics are available at what VOLTTRON platform. - -#. Even though each VOLTTRON platform will have knowledge of what topics - are available around itself, no actual messages will be forwarded - between VOLTTRON platforms until an agent on a specific platform - subscribes to a topic. When an agent subscribes to a topic that has a - publisher on a different VOLTTRON platform, the VIP router will send - a request to its peer routers so that the messages sent to that topic - will be forwarded. There will be cases (such as clean energy - transactive project) where the publisher to a topic may be multiple - hops away. In this case, the subscribe request will be sent towards - the publisher through other VIP routers. In order to find the most - efficient path, we may need to keep track of the total number of hops - (in terms of number of VIP routers). - -#. The model described in steps 5/6/7 applies to data collection. For - control applications, VOLTTRON team only allows control actions to be - originated from the VOLTTRON instance that is directly connected to - that controlled device. This decision is made to increase the - robustness of the control agent and to encourage truly distributed - applications to be developed. - -#. Direct agent to agent communication will be supported by creation of - an ephemeral topic under the topic hierarchy. Our measurements have - shown repeatedly that the overhead of using the ZeroMQ message - pub/sub is minimal and has zero impact on communications throughput. - -In summary, by making small changes to the way VIP operates, I believe -that we can significantly increase the usability of the platform and -also correct the mixing of two communication platforms into VIP. -VOLTTRON message bus will return to being a pub/sub messaging system -going forward. Direct agent to agent communication will be supported -through the message bus. diff --git a/docs/source/core_services/messagebus/VIP/index.rst b/docs/source/core_services/messagebus/VIP/index.rst deleted file mode 100644 index 813f89125f..0000000000 --- a/docs/source/core_services/messagebus/VIP/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -============================== -VOLTTRON Interconnect Protocol -============================== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/messagebus/index.rst b/docs/source/core_services/messagebus/index.rst deleted file mode 100644 index 2e0e578946..0000000000 --- a/docs/source/core_services/messagebus/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. _messagebus index: - -==================== -VOLTTRON Message Bus -==================== - -.. toctree:: - :glob: - :maxdepth: 2 - - VIP/index - - * diff --git a/docs/source/core_services/messagebus/topics.rst b/docs/source/core_services/messagebus/topics.rst deleted file mode 100644 index 3ddf38cc35..0000000000 --- a/docs/source/core_services/messagebus/topics.rst +++ /dev/null @@ -1,60 +0,0 @@ -.. _Topics: - -Messaging and Topics -==================== - -Introduction ------------- - -Agents in |VOLTTRON| communicate with each other using a -publish/subscribe mechanism built on the Zero MQ Python library. This -allows for great flexibility as topics can be created dynamically and -the messages sent can be any format as long as the sender and receiver -understand it. An agent with data to share publishes to a topic, then -any agents interested in that data subscribe to that topic. - -While this flexibility is powerful, it also could also lead to confusion -if some standard is not followed. The current conventions for -communicating in the VOLTTRON are: - -- Topics and subtopics follow the format: topic/subtopic/subtopic -- Subscribers can subscribe to any and all levels. Subscriptions to - "topic" will include messages for the base topic and all subtopics. - Subscriptions to "topic/subtopic1" will only receive messages for - that subtopic and any children subtopics. Subscriptions to empty - string ("") will receive ALL messages. This is not recommended. - - - All agents should subscribe to the "platform" topic. This is the - topic the VOLTTRON will use to send messages to agents, such as - "shutdown". - -Agents should set the "From" header. This will allow agents to filter on -the "To" message sent back. This is especially useful for requests to -the ArchiverAgent so agents do not receive replies not meant for their -request. - -Topics ------- - -In VOLTTRON -~~~~~~~~~~~ - -- platform - Base topic used by the platform to inform agents of - platform events -- platform/shutdown - General shutdown command. All agents should exit - upon receiving this. Message content will be a reason for the - shutdown -- platform/shutdown\_agent - This topic will provide a specific agent - id. Agents should subscribe to this topic and exit if the id in the - message matches their id. - -- devices - Base topic for data being published by drivers -- datalogger - Base topic for agents wishing to record time series data -- record - Base topic for agents to record data in an arbitrary format. - -Controller Agent Topics -~~~~~~~~~~~~~~~~~~~~~~~ - -See the documentation for the [[ActuatorAgent]]. - -.. |VOLTTRON| unicode:: VOLTTRON U+2122 diff --git a/docs/source/core_services/messagebus_refactor/RabbitMQ-Overview.rst b/docs/source/core_services/messagebus_refactor/RabbitMQ-Overview.rst deleted file mode 100644 index f5e7291586..0000000000 --- a/docs/source/core_services/messagebus_refactor/RabbitMQ-Overview.rst +++ /dev/null @@ -1,225 +0,0 @@ - .. _RabbitMQ-Overview: - -================= -RabbitMQ Overview -================= - -.. NOTE:: Some of the RabbitMQ summary/overview documentation and supporting images added here are taken from `RabbitMQ official documentation `_. - -RabbitMQ is the most popular messaging library with over 35,000 production deployments. -It is highly scalable, easy to deploy, runs on many operating systems and cloud -environments. It supports many kinds of distributed deployment methodologies such as -clusters, federation and shovels. - - -RabbitMQ uses Advanced Message Queueing Protocol (AMQP) and works on the basic -producer consumer model. A consumer is a program that consumes/receives messages and -producer is a program that sends the messages. Following are some important -definitions that we need to know before we proceed. - -* Queue - Queues can be considered like a post box that stores messages until consumed by the consumer. Each consumer must create a queue to receives messages that it is interested in receiving. We can set properties to the queue during it's declaration. The queue properties are - - * Name - Name of the queue - * Durable - Flag to indicate if the queue should survive broker restart. - * Exclusive - Used only for one connection and it will be removed when connection is closed. - * Auto-queue - Flag to indicate if auto-delete is needed. The queue is deleted when last consumer un-subscribes from it. - * Arguments - Optional, can be used to set message TTL (Time To Live), queue limit etc. - -* Bindings - Consumers bind the queue to an exchange with binding keys or routing patterns. Producers send messages and associate them with a routing key. Messages are routed to one or many queues based on a pattern matching between a message routing key and binding key. - -* Exchanges - Exchanges are entities that are responsible for routing messages to the queues based on the routing pattern/binding key used. They look at the routing key in the message when deciding how to route messages to queues. There are different types of exchanges and one must choose the type of exchange depending on the application design requirements - - #. Fanout - It blindly broadcasts the message it receives to all the queues it knows. - - #. Direct - Here, the message is routed to a queue if the routing key of the message exactly matches the binding key of the queue. - - #. Topic - Here, the message is routed to a queue based on pattern matching of the routing key with the binding key. The binding key and the routing key pattern must be a list of words delimited by dots, for example, "car.subaru.outback" or "car.subaru.*", "car.#". A message sent with a particular routing key will be delivered to all the queues that are bound with a matching binding key with some special rules as - - '*' (star) - can match exactly one word in that position. - '#' (hash) - can match zero or more words - - #. Headers - If we need more complex matching then we can add a header to the message with all the attributes set to the values that need to be matched. The message is considered matching if the values of the attributes in the header is equal to that of the binding. Header exchange ignore the routing key. - - We can set some properties to the exchange during it's declaration. - - * Name - Name of the exchange - * Durable - Flag to indicate if the exchange should survive broker restart. - * Auto-delete - Flag indicates if auto-delete is needed. If set to true, the exchange is deleted when the last queue is unbound from it. - * Arguments - Optional, used by plugins and broker-specific features - -Lets use an example to understand how they all fit together. Consider an example where there -are four consumers (Consumer 1 - 4) interested in receiving messages matching the pattern -"green", "red" or "yellow". In this example, we are using a direct exchange that will route -the messages to the queues only when there is an exact match of the routing key of the message -with the binding key of the queues. Each of the consumers declare a queue and bind the queue -to the exchange with a binding key of interest. Lastly, we have a producer that is continuously -sending messages to exchange with routing key "green". The exchange will check for an exact -match and route the messages to only Consumer 1 and Consumer 3. - -.. image:: files/rabbitmq_exchange.png - - -For more information about queues, bindings, exchanges, please refer to -`RabbitMQ tutorial `_. - - -Distributed RabbitMQ Brokers -============================ -RabbitMQ allows multiple distributed RabbitMQ brokers to be connected in three different ways - -with clustering, with federation and using shovel. We take advantage of these built-in plugins -for multi-platform VOLTTRON communication. For more information about the differences between clustering, -federation, and shovel, please refer to RabbitMQ documentation -`Distributed RabbitMQ brokers `_. - -Clustering ----------- -Clustering connects multiple brokers residing in multiple machines to form a single logical broker. -It is used in applications where tight coupling is necessary i.e, where each node shares the data -and knows the state of all other nodes in the cluster. A new node can connect to the cluster through -a peer discovery mechanism if configured to do so in the RabbitMQ config file. For all the nodes to -be connected together in a cluster, it is necessary for them to share the same Erlang cookie and be -reachable through it's DNS hostname. A client can connect to any one of the nodes in the cluster and -perform any operation (to send/receive messages from other nodes etc.), the nodes will route the operation -internally. In case of a node failure, clients should be able to reconnect to a different node, -recover their topology and continue operation. - -Please note, this feature is not integrated into VOLTTRON. But we hope to support it in the future. -For more detailed information about clustering, please refer to RabbitMQ documentation -`Clustering plugin `_. - -.. _Federation: - -Federation ----------- -Federation plugin is used in applications that does not require as much of tight coupling as clustering. -Federation has several useful features. - -* Loose coupling - The federation plugin can transmit messages between brokers (or clusters) in different administrative domains: - - * they may have different users and virtual hosts; - * they may run on different versions of RabbitMQ and Erlang. - -* WAN friendliness - They can tolerate network intermittent connectivity. - -* Specificity - Not everything needs to be federated ( made available to other brokers ). There can be local-only components. - -* Scalability - Federation does not require O(n2) connections for *n* brokers, so it scales better. - -The federation plugin allows you to make exchanges and queues federated. A federated exchange or queue can -receive messages from one or more upstreams (remote exchanges and queues on other brokers). A federated -exchange can route messages published upstream to a local queue. A federated queue lets a local consumer -receive messages from an upstream queue. - -Before we move forward, let's define upstream and downstream servers. - -* Upstream server - The node that is publishing some message of interest -* Downstream server - The node connected to a different broker that wants to receive messages from the upstream server - -A federation link needs to be established from downstream server to the upstream server. The data flows in -single direction from upstream server to downstream server. For bi-directional data flow, we would need to -create federation links on both the nodes. - -We can receive messages from upstream server to downstream server by either making an exchange or a queue -*federated*. - -For more detailed information about federation, please refer to RabbitMQ documentation -`Federation plugin `_. - -Federated Exchange ------------------- -When we make an exchange on the downstream server *federated*, the messages published to the upstream -exchanges are copied to the federated exchange, as though they were published directly to it. - -.. image:: files/federation.png - -Above figure explains message transfer using federated exchange. The box on the right acts as the downstream server -and the box on the left acts as the upstream server. A federation/upstream link is established between -the downstream server and the upstream server by using federation management plugin. An exchange on the -downstream server is made *federated* using federation policy configuration. The federated exchange only -receives the messages for which it has subscribed for. An upstream queue is created on the upstream -server with a binding key same as subscription made on the federated exchange. For example, if an upstream -server is publishing messages with binding key "foo" and a client on the downstream server is interested -in receiving messages of the binding key "foo", then it creates a queue and binds the queue to the federated -with the same binding key. This binding is sent to the upstream and the upstream queue binds to the -upstream exchange with that key. - - -Publications to either exchange may be received by queues bound to the federated exchange, but publications -directly to the federated exchange cannot be received by queues bound to the upstream exchange. - -For more information about federated exchanges and different federation topologies, please read -`Federated Exchanges `_. - -Federated Queue ---------------- -Federated queue provides a way of balancing load of a single queue across nodes or clusters. -A federated queue lets a local consumer receive messages from an upstream queue. A typical -use would be to have the same "logical" queue distributed over many brokers. Such a logical -distributed queue is capable of having higher capacity than a single queue. A federated queue -links to other upstream queues. - -A federation or upstream link needs to be created like before and a federated queue needs -to be setup on the downstream server using federation policy configuration. The federated -queue will only retrieve messages when it has run out of messages locally, it has consumers -that need messages, and the upstream queue has "spare" messages that are not being consumed. - -For more information about federated queues, please read -`Federated Queues `_. - -.. _Shovel: - -Shovel ------- -Shovel plugin allows you to reliably and continually move messages from a source in one -broker to destination in another broker. A shovel behaves like a well-written client application, that - -* connects to it's source and destination broker -* consumes messages from the source queue -* re-publishes messages to the destination if the messages match the routing key. - -Shovel plugin uses Erlang client under the hood. In case of shovel, apart from configuring -the hostname, port and virtual host of the remote node, we will also have to provide list -of routing keys that we want to forward to remote node.The primary advantages of shovels are - -* Loose coupling - A shovel can move messages between brokers (or clusters) in different administrative domains: - - * they may have different users and virtual hosts; - * they may run on different versions of RabbitMQ and Erlang. -* WAN friendliness - They can tolerate network intermittent connectivity. - -Shovels are also useful in case if one of the nodes is behind NAT. We can setup shovel on -the node behind NAT to forward messages to the node outside NAT. -Shovels do not allow you to adapt to subscriptions like a federation link and we need to a -create a new shovel per subscription. - -For more detailed information about shovel, please refer to RabbitMQ documentation -`Shovel plugin `_. - - -Authentication in RabbitMQ -========================== - -By default RabbitMQ supports SASL PLAIN authentication with user name and password. RabbitMQ supports other SASL authentication mechanism using plugins. In VOLTTRON we use one such external plugin based on x509 certifcates(``_). This authentication is based on a techique called public key cryptography which consists of a key pair - a public key and a private key. Data that has been encrypted with a public key can only be decrypted with the corresponding private key and vice versa. The owner of key pair makes the public key available and keeps the private confidential. To send a secure data to a receiver, a sender encrypts the data with the receiver's public key. Since only the receiver has access to his own private key only the receiver can decrypted. This ensures that others, even if they can get access to the encrypted data, cannot decrypt it. This is how public key cryptography achieves confidentiality. - - -Digital certificate is a digital file that is used to prove ownership of a public key. Certificates act like identification cards for it owner/entity. Certificates are hence crucial to determine that a sender is using the right public key to encrypt the data in the first place. Digital Certificates are issued by Certification Authorities(CA). Certification Authorities fulfil the role of the Trusted Third Party by accepting Certificate applications from entities, authenticating applications, issuing Certificates and maintaining status information about the Certificates issued. Each CA has its own public private key pair and its public key certificate is called a root CA certificate. The CA attests to the identity of a Certificate applicant when it signs the Digital Certificate using its private key. In x509 based authentication, a signed certificate is presented instead of username/password for authentication and if the server recognizes the the signer of the certificate as a trusted CA, accepts and allows the connection. Each server/system can maintain its own list of trusted CAs (i.e. list of public certificates of CAs). Certificates signed by any of the trusted CA would be considered trusted. Certificates can also be signed by intermediate CAs that are in turn signed by a trusted. - -This section only provides a breif overview about the SSL based authentication. Please refer to the vast material available online for detailed description. Some useful links to start: - - * ``_ - * ``_ - -Management Plugin -================= -The rabbitmq-management plugin provides an HTTP-based API for management and monitoring of RabbitMQ -nodes and clusters, along with a browser-based UI and a command line tool, *rabbitmqadmin*. The management -interface allows you to - -* Create, Monitor the status and delete resources such as virtual hosts, users, exchanges, queues etc. -* Monitor queue length, message rates and connection information and more -* Manage users and add permissions (read, write and configure) to use the resources -* Manage policies and runtime parameters -* Send and receive messages (for trouble shooting) - -For more detailed information about the management plugin, please refer to RabbitMQ documentation -`Management Plugin `_. diff --git a/docs/source/core_services/messagebus_refactor/RabbitMQ-Refactor-VOLTTRON.rst b/docs/source/core_services/messagebus_refactor/RabbitMQ-Refactor-VOLTTRON.rst deleted file mode 100644 index a044775c16..0000000000 --- a/docs/source/core_services/messagebus_refactor/RabbitMQ-Refactor-VOLTTRON.rst +++ /dev/null @@ -1,393 +0,0 @@ -.. _RabbitMQ-VOLTTRON: - -======================= -RabbitMQ Based VOLTTRON -======================= -RabbitMQ VOLTTRON uses Pika library for RabbitMQ message bus implementation. To setup VOLTTRON -instance to use RabbitMQ message bus, we need to first configure VOLTTRON to use RabbitMQ message -library. The contents of the RabbitMQ configuration file looks like below. - -Path: $VOLTTRON_HOME/rabbitmq_config.yml - -.. code-block:: yaml - - #host parameter is mandatory parameter. fully qualified domain name - host: mymachine.pnl.gov - - # mandatory. certificate data used to create root ca certificate. Each volttron - # instance must have unique common-name for root ca certificate - certificate-data: - country: 'US' - state: 'Washington' - location: 'Richland' - organization: 'PNNL' - organization-unit: 'VOLTTRON Team' - # volttron1 has to be replaced with actual instance name of the VOLTTRON - common-name: 'volttron1_root_ca' - # - # optional parameters for single instance setup - # - virtual-host: 'volttron' # defaults to volttron - - # use the below four port variables if using custom rabbitmq ports - # defaults to 5672 - amqp-port: '5672' - - # defaults to 5671 - amqp-port-ssl: '5671' - - # defaults to 15672 - mgmt-port: '15672' - - # defaults to 15671 - mgmt-port-ssl: '15671' - - # defaults to true - ssl: 'true' - - # defaults to ~/rabbitmq_server/rabbbitmq_server-3.7.7 - rmq-home: "~/rabbitmq_server/rabbitmq_server-3.7.7" - -Each VOLTTRON instance resides within a RabbitMQ virtual host. The name of the virtual -host needs to be unique per VOLTTRON instance if there are multiple virtual instances -within a single host/machine. The hostname needs to be able to resolve to a valid IP. -The default port of AMQP port without authentication is 5672 and with authentication -is 5671. The default management HTTP port without authentication is 15672 and -with authentication is 15671. These needs to be set appropriately if default ports are -not used. The 'ssl' flag indicates if SSL based authentication is required or not. -If set to True, information regarding SSL certificates needs to be also provided. -SSL based authentication is described in detail in `Authentication And Authorization With RabbitMQ Message Bus `_. - - -To configure the VOLTTRON instance to use RabbitMQ message bus, run the following command. - - vcfg --rabbitmq single [optional path to rabbitmq_config.yml] - -At the end of the setup process, RabbitMQ broker is setup to use the configuration provided. -A new topic exchange for the VOLTTRON instance is created within the configured virtual host. - - -On platform startup, VOLTTRON checks for the type of message bus to be used. If using RabbitMQ -message bus, the RabbitMQ platform router is instantiated. The RabbitMQ platform router, - -* Connects to RabbitMQ broker (with or without authentication) -* Creates a VIP queue and binds itself to the "VOLTTRON" exchange with binding key ".router". This binding key makes it unique across multiple VOLTTRON instances in a single machine as long as each instance has a unique instance name. -* Handles messages intended for router module such as "hello", "peerlist", "query" etc. -* Handles unrouteable messages - Messages which cannot be routed to any destination agent are captured and an error message indicating "Host Unreachable" error is sent back to the caller. -* Disconnects from the broker when the platform shuts down. - - -When any agent is installed and started, the Agent Core checks for the type of message bus used. -If it is RabbitMQ message bus then - -* It creates a RabbitMQ user for the agent. -* If SSL based authentication is enabled, client certificates for the agent is created. -* Connect to the RabbitQM broker with appropriate connection parameters -* Creates a VIP queue and binds itself to the "VOLTTRON" exchange with binding key ".". -* Sends and receives messages using Pika library methods. -* Checks for the type of subsystem in the message packet that it receives and calls the appropriate subsystem message handler. -* Disconnects from the broker when the agent stops or platform shuts down. - - -RPC In RabbitMQ VOLTTRON -======================== -The agent functionality remain unchanged irrespective of the underlying message bus used. -That means they can continue to use the same RPC interfaces without any change. - -.. image:: files/rpc.png - -Consider two agents with VIP identities "agent_a" and "agent_b" connected to VOLTTRON platform -with instance name "volttron1". Agent A and B each have a VIP queue with binding key "volttron1.agent_a" -and "volttron1.agent_b". Following is the sequence of operation when Agent A wants to make RPC -call to Agent B. - -1. Agent A make RPC call to Agent B. - agent_a.vip.rpc.call("agent_b", set_point, "point_name", 2.5) - -2. RPC subsystem wraps this call into a VIP message object and sends it to Agent B. -3. The VOLTTRON exchange routes the message to Agent B as the destination routing in the VIP message object matches with the binding key of Agent B. -4. Agent Core on Agent B receives the message, unwraps the message to find the subsystem type and calls the RPC subsystem handler. -5. RPC subsystem makes the actual RPC call "set_point()" and gets the result. It then wraps into VIP message object and sends it back to the caller. -6. The VOLTTRON exchange routes it to back to Agent A. -7. Agent Core on Agent A calls the RPC subsystem handler which in turn hands over the RPC result to Agent A application. - - -PUBSUB In RabbitMQ VOLTTRON -=========================== -The agent functionality remains unchanged irrespective of the platform using ZeroMQ based pubsub or -RabbitMQ based pubsub i,e, agents continue to use the same PubSub interfaces and use the same topic -format delimited by “/”. Since RabbitMQ expects binding key to be delimited by '.', RabbitMQ PUBSUB -internally replaces '/' with ".". Additionally, all agent topics converted to -“__pubsub__..” to differentiate from main Agent VIP queue binding. - -.. image:: files/pubsub.png - -Consider two agents with VIP identities "agent_a" and "agent_b" connected to VOLTTRON platform -with instance name "volttron1". Agent A and B each have a VIP queue with binding key "volttron1.agent_a" -and "volttron1.agent_b". Following is the sequence of operation when Agent A subscribes to a topic and Agent B -publishes to same the topic. - -1. Agent B makes subscribe call for topic "devices". - agent_b.vip.pubsub.subscribe("pubsub", prefix="devices", callback=self.onmessage) - -2. Pubsub subsystem creates binding key from the topic “__pubsub__.volttron1.devices.#” - -3. It creates a queue internally and binds the queue to the VOLTTRON exchange with the above binding key. - -4. Agent B is publishing messages with topic: "devices/hvac1". - agent_b.vip.pubsub.publish("pubsub", topic="devices/hvac1", headers={}, message="foo"). - -5. PubSub subsystem internally creates a VIP message object and publishes on the VOLTTRON exchange. - -6. RabbitMQ broker routes the message to Agent B as routing key in the message matches with the binding key of the topic subscription. - -7. The pubsub subsystem unwraps the message and calls the appropriate callback method of Agent A. - -If agent wants to subscribe to topic from remote instances, it uses -agent.vip.subscribe(“pubsub”, “devices.hvac1”, all_platforms=True”) -It is internally set to “__pubsub__.*.” - -Pubsub subsystem for ZeroMQ message bus performs O(N) comparisons where N is the number of unique -subscriptions. RabbitMQ Topic Exchange was enhanced in version 2.6.0 to reduce the overhead of -additional unique subscriptions to almost nothing in most cases. We speculate they are using a tree -structure to store the binding keys which would reduce the search time to O(1) in most cases -and O(ln) in the worst case. VOLTTRON PUBSUB with ZeroMQ could be updated to match this performance -scalability with some effort. - -Multi-Platform Communication In RabbitMQ VOLTTRON -================================================= -With ZeroMQ based VOLTTRON, multi-platform communication was accomplished in three different ways. - -1. Direct connection to remote instance - Write an agent that would connect to remote instance directly. - -2. Special agents - Use special agents such as forward historian/data puller agents that would -forward/receive messages to/from remote instances. -In RabbitMQ-VOLTTRON, we make use of shovel plugin to achieve this behavior. Please refer to -:ref:`Shovel Plugin ` to get an overview of shovels. - -3. Multi-Platform RPC and PubSub - Configure VIP address of all remote instances that an instance has -to connect to in it's $VOLTTRON_HOME/external_discovery.json and let the router module in each -instance manage the connection and take care of the message routing for us. In RabbitMQ-VOLTTRON, we -make use of federation plugin to achieve this behavior. Please refer to -:ref:`Federation Plugin ` get an overview of federation. - -Using Federation Plugin ------------------------ -We can connect multiple VOLTTRON instances using the federation plugin. Before setting up federation -links, we need to first identify upstream server and downstream server. Upstream Server is the node -that is publishing some message of interest and downStream server is the node that wants to receive -messages from the upstream server. A federation link needs to be established from a downstream VOLTTRON -instance to upstream VOLTTRON instance. To setup a federation link, we will need to add upstream server -information in a RabbitMQ federation configuration file - -Path: $VOLTTRON_HOME/rabbitmq_federation_config.yml - -.. code-block:: yaml - - # Mandatory parameters for federation setup - federation-upstream: - rabbit-4: - port: '5671' - virtual-host: volttron4 - rabbit-5: - port: '5671' - virtual-host: volttron5 - -To configure the VOLTTRON instance to setup federation, run the following command. - - vcfg --rabbitmq federation [optional path to rabbitmq_federation_config.yml] - -This will setup federation links to upstream servers and sets policy to make the VOLTTRON -exchange *federated*. Once a federation link is established to remote instance, the messages -published on the remote instance become available to local instance as if it were published on -the local instance. - -For detailed instructions to setup federation, please refer to README section <>. - -Multi-Platform RPC With Federation ----------------------------------- -For multi-platform RPC communication, federation links need to be established on both the VOLTTRON -nodes. Once the federation links are established, RPC communication becomes fairly simple. - -.. image:: files/multiplatform_rpc.png - -Consider Agent A on volttron instance "volttron1" on host "host_A" wants to make RPC call on Agent B -on VOLTTRON instance "volttron2" on host "host_B". - -1. Agent A makes RPC call. - -.. code-block:: Python - - kwargs = {"external_platform": self.destination_instance_name} - agent_a.vip.rpc.call("agent_b", set_point, "point_name", 2.5, \**kwargs) - -2. The message is transferred over federation link to VOLTTRON instance "volttron2" as both the exchanges are made *federated*. - -3. RPC subsystem of Agent B calls the actual RPC method and gets the result. It encapsulates the message result into VIP message object and sends it back to Agent A on VOLTTRON instance "volttron1". - -4. The RPC subsystem on Agent A receives the message result and gives it to Agent A application. - -Multi-Platform PubSub With Federation -------------------------------------- -For multi-platform PubSub communication, it is sufficient to have federation link from downstream server -to upstream server. In case of bi-directional data flow, links have to established in both the directions. - -.. image:: files/multiplatform_pubsub.png - -Consider Agent B on volttron instance "volttron2" on host "host_B" wants to subscribe to messages from -VOLTTRON instance "volttron2" on host "host_B". Firstly, federation link needs to be established from -"volttron2" to "volttron1". - -1. Agent B makes a subscribe call. - - agent_b.vip.subscribe.call("pubsub", prefix="devices", all_platforms=True) - -2. The PubSub subsystem converts the prefix to "__pubsub__.*.devices.#". Here, "*" indicates that agent is subscribing to "devices" topic from all the VOLTTRON platforms. - -3. A new queue is created and bound to VOLTTRON exchange with above binding key. Since the VOLTTRON exchange is a *federated exchange*, any subscribed message on the upstream server becomes available on the federated exchange and Agent B will be able to receive it. - -4. Agent A publishes message to topic "devices/pnnl/isb1/hvac1" - -5. PubSub subsystem publishes this messgae on it's VOLTTRON exchange. - -6. Due to the federation link, message is received by the Pubsub subsytem of Agent A. - -Using Shovel Plugin -------------------- -Shovels act as well written client application which moves messages from source to destination broker. -Below configuration shows how to setup a shovel to forward PubSub messages or perform -multi-platform RPC communication from local to a remote instance. It expects hostname, -port and virtual host of remote instance. - -Path: $VOLTTRON_HOME/rabbitmq_shovel_config.yml - -.. code-block:: yaml - - # Mandatory parameters for shovel setup - shovel: - rabbit-2: - port: '5671' - virtual-host: volttron - # Configuration to forward pubsub topics - pubsub: - # Identity of agent that is publishing the topic - platform.driver: - - devices - # Configuration to make remote RPC calls - rpc: - # Remote instance name - volttron2: - # List of pair of agent identities (local caller, remote callee) - - [scheduler, platform.actuator] - -To forward PubSub messages, the topic and agent identity of the publisher agent is needed. -To perform RPC, instance name of the remote instance and agent identities of the local agent -and remote agent are needed. - -To configure the VOLTTRON instance to setup shovel, run the following command. - - vcfg --rabbitmq shovel [optional path to rabbitmq_shovel_config.yml] - -This setups up a shovel that forwards messages (either PubSub or RPC) from local exchange -to remote exchange. - -Multi-Platform PubSub With Shovel ---------------------------------- -After the shovel link is established for Pubsub, the below figure shows how the communication happens. -Please note, for bi-directional pubsub communication, shovel links need to be created on -both the nodes. The "blue" arrows show the shovel binding key. The pubsub topic configuration -in `$VOLTTRON_HOME/rabbitmq_shovel_config.yml` get internally converted to shovel binding key, -`"__pubsub__.."`. - -.. image:: files/multiplatform_shovel_pubsub.png - -Now consider a case where shovels are setup in both the directions for forwarding "devices" -topic. - -1. Agent B makes a subscribe call to receive messages with topic "devices" from all connected platforms. - - agent_b.vip.subscribe.call("pubsub", prefix="devices", all_platforms=True) - -2. The PubSub subsystem converts the prefix to "__pubsub__.*.devices.#" -"*" indicates that agent is subscribing to "devices" topic from all the VOLTTRON platforms. - -3. A new queue is created and bound to VOLTTRON exchange with above binding key. - -4. Agent A publishes message to topic "devices/pnnl/isb1/hvac1" - -5. PubSub subsystem publishes this message on it's VOLTTRON exchange. - -6. Due to a shovel link from VOLTTRON instance "volttron1" to "volttron2", the message is forwarded from volttron exchange "volttron1" to "volttron2" and picked up by Agent A on "volttron2". - -Multi-Platform RPC With Shovel ------------------------------- -After the shovel link is established for multi-platform RPC, the below figure shows how the -RPC communication happens. Please note it is mandatory to have shovel links on both directions -as it is request-response type of communication. We will need to set the agent identities for -caller and callee in the `$VOLTTRON_HOME/rabbitmq_shovel_config.yml`. The "blue" arrows show -the resulting the shovel binding key. - -.. image:: files/multiplatform_shovel_rpc.png - -Consider Agent A on volttron instance "volttron1" on host "host_A" wants to make RPC call on Agent B -on VOLTTRON instance "volttron2" on host "host_B". - -1. Agent A makes RPC call. - -.. code-block:: Python - - kwargs = {"external_platform": self.destination_instance_name} - agent_a.vip.rpc.call("agent_b", set_point, "point_name", 2.5, \**kwargs) - -2. The message is transferred over shovel link to VOLTTRON instance "volttron2". - -3. RPC subsystem of Agent B calls the actual RPC method and gets the result. It encapsulates the message result into VIP message object and sends it back to Agent A on VOLTTRON instance "volttron1". - -4. The RPC subsystem on Agent A receives the message result and gives it to Agent A application. - -RabbitMQ Management Tool Integrated Into VOLTTRON -================================================= -Some of the important native RabbitMQ control and management commands are now integrated with -"volttron-ctl" (vctl) utility. Using volttron-ctl RabbitMQ management utility, we can control and -monitor the status of RabbitMQ message bus. - -:: - - vctl rabbitmq --help - usage: vctl command [OPTIONS] ... rabbitmq [-h] [-c FILE] [--debug] - [-t SECS] - [--msgdebug MSGDEBUG] - [--vip-address ZMQADDR] - ... - subcommands: - - add-vhost add a new virtual host - add-user Add a new user. User will have admin privileges - i.e,configure, read and write - add-exchange add a new exchange - add-queue add a new queue - list-vhosts List virtual hosts - list-users List users - list-user-properties - List users - list-exchanges add a new user - list-exchange-properties - list exchanges with properties - list-queues list all queues - list-queue-properties - list queues with properties - list-bindings list all bindings with exchange - list-federation-parameters - list all federation parameters - list-shovel-parameters - list all shovel parameters - list-policies list all policies - remove-vhosts Remove virtual host/s - remove-users Remove virtual user/s - remove-exchanges Remove exchange/s - remove-queues Remove queue/s - remove-federation-parameters - Remove federation parameter - remove-shovel-parameters - Remove shovel parameter - remove-policies Remove policy diff --git a/docs/source/core_services/multiplatform/index.rst b/docs/source/core_services/multiplatform/index.rst deleted file mode 100644 index 45d5303e9f..0000000000 --- a/docs/source/core_services/multiplatform/index.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. _VOLTTRON-MultiPlatform: - -======================================= -MultiPlatform Message Bus Communication -======================================= - -The multi platform message bus communication allows the user to connect to remote VOLTTTRON platforms seamlessly. This -bypasses the need for an agent wanting to send/receive messages to/from remote platforms from having to setup the -connection to remote platform directly. Instead, the router module in each platform will maintain connections to the -remote platforms internally, that means it will connect, disconnect and monitor the status of each connection. - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/openadr/VenAgentConfig.rst b/docs/source/core_services/openadr/VenAgentConfig.rst deleted file mode 100644 index 5736d416f9..0000000000 --- a/docs/source/core_services/openadr/VenAgentConfig.rst +++ /dev/null @@ -1,140 +0,0 @@ -.. _VenAgentConfig: - -OpenADR VEN Agent: Installation and Configuration -================================================= - -The VEN agent can be configured, built and launched using the VOLTTRON agent installation -process described in -http://volttron.readthedocs.io/en/develop/devguides/agent_development/Agent-Development.html#agent-development. - -The VEN agent depends on some third-party libraries that are not in the standard -VOLTTRON installation. They should be installed in the VOLTTRON virtual environment prior to building the agent: -:: - - (volttron) $ cd $VOLTTRON_ROOT/services/core/OpenADRVenAgent - (volttron) $ pip install -r requirements.txt - -where **$VOLTTRON_ROOT** is the base directory of the cloned VOLTTRON code repository. - -The VEN agent is designed to work in tandem with a “control agent,” another -VOLTTRON agent that uses VOLTTRON RPC calls to manage events and supply report data. -A sample control agent has been provided in the **test/ControlAgentSim** subdirectory -under OpenADRVenAgent. - -The VEN agent maintains a persistent store of event and report data in -**$VOLTTRON_HOME/data/openadr.sqlite**. Some care should be taken in managing the -disk consumption of this data store. If no events or reports are active, -it is safe to take down the VEN agent and delete the file; the persistent -store will be reinitialized automatically on agent startup. - -Configuration Parameters ------------------------- - -The VEN agent’s configuration file contains JSON that includes several parameters -for configuring VTN server communications and other behavior. A sample configuration -file, **openadrven.config**, has been provided in the agent directory. - -The VEN agent supports the following configuration parameters: - -========================= ======================== ==================================================== -Parameter Example Description -========================= ======================== ==================================================== -db_path “$VOLTTRON_HOME/data/ Pathname of the agent's sqlite database. Shell - openadr.sqlite” variables will be expanded if they are present - in the pathname. -ven_id “0” The OpenADR ID of this virtual end node. Identifies - this VEN to the VTN. If automated VEN registration - is used, the ID is assigned by the VTN at that - time. If the VEN is registered manually with the - VTN (i.e., via configuration file settings), then - a common VEN ID should be entered in this config - file and in the VTN's site definition. -ven_name "ven01" Name of this virtual end node. This name is used - during automated registration only, identiying - the VEN before its VEN ID is known. -vtn_id “vtn01” OpenADR ID of the VTN with which this VEN - communicates. -vtn_address “http://openadr-vtn. URL and port number of the VTN. - ki-evi.com:8000” -send_registration “False” (“True” or ”False”) If “True”, the VEN sends - a one-time automated registration request to - the VTN to obtain the VEN ID. If automated - registration will be used, the VEN should be run - in this mode initially, then shut down and run - with this parameter set to “False” thereafter. -security_level “standard” If 'high', the VTN and VEN use a third-party - signing authority to sign and authenticate each - request. The default setting is “standard”: the - XML payloads do not contain Signature elements. -poll_interval_secs 30 (integer) How often the VEN should send an OadrPoll - request to the VTN. The poll interval cannot be - more frequent than the VEN’s 5-second process - loop frequency. -log_xml “False” (“True” or “False”) Whether to write each - inbound/outbound request’s XML data to the - agent's log. -opt_in_timeout_secs 1800 (integer) How long to wait before making a - default optIn/optOut decision. -opt_in_default_decision “optOut” (“True” or “False”) Which optIn/optOut choice - to make by default. -request_events_on_startup "False" ("True" or "False") Whether to ask the VTN for a - list of current events during VEN startup. -report_parameters (see below) A dictionary of definitions of reporting/telemetry - parameters. -========================= ======================== ==================================================== - -Reporting Configuration ------------------------ - -The VEN’s reporting configuration, specified as a dictionary in the agent configuration, -defines each telemetry element (metric) that the VEN can report to the VTN, if requested. -By default, it defines reports named “telemetry” and "telemetry_status", with a report -configuration dictionary containing the following parameters: - -======================================================= =========================== ==================================================== -"telemetry" report: parameters Example Description -======================================================= =========================== ==================================================== -report_name "TELEMETRY_USAGE" Friendly name of the report. -report_name_metadata "METADATA_TELEMETRY_USAGE" Friendly name of the report’s metadata, when sent - by the VEN’s oadrRegisterReport request. -report_specifier_id "telemetry" Uniquely identifies the report’s data set. -report_interval_secs_default "300" How often to send a reporting update to the VTN. -telemetry_parameters (baseline_power_kw): r_id "baseline_power" (baseline_power) Unique ID of the metric. -telemetry_parameters (baseline_power_kw): report_type "baseline" (baseline_power) The type of metric being reported. -telemetry_parameters (baseline_power_kw): reading_type "Direct Read" (baseline_power) How the metric was calculated. -telemetry_parameters (baseline_power_kw): units "powerReal" (baseline_power) The reading's data type. -telemetry_parameters (baseline_power_kw): method_name "get_baseline_power" (baseline_power) The VEN method to use when - extracting the data for reporting. -telemetry_parameters (baseline_power_kw): min_frequency 30 (baseline_power) The metric’s minimum sampling - frequency. -telemetry_parameters (baseline_power_kw): max_frequency 60 (baseline_power) The metric’s maximum sampling - frequency. -telemetry_parameters (current_power_kw): r_id "actual_power" (current_power) Unique ID of the metric. -telemetry_parameters (current_power_kw): report_type "reading" (current_power) The type of metric being reported. -telemetry_parameters (current_power_kw): reading_type "Direct Read" (current_power) How the metric was calculated. -telemetry_parameters (current_power_kw): units "powerReal" (baseline_power) The reading's data type. -telemetry_parameters (current_power_kw): method_name "get_current_power" (current_power) The VEN method to use when - extracting the data for reporting. -telemetry_parameters (current_power_kw): min_frequency 30 (current_power) The metric’s minimum sampling - frequency. -telemetry_parameters (current_power_kw): max_frequency 60 (current_power) The metric’s maximum sampling - frequency. -======================================================= =========================== ==================================================== - -======================================================= =========================== ==================================================== -"telemetry_status" report: parameters Example Description -======================================================= =========================== ==================================================== -report_name "TELEMETRY_STATUS" Friendly name of the report. -report_name_metadata "METADATA_TELEMETRY_STATUS" Friendly name of the report’s metadata, when sent - by the VEN’s oadrRegisterReport request. -report_specifier_id "telemetry_status" Uniquely identifies the report’s data set. -report_interval_secs_default "300" How often to send a reporting update to the VTN. -telemetry_parameters (Status): r_id "Status" Unique ID of the metric. -telemetry_parameters (Status): report_type "x-resourceStatus" The type of metric being reported. -telemetry_parameters (Status): reading_type "x-notApplicable" How the metric was calculated. -telemetry_parameters (Status): units "" The reading's data type. -telemetry_parameters (Status): method_name "" The VEN method to use when extracting the data - for reporting. -telemetry_parameters (Status): min_frequency 60 The metric’s minimum sampling frequency. -telemetry_parameters (Status): max_frequency 120 The metric’s maximum sampling frequency. -======================================================= =========================== ==================================================== diff --git a/docs/source/core_services/openadr/VenAgentGuide.rst b/docs/source/core_services/openadr/VenAgentGuide.rst deleted file mode 100644 index 42b009b0e2..0000000000 --- a/docs/source/core_services/openadr/VenAgentGuide.rst +++ /dev/null @@ -1,183 +0,0 @@ -.. _VenAgentGuide: - -OpenADR VEN Agent: Operation -============================ - -Events: - -- The VEN maintains a persistent record of DR events. -- Event updates (including creation) trigger publication of event JSON on the VOLTTRON message bus. -- Another VOLTTRON agent (a “control agent”) can get notified immediately of event updates by subscribing to event publication. It can also call get_events() to retrieve the current status of each active DR event. - -Reporting: - -- The VEN reports device status and usage telemetry to the VTN, relying on information received periodically from other VOLTTRON agents. -- The VEN config defines telemetry values (data points) that can be reported to the VTN. -- The VEN maintains a persistent record of telemetry values over time. -- Other VOLTTRON agents are expected to call report_telemetry() to supply the VEN with a regular stream of telemetry values for reporting. -- The VTN can identify which of the VEN’s supported data points needs to be actively reported at a given time, including their reporting frequency. -- Another VOLTTRON agent (a “control agent”) can get notified immediately of changes in telemetry reporting requirements by subscribing to publication of “telemetry parameters.” It can also call get_telemetry_parameters() to retrieve the current set of reporting requirements. -- The VEN persists these reporting requirements so that they survive VOLTTRON restarts. - -VOLTTRON Agent Interface ------------------------- - -The VEN implements the following VOLTTRON PubSub and RPC calls. - -PubSub: Event Update --------------------- - -When an event is created/updated, the event is published with a topic that includes 'openadr/event/{ven_id}'. - -Event JSON structure: -:: - - { - "event_id" : String, - "creation_time" : DateTime - UTC, - "start_time" : DateTime - UTC, - "end_time" : DateTime - UTC, - "priority" : Integer, # Values: 0, 1, 2, 3. Usually expected to be 1. - "signals" : String, # Values: json string describing one or more signals. - "status" : String, # Values: unresponded, far, near, active, completed, canceled. - "opt_type" : String # Values: optIn, optOut, none. - } - -If an event status is 'unresponded', the VEN is awaiting a decision on whether to optIn or optOut. -The downstream agent that subscribes to this PubSub message should communicate that choice -to the VEN by calling respond_to_event() (see below). The VEN then relays the choice to the VTN. - - -PubSub: Telemetry Parameters Update ------------------------------------ - -When the VEN telemetry reporting parameters have been updated (by the VTN), they -are published with a topic that includes 'openadr/status/{ven_id}'. - -These parameters include state information about the current report. - -Telemetry parameters structure: -:: - - { - 'telemetry': '{ - "baseline_power_kw": { - "r_id" : "baseline_power", # ID of the reporting metric - "report_type" : "baseline", # Type of reporting metric, e.g. baseline or reading - "reading_type" : "Direct Read", # (per OpenADR telemetry_usage report requirements) - "units" : "powerReal", # (per OpenADR telemetry_usage reoprt requirements) - "method_name" : "get_baseline_power", # Name of the VEN agent method that gets the metric - "min_frequency" : (Integer), # Data capture frequency in seconds (minimum) - "max_frequency" : (Integer) # Data capture frequency in seconds (maximum) - }, - "current_power_kw": { - "r_id" : "actual_power", # ID of the reporting metric - "report_type" : "reading", # Type of reporting metric, e.g. baseline or reading - "reading_type" : "Direct Read", # (per OpenADR telemetry_usage report requirements) - "units" : "powerReal", # (per OpenADR telemetry_usage report requirements) - "method_name" : "get_current_power", # Name of the VEN agent method that gets the metric - "min_frequency" : (Integer), # Data capture frequency in seconds (minimum) - "max_frequency" : (Integer) # Data capture frequency in seconds (maximum) - } - }' - 'report parameters': '{ - "status" : (String), # active, inactive, completed, or cancelled - "report_specifier_id" : "telemetry", # ID of the report definition - "report_request_id" : (String), # ID of the report request; supplied by the VTN - "request_id" : (String), # Request ID of the most recent VTN report modification - "interval_secs" : (Integer), # How often a report update is sent to the VTN - "granularity_secs" : (Integer), # How often a report update is sent to the VTN - "start_time" : (DateTime - UTC), # When the report started - "end_time" : (DateTime - UTC), # When the report is scheduled to end - "last_report" : (DateTime - UTC), # When a report update was last sent - "created_on" : (DateTime - UTC) # When this set of information was recorded in the VEN db - }', - 'manual_override' : (Boolean) # VEN manual override status, as supplied by Control Agent - 'online' : (Boolean) # VEN online status, as supplied by Control Agent - } - -Telemetry value definitions such as baseline_power_kw and current_power_kw come from the VEN agent config. - -RPC Calls ---------- - -respond_to_event() -:: - - @RPC.export - def respond_to_event(self, event_id, opt_in=True): - """ - Respond to an event, opting in or opting out. - - If an event's status=unresponded, it is awaiting this call. - When this RPC is received, the VEN sends an eventResponse to - the VTN, indicating whether optIn or optOut has been chosen. - If an event remains unresponded for a set period of time, - it times out and automatically opts in to the event. - - Since this call causes a change in the event's status, it triggers - a PubSub call for the event update, as described above. - - @param event_id: (String) ID of an event. - @param opt_type: (Boolean) Whether to opt in to the event (default True). - """ - -get_events() -:: - - @RPC.export - def get_events(self, active_only=True, started_after=None, end_time_before=None): - """ - Return a list of events. - - By default, return only event requests with status=active or status=unresponded. - - If an event's status=active, a DR event is currently in progress. - - @param active_only: (Boolean) Default True. - @param started_after: (DateTime) Default None. - @param end_time_before: (DateTime) Default None. - @return: (JSON) A list of events -- see 'PubSub: event update'. - """ - -get_telemetry_parameters() -:: - - @RPC.export - def get_telemetry_parameters(self): - """ - Return the VEN's current set of telemetry parameters. - - @return: (JSON) Current telemetry parameters -- see 'PubSub: telemetry parameters update'. - """ - -set_telemetry_status() -:: - - @RPC.export - def set_telemetry_status(self, online, manual_override): - """ - Update the VEN's reporting status. - - @param online: (Boolean) Whether the VEN's resource is online. - @param manual_override: (Boolean) Whether resource control has been overridden. - """ - -report_telemetry() -:: - - @RPC.export - def report_telemetry(self, telemetry_values): - """ - Update the VEN's report metrics. - - Examples of telemetry_values are: - { - 'baseline_power_kw': '6.2', - 'current_power_kw': '6.145', - 'start_time': '2017-12-05 16:11:42.977298+00:00', - 'end_time': '2017-12-05 16:12:12.977298+00:00' - } - - @param telemetry_values: (JSON) Current value of each report metric. - """ diff --git a/docs/source/core_services/security/Key-Stores.rst b/docs/source/core_services/security/Key-Stores.rst deleted file mode 100644 index 33e9b6a9b6..0000000000 --- a/docs/source/core_services/security/Key-Stores.rst +++ /dev/null @@ -1,41 +0,0 @@ -.. _Key-Stores: - -========== -Key Stores -========== - -*Note: most VOLTTRON users should not need to directly interact with -agent key stores. These are notes for VOLTTRON platform developers. -This is not a stable interface and the implementation details are -subject to change.* - -Each agent has its own encryption key-pair that is used to -:ref:`authenticate` itself to the VOLTTRON -platform. A key-pair comprises a public key and a private (secret) key. -These keys are saved in a key store, which is implemented by the -:py:class:`KeyStore class`. -Each agent has its own key store. - -Key Store Locations -------------------- - -There are two main locations key stores will be saved. Installed agents' -key stores are in the the agent's data directory:: - - $VOLTTRON_HOME/agents///keystore.json - -Agents that are not installed, such as platform services and stand-alone -agents, store their key stores here:: - - $VOLTTRON_HOME/keystores//keystore.json - -Generating a Key Store ----------------------- - -Agents automatically retrieve keys from their key store unless -both the ``publickey`` and ``secretkey`` parameters are specified -when the agent is initialized. If an agent's key store does not exist -it will automatically be generated upon access. - -Users can generate a key pair by running the -``vctl auth keypair`` command. diff --git a/docs/source/core_services/security/Known-Hosts-File.rst b/docs/source/core_services/security/Known-Hosts-File.rst deleted file mode 100644 index ca6f705ab0..0000000000 --- a/docs/source/core_services/security/Known-Hosts-File.rst +++ /dev/null @@ -1,70 +0,0 @@ -.. _Known-Hosts-File: - -================ -Known Hosts File -================ - -Before an agent can connect to a VOLTTRON platform that agent must know the -platform's VIP address and public key (known as the *server key*). -It can be tedious to manually keep -track of server keys and match them with their corresponding -addresses. - -The purpose of the known-hosts file is to save a mapping of platform addresses -to server keys. This way the user only has to specify a server key one time. - -Saving a Server Key -------------------- - -Suppose a user wants to connect to a platform at ``192.168.0.42:22916``, and the -platform's public key is ``uhjbCUm3kT5QWj5Py9w0XZ7c1p6EP8pdo4Hq4dNEIiQ``. -To save this address-to-server-key association, the user can run:: - - volttron-ctl auth add-known-host --host 192.168.0.42:22916 --serverkey uhjbCUm3kT5QWj5Py9w0XZ7c1p6EP8pdo4Hq4dNEIiQ - -Now agents on this system will automatically use the correct server key when -connecting to the platform at ``192.168.0.42:22916``. - -Server Key for Local Platforms ------------------------------- - -When a platform starts it automatically adds its public key to the -known-hosts file. Thus agents connecting to the local VOLTTRON platform -(on the same system and using the same ``$VOLTTRON_HOME``) will automatically -be able to retrieve the platform's public key. - -Know-Host-File Details ----------------------- - -*Note: the following details regarding the known-hosts file are subject to -change. These notes are primarily for developers, but the may be helpful -if troubleshooting an issue.* **The known-hosts file should not be edited -directly.** - -File Location -~~~~~~~~~~~~~ - -The known-hosts-file is stored at ``$VOLTTRON_HOME/known_hosts``. - -File Contents -~~~~~~~~~~~~~ - -Here are the contents of an example known-hosts file: - -.. code:: JSON - - { - "@": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", - "127.0.0.1:22916": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", - "127.0.0.2:22916": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", - "127.0.0.1:12345": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", - "192.168.0.42:22916": "uhjbCUm3kT5QWj5Py9w0XZ7c1p6EP8pdo4Hq4dNEIiQ" - } - -The first four entries are for the local platform. (They were automatically -added when the platform started.) The first entry with the ``@`` key is for -IPC connections, and the entries with the ``127.0.0.*`` keys are for -local TCP connections. Note that a single VOLTTRON platform can bind to -multiple TCP addresses, and each address will be automatically added -to the known-hosts file. The last entry is for a remote VOLTTRON platform. -(It was added in the `Saving a Server Key`_ section.) diff --git a/docs/source/core_services/security/Protecting-Pub-Sub-Topics.rst b/docs/source/core_services/security/Protecting-Pub-Sub-Topics.rst deleted file mode 100644 index 7a89b2e217..0000000000 --- a/docs/source/core_services/security/Protecting-Pub-Sub-Topics.rst +++ /dev/null @@ -1,67 +0,0 @@ -.. _Protected-Topics: - -Protecting Pub/Sub Topics -========================= - -VIP :ref:`authorization ` enables -VOLTTRON platform owners to protect pub/sub topics. More -specifically, a platform owner can limit who can publish to a given -topic. This protects subscribers on that platform from receiving -messages (on the protected topic) from unauthorized agents. - -Example -------- - -To protect a topic, add the topic name to -``$VOLTTRON_HOME/protected_topics.json``. For example, the following -protected-topics file declares that the topic ``foo`` is protected: - -.. code:: JSON - - { - "write-protect": [ - {"topic": "foo", "capabilities": ["can_publish_to_foo"]} - ] - } - -**Note:** The capability name ``can_publish_to_foo`` is not special. It -can be any string, but it is easier to manage capabilities with -meaningful names. - -Now only agents with the capability ``can_publish_to_foo`` can publish -to the topic ``foo``. To add this capability to authenticated agents, -run ``vctl auth update`` (or ``volttron-ctl auth add`` for new -authentication entries), and enter ``can_publish_to_foo`` in the capabilities -field: - -.. code:: Bash - - capabilities (delimit multiple entries with comma) []: can_publish_to_foo - -Agents that have the ``can_publish_to_foo`` capabilites can publish to topic ``foo``. -That is, such agents can call: - -.. code:: Python - - self.vip.pubsub.publish('pubsub', 'foo', message='Here is a message') - -If unauthorized agents try to publish to topic ``foo`` they will get an exception: - -``to publish to topic "foo" requires capabilities ['can_publish_to_foo'], but capability list [] was provided`` - -Regular Expressions -------------------- - -Topic names in ``$VOLTTRON_HOME/protected_topics.json`` can be specified -as regular expressions. In order to use a regular expression, the topic name -must begin and end with a "/". For example: - -.. code:: JSON - - { - "write-protect": [ - {"topic": "/foo/*.*/", "capabilities": ["can_publish_to_foo"]} - ] - } - -This protects topics such as ``foo/bar`` and ``foo/anything``. diff --git a/docs/source/core_services/security/index.rst b/docs/source/core_services/security/index.rst deleted file mode 100644 index 57a072d05c..0000000000 --- a/docs/source/core_services/security/index.rst +++ /dev/null @@ -1,33 +0,0 @@ -================= -VOLTTRON Security -================= - -There are various security-related topics throughout VOLTTRON's documentation. -This is a quick roadmap for finding security documentation. - -A core component of VOLTTRON is its :ref:`message bus`. -The security of this message bus is crucial to the entire system. -The :ref:`VOLTTRON Interconnect Protocol` provides -communication over the message bus. VIP was built with security in mind -from the ground up. VIP uses encrypted channels and enforces agent -:ref:`authentication` by default for all network communication. -VIP's :ref:`authorization` mechanism allows system -administrators to limit agent capabilities with fine granularity. - -Even with these security mechanisms built into VOLTTRON, it is important -for system administrators to -:ref:`harden VOLTTRON's underlying OS`. - -The VOLTTRON team has engaged with PNNL's Secure Software Central team to create -a threat profile document. You can read about the threat assessment findings and -how the VOLTTRON team is addressing them here: `SSC Threat Profile -`_ - -Additional documentation related to VIP authentication and authorization -is available here: - -.. toctree:: - :glob: - :maxdepth: 1 - - * diff --git a/docs/source/core_services/security/running_agent_as_user.rst b/docs/source/core_services/security/running_agent_as_user.rst deleted file mode 100644 index 29e4c1b58c..0000000000 --- a/docs/source/core_services/security/running_agent_as_user.rst +++ /dev/null @@ -1,100 +0,0 @@ -.. _Running Agents as unique Unix user: - -============================== -Running Agents as unique users -============================== - -This VOLTTRON feature will cause the platform to create a new, unique Unix user(agent users) -on the host machine for each agent installed on the platform. This user will -have restricted permissions for the file system, and will be used to run the -agent process. The Unix user starting the VOLTTRON platform will be given -limited sudo access to create and delete agent users. - -Since this feature require system level changes (sudo access, user creation, file permission changes) the initial step -needs to be run as root or user with sudo access. This can be a user other than Unix user used to run volttron platform. -All files and folder created by VOLTTRON process in this mode would by default not have any access to others. -Permission for Unix group others would be provided to specific files and folder based on VOLTTRON process requirement. -It is recommended that you use a new volttron home to run volttron in secure mode. Converting a existing VOLTTRON -instance to secure mode is also possible but would involve some manual changes. Please see section -`Porting existing volttron home to secure mode`_. -Please note VOLTTRON has to be bootstrapped as prerequisite to running agents as unique users. - - -Setup agents to run using unique users ---------------------------------------- - -1. **This feature requires acl to be installed.** - - Make sure acl library is installed. If you are running on docker image acl might not be installed by default - - **apt-get install acl** - -2. Agents now run as a user different from VOLTTRON platform user. Agent users should have read and execute permissions - to all directories in the path to the python executable used by VOLTTRON. For example, if VOLTTRON is using a virtual - environment, then agent users should have read permissions to /bin/python and read - and execute permission to all the directories in the path /bin. This can be achieved by running - **chmod -R o+rx /bin** - -3. **Run scripts/secure_user_permissions.sh as root or using sudo** - - This script should be run as root or using sudo. This script gives the VOLTTRON platform user limited sudo access to - create a new unix user for each agent. All users created will be of the format volttron_. - - This script prompts for: - - a. **volttron platform user** - Unix user who would be running VOLTTRON platform. This should be an existing unix user. - On a development machine this could be the unix user you logged in as to check out VOLTTRON source - - b. **VOLTTRON_HOME directory** - The absolute path of volttron home directory. - - c. **Volttron instance name if VOLTTRON_HOME/config does not exist** - - - If VOLTTRON_HOME/config file exists instance name is got from config file. If not user will be prompted for - instance name. volttron_ must be a 23 characters or shorter containing only characters valid as Unix user names. - - This script will create necessary entries in /etc/sudoers.d/volttron to allow the volttron platform user to create - and delete agent users, Volttron agent group, and run any non-sudo command as agent users. - - This script will also create VOLTTRON_HOME and the config file if given a new volttron home directory when prompted. - -4. **Continue with VOLTTRON bootstrap and setup as normal** - point to the VOLTTRON_HOME that you provided in step 2. - -5. **On agent install (or agent start for existing agents)** - a unique agent user(Unix user) is created and the agent - is started as this user. The agent user name is recorded in USER_ID file under the agent install directory - (VOLTTRON_HOME/agents//USER_ID). Subsequent agent restarts will read content of USER_ID file and start - the agent process as that user. - -6. **On agent uninstall** - The agent user is deleted and the agent install directory is deleted. - -Creating new Agents -------------------- - -In this secure mode, agents will only have read write access to agent-data directory under the agent install -directory - VOLTTRON_HOME/agents///.agent-data. Attempting to write in any other -folder under VOLTTRON_HOME will result in permission errors. - -Changes to existing agents in secure mode ------------------------------------------ - -Due to the above change, **SQL historian has been modified to create its database by default under its agent-data directory** -if no path is given in the config file. If providing a path to the database in the config file, please provide a -directory where agent will have write access. This can be an external directory for which agent user (recorded in -VOLTTRON_HOME/agents//USER_ID) has read, write, and execute access. - - -Porting existing volttron home to secure mode ----------------------------------------------- - -When running scripts/secure_users_permissions.sh you will be prompted for a VOLTTRON_HOME directory. If this directory -exists and contains a volttron config file. The script will update the file locations and permissions of existing -volttron files including installed directories. However this step has the following limitations - -#. **You will NOT be able to revert to insecure mode once the changes are done.** Once setup is complete, changing the - config file manually to make parameter "secure-agent-users" to False, may result inconsistent volttron behavior -#. Volttron process and all agents have to be restarted to take effect. -#. **Agents can only to write to its own agent-data dir.** So if your agents writes to any directory outside - vhome/agents///agent-name.agent-data move existing files and update configuration such that - agent writes to agent-name.agent-data dir. For example, if you have SQLHistorian in writing .sqlite file to a - subdirectory under VOLTTRON_HOME that is not vhome/agents///agent-name.agent-data this needs - to be manually updated. - diff --git a/docs/source/core_services/service_agents/actuator/ActuatorAgentProgrammingNotes.rst b/docs/source/core_services/service_agents/actuator/ActuatorAgentProgrammingNotes.rst deleted file mode 100644 index f655a36ebf..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorAgentProgrammingNotes.rst +++ /dev/null @@ -1,46 +0,0 @@ -.. _ActuatorAgentProgrammingNotes: - -Notes on Working With the ActuatorAgent ---------------------------------------- - -- An agent can watch the window value from `device state - updates `__ to perform scheduled actions - within a timeslot. - - - If an Agent's Task is LOW\_PREEMPT priority it can watch for - `device state updates `__ where the window - is less than or equal to the grace period (default 60.0). - -- When considering if to schedule long or multiple short time slots on - a single device: - - - Do we need to ensure the device state for the duration between - slots? - - Yes. Schedule one long time slot instead. - - No. Is it all part of the same Task or can we break it up in case - there is a conflict with one of our time slots? - -- When considering time slots on multiple devices for a single Task: - - - Is the Task really dependent on all devices or is it actually - multiple Tasks? - -- When considering priority: - - - Does the Task have to happen **on an exact day**? - - No. Consider LOW and reschedule if preempted. - - Yes. Use HIGH. - - Is it problematic to prematurely stop a Task once started? - - No. Consider LOW\_PREEMPT and watch the `device state - updates `__ for a small window value. - - Yes. Consider LOW or HIGH. - -- If an agent is only observing but needs to assure that no another - Task is going on while taking readings it can schedule the time to - prevent other agents from messing with a devices state. The schedule - updates can be used as a reminder as to when to start watching. -- **Any** device, existing or not, can be scheduled. This allows for - agents to schedule fake devices to create reminders to start working - later rather then setting up their own internal timers and schedules. - - diff --git a/docs/source/core_services/service_agents/actuator/ActuatorConfig.rst b/docs/source/core_services/service_agents/actuator/ActuatorConfig.rst deleted file mode 100644 index 0c9e92cc83..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorConfig.rst +++ /dev/null @@ -1,23 +0,0 @@ -.. _ActuatorConfig: - -ActuatorAgent Configuration ---------------------------- - -| ``schedule_publish_interval:: Interval between ``\ ```published`` - ``schedule`` - - ``announcements`` `__\ `` in seconds. Defaults to 30.`` - -| ``preempt_grace_time:: Minimum time given to Tasks which have been preempted  - to clean up in seconds. Defaults to 60.`` -| ``schedule_state_file:: File used to save and restore Task states if the  - ActuatorAgent restarts for any reason. File will be created if it does not  - exist when it is needed.`` - -Sample configuration file -~~~~~~~~~~~~~~~~~~~~~~~~~ - -| { -| "schedule\_publish\_interval": 30, -| "schedule\_state\_file": "actuator\_state.pickle" -| } diff --git a/docs/source/core_services/service_agents/actuator/ActuatorHeartbeat.rst b/docs/source/core_services/service_agents/actuator/ActuatorHeartbeat.rst deleted file mode 100644 index d1ef00cdf5..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorHeartbeat.rst +++ /dev/null @@ -1,21 +0,0 @@ -.. _ActuatorHeartbeat: - -Heartbeat Signal ----------------- - -The ActuatorAgent can be configured to send a heartbeat message to the -device to indicate the platform is running. Ideally, if the heartbeat -signal is not sent the device should take over and resume normal -operation. - -The configuration has two parts, the interval (in seconds) for sending -the heartbeat and the specific point that should be modified each -iteration. - -The heart beat interval is specified with a global "heartbeat\_interval" -setting. The ActuatorAgent will automatically set the heartbeat point to -alternating "1" and "0" values. Changes to the heartbeat point will be -published like any other value change on a device. - -The heartbeat points are specified in the driver configuration file of -individual devices diff --git a/docs/source/core_services/service_agents/actuator/ActuatorSchedulePreemption.rst b/docs/source/core_services/service_agents/actuator/ActuatorSchedulePreemption.rst deleted file mode 100644 index 80cecd631e..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorSchedulePreemption.rst +++ /dev/null @@ -1,46 +0,0 @@ -.. _ActuatorSchedulePreemption: - -Task Preemption ---------------- - -Both LOW and LOW\_PREEMPT priority Tasks can be preempted. LOW priority -Tasks may be preempted by a conflicting HIGH priority Task before it -starts. LOW\_PREEMPT priority Tasks can be preempted by HIGH priority -Tasks even after they start. - -When a Task is preempted the ActuatorAgent will publish to -"devices/actuators/schedule/response" with the following header: - -:: - - #python - { - 'type': 'CANCEL_SCHEDULE', - 'requesterID': , - 'taskID': - } - -And the message (after parsing the json): - -:: - - #python - { - 'result': 'PREEMPTED', - 'info': '', - 'data': - { - 'agentID': , - 'taskID': - } - } - -Preemption Grace Time -~~~~~~~~~~~~~~~~~~~~~ - -If a LOW\_PREEMPT priority Task is preempted while it is running the -Task will be given a grace period to clean up before ending. For every -device which has a current time slot the window of remaining time will -be reduced to the grace time. At the end of the grace time the Task will -finish. If the Task has no currently open time slots on any devices it -will end immediately. diff --git a/docs/source/core_services/service_agents/actuator/ActuatorScheduleRequest.rst b/docs/source/core_services/service_agents/actuator/ActuatorScheduleRequest.rst deleted file mode 100644 index d22ed22464..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorScheduleRequest.rst +++ /dev/null @@ -1,122 +0,0 @@ -.. _ActuatorScheduleRequest: - -Requesting Schedule Changes ---------------------------- - -For information on responses see `AcutatorAgent responses to a schedule -or cancel requests. `__ - -**For 2.0 Agents using the pubsub interface: The actuator agent expects -all messages to be JSON and will parse them accordingly. Use -publish\_json to send messages where possible.** - -3.0 agents using pubsub for scheduling and setting point values should -publish python objects like normal. - -Scheduling a Task -~~~~~~~~~~~~~~~~~ - -An agent can request a task schedule by publishing to the -"devices/actuators/schedule/request" topic with the following header: - -:: - - #python - { - 'type': 'NEW_SCHEDULE', - 'requesterID': - 'taskID': , #The desired task ID for this task. It must be unique among all other scheduled tasks. - 'priority': , #The desired task priority, must be 'HIGH', 'LOW', or 'LOW_PREEMPT' - } - -with the following message: - -:: - - #python - [ - ["campus/building/device1", #First time slot. - "2013-12-06 16:00:00", #Start of time slot. - "2013-12-06 16:20:00"], #End of time slot. - ["campus/building/device1", #Second time slot. - "2013-12-06 18:00:00", #Start of time slot. - "2013-12-06 18:20:00"], #End of time slot. - ["campus/building/device2", #Third time slot. - "2013-12-06 16:00:00", #Start of time slot. - "2013-12-06 16:20:00"], #End of time slot. - #etc... - ] - -.. warning:: - - If time zones are not included in schedule requests then the Actuator will - interpret them as being in local time. This may cause remote interaction - with the actuator to malfunction. - -Points on Task Scheduling -^^^^^^^^^^^^^^^^^^^^^^^^^ - -- Everything in the header is required. -- Task id and requester id (agentid) should be a non empty value of - type string -- A Task schedule must have at least one time slot. -- The start and end times are parsed with `dateutil's date/time - parser `__. - **The default string representation of a python datetime object will - parse without issue.** -- Two Tasks are considered conflicted if at least one time slot on a - device from one task overlaps the time slot of the other on the same - device. -- The end time of one time slot can be the same as the start time of - another time slot for the same device. This will not be considered a - conflict. For example, time\_slot1(device0, time1, **time2**) and - time\_slot2(device0,\ **time2**, time3) are not considered a conflict -- A request must not conflict with itself. -- If something goes wrong see `this failure string - list `__ for an explanation - of the error. - -Task Priorities -^^^^^^^^^^^^^^^ - -HIGH: - This Task cannot be preempted under any circumstance.  - This task may preempt other conflicting preemptable Tasks. - -LOW: - This Task cannot be preempted \ **once it has started**\ .  - A Task is considered started once the earliest time slot on any device  - has been reached. This Task may **not** preempt other Tasks. - -LOW\_PREEMPT: - This Task may be preempted at any time. If the Task is preempted  - once it has begun running any current time slots will be given a grace period  - (configurable in the ActuatorAgent configuration file, defaults to 60 seconds) - before being revoked. This Task may **not** preempt other Tasks. - -Canceling a Task -~~~~~~~~~~~~~~~~ - -A task may be canceled by publishing to the -"devices/actuators/schedule/request" topic with the following header: - -:: - - #python - { - 'type': 'CANCEL_SCHEDULE', - 'requesterID': - 'taskID': , #The desired task ID for this task. It must be unique among all other scheduled tasks. - } - -Points on Task Canceling -^^^^^^^^^^^^^^^^^^^^^^^^ - -- The requesterID and taskID must match the original values from the - original request header. -- After a Tasks time has passed there is no need to cancel it. Doing so - will result in a "TASK\_ID\_DOES\_NOT\_EXIST" error. -- If something goes wrong see `this failure string - list `__ for an explanation - of the error. - diff --git a/docs/source/core_services/service_agents/actuator/ActuatorScheduleResponse.rst b/docs/source/core_services/service_agents/actuator/ActuatorScheduleResponse.rst deleted file mode 100644 index 3109f09d1c..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorScheduleResponse.rst +++ /dev/null @@ -1,94 +0,0 @@ -.. _ActuatorScheduleResponse: - -ActuatorAgent Response ----------------------- - -In response to a `Task schedule request `__ the -ActuatorAgent will respond on the topic -"devices/actuators/schedule/result" with the header: - -:: - - #python - { - 'type': <'NEW_SCHEDULE', 'CANCEL_SCHEDULE'> - 'requesterID': , - 'taskID': - } - -And the message (after parsing the json): - -:: - - #python - { - 'result': <'SUCCESS', 'FAILURE', 'PREEMPTED'>, - 'info': , - 'data': - } - -The ActuatorAgent may publish cancellation notices for preempted Tasks -using the "PREEMPTED" result. - -Preemption Data -~~~~~~~~~~~~~~~ - -Preemption data takes the form: - -:: - - #python - { - 'agentID': , - 'taskID': - } - -Failure Reasons -~~~~~~~~~~~~~~~ - -In many cases the ActuatorAgent will try to give good feedback as to why -a request failed. - -General Failures -^^^^^^^^^^^^^^^^ - -| ``INVALID_REQUEST_TYPE:: Request type was not "NEW_SCHEDULE" or "CANCEL_SCHEDULE".`` -| ``MISSING_TASK_ID:: Failed to supply a taskID.`` -| ``MISSING_AGENT_ID:: AgentID not supplied.`` - -Task Schedule Failures -^^^^^^^^^^^^^^^^^^^^^^ - -| ``TASK_ID_ALREADY_EXISTS: The supplied taskID already belongs to an existing task.`` -| ``MISSING_PRIORITY: Failed to supply a priority for a Task schedule request.`` -| ``INVALID_PRIORITY: Priority not one of "HIGH", "LOW", or "LOW_PREEMPT".`` -| ``MALFORMED_REQUEST_EMPTY: Request list is missing or empty.`` -| ``REQUEST_CONFLICTS_WITH_SELF: Requested time slots on the same device overlap.`` - ``MALFORMED_REQUEST: Reported when the request parser raises an unhandled exception. The exception name and info are appended to this info string.`` - ``CONFLICTS_WITH_EXISTING_SCHEDULES: This schedule conflict with an existing schedules that it cannot preempt. The data item for the results will contain info about the conflicts in this form (after parsing json):`` - -:: - - #python - { - '': - { - '': - [ - ["campus/building/device1", - "2013-12-06 16:00:00", - "2013-12-06 16:20:00"], - ["campus/building/device1", - "2013-12-06 18:00:00", - "2013-12-06 18:20:00"] - ] - '':[...] - } - '': {...} - } - -Task Cancel Failures -^^^^^^^^^^^^^^^^^^^^ - -``TASK_ID_DOES_NOT_EXIST:: Trying to cancel a Task which does not exist. This error can also occur when trying to cancel a finished Task.`` -``AGENT_ID_TASK_ID_MISMATCH:: A different agent ID is being used when trying to cancel a Task.`` diff --git a/docs/source/core_services/service_agents/actuator/ActuatorScheduleState.rst b/docs/source/core_services/service_agents/actuator/ActuatorScheduleState.rst deleted file mode 100644 index bda8f84b2c..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorScheduleState.rst +++ /dev/null @@ -1,28 +0,0 @@ -.. _ActuatorScheduleState: - -Schedule State Broadcast ------------------------- - -Periodically the ActuatorAgent will publish the state of all currently -used devices. - -For each device the ActuatorAgent will publish to an associated topic: - -:: - - #python - 'devices/actuators/schedule/announce/' - -With the following header: - -:: - - #python - { - 'requesterID': , - 'taskID': - 'window': - } - -The frequency of the updates is configurable with the -"schedule\_publish\_interval" setting. diff --git a/docs/source/core_services/service_agents/actuator/ActuatorValueRequest.rst b/docs/source/core_services/service_agents/actuator/ActuatorValueRequest.rst deleted file mode 100644 index aa0020cca7..0000000000 --- a/docs/source/core_services/service_agents/actuator/ActuatorValueRequest.rst +++ /dev/null @@ -1,113 +0,0 @@ -.. _ActuatorValueRequest: - -ActuatorAgent Interaction -------------------------- - -Once an Task has been scheduled and the time slot for one or more of the -devices has started an agent may interact with the device using the -**get** and **set** topics. - -Both **get** and **set** are responded to the same way. See -[#ActuatorReply Actuator Reply] below. - -Getting values -~~~~~~~~~~~~~~ - -While the sMap driver for a device should always be setup to -periodically broadcast the state of a device you may want an up to the -moment value for an actuation point on a device. - -To request a value publish a message to the following topic: - -:: - - #python - 'devices/actuators/get//' - -Setting Values -~~~~~~~~~~~~~~ - -Value are set in a similar manner: - -To set a value publish a message to the following topic: - -:: - - #python - 'devices/actuators/set//' - -With this header: - -:: - - #python - { - 'requesterID': - } - -And the message contents being the new value of the actuator. - -**The actuator agent expects all messages to be JSON and will parse them -accordingly. Use publish\_json to send messages where possible. This is -significant for Boolean values especially.** - -Actuator Reply -~~~~~~~~~~~~~~ - -#ActuatorReply The ActuatorAgent will reply to both **get** and *set*' -on the **value** topic for an actuator: - -:: - - #python - 'devices/actuators/value//' - -With this header: - -:: - - #python - { - 'requesterID': - } - -With the message containing the value encoded in JSON. - -Actuator Error Reply -~~~~~~~~~~~~~~~~~~~~ - -If something goes wrong the ActuatorAgent will reply to both **get** and -*set*' on the **error** topic for an actuator: - -:: - - #python - 'devices/actuators/error//' - -With this header: - -:: - - #python - { - 'requesterID': - } - -The message will be in the following form: - -:: - - #python - { - 'type': - 'value': - } - -Common Error Types -^^^^^^^^^^^^^^^^^^ - -| ``LockError:: Returned when a request is made when we do not have permission to use a device. (Forgot to schedule, preempted and we did not handle the preemption message correctly, ran out of time in time slot, etc...)`` -| ``ValueError:: Message missing or could not be parsed as JSON.`` - -Other error types involve problem with communication between the -ActuatorAgent and sMap. diff --git a/docs/source/core_services/service_agents/actuator/index.rst b/docs/source/core_services/service_agents/actuator/index.rst deleted file mode 100644 index fb721625f3..0000000000 --- a/docs/source/core_services/service_agents/actuator/index.rst +++ /dev/null @@ -1,40 +0,0 @@ -.. _ActuatorAgent: - -ActuatorAgent -============= - -This agent is used to manage write access to devices. Agents -may request scheduled times, called Tasks, to interact with one or more -devices. - -Actuator Agent Communication ----------------------------- - -:doc:`Scheduling and canceling a Task. ` - -:doc:`Interacting with a device via the -ActuatorAgent. ` - -:doc:`AcutatorAgent responses to a schedule or cancel -request. ` - -:doc:`Schedule state announcements. ` - -:doc:`What happens when a running Task is -preempted. ` - -:doc:`Setup heartbeat signal for a device. ` - -:doc:`ActuatorAgent configuration. ` - -:doc:`Notes on programming agents to work with the -ActuatorAgent ` - - - - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/central_management/index.rst b/docs/source/core_services/service_agents/central_management/index.rst deleted file mode 100644 index 73997edec5..0000000000 --- a/docs/source/core_services/service_agents/central_management/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -=========================== -VOLTTRON Central Management -=========================== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/datamover/datamover.rst b/docs/source/core_services/service_agents/datamover/datamover.rst deleted file mode 100644 index fa46b06b64..0000000000 --- a/docs/source/core_services/service_agents/datamover/datamover.rst +++ /dev/null @@ -1,57 +0,0 @@ -.. _DataMover_Historian: - -=================== -DataMover Historian -=================== - -The DataMover Historian is used to send data from one instance of VOLTTRON to -another. This agent is similar to the Forward Historian but does not publish -data on the target platform's message bus. Messages are instead inserted into -the backup queue in the target's historian. This helps to ensure that messages -are recorded. - -If the target instance becomes unavailable or the target historian is stopped -then this agent's cache will build up until it reaches it's maximum capacity -or the instance and agent comes back online. - -The DataMover now uses the configuration store for storing its -configurations. This allows dynamic updating of configuration without having -to rebuild the agent. - -Configuration Options ---------------------- - -The following JSON configuration file shows all the options currently supported -by the DataMover agent. - -.. code-block:: python - - { - # destination-serverkey - # The destination instance's publickey. Required if the - # destination-vip-address has not been added to the known-host file. - # See vctl auth --help for all instance security options. - # - # This can be retrieved either through the command: - # vctl auth serverkey - # Or if the web is enabled on the destination through the browser at: - # http(s)://hostaddress:port/discovery/ - "destination-serverkey": null, - - # destination-vip-address - REQUIRED - # Address of the target platform. - # Examples: - # "destination-vip": "ipc://@/home/volttron/.volttron/run/vip.socket" - # "destination-vip": "tcp://127.0.0.1:23916" - "destination-vip": "tcp://:", - - # destination_historian_identity - # Identity of the historian to send data to. Only needed if data - # should be sent an agent other than "platform.historian" - "destination-historian-identity": "platform.historian", - - # remote_identity - OPTIONAL - # identity that will show up in peers list on the remote platform - # By default this identity is randomly generated - "remote-identity": "22916.datamover" - } diff --git a/docs/source/core_services/service_agents/datamover/index.rst b/docs/source/core_services/service_agents/datamover/index.rst deleted file mode 100644 index 4228e73d2a..0000000000 --- a/docs/source/core_services/service_agents/datamover/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -================ -Data Mover Agent -================ - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/emailer/index.rst b/docs/source/core_services/service_agents/emailer/index.rst deleted file mode 100644 index 3e32da90bb..0000000000 --- a/docs/source/core_services/service_agents/emailer/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -============= -Emailer Agent -============= - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/externaldata/index.rst b/docs/source/core_services/service_agents/externaldata/index.rst deleted file mode 100644 index a1552724fd..0000000000 --- a/docs/source/core_services/service_agents/externaldata/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -=================== -External Data Agent -=================== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/failover/index.rst b/docs/source/core_services/service_agents/failover/index.rst deleted file mode 100644 index 283ff53654..0000000000 --- a/docs/source/core_services/service_agents/failover/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -============== -Failover Agent -============== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/file_watch_publisher/index.rst b/docs/source/core_services/service_agents/file_watch_publisher/index.rst deleted file mode 100644 index fd5d85fd89..0000000000 --- a/docs/source/core_services/service_agents/file_watch_publisher/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -=========================== -File Watch Publisher Agent -=========================== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/index.rst b/docs/source/core_services/service_agents/index.rst deleted file mode 100644 index 2c255279c4..0000000000 --- a/docs/source/core_services/service_agents/index.rst +++ /dev/null @@ -1,19 +0,0 @@ -============== -Service Agents -============== - -.. toctree:: - :glob: - :maxdepth: 2 - - actuator/index - alerter/index - emailer/index - failover/index - file_watch_publisher/index - platform/index - market_service/index - threshold/index - central_management/index - weather/index - * diff --git a/docs/source/core_services/service_agents/market_service/index.rst b/docs/source/core_services/service_agents/market_service/index.rst deleted file mode 100644 index c62f5e35c4..0000000000 --- a/docs/source/core_services/service_agents/market_service/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -==================== -Market Service Agent -==================== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/platform/Platform-Agent.rst b/docs/source/core_services/service_agents/platform/Platform-Agent.rst deleted file mode 100644 index b2d44094cf..0000000000 --- a/docs/source/core_services/service_agents/platform/Platform-Agent.rst +++ /dev/null @@ -1,28 +0,0 @@ -.. _Platform-Agent: - -Platform Agent -~~~~~~~~~~~~~~ - -Introduction -============ - -The Platform Agent allows communication from a VOLTTRON Central -instance. Each VOLTTRON instance that is to be controlled through the -VOLTTRON Central agent should have one and only one Platform Agent. The -Platform Agent must have the VIP identity of platform.agent. - -Configuration -------------- - -The minimal configuration (and most likely the only used) for a Platform -Agent is as follows - -:: - - { - # Agent id is used in the display on volttron central. - "agentid": "Platform 1", - } - -The other options for the Platform Agent configuration can be found in -the Platform Agent source directory. diff --git a/docs/source/core_services/service_agents/platform/index.rst b/docs/source/core_services/service_agents/platform/index.rst deleted file mode 100644 index ea5b1fe283..0000000000 --- a/docs/source/core_services/service_agents/platform/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -============== -Platform Agent -============== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/sysmon/index.rst b/docs/source/core_services/service_agents/sysmon/index.rst deleted file mode 100644 index 625baed63b..0000000000 --- a/docs/source/core_services/service_agents/sysmon/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -======================= -System Monitoring Agent -======================= - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/threshold/index.rst b/docs/source/core_services/service_agents/threshold/index.rst deleted file mode 100644 index 86843a54a5..0000000000 --- a/docs/source/core_services/service_agents/threshold/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -========================= -Threshold Detection Agent -========================= - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/core_services/service_agents/topic_watcher/index.rst b/docs/source/core_services/service_agents/topic_watcher/index.rst deleted file mode 100644 index 46f6af7a7f..0000000000 --- a/docs/source/core_services/service_agents/topic_watcher/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -=================== -Topic Watcher Agent -=================== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/setup/bootstrap-options.rst b/docs/source/deploying-volttron/bootstrap-options.rst similarity index 100% rename from docs/source/setup/bootstrap-options.rst rename to docs/source/deploying-volttron/bootstrap-options.rst diff --git a/docs/source/deploying-volttron/deployment-planning-options.rst b/docs/source/deploying-volttron/deployment-planning-options.rst new file mode 100644 index 0000000000..5bdbb177d7 --- /dev/null +++ b/docs/source/deploying-volttron/deployment-planning-options.rst @@ -0,0 +1,277 @@ +.. _planning-install: + +=========================== +Planning a VOLTTRON Install +=========================== + +The 3 major installation types for VOLTTRON are doing development, doing research using VOLTTRON, and +collecting and managing physical devices. + +Development and Research installation tend to be smaller footprint installations. For development, the +data is usually synthetic or copied from another source. The existing documentation covers development +installs in significant detail. + +Other deployments will have a better installation experience if they consider certain kinds of questions +while they plan their installation. + + +Questions +========= + + * Do you want to send commands to the machines ? + * Do you want to store the data centrally ? + * How many machines do you expect to collect data from on each "collector" ? + * How often will the machines collect data ? + * Are all the devices visible to the same network ? + * What types of VOLTTRON applications do you want to run ? + + +Commands +-------- + +If you wish to send commands to the devices, you will want to install and configure the Volttron Central +agent. If you are only using VOLTTRON to securely collect the data, you can turn off the extra agents +to reduce the footprint. + + +Storing Data +------------ + +VOLTTRON supports multiple historians. mySQL and MongoDB are the most commonly used. As you plan your +installation, you should consider how quickly you need access to the data and where. If you are looking +at the health and well-being of an entire suite of devices, its likely that you want to do that from a +central location. Analytics can be performed at the edge by VOLTTRON applications or can be performed +across the data usually from a central data repository. The latency that you can tolerate in your data +being available will also determine choices in different agents (ForwardHistorian versus Data Mover) + + +How Many +-------- + +The ratio of how many devices-to-collector machine is based on several factors. These include: + + * how much memory and network bandwidth the collection machine has. More = More devices + * how fast the local storage is can affect how fast the data cache can be written. Very slow + storage devices can fall behind + +The second half of the "how many" question is how many collector paltforms are writing to a single +VOLTTRON platform to store data - and whether that storage is local, remote, big enough, etc. + +If you are storing more than moderate amount of data, you will probably benefit from installing +your database on a different machine than your concreate historian machine. Note: This is +contra-indicated if you have a slow network connection between you concrete historian and your database machine. + +In synthetic testing up to 6 virtual machines hosting 500 devices each ( 18 points) were easily +supported by a single centralized platform writing to a Mongo database - using a high speed network. +That central platform experienced very little CPU or memory load when the VOLTTRON Central agent was disabled. + + +How Often +--------- + +This question is closely related to the last. A higher sampling frequency will create more data. This +wil place more work in the storage phase. + + +Networks +-------- + +In many cases, there are constraints on how networks can interact with each other. In many cases, +these include security considerations. On some sites, the primary network will be protected from less +secure networks and may require different installation considerations. For example, if a data collector +machine and the database machine are on the same network with sufficient security, you may choose +to have the data collector write directly to the database. If the collector is on an isolated building +network then you will likely need to use the ForwardHistorian to bridge the two networks. + + +Other Considerations +-------------------- + +Physical location and maintenance of collector machines must be considered in all live deployments. +Although the number of data points may imply a heavy load on a data collection box, the physical constraints +may limit the practicality of having more than a single box. The other side of that discussion is deploying +many collector boxes may be simpler initially, but may create a maintenance challenge if you don't +plan ahead on how you apply patches, etc. + +Naming conventions should also be considered. The ability to trace data through the system and identify +the collector machine and device can be invaluable in debugging and analysis. + + +.. _Deployment-Options: + +Deployment Options +================== + +There are several ways to deploy the VOLTTRON platform in a Linux environment. It is up to the user to determine which +is right for them. The following assumes that the platform has already been bootstrapped and is ready to run. + + +Simple Command Line +------------------- + +With the VOLTTRON environment activated the platform can be started simply by running VOLTTRON on the command +line. + +:: + + $volttron -vv + +This will start the platform in the current terminal with very verbose logging turned on. This +is most appropriate for testing Agents or testing a deployment for problems before switching to a +more long term solution. This will print all log messages to the console in real time. + +This should not be used for long term deployment. As soon as an SSH session is terminated for whatever reason +the processes attached to that session will be killed. This also will not capture log message to a file. + + +Running VOLTTRON as a Background Process +---------------------------------------- + +A simple, more long term solution, is to run volttron in the background and disown it from the current terminal. + +.. warning:: + If you plan on running VOLTTRON in the background and detaching it from the + terminal with the ``disown`` command be sure to redirect stderr and stdout to ``/dev/null``. + Even if logging to a file is used some libraries which VOLTTRON relies on output + directly to stdout and stderr. This will cause problems if those file descriptors + are not redirected to ``/dev/null``. + +.. code-block:: bash + + $volttron -vv -l volttron.log > /dev/null 2>&1& + +Alternatively: + +.. code-block:: bash + + ``./start-volttron`` + +.. note:: + + If you are not in an activated environment, this script will start the platform running in the background in the + correct environment, however the environment will not be activated for you, you must activate it yourself. + +**If there are other jobs running in your terminal be sure to disown the correct one.** + +.. code-block:: console + + $jobs + [1]+ Running something else + [2]+ Running ./start-volttron + + #Disown VOLTTRON + $disown %2 + +This will run the VOLTTRON platform in the background and turn it into a daemon. The log output will be directed +to a file called ``volttron.log`` in the current directory. + +To keep the size of the log under control for more longer term deployments us the rotating log configuration file +``examples/rotatinglog.py``. + +.. code-block:: bash + + $volttron -vv --log-config examples/rotatinglog.py > /dev/null 2>&1& + +This will start a rotate the log file at midnight and limit the total log data to seven days worth. + +The main downside to this approach is that the VOLTTRON platform will not automatically +resume if the system is restarted. It will need to be restarted manually after reboot. + + +Setting up VOLTTRON as a System Service +--------------------------------------- + + +Systemd +^^^^^^^ + +An example service file ``scripts/admin/volttron.service`` for systemd cas be used as a starting point +for setting up VOLTTRON as a service. Note that as this will redirect all the output that would +be going to stdout - to the syslog. This can be accessed using `journalctl`. For systems that run +all the time or have a high level of debugging turned on, we recommend checking the system's +logrotate settings. + +.. code-block:: console + + [Unit] + Description=VOLTTRON Platform Service + After=network.target + + [Service] + Type=simple + + #Change this to the user that VOLTTRON will run as. + User=volttron + Group=volttron + + #Uncomment and change this to specify a different VOLTTRON_HOME + #Environment="VOLTTRON_HOME=/home/volttron/.volttron" + + #Change these to settings to reflect the install location of VOLTTRON + WorkingDirectory=/var/lib/volttron + ExecStart=/var/lib/volttron/env/bin/volttron -vv + ExecStop=/var/lib/volttron/env/bin/volttron-ctl shutdown --platform + + + [Install] + WantedBy=multi-user.target + +After the file has been modified to reflect the setup of the platform you can install it with the +following commands. These need to be run as root or with sudo as appropriate. + +.. code-block:: console + + #Copy the service file into place + cp scripts/admin/volttron.service /etc/systemd/system/ + + #Set the correct permissions if needed + chmod 644 /etc/systemd/system/volttron.service + + #Notify systemd that a new service file exists (this is crucial!) + systemctl daemon-reload + + #Start the service + systemctl start volttron.service + + +Init.d +^^^^^^ + +An example init script ``scripts/admin/volttron`` can be used as a starting point for +setting up VOLTTRON as a service on init.d based systems. + +Minor changes may be needed for the file to work on the target system. Specifically +the ``USER``, ``VLHOME``, and ``VOLTTRON_HOME`` variables may need to be changed. + +.. code-block:: console + + ... + #Change this to the user VOLTTRON will run as. + USER=volttron + #Change this to the install location of VOLTTRON + VLHOME=/var/lib/volttron + + ... + + #Uncomment and change this to specify a different VOLTTRON_HOME + #export VOLTTRON_HOME=/home/volttron/.volttron + + +The script can be installed with the following commands. These need to be run as root or with `sudo` as appropriate. + +.. code-block:: console + + #Copy the script into place + cp scripts/admin/volttron /etc/init.d/ + + #Make the file executable + chmod 755 /etc/init.d/volttron + + #Change the owner to root + chown root:root /etc/init.d/volttron + + #These will set it to startup automatically at boot + update-rc.d volttron defaults + + #Start the service + /etc/init.d/volttron start diff --git a/docs/source/devguides/walkthroughs/Deployment-Walkthrough.rst b/docs/source/deploying-volttron/deployment-walk-through.rst similarity index 85% rename from docs/source/devguides/walkthroughs/Deployment-Walkthrough.rst rename to docs/source/deploying-volttron/deployment-walk-through.rst index e45852e4a5..72432594fd 100644 --- a/docs/source/devguides/walkthroughs/Deployment-Walkthrough.rst +++ b/docs/source/deploying-volttron/deployment-walk-through.rst @@ -1,7 +1,8 @@ -.. _Deployment-Walkthrough: +.. _Deployment-Walk-through: -Deployment Walkthrough -~~~~~~~~~~~~~~~~~~~~~~ +======================= +Deployment Walk-through +======================= This page is meant as an overview of setting up a VOLTTRON deployment which consists of one or more platforms collecting data and being @@ -9,18 +10,21 @@ managed by another platform running the VOLTTRON Central agent. High level instructions are included but for more details on each step, please follow links to that section of the wiki. -Assumptions: + +Assumptions +----------- - “Data Collector” is the box that has the drivers and is collecting data it needs to forward. - “Volttron Central/VC” is the box that has the historian which will save data to the database. -- VOLTTRON_HOME is assumed to the default on both boxes which is: /home//.volttron +- VOLTTRON_HOME is assumed to the default on both boxes which is: `/home//.volttron` -Notes/Tips: +Notes/Tips +---------- - Aside from installing the required packages with apt-get, ``sudo`` is not required and should not be used. VOLTTRON is designed to be run as a non-root user and running with sudo is not supported. -- The convenience :ref:`scripts ` have been developed to simplify +- The convenience :ref:`scripts ` have been developed to simplify many of the repetitive multi-step processes. For instance, ``scripts/core/make-listener`` can be modified for any agent and make it one command to stop, remove, build, install, configure, tag, @@ -34,29 +38,34 @@ Notes/Tips: - Double check firewall rules/policies when setting up a multi-node deployment to ensure that platforms can communicate -On all machines: -================ + +On all machines +=============== On all machines in the deployment, setup the platform, setup encryption, authentication, and authorization. Also, build the basic agents for the deployment. All platforms will need a PlatformAgent and a Historian. -Using :ref:`scripts ` will help simplify this project. +Using :ref:`scripts ` will help simplify this project. + +:ref:`Install required packages ` -:ref:`Install required packages ` ---------------------------------------------------------- - ``sudo apt-get install build-essential python-dev openssl libssl-dev libevent-dev git`` -:ref:`Build the project ` +:ref:`Build the project ` + ---------------------------------------------- - Clone the repository and build using ``python bootstrap.py`` + Configuring Platform --------------------- -On VC: -====== + +On VC +===== - Run :ref:`vcfg ` - Setup as VOLTTRON Central. @@ -66,16 +75,17 @@ On VC: 2016-05-19 08:42:58,062 () volttron.platform.main INFO: public key: -On the data collector: -====================== - -Setup :ref:`drivers ` ---------------------------------------- +On the data collector +===================== -For a simple case, follow instructions to install a :ref:`Fake Driver`` + +Driver Setup +------------ + +For a simple case, follow instructions to install a :ref:`Fake Driver ` for testing purposes. For an actual deployment against real devices see the following: -- Create a :ref:`Master Driver Agent ` to coordinate +- Create a :ref:`Master Driver Agent ` to coordinate drivers for the devices controlled by this platform. - For :ref:`MODBUS ` devices, create config files and point configuration files. @@ -85,13 +95,14 @@ for testing purposes. For an actual deployment against real devices see the foll Setup the Forwarder ------------------- + Now that data is being published to the bus, a :ref:`Forward Historian` can be configured to send this data to the VC instance for storage. - Use: vctl keypair to generate a keypair - cat VOLTTRON_HOME/keypair to get the public and secret keys - Create a config directory in the main project directory -- Setup a :ref:`Forward Historian` +- Setup a :ref:`Forward Historian` - cp services/core/ForwardHistorian/config config/forwarder.config - Edit forwarder.config using the VC’s VIP address, the public server key, and the keypair @@ -110,6 +121,7 @@ Start a listener agent on VC, you should see data from the data collector appear In the log for VC, check for credentials success for the ip of data collector. + Registering the collection platform ==================================== @@ -118,6 +130,7 @@ Registering the collection platform - Enter a name for the collection platform and the ip configured http://: - Open the tree upper left of the UI and find your platform. + Troubleshooting: ================ @@ -126,3 +139,9 @@ Troubleshooting: ipc:\ //@/home/volttron/.volttron/run/vip.socket Change password by putting pw hash in config file Add remote ip address to config file + + +.. toctree:: + + deployment-planning-options + bootstrap-options diff --git a/docs/source/devguides/walkthroughs/files/add-charts-button.png b/docs/source/deploying-volttron/files/add-charts-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/add-charts-button.png rename to docs/source/deploying-volttron/files/add-charts-button.png diff --git a/docs/source/devguides/walkthroughs/files/add-charts.png b/docs/source/deploying-volttron/files/add-charts.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/add-charts.png rename to docs/source/deploying-volttron/files/add-charts.png diff --git a/docs/source/devguides/walkthroughs/files/chart-multiple-lines.png b/docs/source/deploying-volttron/files/chart-multiple-lines.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/chart-multiple-lines.png rename to docs/source/deploying-volttron/files/chart-multiple-lines.png diff --git a/docs/source/devguides/walkthroughs/files/chart-type.png b/docs/source/deploying-volttron/files/chart-type.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/chart-type.png rename to docs/source/deploying-volttron/files/chart-type.png diff --git a/docs/source/devguides/walkthroughs/files/charts-window.png b/docs/source/deploying-volttron/files/charts-window.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/charts-window.png rename to docs/source/deploying-volttron/files/charts-window.png diff --git a/docs/source/devguides/walkthroughs/files/filter-and-select.png b/docs/source/deploying-volttron/files/filter-and-select.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/filter-and-select.png rename to docs/source/deploying-volttron/files/filter-and-select.png diff --git a/docs/source/devguides/walkthroughs/files/filter-button.png b/docs/source/deploying-volttron/files/filter-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/filter-button.png rename to docs/source/deploying-volttron/files/filter-button.png diff --git a/docs/source/devguides/walkthroughs/files/filter-name.png b/docs/source/deploying-volttron/files/filter-name.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/filter-name.png rename to docs/source/deploying-volttron/files/filter-name.png diff --git a/docs/source/devguides/walkthroughs/files/filter-status.png b/docs/source/deploying-volttron/files/filter-status.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/filter-status.png rename to docs/source/deploying-volttron/files/filter-status.png diff --git a/docs/source/devguides/walkthroughs/files/go-to-charts.png b/docs/source/deploying-volttron/files/go-to-charts.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/go-to-charts.png rename to docs/source/deploying-volttron/files/go-to-charts.png diff --git a/docs/source/devguides/walkthroughs/files/inspect-charts.png b/docs/source/deploying-volttron/files/inspect-charts.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/inspect-charts.png rename to docs/source/deploying-volttron/files/inspect-charts.png diff --git a/docs/source/devguides/walkthroughs/files/load-chart.png b/docs/source/deploying-volttron/files/load-chart.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/load-chart.png rename to docs/source/deploying-volttron/files/load-chart.png diff --git a/docs/source/devguides/walkthroughs/files/load-topics.png b/docs/source/deploying-volttron/files/load-topics.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/load-topics.png rename to docs/source/deploying-volttron/files/load-topics.png diff --git a/docs/source/devguides/walkthroughs/files/load-tree-item.png b/docs/source/deploying-volttron/files/load-tree-item.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/load-tree-item.png rename to docs/source/deploying-volttron/files/load-tree-item.png diff --git a/docs/source/devguides/walkthroughs/files/login-screen.png b/docs/source/deploying-volttron/files/login-screen.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/login-screen.png rename to docs/source/deploying-volttron/files/login-screen.png diff --git a/docs/source/devguides/walkthroughs/files/logout-button.png b/docs/source/deploying-volttron/files/logout-button.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/logout-button.png rename to docs/source/deploying-volttron/files/logout-button.png diff --git a/docs/source/devguides/walkthroughs/files/manage-platforms.png b/docs/source/deploying-volttron/files/manage-platforms.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/manage-platforms.png rename to docs/source/deploying-volttron/files/manage-platforms.png diff --git a/docs/source/devguides/walkthroughs/files/multiplatform-config.png b/docs/source/deploying-volttron/files/multiplatform-config.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/multiplatform-config.png rename to docs/source/deploying-volttron/files/multiplatform-config.png diff --git a/docs/source/devguides/walkthroughs/files/multiplatform-discovery-config.png b/docs/source/deploying-volttron/files/multiplatform-discovery-config.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/multiplatform-discovery-config.png rename to docs/source/deploying-volttron/files/multiplatform-discovery-config.png diff --git a/docs/source/devguides/walkthroughs/files/multiplatform-external-address.png b/docs/source/deploying-volttron/files/multiplatform-external-address.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/multiplatform-external-address.png rename to docs/source/deploying-volttron/files/multiplatform-external-address.png diff --git a/docs/source/devguides/walkthroughs/files/multiplatform-pubsub.png b/docs/source/deploying-volttron/files/multiplatform-pubsub.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/multiplatform-pubsub.png rename to docs/source/deploying-volttron/files/multiplatform-pubsub.png diff --git a/docs/source/devguides/walkthroughs/files/multiplatform-setupmode-auth-screen.png b/docs/source/deploying-volttron/files/multiplatform-setupmode-auth-screen.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/multiplatform-setupmode-auth-screen.png rename to docs/source/deploying-volttron/files/multiplatform-setupmode-auth-screen.png diff --git a/docs/source/devguides/walkthroughs/files/multiplatform-terminator-setup.png b/docs/source/deploying-volttron/files/multiplatform-terminator-setup.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/multiplatform-terminator-setup.png rename to docs/source/deploying-volttron/files/multiplatform-terminator-setup.png diff --git a/docs/source/devguides/walkthroughs/files/pin-chart.png b/docs/source/deploying-volttron/files/pin-chart.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/pin-chart.png rename to docs/source/deploying-volttron/files/pin-chart.png diff --git a/docs/source/devguides/walkthroughs/files/platforms.png b/docs/source/deploying-volttron/files/platforms.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/platforms.png rename to docs/source/deploying-volttron/files/platforms.png diff --git a/docs/source/devguides/walkthroughs/files/problems-found.png b/docs/source/deploying-volttron/files/problems-found.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/problems-found.png rename to docs/source/deploying-volttron/files/problems-found.png diff --git a/docs/source/devguides/walkthroughs/files/side-panel-closed.png b/docs/source/deploying-volttron/files/side-panel-closed.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/side-panel-closed.png rename to docs/source/deploying-volttron/files/side-panel-closed.png diff --git a/docs/source/devguides/walkthroughs/files/side-panel-open.png b/docs/source/deploying-volttron/files/side-panel-open.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/side-panel-open.png rename to docs/source/deploying-volttron/files/side-panel-open.png diff --git a/docs/source/devguides/walkthroughs/files/terminator-setup.png b/docs/source/deploying-volttron/files/terminator-setup.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/terminator-setup.png rename to docs/source/deploying-volttron/files/terminator-setup.png diff --git a/docs/source/devguides/walkthroughs/files/vc-agents.png b/docs/source/deploying-volttron/files/vc-agents.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-agents.png rename to docs/source/deploying-volttron/files/vc-agents.png diff --git a/docs/source/devguides/walkthroughs/files/vc-cert-warning-1.png b/docs/source/deploying-volttron/files/vc-cert-warning-1.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-cert-warning-1.png rename to docs/source/deploying-volttron/files/vc-cert-warning-1.png diff --git a/docs/source/devguides/walkthroughs/files/vc-cert-warning-2.png b/docs/source/deploying-volttron/files/vc-cert-warning-2.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-cert-warning-2.png rename to docs/source/deploying-volttron/files/vc-cert-warning-2.png diff --git a/docs/source/devguides/walkthroughs/files/vc-cert-warning-3.png b/docs/source/deploying-volttron/files/vc-cert-warning-3.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-cert-warning-3.png rename to docs/source/deploying-volttron/files/vc-cert-warning-3.png diff --git a/docs/source/devguides/walkthroughs/files/vc-cert-warning-4.png b/docs/source/deploying-volttron/files/vc-cert-warning-4.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-cert-warning-4.png rename to docs/source/deploying-volttron/files/vc-cert-warning-4.png diff --git a/docs/source/devguides/walkthroughs/files/vc-dashboard.png b/docs/source/deploying-volttron/files/vc-dashboard.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-dashboard.png rename to docs/source/deploying-volttron/files/vc-dashboard.png diff --git a/docs/source/devguides/walkthroughs/files/vc-login.png b/docs/source/deploying-volttron/files/vc-login.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-login.png rename to docs/source/deploying-volttron/files/vc-login.png diff --git a/docs/source/devguides/walkthroughs/files/vc-platform.png b/docs/source/deploying-volttron/files/vc-platform.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/vc-platform.png rename to docs/source/deploying-volttron/files/vc-platform.png diff --git a/docs/source/devguides/walkthroughs/files/volttron-admin-page.png b/docs/source/deploying-volttron/files/volttron-admin-page.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/volttron-admin-page.png rename to docs/source/deploying-volttron/files/volttron-admin-page.png diff --git a/docs/source/deploying-volttron/forward-historian-deployment.rst b/docs/source/deploying-volttron/forward-historian-deployment.rst new file mode 100644 index 0000000000..e3c67b7f46 --- /dev/null +++ b/docs/source/deploying-volttron/forward-historian-deployment.rst @@ -0,0 +1,86 @@ +.. _Forward-Historian-Deployment: + +============================ +Forward Historian Deployment +============================ + +This guide describes a simple setup where one VOLTTRON instance collects data from a fake devices and sends to another +instance . Lets consider the following example. + +We are going to create two VOLTTRON instances and send data from one VOLTTRON instance running a fake driver(subscribing +values from a fake device) and sending the values to the second VOLTTRON instance. + + +VOLTTRON instance 1 forwards data to VOLTTRON instance 2 +-------------------------------------------------------- + + +VOLTTRON instance 1 +^^^^^^^^^^^^^^^^^^^ + +- ``vctl shutdown –platform`` (if the platform is already working) +- ``vcfg`` (this helps in configuring the volttron instance + http://volttron.readthedocs.io/en/releases-4.1/core_services/control/VOLTTRON-Config.html + + - Specify the IP of the machine: ``tcp://130.20.*.*:22916`` + - Specify the port you want to use + - Specify if you want to run VC(Volttron Central) here or this this instance would be controlled + by a VC and the IP and port of the VC + + - Then install agents like Master Driver Agent with a fake driver for the instance. + - Install a listener agent so see the topics that are coming from the diver agent + - Then run the volttron instance by using the following command: ``./start-volttron`` + +- Volttron authentication: We need to add the IP of the instance 2 in the `auth.config` file of the VOLTTRON agent. + This is done as follows: + + - ``vctl auth-add`` + - We specify the IP of the instance 2 and the credentials of the agent (read + :ref:`Agent Authentication ` + - For specifying authentication for all the agents , we specify ``/.*/`` + - This should enable authentication for all the volttron-instance based on the IP you specify here + + +For this documentation, the topics from the driver agent will be send to the instance 2 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- We use the existing agent called the Forward Historian for this purpose which is available in service/core in the + VOLTTRON directory. +- In the config file under the Forward Historian directory, we modify the following fields: + + - Destination-vip: the IP of the volttron instance to which we have to forward the data to along with the port + number. Example : ``tcp://130.20.*.*:22916`` + - Destination-serverkey: The server key of the VOLTTRON instance to which we need to forward the data to. + This can be obtained at the VOLTTRON instance by typing ``vctl auth serverkey`` + +- Service_topic_list: specify the topics you want to forward specifically instead of all the values. +- Once the above values are set, your forwarder is all set . +- You can create a script file for the same and execute the agent. + + +VOLTTRON instance 2 +^^^^^^^^^^^^^^^^^^^ + +- ``vctl shutdown –platform`` (if the platform is already working) +- ``volttron-cfg`` (this helps in configuring the volttron instance) + http://volttron.readthedocs.io/en/releases-4.1/core_services/control/VOLTTRON-Config.html + + - Specify the IP of the machine : ``tcp://130.20.*.*:22916`` + - Specify the port you want to use. + - Install the listener agent (this will show the connection from instance 1 if its successful + and then show all the topics from instance 1. + +- Volttron authentication: We need to add the IP of the instance 1 in the auth.config file of the VOLTTRON agent . This + is done as follows: + + - ``vctl auth-add`` + - We specify the IP of the instance 1 and the credentials of the agent + - For specifying authentication for all the agents , we specify ``/.*/`` + - This should enable authentication for all the volttron-instance based on the IP you specify here + + +Listener Agent +^^^^^^^^^^^^^^ + +Run the listener agent on this instance to see the values being forwarded from instance 1. Once the above setup is +done, you should be able to see the values from instance 1 on the listener agent of instance 2. diff --git a/docs/source/devguides/walkthroughs/Multiplatform-Walkthrough.rst b/docs/source/deploying-volttron/multi-platform-walk-through.rst similarity index 76% rename from docs/source/devguides/walkthroughs/Multiplatform-Walkthrough.rst rename to docs/source/deploying-volttron/multi-platform-walk-through.rst index ac0027704f..1402a5fd5a 100644 --- a/docs/source/devguides/walkthroughs/Multiplatform-Walkthrough.rst +++ b/docs/source/deploying-volttron/multi-platform-walk-through.rst @@ -1,17 +1,18 @@ -.. _Multi-Platform-Walkthrough: +.. _Multi-Platform-Walk-through: -Multi-Platform Connection Walkthrough -===================================== +====================================== +Multi-Platform Connection Walk-through +====================================== Multi-Platform message bus communication alleviates the need for an agent in one platform to connect to another platform -directly in order for it to send/receive messages from the other platform. With multi-platform communication, connections -to external platforms will be maintained by the platforms itself and agents do not have the burden to manage the -connections directly. This guide will show how to connect three VOLTTRON instances with a fake driver running on VOLTTRON -instance 1 publishing to topic with prefix="devices" and listener agents running on other 2 VOLTTRON instances +directly in order for it to send/receive messages from the other platform. With multi-platform communication, +connections to external platforms will be maintained by the platforms itself and agents do not have the burden to manage +the connections directly. This guide will show how to connect three VOLTTRON instances with a fake driver running on +VOLTTRON instance 1 publishing to topic with prefix="devices" and listener agents running on other 2 VOLTTRON instances subscribed to topic "devices". -- `Getting Started <#getting-started>`__ +- :ref:`Getting Started ` - `Multi-Platform Configuration <#multi-platform-configuration>`__ - `Configuration and Authentication in Setup Mode <#configuration-and-authentication-in-setup-mode>`__ - `Setup Configuration and Authentication Manually <#setup-configuration-and-authentication-manually>`__ @@ -46,8 +47,8 @@ or add below line in the `onstart` method .. note:: If using the onstart method remove the @PubSub.subscribe('pubsub', '') from the top of the method. -After :ref:`building VOLTTRON `, open three shells with the current directory the root of the -VOLTTRON repository. Then activate the VOLTTRON environment and export the VOLTTRON\_HOME variable. The home +After :ref:`installing VOLTTRON `, open three shells with the current directory the root of the +VOLTTRON repository. Then activate the VOLTTRON environment and export the `VOLTTRON_HOME` variable. The home variable needs to be different for each instance. .. code-block:: console @@ -56,8 +57,8 @@ variable needs to be different for each instance. $ export VOLTTRON_HOME=~/.volttron1 Run `vcfg` in all the three shells. This command will ask how the instance -should be set up. Many of the options have defaults and that will be sufficient. Enter a different VIP address for each -platform. Configure fake master driver in the first shell and listener agent in second and third shell. +should be set up. Many of the options have defaults and that will be sufficient. Enter a different VIP address for each +platform. Configure fake master driver in the first shell and listener agent in second and third shell. |Terminator Setup| @@ -69,7 +70,7 @@ If the platform supports web server, add the ``bind-web-address`` as well. Here is an example, -Path of the config: $VOLTTRON_HOME/config +Path of the config: `$VOLTTRON_HOME/config` .. code-block:: console @@ -82,14 +83,17 @@ Instance name and bind web address entries added into each VOLTTRON platform's c |Multi-Platform Config| -Next, each instance needs to know the VIP address, platform name and server keys of the remote platforms that it is connecting -to. In addition, each platform has to authenticate or accept the connecting instances' public keys. We can do this step -either by running VOLTTRON in setup mode or configure the information manually. +Next, each instance needs to know the VIP address, platform name and server keys of the remote platforms that it is +connecting to. In addition, each platform has to authenticate or accept the connecting instances' public keys. We can +do this step either by running VOLTTRON in setup mode or configure the information manually. + Configuration and Authentication in Setup Mode ---------------------------------------------- -.. note:: It is necessary for **each** platform to have a web server if running in setup mode +.. note:: + + It is necessary for **each** platform to have a web server if running in setup mode Add list of web addresses of remote platforms in ``$VOLTTRON_HOME/external_address.json`` @@ -122,11 +126,12 @@ After all the connections are authenticated, we can start the instances in norma Setup Configuration and Authentication Manually ----------------------------------------------- + If you do not need web servers in your setup, then you will need to build the platform discovery config file manually. The config file should contain an entry containing VIP address, instance name and serverkey of each remote platform connection. -Name of the file: external_platform_discovery.json +Name of the file: `external_platform_discovery.json` Directory path: Each platform’s VOLTTRON_HOME directory. @@ -157,9 +162,9 @@ Contents of ``external_platform_discovery.json`` of VOLTTRON instance 1, 2, 3 is |Multi-Platform Discovery Config| -After this, you will need to add the server keys of the connecting platforms using the ``vctl`` utility. Type +After this, you will need to add the server keys of the connecting platforms using the ``vctl`` utility. Type **vctl auth add** command on the command prompt and simply hit Enter to select defaults on all fields -except **credentials**. Here, we can either add serverkey of connecting platform or type `/.*/` to allow ALL +except **credentials**. Here, we can either add serverkey of connecting platform or type `/.*/` to allow ALL connections. .. warning:: `/.*/` allows ALL agent and platform connections without authentication. @@ -181,7 +186,6 @@ connections. For more information on authentication see :ref:`authentication`. - Once the initial configuration are setup, you can start all the VOLTTRON instances in normal mode. .. code-block:: console @@ -190,8 +194,10 @@ Once the initial configuration are setup, you can start all the VOLTTRON instanc Next step is to start agents in each platform to observe the multi-platform PubSub communication behavior. + Start Master driver on VOLTTRON instance 1 ------------------------------------------ + If master driver is not configured to auto start when the instance starts up, we can start it explicitly with this command. @@ -202,6 +208,7 @@ command. Start Listener agents on VOLTTRON instance 2 and 3 -------------------------------------------------- + If the listener agent is not configured to auto start when the instance starts up, we can start it explicitly with this command. @@ -236,3 +243,29 @@ We can stop all the VOLTTRON instances by executing below command in each termin :target: ../../_images/multiplatform-discovery-config.png .. |Multi-Platform PubSub| image:: files/multiplatform-pubsub.png :target: ../../_images/multiplatform-pubsub.png + + +.. _External-Address-Configuration: + +Platform External Address Configuration +======================================= + +In the configuration file located in `$VOLTTRON_HOME/config` add ``vip-address=tcp://ip:port`` for each address you want +to listen on: + +:: + + Example + vip-address=tcp://127.0.0.102:8182 + vip-address=tcp://127.0.0.103:8083 + vip-address=tcp://127.0.0.103:8183 + +.. note:: + + The config file is generated after running the `vcfg` command. The VIP-address is for the local platform, NOT the + remote platform. + + +.. toctree:: + + forward-historian-deployment diff --git a/docs/source/devguides/deployment/Linux-Platform-Hardening-Recommendations-for-VOLTTRON-users.rst b/docs/source/deploying-volttron/platform-hardening.rst similarity index 98% rename from docs/source/devguides/deployment/Linux-Platform-Hardening-Recommendations-for-VOLTTRON-users.rst rename to docs/source/deploying-volttron/platform-hardening.rst index 6129b0803e..1e68288c82 100644 --- a/docs/source/devguides/deployment/Linux-Platform-Hardening-Recommendations-for-VOLTTRON-users.rst +++ b/docs/source/deploying-volttron/platform-hardening.rst @@ -1,14 +1,15 @@ -.. _Platform-Hardening-for-VOLTTRON: +.. _Platform-Hardening: -Platform Hardening for VOLTTRON -=============================== +================== +Platform Hardening +================== Rev. 0 \| 1/29/2015 \| Initial Document Development Rev. 1 \| 2/5/2015 \| Integrate comments from extended VOLTTRON team. Introduction -============ +------------ VOLTTRON is an agent-based application development platform for distributed control systems. VOLTTRON itself is built with modern @@ -23,7 +24,7 @@ cyber security strategy that is recommended in this document is based on risk management. Linux System Hardening -====================== +---------------------- Here are the non-exhaustive recommendations for Linux hardening from the VOLTTRON team: @@ -178,7 +179,7 @@ hardening from the VOLTTRON team: e recommend using mod\_security and mod\_evasive modules. System Monitoring -================= +----------------- - Monitor system state and resources. Use a monitoring tool such as Xymon (http://xymon.sourceforge.net) or big brother @@ -207,7 +208,7 @@ System Monitoring (http://cipherdyne.org/psad/) can be used to look for intrusions as well. Security Testing -================ +---------------- Every security control discussed in the previous sections must be tested to determine correct operation and impact. @@ -220,7 +221,7 @@ tools such as Nessus (http://www.tenable.com/products/nessus) and nmap (http://nmap.org) should be used to perform cyber security testing. Conclusion -========== +---------- No system is 100% secure unless it is disconnected from the network and is in a physically secure location. VOLTTRON team recommends a diff --git a/docs/source/devguides/walkthroughs/SingleMachine-Walkthrough.rst b/docs/source/deploying-volttron/single-machine-walk-through.rst similarity index 96% rename from docs/source/devguides/walkthroughs/SingleMachine-Walkthrough.rst rename to docs/source/deploying-volttron/single-machine-walk-through.rst index 940a671e63..5ffd657274 100644 --- a/docs/source/devguides/walkthroughs/SingleMachine-Walkthrough.rst +++ b/docs/source/deploying-volttron/single-machine-walk-through.rst @@ -1,14 +1,15 @@ -.. _SingleMachine-Walkthrough: +.. _Single-Machine-Walk-through: Single Machine Deployment ========================= -The purpose of this demonstration is to show the process of setting up a simple VOLTTRON instance for use on a single machine. +The purpose of this demonstration is to show the process of setting up a simple VOLTTRON instance for use on a single +machine. Install and Build VOLTTRON -------------------------- -First, :ref:`install ` and :ref:`build ` VOLTTRON: +First, :ref:`install ` VOLTTRON: For a quick reference: @@ -214,4 +215,4 @@ You will now see a list of agents. They should all be running. For more information on VOLTTRON Central, please see: * :ref:`VOLTTRON Central Management ` -* :ref:`VOLTTRON Central Demo ` +* :ref:`VOLTTRON Central Demo ` diff --git a/docs/source/devguides/walkthroughs/VOLTTRON-Central-Demo.rst b/docs/source/deploying-volttron/volttron-central.rst similarity index 85% rename from docs/source/devguides/walkthroughs/VOLTTRON-Central-Demo.rst rename to docs/source/deploying-volttron/volttron-central.rst index bd7d89825e..4025b0df30 100644 --- a/docs/source/devguides/walkthroughs/VOLTTRON-Central-Demo.rst +++ b/docs/source/deploying-volttron/volttron-central.rst @@ -1,5 +1,6 @@ .. _VOLTTRON-Central-Demo: +===================== VOLTTRON Central Demo ===================== @@ -29,12 +30,13 @@ interface. - `Dashboard Charts <#dashboard-charts>`__ - `Remove Charts <#remove-charts>`__ + Getting Started --------------- -After :ref:`building VOLTTRON `, open three shells +After :ref:`installing VOLTTRON `, open three shells with the current directory the root of the VOLTTRON repository. Then activate -the VOLTTRON environment and export the VOLTTRON\_HOME variable. The home +the VOLTTRON environment and export the `VOLTTRON_HOME` variable. The home variable needs to be different for each instance. If you are using Terminator you can right click and select "Split Vertically". @@ -86,11 +88,10 @@ and localhost is volttron-pc. Creating new web server certificate. Is this an instance of volttron central? [N]: y Configuring /home/user/volttron/services/core/VolttronCentral. - Enter volttron central admin user name: - Enter volttron central admin password: - Retype password: Installing volttron central. Should the agent autostart? [N]: y + VC admin and password are set up using the admin web interface. + After starting VOLTTRON, please go to https://volttron-pc:8443/admin/login.html to complete the setup. Will this instance be controlled by volttron central? [Y]: y Configuring /home/user/volttron/services/core/VolttronCentralPlatform. What is the name of this instance? [volttron1]: @@ -130,17 +131,18 @@ to select defaults on all fields except **credentials**, where we will type For more information on authorization see :ref:`authentication`. + Remote Platform Configuration ----------------------------- The next step is to configure the instances that will connect to VOLTTRON Central. In the second and third terminal windows run `vcfg`. Like -the VOLTTRON\_HOME variable, these instances need to have unique addresses. +the VOLTTRON\_HOME variable, these instances need to have unique VIP addresses and unique instance names. Install a platform agent and a historian as before. Since we used the default options when configuring VOLTTRON Central, we can use the default options when configuring these platform agents as well. The configuration will be a little -different. +different. The example below is for the second volttron instance. Note the unique VIP address and instance name. .. code-block:: console @@ -153,11 +155,10 @@ different. What type of message bus (rmq/zmq)? [zmq]: What is the vip address? [tcp://127.0.0.1]: tcp://127.0.0.2 What is the port for the vip address? [22916]: - Is this instance web enabled? [N]: - Is this an instance of volttron central? [N]: - Will this instance be controlled by volttron central? [Y]: y + Is this instance web enabled? [N]: + Will this instance be controlled by volttron central? [Y]: Configuring /home/user/volttron/services/core/VolttronCentralPlatform. - What is the name of this instance? [volttron1]: + What is the name of this instance? [volttron1]: volttron2 What is the hostname for volttron central? [https://volttron-pc]: What is the port for volttron central? [8443]: Should the agent autostart? [N]: y @@ -175,12 +176,21 @@ different. (volttron)user@volttron-pc:~/volttron$ + Starting the Demo ----------------- -Start each Volttron instance after configuration. The "-l" option in the -following command tells volttron to log to a file. The file name -should be different for each instance. +Start each Volttron instance after configuration. You have two options. + +Option 1: The following command starts the volttron process in the background. The "-l" option tells volttron to log +to a file. The file name should be different for each instance. + +.. code-block:: console + + $ volttron -vv -l volttron.log& + +Option 2: Use the utility script start-volttron. + .. code-block:: console @@ -214,14 +224,17 @@ or In each of the above examples one could use * suffix to match more than one agent. -Open your browser to `localhost:8443/vc/index.hmtl` and and log in with the -credentials you provided. The platform agents should be automatically register -with VOLTTRON central. +Open your browser to the web address that you specified for the VOLTTRON Central agent that you configured for the +first instance. The web address is the admin web interface to Volttron; it is defined in the configuration of the +first instance. In the above examples, the configuration file would be located at `~/.volttron1/config` and the +admin web interface would be defined in the "volttron-central-address" field. The address takes the pattern: +`https://:8443/index.html`, where localhost is the local host of your machine. +In the above examples, our localhost is `volttron-pc`; thus our admin web interface would be `https://volttron-pc:8443/index.html`. -.. note:: - - localhost is the local host of your machine. In the above examples, - this was volttron-pc. +When you open the page for the first time, Volttron will prompt you to create a new user and password. Recall when you +ran `vcfg` in the first shell to configure the VOLTTRON Central agent, the script displayed +the message "VC admin and password are set up using the admin web interface." This is where you setup you admin name +and password. Stopping the Demo @@ -239,13 +252,18 @@ Once the demo is complete you may wish to see the :ref:`VOLTTRON Central Management Agent ` page for more details on how to configure the agent for your specific use case. + Log In ------ -To log in to VOLTTRON Central, navigate in a browser to localhost:8443/vc/index.html, and enter the user name and password on the login screen. +To log in to VOLTTRON Central, open a browser and login to the Volttron web interface, which takes the form +`https://localhost:8443/vc/index.html` where localhost is the local host of your machine. In the above example, we open the +following URL in which our localhost is "volttron-pc": https://volttron-pc:8443/vc/index.html and enter the user name +and password on the login screen. |Login Screen| + Log Out ------- @@ -254,8 +272,9 @@ of the screen. |Logout Button| + Platforms Tree -~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ The side panel on the left of the screen can be extended to reveal the tree view of registered platforms. @@ -268,6 +287,7 @@ Top-level nodes in the tree are platforms. Platforms can be expanded in the tree to reveal installed agents, devices on buildings, and performance statistics about the platform instances. + Loading the Tree ---------------- @@ -279,6 +299,7 @@ node is expanded is when the items for that platform are loaded. After a platform has been loaded in the tree, all the items under a node can be quickly expanded by double-clicking on the node. + Health Status ------------- @@ -292,6 +313,7 @@ cursor over the item. |Status Tooltips| + Filter the Tree --------------- @@ -308,8 +330,9 @@ or "unknown." |Filter Status| + Platforms Screen -~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^ This screen lists the registered VOLTTRON platforms and allows new platforms to be registered by clicking the Register Platform button. @@ -319,8 +342,9 @@ to go to the platform management view. |Platforms| + Platform View -~~~~~~~~~~~~~ +^^^^^^^^^^^^^ From the platforms screen, click on the name link of a platform to manage it. Managing a platform includes installing, starting, stopping, @@ -338,8 +362,9 @@ on a specific type of agent. For instance, platform agents and VOLTTRON Central agents can't be removed or stopped, but they can be restarted if they've been interrupted. + Add Charts -~~~~~~~~~~ +^^^^^^^^^^ Performance statistics and device points can be added to charts either from the Charts page or from the platforms tree in the side panel. diff --git a/docs/source/community_resources/index.rst b/docs/source/developing-volttron/contributing/community.rst similarity index 83% rename from docs/source/community_resources/index.rst rename to docs/source/developing-volttron/contributing/community.rst index 1c174ac23f..bb53778245 100644 --- a/docs/source/community_resources/index.rst +++ b/docs/source/developing-volttron/contributing/community.rst @@ -1,33 +1,38 @@ -.. _community: +.. _Community: + +================== +Join the Community +================== -=================== -Join the Community -=================== The VOLTTRON team aims to work with users and contributors to continuously improve the platform with features requested by the community as well as architectural features that improve robustness, security, and scalability. Contributing back to the project, which is encouraged but not required, enhances its capabilities for the whole community. To -learn more, check out :ref:`Contributing ` and :ref:`Documentation `. +learn more, check out :ref:`Contributing ` and :ref:`Documentation `. + Slack Channel -^^^^^^^^^^^^^ +============= volttron-community.slack.com is where the |VOLTTRON| community at large can ask questions and meet with others using |VOLTTRON|. To be added to Slack please email the VOLTTRON team at `volttron@pnnl.gov `__. + Mailing List -^^^^^^^^^^^^ +============ Join the mailing list by emailing `volttron@pnnl.gov `__. + Stack Overflow -^^^^^^^^^^^^^^ +============== The VOLTTRON community supports questions being asked and answered through Stack Overflow. The questions tagged with the `volttron` tag can be found at http://stackoverflow.com/questions/tagged/volttron. + Office Hours -^^^^^^^^^^^^ +============ PNNL hosts office hours every other week on Fridays at 11 AM (PST). These meetings are designed to be very informal where VOLTTRON developers can answer specific questions about the inner workings of VOLTTRON. These meetings are also @@ -37,15 +42,13 @@ available through a Zoom meeting. To be invited to the link meeting, contact the Meetings are recorded and can be reviewed `here `__. + Contributing Back -^^^^^^^^^^^^^^^^^^ +================= .. toctree:: - :glob: - :maxdepth: 1 - contributing - documentation - * + contributing-code + contributing-documentation .. |VOLTTRON| unicode:: VOLTTRON U+2122 diff --git a/docs/source/community_resources/contributing.rst b/docs/source/developing-volttron/contributing/contributing-code.rst similarity index 64% rename from docs/source/community_resources/contributing.rst rename to docs/source/developing-volttron/contributing/contributing-code.rst index c977f9960a..03960970fc 100644 --- a/docs/source/community_resources/contributing.rst +++ b/docs/source/developing-volttron/contributing/contributing-code.rst @@ -1,33 +1,34 @@ -.. _contributing: +.. _Contributing-Code: +======================== Contributing to VOLTTRON ======================== As an open source project VOLTTRON requires input from the community to keep development focused on new and useful -features. To that end were are revising our commit process to hopefully allow more committers to be apart of the +features. To that end we are revising our commit process to hopefully allow more contributors to be apart of the community. The following document outlines the process for source code and documentation to be submitted. There are GUI tools that may make this process easier, however this document will focus on what is required from the command line. The only requirements for contributing are Git (Linux version control software) and your favorite web browser. -Getting Started -~~~~~~~~~~~~~~~ Forking the main VOLTTRON repository ------------------------------------- +==================================== The first step to editing the repository is to fork it into your own user space. This is done by pointing -your favorite web browser to -http://github.com/VOLTTRON/volttron and then clicking "Fork" on the upper right of the screen. (Note: You must have a -GitHub account to fork the repository. If you don't have one, we encourage you to `sign up https://github.com/join?source_repo=VOLTTRON%2Fvolttron`.) +your favorite web browser to http://github.com/VOLTTRON/volttron and then clicking "Fork" on the upper right of the +screen. (Note: You must have a GitHub account to fork the repository. If you don't have one, we encourage you to +`sign up https://github.com/join?source_repo=VOLTTRON%2Fvolttron`.) + Cloning 'YOUR' VOLTTRON forked repository ------------------------------------------ +========================================= -The next step in the process is to get your forked repository down to your computer to work on. -This will create an identical copy of the GitHub repository on your local machine. To do this you need to know the address of -your repository. The URL to your repository address will be "https://github.com//volttron.git". From a terminal execute the following commands which will create a directory "git" in your home directory and then change to that directory, clone from your repository, and finally change into the cloned repository. +The next step in the process is to copy your forked repository onto your computer to work on. This will create an +identical copy of the GitHub repository on your local machine. To do this you need to know the address of your +repository. The URL to your repository address will be "https://github.com//volttron.git". From a +terminal execute the following commands: .. note:: @@ -41,14 +42,16 @@ your repository. The URL to your repository address will be "https://github.com git clone -b develop https://github.com//volttron.git cd volttron + Adding and Committing files ---------------------------- +=========================== Now that you have your repository cloned, it's time to start doing some modifications. Using a simple text editor you can create or modify any file in the volttron directory. After making a modification or creating a file it is time to move it to the stage for review before committing to the local repository. For this example let's assume we have made a change to `README.md` in the root of the volttron directory and added a new file called `foo.py`. To get -those files in the staging area (preparing for committing to the local repository) we would execute the following commands +those files in the staging area (preparing for committing to the local repository) we would execute the following +commands: .. code-block:: bash @@ -58,63 +61,72 @@ those files in the staging area (preparing for committing to the local repositor # Alternatively in one command git add foo.py README.md -After adding the files to the stage you can review the staged files by executing +After adding the files to the stage you can review the staged files by executing: .. code-block:: bash git status -Finally in order to commit to the local repository we need to think of what change we actually did and be able to +Finally, in order to commit to the local repository we need to think of what change we actually did and be able to document it. We do that with a commit message (the -m parameter) such as the following. .. code-block:: bash git commit -m "Added new foo.py and updated copyright of README.md" + Pushing to the remote repository --------------------------------- +================================ The next step is to share our changes with the world through GitHub. We can do this by pushing the commits -from your local repository out to your GitHub repository. This is done by the following command. +from your local repository out to your GitHub repository. This is done by the following command: .. code-block:: bash git push - # alternative where origin is the name of the remote you are pushing to - # more on that later. - git push origin -Getting modifications to the main VOLTTRON repository ------------------------------------------------------ -Now we want our changes to be added into the main VOLTTRON repository. After all our `foo.py` can cure a lot of the +Creating a Pull Request to the main VOLTTRON repository +======================================================= + +Now we want our changes to be added into the main VOLTTRON repository. After all, our `foo.py` can cure a lot of the world's problems and of course it is always good to have a copyright with the correct year. Open your browser to https://github.com/VOLTTRON/volttron/compare/develop...YOUR_USERNAME:develop. -On that page the base fork should always be VOLTTRON/volttron with the base develop whilst the head fork should +On that page the base fork should always be VOLTTRON/volttron with the base develop, the head fork should be /volttron and the compare should be the branch in your repository to pull from. Once you have -verified that you have got the right changes made then, click on create pull request, enter a title and description that represent your changes and submit the pull request. +verified that you have got the right changes made then, click on create pull request, enter a title and description that +represent your changes and submit the pull request. + +.. note:: + + The VOLTTRON repository includes a stub for completing your pull request. Please follow the stub to facilitate the + reviewing and merging processes. + What happens next? ------------------- +================== + Once you create a pull request, one or more VOLTTRON team members will review your changes and either accept them as is -or ask for modifications in order to have your commits accepted. You will be automatically emailed through the GitHub +ask for modifications in order to have your commits accepted. You will be automatically emailed through the GitHub notification system when this occurs (assuming you haven't changed your GitHub preferences). + Next Steps -~~~~~~~~~~ +---------- + Merging changes from the main VOLTTRON repository -------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As time goes on the VOLTTRON code base will continually be modified so the next time you want to work on a change to your files the odds are your local and remote repository will be out of date. In order to get your remote VOLTTRON repository up to date with the main VOLTTRON repository you could simply do a pull request to your remote repository -from the main repository. That would involve pointing your browser at -"https://github.com/YOUR_USERNAME/volttron/compare/develop...VOLTTRON:develop". +from the main repository. To do so, navigate your browser to +https://github.com/YOUR_USERNAME/volttron/compare/develop...VOLTTRON:develop. -Click the 'Create Pull Request' button. On the following page click the -'Create Pull Request' button. On the next page click 'Merge Pull Request' button. +Click the 'Create Pull Request' button. On the following page click the 'Create Pull Request' button. On the next page +click 'Merge Pull Request' button. Once your remote is updated you can now pull from your remote repository into your local repository through the following command: @@ -126,20 +138,20 @@ following command: The other way to get the changes into your remote repository is to first update your local repository with the changes from the main VOLTTRON repository and then pushing those changes up to your remote repository. To do that you need to first create a second remote entry to go along with the origin. A remote is simply a pointer to the url of a -different repository than the current one. Type the following command to create a new remote called 'upstream' +different repository than the current one. Type the following command to create a new remote called 'upstream': .. code-block:: bash git remote add upstream https://github.com/VOLTTRON/volttron To update your local repository from the main VOLTTRON repository then execute the following command where upstream is -the remote and develop is the branch to pull from. +the remote and develop is the branch to pull from: .. code-block:: bash git pull upstream develop -Finally to get the changes into your remote repository you can execute +Finally to get the changes into your remote repository you can execute: .. code-block:: bash @@ -147,35 +159,39 @@ Finally to get the changes into your remote repository you can execute Other commands to know -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^ At this point in time you should have enough information to be able to update both your local and remote repository and create pull requests in order to get your changes into the main VOLTTRON repository. The following commands are -other commands to give you more information that the preceeding tutorial went through +other commands to give you more information that the preceding tutorial went through + Viewing what the remotes are in our local repository ----------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash git remote -v + Stashing changed files so that you can do a merge/pull from a remote --------------------------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash - git stash save 'A commment to be listed' + git stash save 'A comment to be listed' + Applying the last stashed files to the current repository ---------------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash git stash pop + Finding help about any git command ----------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash @@ -185,40 +201,45 @@ Finding help about any git command git help push git help merge + Creating a branch from the branch and checking it out ------------------------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash git checkout -b newbranchname + Checking out a branch (if not local already will look to the remote to checkout) --------------------------------------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash git checkout branchname + Removing a local branch (cannot be current branch) --------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash git branch -D branchname + Determine the current and show all local branches -------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash git branch -Hooking into other services -~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using Travis Continuous Integration Tools +----------------------------------------- The main VOLTTRON repository is hooked into an automated build tool called travis-ci. Your remote repository can be -automatically built with the same tool by hooking your account into travis-ci's environment. -To do this go to https://travis-ci.org and create an account. You can using your GitHub login directly to this -service. Then you will need to enable the syncing of your repository through the travis-ci service. Finally you need -to push a new change to the repository. If the build fails you will receive an email notifying you of that fact and -allowing you to modify the source code and then push new changes out. +automatically built with the same tool by hooking your account into travis-ci's environment. To do this go to +https://travis-ci.org and create an account. You can using your GitHub login directly to this service. Then you will +need to enable the syncing of your repository through the travis-ci service. Finally you need to push a new change to +the repository. If the build fails you will receive an email notifying you of that fact and allowing you to modify the +source code and then push new changes out. diff --git a/docs/source/developing-volttron/contributing/contributing-documentation.rst b/docs/source/developing-volttron/contributing/contributing-documentation.rst new file mode 100644 index 0000000000..b65ad56056 --- /dev/null +++ b/docs/source/developing-volttron/contributing/contributing-documentation.rst @@ -0,0 +1,216 @@ +.. _Contributing-Documentation: + +========================== +Contributing Documentation +========================== + +The Community is encouraged to contribute documentation back to the project as they work through use cases the +developers may not have considered or documented. By contributing documentation back, the community can +learn from each other and build up a more extensive knowledge base. + +|VOLTTRON| documentation utilizes ReadTheDocs: http://volttron.readthedocs.io/en/develop/ and is built +using the `Sphinx `_ Python library with static content in +`Restructured Text `_. + + +Building the Documentation +========================== + +Static documentation can be found in the `docs/source` directory. Edit or create new .rst files to add new content +using the `Restructured Text `_ format. To see the results +of your changes the documentation can be built locally through the command line using the following instructions: + +If you've already :ref:`bootstrapped ` |VOLTTRON|, do the following while activated. If not, +this will also pull down the necessary |VOLTTRON| libraries. + +.. code-block:: bash + + python bootstrap.py --documentation + cd docs + make html + +Then, open your browser to the created local files: + +.. code-block:: bash + + file:///home//git/volttron/docs/build/html/overview/index.html + + +When complete, changes can be contributed back using the same process as code :ref:`contributions ` +by creating a pull request. When the changes are accepted and merged, they will be reflected in the ReadTheDocs site. + +.. |VOLTTRON| unicode:: VOLTTRON U+2122 + + +.. _Documentation-Styleguide: + +Documentation Styleguide +======================== + + +Naming Conventions +------------------ + +* File names and directories should be all lower-case and use only dashes/minus signs (-) as word separators + +:: + + index.rst + ├── first-document.rst + ├── more-documents + │ ├──second-document.rst + +* Reference Labels should be Capitalized and dash/minus separated: + +:: + + .. _Reference-Label: + +* Headings and Sub-headings should be written like book titles: + +:: + + ============== + The Page Title + ============== + + +Headings +-------- + +Each page should have a main title: + +:: + + ================================== + This is the Main Title of the Page + ================================== + +It can be useful to include reference labels throughout the document to use to refer back to that section of +documentation. Include reference labels above titles and important headings: + +:: + + .. _Main-Title: + + ================================== + This is the main title of the page + ================================== + + +Heading Levels +^^^^^^^^^^^^^^ + +* Page titles and documentation parts should use over-line and underline hashes: + +:: + + ===== + Title + ===== + +* Chapter headings should be over-lined and underlined with asterisks + +:: + + ******* + Chapter + ******* + +* For sections, subsections, sub-subsections, etc. underline the heading with the following: + + * =, for sections + * -, for subsections + * ^, for sub-subsections + * “, for paragraphs + + +In addition to following guidelines for styling, please separate headers from previous content by two newlines. + +:: + + ===== + Title + ===== + + Content + + + Subheading + ========== + + +Example Code Blocks +-------------------- + +Use bash for commands or user actions: + +.. code-block:: bash + + ls -al + + +Use this for the results of a command: + +.. code-block:: console + + total 5277200 + drwxr-xr-x 22 volttron volttron 4096 Oct 20 09:44 . + drwxr-xr-x 23 volttron volttron 4096 Oct 19 18:39 .. + -rwxr-xr-x 1 volttron volttron 164 Sep 29 17:08 agent-setup.sh + drwxr-xr-x 3 volttron volttron 4096 Sep 29 17:13 applications + + +Use this when Python source code is displayed + +.. code-block:: python + + @RPC.export + def status_agents(self): + return self._aip.status_agents() + + +Directives +---------- + +.. DANGER:: + + Something very bad! + +.. tip:: + + This is something good to know + + +Some other directives +^^^^^^^^^^^^^^^^^^^^^ + +"attention", "caution", "danger", "error", "hint", "important", "note", "tip", "warning", "admonition" + + +Links +----- + +Linking to external sites is simple: + +:: + + Link to `Google `_ + + +References +---------- + +You can reference other sections of documentation using the `ref` directive: + +:: + + This will reference the :ref:`platform installation ` + + +Other resources +--------------- + +- http://pygments.org/docs/lexers/ +- http://documentation-style-guide-sphinx.readthedocs.io/en/latest/style-guide.html +- http://www.sphinx-doc.org/en/stable/markup/code.html diff --git a/docs/source/devguides/agent_development/Agent-Configuration-Store.rst b/docs/source/developing-volttron/developing-agents/agent-configuration-store.rst similarity index 91% rename from docs/source/devguides/agent_development/Agent-Configuration-Store.rst rename to docs/source/developing-volttron/developing-agents/agent-configuration-store.rst index c019a8f295..6c5e84404e 100644 --- a/docs/source/devguides/agent_development/Agent-Configuration-Store.rst +++ b/docs/source/developing-volttron/developing-agents/agent-configuration-store.rst @@ -18,6 +18,7 @@ Updates from the platform will usually trigger callbacks on the agent. Agent access to the Configuration Store is managed through the `self.vip.config` object in the Agent class. + The "config" Configuration ************************** @@ -29,9 +30,7 @@ change to another configuration triggers any callbacks for `config`. Configuration Callbacks *********************** -Agents may setup callbacks for different configuration events. - -The callback method must have the following signature: +Agents may setup callbacks for different configuration events. The callback method must have the following signature: .. code-block:: python @@ -42,18 +41,24 @@ The callback method must have the following signature: The example above is for a class member method, however the method does not need to be a member of the agent class. - **config_name** - The method to call when a configuration event occurs. -- **action** - The specific configuration event type that triggered the callback. Possible values are "NEW", "UPDATE", "DELETE". See `Configuration Events`_ -- **contents** - The actual contents of the configuration. Will be a string, list, or dictionary for the actions "NEW" and "UPDATE". None if the action is "DELETE". +- **action** - The specific configuration event type that triggered the callback. Possible values are "NEW", "UPDATE", + "DELETE". See :ref:`Configuration Events ` +- **contents** - The actual contents of the configuration. Will be a string, list, or dictionary for the actions "NEW" + and "UPDATE". None if the action is "DELETE". .. note:: - All callbacks which are connected to the "NEW" event for a configuration will called during agent startup with the initial state of the configuration. + All callbacks which are connected to the "NEW" event for a configuration will called during agent startup with the + initial state of the configuration. +.. _Configuration-Store-Events: + Configuration Events -------------------- -- **NEW** - This event happens for every existing configuration at Agent startup and whenever a new configuration is added to the Configuration Store. +- **NEW** - This event happens for every existing configuration at Agent startup and whenever a new configuration is + added to the Configuration Store. - **UPDATE** - This event happens every time a configuration is changed. - **DELETE** - The event happens every time a configuration is removed from the store. @@ -73,17 +78,21 @@ A callback is setup with the `self.vip.config.subscribe` method. subscribe(callback, actions=["NEW", "UPDATE", "DELETE"], pattern="*") - **callback** - The method to call when a configuration event occurs. -- **actions** - The specific configuration event that will trigger the callback. May be a string with the name of a single action or a list of actions. +- **actions** - The specific configuration event that will trigger the callback. May be a string with the name of a + single action or a list of actions. - **pattern** - The pattern used to match configuration names to trigger the callback. + Configuration Name Pattern Matching ----------------------------------- -Configuration name matching uses Unix file name matching semantics. Specifically the python module :py:mod:`fnmatch` is used. +Configuration name matching uses Unix file name matching semantics. Specifically the python module :py:mod:`fnmatch` is +used. Name matching is not case sensitive regardless of the platform VOLTTRON is running on. -For example, the pattern `devices/*` will trigger the supplied callback for any configuration name that starts with `devices/`. +For example, the pattern `devices/*` will trigger the supplied callback for any configuration name that starts with +`devices/`. The default pattern matches all configurations. @@ -91,9 +100,8 @@ The default pattern matches all configurations. Getting a Configuration *********************** -Once RPC methods are available to an agent (once onstart methods have been called or -from any configuration callback) the contents of any configuration may be acquired -with the `self.vip.config.get` method. +Once RPC methods are available to an agent (once onstart methods have been called or from any configuration callback) +the contents of any configuration may be acquired with the `self.vip.config.get` method. .. code-block:: python @@ -101,12 +109,14 @@ with the `self.vip.config.get` method. If the Configuration Subsystem has not been initialized with the starting values of the agent configuration that will happen in order to satisfy the request. + If initialization occurs to satisfy the request callbacks will *not* be called before returning the results. Typically an Agent will only obtain the contents of a configuration via a callback. This method is included for agents that want to save state in the store and only need to retrieve the contents of a configuration at startup and ignore any changes to the configuration going forward. + Setting a Configuration *********************** @@ -163,8 +173,8 @@ via the tradition method of a bundled configuration file the `self.vip.config.se .. warning:: - This method may **not** be called once the Agent Configuration Store Subsystem has been initialized. This method should - only be called from `__init__` or an `onsetup` method. + This method may **not** be called once the Agent Configuration Store Subsystem has been initialized. This method + should only be called from `__init__` or an `onsetup` method. The `set_default` method adds a temporary configuration to the Agents Configuration Subsystem. Nothing is sent to the platform. If a configuration with the same name exists in the platform store it will be presented to @@ -183,21 +193,25 @@ If a configuration is deleted from the store and a default configuration exists the Agent Configuration Subsystem will call the `UPDATE` callback for that configuration with the contents of the default configuration. + Other Methods ************* In a well thought out configuration scheme these methods should not be needed but are included for completeness. + List Configurations ------------------- A current list of all configurations for the Agent may be called with the `self.vip.config.list` method. + Unsubscribe ----------- All subscriptions can be removed with a call to the `self.vip.config.unsubscribe_all` method. + Delete ------ @@ -211,6 +225,7 @@ A configuration can be deleted with a call to the `self.vip.config.delete` metho This method may **not** be called from a callback for the same reason as the `self.vip.config.set` method. + Delete Default -------------- @@ -225,6 +240,7 @@ A default configuration can be deleted with a call to the `self.vip.config.delet This method may **not** be called once the Agent Configuration Store Subsystem has been initialized. This method should only be called from `__init__` or an `onsetup` method. + Example Agent ************* @@ -280,4 +296,3 @@ The following example shows how to use set_default with a basic configuration an def configure_delete(self, config_name, action, contents): _log.debug("Removing {}".format(config_name)) #Do something in response to the removed configuration. - diff --git a/docs/source/devguides/agent_development/Agent-Development-Cheatsheet.rst b/docs/source/developing-volttron/developing-agents/agent-development-cheatsheet.rst similarity index 61% rename from docs/source/devguides/agent_development/Agent-Development-Cheatsheet.rst rename to docs/source/developing-volttron/developing-agents/agent-development-cheatsheet.rst index 70411168cf..f3241277e9 100644 --- a/docs/source/devguides/agent_development/Agent-Development-Cheatsheet.rst +++ b/docs/source/developing-volttron/developing-agents/agent-development-cheatsheet.rst @@ -1,5 +1,6 @@ .. _Agent-Development-Cheatsheet: +============================= Agent Development Cheat Sheet ============================= @@ -8,12 +9,15 @@ that are frequently useful in agent development. Utilities ---------- -These functions can be found in the *volttron.platform.agent.utils* module. -*logging* also needs to be imported to use the logger. +========= + +These functions can be found in the `volttron.platform.agent.utils` module. `logging` also needs to be imported to use +the logger. + setup_logging -~~~~~~~~~~~~~ +------------- + You'll probably see the following lines near the top of agent files: .. code-block:: python @@ -21,35 +25,37 @@ You'll probably see the following lines near the top of agent files: utils.setup_logging() _log = logging.getLogger(__name__) -This code sets up the logger for this module so it can provide more useful -output. In most cases it will be better to use the logger in lieu of simply -printing messages with print. +This code sets up the logger for this module so it can provide more useful output. In most cases it will be better to +use the logger in lieu of simply printing messages with print. + load_config -~~~~~~~~~~~ -load_config does just that. Give it the path to your config file and it will -parse the json and return a dictionary. +----------- + +Given the path to your config file `load_config` will parse the json and return a dictionary. + vip_main -~~~~~~~~ -This is the function that is called to start your agent. You'll likely -see it in the main methods at the bottom of agents' files. Whatever is -passed to it (a class name or a function that returns an instance of -your agent) should accept a file path that can be parsed with load_config. +-------- + +This is the function that is called to start your agent. You'll likely see it in the main methods at the bottom of +agents' files. Whatever is passed to it (a class name or a function that returns an instance of your agent) should +accept a file path that can be parsed with `load_config`. Core Agent Functionality ------------------------- -These tools volttron.platform.vip.agent module. -Try importing +======================== + +These tools are included in the `volttron.platform.vip.agent` module; Import the module to use the included agent +functions. Agent Lifecycle Events -~~~~~~~~~~~~~~~~~~~~~~ -Each agent has four events that are triggered at different stages -of its life. These are onsetup, onstart, onstop, and onfinish. Registering -callbacks to these events are commonplace in agent development, with onstart -being the most frequently used. +---------------------- + +Each agent has four events that are triggered at different stages of its life. These are `onsetup`, `onstart`, `onstop`, +and `onfinish`. Registering callbacks to these events are commonplace in agent development, with onstart being the most +frequently used. The easiest way to register a callback is with a function decorator: @@ -59,12 +65,13 @@ The easiest way to register a callback is with a function decorator: def function(self, sender, **kwargs): function_body + Periodic and Scheduled Function Calls -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Functions and agent methods can be registered to be called periodically or scheduled -to run at a particular time using the Core.schedule decorator or by calling an agent's -core.schedule() method. The latter is especially useful if, for example, a decision -needs to be made in an agent's onstart method as to whether a call should be scheduled. +===================================== + +Functions and agent methods can be registered to be called periodically or scheduled to run at a particular time using +the `Core.schedule` decorator or by calling an agent's `core.schedule()` method. The latter is especially useful if, for +example, a decision needs to be made in an agent's onstart method as to whether a call should be scheduled. .. code-block:: python @@ -93,11 +100,14 @@ or Subsystem ----------- +========= + These features are available to all Agent subclasses. No extra imports are required. + Remote Procedure Calls -~~~~~~~~~~~~~~~~~~~~~~ +---------------------- + Remote Procedure Calls, or RPCs are a powerful way to interact with other agents. To make a function available to call by a remote agent just add the export decorator: @@ -115,11 +125,12 @@ To make a function available to call by a remote agent just add the export decor # where function() is defined agent.vip.rpc.call(vip, 'function').get(timeout=t) + Pubsub -~~~~~~ -Agents can publish and subscribe to topics. Like RPC, pubsub functions can be invoked -via decorators or inline through vip. The following function is called whenever -the agent sees a message starting with *topic_prefix*. +------ + +Agents can publish and subscribe to topics. Like RPC, pubsub functions can be invoked via decorators or inline through +vip. The following function is called whenever the agent sees a message starting with *topic_prefix*. .. code-block:: python @@ -127,40 +138,41 @@ the agent sees a message starting with *topic_prefix*. def function(self, peer, sender, bus, topic, headers, message): function_body -An agent can publish to a topic *topic* with the *self.vip.pubsub.publish* method. +An agent can publish to a topic *topic* with the `self.vip.pubsub.publish` method. + +An agent can remove a subscriptions with `self.vip.pubsub.unsubscribe`. Giving None as values for the prefix and +callback argument will unsubscribe from everything on that bus. This is handy for subscriptions that must be updated +base on a configuration setting. -An agent can remove a subscriptions with *self.vip.pubsub.unsubscribe*. Giving None as values -for the prefix and callback argument will unsubscribe from everything on that bus. This -is handy for subscriptions that must be updated base on a configuration setting. Configuration Store -~~~~~~~~~~~~~~~~~~~ +------------------- -Support for the configuration store is done by subscribing to configuration changes -with *self.vip.config.subscribe*. +Support for the configuration store is done by subscribing to configuration changes with `self.vip.config.subscribe`. .. code-block:: python self.vip.config.subscribe(self.configure_main, actions=["NEW", "UPDATE"], pattern="config") -See :doc:`Agent Configuration Store ` +See :ref:`Agent Configuration Store ` + Heartbeat -~~~~~~~~~ -The heartbeat subsystem provides access to a periodic publish so that others -can observe the agent's status. Other agents can subscibe to the -*heartbeat* topic to see who is actively publishing to it. +--------- + +The heartbeat subsystem provides access to a periodic publish so that others can observe the agent's status. Other +agents can subscribe to the *heartbeat* topic to see who is actively publishing to it. It it turned off by default. -It it turned off by default. Health -~~~~~~ -The health subsystem adds extra status information to the an agent's heartbeat. -Setting the status will start the heartbeat if it wasn't already. +------ + +The health subsystem adds extra status information to the an agent's heartbeat. Setting the status will start the +heartbeat if it wasn't already. -Agent Skeleton --------------- +Agent Skeleton Code +=================== .. code-block:: python diff --git a/docs/source/devguides/agent_development/Agent-Development.rst b/docs/source/developing-volttron/developing-agents/agent-development-walk-through.rst similarity index 61% rename from docs/source/devguides/agent_development/Agent-Development.rst rename to docs/source/developing-volttron/developing-agents/agent-development-walk-through.rst index e3bc8cf52d..e66633d1e7 100644 --- a/docs/source/devguides/agent_development/Agent-Development.rst +++ b/docs/source/developing-volttron/developing-agents/agent-development-walk-through.rst @@ -1,30 +1,34 @@ -.. _Agent-Development: +.. _Agent-Development-Walk-through: -Agent Creation Walkthrough --------------------------- +============================== +Agent Development Walk-through +============================== -The VOLTTRON platform now has utilities to speed the creation and installation -of new agents. To use these utilities the VOLTTRON environment must be activated. +The VOLTTRON platform now has utilities to speed the creation and installation of new agents. To use these utilities the +VOLTTRON environment must be activated. From the project directory, activate the VOLTTRON environment with: -``. env/bin/activate`` +.. code-block:: bash + + source env/bin/activate + Create Agent Code -~~~~~~~~~~~~~~~~~ +================= Run the following command to start the Agent Creation Wizard: -``vpkg init TestAgent tester`` +.. code-block:: bash -`TestAgent` is the directory that the agent code will be placed in. The directory must -not exist when the command is run. + vpkg init TestAgent tester -`tester` is the name of the agent module created by wizard. +`TestAgent` is the directory that the agent code will be placed in. The directory must not exist when the command is +run. `tester` is the name of the agent module created by wizard. The Wizard will prompt for the following information: -:: +.. code-block:: console Agent version number: [0.1]: 0.5 Agent author: []: VOLTTRON Team @@ -34,7 +38,7 @@ The Wizard will prompt for the following information: Once the last question is answered the following will print to the console: -:: +.. code-block:: console 2018-08-02 12:20:56,604 () volttron.platform.packaging INFO: Creating TestAgent 2018-08-02 12:20:56,604 () volttron.platform.packaging INFO: Creating TestAgent/tester @@ -45,8 +49,9 @@ Once the last question is answered the following will print to the console: The TestAgent directory is created with the new Agent inside. + Agent Directory -~~~~~~~~~~~~~~~ +=============== At this point, the contents of the TestAgent directory should look like: @@ -61,22 +66,19 @@ At this point, the contents of the TestAgent directory should look like: Examine the Agent Code -~~~~~~~~~~~~~~~~~~~~~~ +---------------------- -The resulting code is well documented with comments and documentation strings. It -gives examples of how to do common tasks in VOLTTRON Agents. +The resulting code is well documented with comments and documentation strings. It gives examples of how to do common +tasks in VOLTTRON Agents (refer to the :ref:`Agent Development Cheatsheet ` for more +information). The main agent code is found in `tester/agent.py`. -The main agent code is found in `tester/agent.py` - -Here we will cover the highlights. Parse Packaged Configuration and Create Agent Instance ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The code to parse a configuration file packaged and installed with the agent -is found in the `tester` function: +The code to parse a configuration file packaged and installed with the agent is found in the `tester` function: -:: +.. code-block:: python def tester(config_path, **kwargs): """Parses the Agent configuration and returns an instance of @@ -103,19 +105,17 @@ is found in the `tester` function: setting2, **kwargs) -The configuration is parsed with the `utils.load_config` function and the results -are stored in the `config` variable. - +The configuration is parsed with the `utils.load_config` function and the results are stored in the `config` variable. An instance of the Agent is created from the parsed values and is returned. + Initialization and Configuration Store Support ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The configuration store is a powerful feature introduced in VOLTTRON 4. -The agent template provides a simple example of setting up default configuration -store values and setting up a configuration handler. +The :ref:`configuration store ` is a powerful feature. The agent template provides +a simple example of setting up default configuration store values and setting up a configuration handler. -:: +.. code-block:: python class Tester(Agent): """ @@ -164,22 +164,20 @@ store values and setting up a configuration handler. self._create_subscriptions(self.setting2) -Values in the default config can be built into the agent or come from the -packaged configuration file. The subscribe method tells our agent which function -to call whenever there is a new or updated config file. For more information -on using the configuration store see :doc:`Agent Configuration Store `. +Values in the default config can be built into the agent or come from the packaged configuration file. The subscribe +method tells our agent which function to call whenever there is a new or updated config file. For more information +on using the configuration store see :ref:`Agent Configuration Store `. + +`_create_subscriptions` (covered in the next section) will use the value in `self.setting2` to create a new subscription. -`_create_subscriptions` (covered in the next section) will use the value in `self.setting2` -to create a new subscription. Setting up a Subscription ^^^^^^^^^^^^^^^^^^^^^^^^^ -The Agent creates a subscription using the value of `self.setting2` in the method -`_create_subscription`. The messages for this subscription are handled with -the `_handle_publish` method: +The Agent creates a subscription to a topic on the message bus using the value of `self.setting2` in the method +`_create_subscription`. The messages for this subscription are handled with the `_handle_publish` method: -:: +.. code-block:: python def _create_subscriptions(self, topic): #Unsubscribe from everything. @@ -194,12 +192,13 @@ the `_handle_publish` method: #By default no action is taken. pass + Agent Lifecycle Events ^^^^^^^^^^^^^^^^^^^^^^ Methods may be setup to be called at agent startup and shutdown: -:: +.. code-block:: python @Core.receiver("onstart") def onstart(self, sender, **kwargs): @@ -225,9 +224,6 @@ Methods may be setup to be called at agent startup and shutdown: """ pass -As the comment mentions, with the new configuration store feature `onstart` methods -are mostly unneeded. However this code does include an example of how to do a Remote -Procedure Call to another agent. Agent Remote Procedure Calls ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -235,7 +231,7 @@ Agent Remote Procedure Calls An agent may receive commands from other agents via a Remote Procedure Call (RPC). This is done with the `@RPC.export` decorator: -:: +.. code-block:: python @RPC.export def rpc_method(self, arg1, arg2, kwarg1=None, kwarg2=None): @@ -247,14 +243,13 @@ This is done with the `@RPC.export` decorator: Packaging Configuration -~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- -The wizard will automatically create a `setup.py` file. This file sets up the -name, version, required packages, method to execute, etc. for the agent based on -your answers to the wizard. The packaging process will also use this +The wizard will automatically create a `setup.py` file. This file sets up the name, version, required packages, method +to execute, etc. for the agent based on your answers to the wizard. The packaging process will also use this information to name the resulting file. -:: +.. code-block:: python from setuptools import setup, find_packages @@ -286,12 +281,12 @@ information to name the resulting file. } ) + Launch Configuration -~~~~~~~~~~~~~~~~~~~~ +-------------------- -In TestAgent, the wizard will automatically create a JSON file called "config". -It contains configuration information for the agent. This file contains -examples of every datatype supported by the configuration system: +In TestAgent, the wizard will automatically create a JSON file called "config". It contains configuration information +for the agent. This file contains examples of every data type supported by the configuration system: :: @@ -307,91 +302,119 @@ examples of every datatype supported by the configuration system: } - +.. _Agent-Packaging-and-Install: Packaging and Installing the Agent -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------------- To install the agent the platform must be running. Start the platform with the command: -``./start-volttron`` +.. code-block:: bash + + ./start-volttron + +.. note:: -.. note:: If you are not in an activated environment, this script will start - the platform running in the background in the correct environment. However - the environment will not be activated for you; you must activate it yourself. + If you are not in an activated environment, this script will start the platform running in the background in the + correct environment. However the environment will not be activated for you; you must activate it yourself. Now we must install it into the platform. Use the following command to install it and add a tag for easily referring to the agent. From the project directory, run the following command: -``python scripts/install-agent.py -s TestAgent/ -c TestAgent/config -t testagent`` +.. code-block:: bash + + python scripts/install-agent.py -s TestAgent/ -c TestAgent/config -t testagent To verify it has been installed, use the following command: -``vctl list`` + +.. code-block:: bash + + vctl list This will result in output similar to the following: .. code-block:: bash - AGENT IDENTITY TAG PRI + AGENT IDENTITY TAG Status Health PRI df testeragent-0.5 testeragent-0.5_1 testagent -The initial number or letter is a unique portion of the full UUID for the agent. AGENT is -the "name" of the agent based on the contents of its class name and the version in its setup.py. IDENTITY is the -agent's identity in the platform. This is automatically assigned based on class name and instance number. This agent's -ID is _1 because it is the first instance. TAG is the name we assigned in the command above. PRI is the priority for -agents which have been "enabled" using the ``vctl enable`` command. +* The first string is a unique portion of the full UUID for the agent +* AGENT is the "name" of the agent based on the contents of its class name and the version in its setup.py. +* IDENTITY is the agent's identity in the platform. This is automatically assigned based on class name and instance + number. This agent's ID is _1 because it is the first instance. +* TAG is the name we assigned in the command above +* Status indicates the running status of an agent - running agents are *running*, agents which are not running will have + no listed status +* Health is an indication of the internal state of the agent. 'Healthy' agents will have GOOD health. If an agent + enters an error state, it will continue to run, but its health will be BAD. +* PRI is the priority for agents which have been "enabled" using the ``vctl enable`` command. When using lifecycle commands on agents, they can be referred to by the UUID (default) or AGENT (name) or TAG. Testing the Agent -~~~~~~~~~~~~~~~~~ +================= -From the Command Line -^^^^^^^^^^^^^^^^^^^^^ -To test the agent, we will start the platform (if not already running), launch the agent, and -check the log file. +From the Command Line +--------------------- +To test the agent, we will start the platform (if not already running), launch the agent, and check the log file. With the VOLTTRON environment activated, start the platform by running (if needed): -``./start-volttron`` +.. code-block:: bash -You can launch the agent in three ways, all of which you can find by using the -``vctl list`` command: + ./start-volttron -- By using the : +You can launch the agent in three ways, all of which you can find by using the `vctl list` command: -``vctl start `` +* By using the : -- By name: +.. code-block:: bash -``vctl start --name testeragent-0.1`` + vctl start -- By tag: +* By name: -``vctl start --tag testagent`` +.. code-block:: bash + + vctl start --name testeragent-0.1 + +* By tag: + +.. code-block:: bash + + vctl start --tag testagent + +Check that it is :ref:`running `: + +.. code-block:: bash + + vctl status -Check that it is :ref:`running `: +* Start the ListenerAgent as in the :ref:`platform installation guide `. +* Check the log file for messages indicating the TestAgent is receiving the ListenerAgents messages: + +.. code-block:: bash -``vctl status`` + TODO -- Start the ListenerAgent as in :ref:`Building VOLTTRON `. -- Check the log file for messages indicating the TestAgent is receiving - the ListenerAgents messages: Automated Test cases and documentation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +-------------------------------------- Before contributing a new agent to the VOLTTRON source code repository, please consider adding two other essential elements. 1. Integration and unit test cases 2. README file that includes details of pre-requisite software, agent setup details (such as setting up databases, - permissions, etc.)and sample configuration + permissions, etc.) and sample configuration -VOLTTRON uses py.test as a framework for executing tests. All unit tests should be based on py.test framework. -py.test is not installed with the distribution by default. To install py.test and it's dependencies execute the +VOLTTRON uses *pytest* as a framework for executing tests. All unit tests should be based on the *pytest* framework. +For instructions on writing unit and integration tests with *pytest*, refer to the +:ref:`Writing Agent Tests ` documentation. + +*pytest* is not installed with the distribution by default. To install py.test and it's dependencies execute the following: .. code-block:: bash @@ -414,9 +437,58 @@ To run a single test module, use the command pytest -To run all of the tests in the volttron repository execute the following in the -root directory using an activated command prompt: +To run all of the tests in the volttron repository execute the following in the root directory using an activated +command prompt: .. code-block:: bash ./ci-integration/run-tests.sh + + +.. _Utility-Scripts: + +Scripts +======= + +In order to make repetitive tasks less repetitive the VOLTTRON team has create several scripts in order to help. These +tasks are available in the `scripts` directory. + +.. note:: + + In addition to the `scripts` directory, the VOLTTRON team has added the config directory to the .gitignore file. By + convention this is where we store customized scripts and configuration that will not be made public. Please feel + free to use this convention in your own processes. + +The `scripts/core` directory is laid out in such a way that we can build scripts on top of a base core. For example the +scripts in sub-folders such as the `historian-scripts` and `demo-comms` use the scripts that are present in the core +directory. + +The most widely used script is `scripts/install-agent.py`. The `install_agent.py` script will remove an agent if the +tag is already present, create a new agent package, and install the agent to `VOLTTRON_HOME`. This script has three +required arguments and has the following signature: + +:: + + # Agent to Package must have a setup.py in the root of the directory. + scripts/install_agent.py + +The `install_agent.py` script will respect the `VOLTTRON_HOME` specified on the command line or set in the global +environment. An example of setting `VOLTTRON_HOME` is as follows. + +:: + + # Sets VOLTTRON_HOME to /tmp/v1home + VOLTTRON_HOME=/tmp/v1home scripts/core/pack_install.sh + + +.. toctree:: + :hidden: + :maxdepth: 1 + + agent-development-cheatsheet + agent-configuration-store + writing-agent-tests + developing-historian-agents + developing-market-agents + example-agents/index + specifications/index diff --git a/docs/source/developing-volttron/developing-agents/developing-historian-agents.rst b/docs/source/developing-volttron/developing-agents/developing-historian-agents.rst new file mode 100644 index 0000000000..4e134a6a9f --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/developing-historian-agents.rst @@ -0,0 +1,108 @@ +.. _Developing-Historian-Agents: + +=========================== +Developing Historian Agents +=========================== + +VOLTTRON provides a convenient base class for developing new historian agents. The base class automatically performs +a number of important functions: + +* subscribes to all pertinent topics +* caches published data to disk until it is successfully recorded to a historian +* creates the public facing interface for querying results +* spells out a simple interface for concrete implementation to meet to make a working Historian Agent +* breaks data to publish into reasonably sized chunks before handing it off to the concrete implementation for + publication. The size of the chunk is configurable +* sets up a separate thread for publication. If publication code needs to block for a long period of time (up to 10s of + seconds) this will no disrupt the collection of data from the bus or the functioning of the agent itself + +The VOLTTRON repository provides several :ref:`historians ` which can be deployed without +modification. + + +BaseHistorian +------------- + +All Historians must inherit from the BaseHistorian class in volttron.platform.agent.base\_historian and implement the +following methods: + + +publish_to_historian(self, to_publish_list) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This method is called by the BaseHistorian class when it has received data from the message bus to be published. +`to_publish_list` is a list of records to publish in the form: + +:: + + [ + { + '_id': 1, + 'timestamp': timestamp, + 'source': 'scrape', + 'topic': 'campus/building/unit/point', + 'value': 90, + 'meta': {'units':'F'} + } + { + ... + } + ] + +- **_id** - ID of the record used for internal record tracking. All IDs in the list are unique +- **timestamp** - Python datetime object of the time data was published at timezone UTC +- **source** - Source of the data: can be scrape, analysis, log, or actuator +- **topic** - Topic data was published on, topic prefix's such as "device" are dropped +- **value** - Value of the data, can be any type. +- **meta** - Metadata for the value, some sources will omit this entirely. + +For each item in the list the concrete implementation should attempt to publish (or discard if non-publishable) every +item in the list. Publication should be batched if possible. For every successfully published record and every record +that is to be discarded because it is non-publishable the agent must call `report_handled` on those records. Records +that should be published but were not for whatever reason require no action. Future calls to `publish_to`_historian` +will include these unpublished records. `publish_to_historian` is always called with the oldest unhandled records. This +allows the historian to no lose data due to lost connections or other problems. + +As a convenience `report_all_handled` can be called if all of the items in `published_list` were successfully handled. + + +query_topic_list(self) +~~~~~~~~~~~~~~~~~~~~~~ + +Must return a list of all unique topics published. + + +query_historian(self, topic, start=None, end=None, skip=0, count=None, order=None) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +This function must return the results of a query in the form: + +:: + + {"values": [(timestamp1: value1), (timestamp2: value2), ...], + "metadata": {"key1": value1, "key2": value2, ...}} + +metadata is not required (The caller will normalize this to {} for you if you leave it out) + +- **topic** - the topic the user is querying for +- **start** - datetime of the start of the query, `None` for the beginning of time +- **end** - datetime of the end of of the query, `None` for the end of time +- **skip** - skip this number of results (for pagination) +- **count** - return at maximum this number of results (for pagination) +- **order** - `FIRST_TO_LAST` for ascending time stamps, `LAST_TO_FIRST` for descending time stamps + + +historian_setup(self) +~~~~~~~~~~~~~~~~~~~~~~ + +Implementing this is optional. This function is run on the same thread as the rest of the concrete implementation at +startup. It is meant for connection setup. + + +Example Historian +----------------- + +An example historian can be found in the `examples/CSVHistorian` directory in the VOLTTRON repository. This example +historian uses a CSV file as the persistent data store. It is recommended to use this agent as a reference for +developing new historian agents. diff --git a/docs/source/devguides/agent_development/Developing-Market-Agents.rst b/docs/source/developing-volttron/developing-agents/developing-market-agents.rst similarity index 56% rename from docs/source/devguides/agent_development/Developing-Market-Agents.rst rename to docs/source/developing-volttron/developing-agents/developing-market-agents.rst index 4ff572a7fc..5bafb13d74 100644 --- a/docs/source/devguides/agent_development/Developing-Market-Agents.rst +++ b/docs/source/developing-volttron/developing-agents/developing-market-agents.rst @@ -4,24 +4,20 @@ Developing Market Agents ======================== -VOLTTRON provides a convenient base class for developing new market -agents. The base class automatically subscribes to all pertinent topics, -and spells out a simple interface for concrete implementation to -make a working Market Agent. +VOLTTRON provides a convenient base class for developing new market agents. The base class automatically subscribes to all pertinent topics, +and spells out a simple interface for concrete implementation to make a working Market Agent. -Markets are implemented by the Market Service Agent which is a core service agent. -The Market Service Agent publishes information on several topics to which the base -agent automatically subscribes. The base agent also provides all the methods you will -need to interact with the Market Service Agent to implement your market transactions. +Markets are implemented by the Market Service Agent which is a core service agent. The Market Service Agent publishes +information on several topics to which the base agent automatically subscribes. The base agent also provides all the +methods you will need to interact with the Market Service Agent to implement your market transactions. MarketAgent =========== -All Market Agents must inherit from the MarketAgent class in -volttron.platform.agent.base_market_agent and call the following -method +All Market Agents must inherit from the MarketAgent class in `volttron.platform.agent.base_market_agent` and call the +following method: -:: +.. code-block:: python self.join_market(market_name, buyer_seller, reservation_callback, offer_callback, aggregate_callback, price_callback, error_callback) @@ -29,14 +25,15 @@ This method causes the market agent to join a single market. If the agent wishe markets it may be called once for each market. The first argument is the name of the market to join and this name must be unique across the entire volttron instance because all markets are implemented by a single market service agent for each volttron instance. The second argument describes the role that this agent wished to play in this market. -The value is imported as +The value is imported as: -:: +.. code-block:: python from volttron.platform.agent.base_market_agent.buy_sell import BUYER, SELLER Arguments 3-7 are callback methods that the agent may implement as needed for the agent's participation in the market. + The Reservation Callback ------------------------ @@ -44,18 +41,19 @@ The Reservation Callback reservation_callback(self, timestamp, market_name, buyer_seller) -This method is called when it is time to reserve a slot in the market for the current market cycle. -If this callback is not registered a slot is reserved for every market cycle. If this callback is registered -it is called for each market cycle and returns True if a reservation is wanted and False if a reservation -is not wanted. The name of the market and the roll being played are provided so that a single callback can handle -several markets. If the agent joins three markets with the same reservation callback routine it will be called three -times with the appropriate market name and buyer/seller role for each call. The MeterAgent example -illustrates the use of this of this method and how to determine whether to make an offer when the reservation is -refused. -A market will only exist if there are reservations for at least one buyer or one seller. -If the market fails to achieve the minimum participation the error callback will be called. -If only buyers or only sellers make reservations any offers will be rejected -with the reason that the market has not formed. +This method is called when it is time to reserve a slot in the market for the current market cycle. If this callback is +not registered a slot is reserved for every market cycle. If this callback is registered it is called for each market +cycle and returns `True` if a reservation is wanted and `False` if a reservation is not wanted. + +The name of the market and the roll being played are provided so that a single callback can handle several markets. +If the agent joins three markets with the same reservation callback routine it will be called three times with the +appropriate market name and buyer/seller role for each call. The MeterAgent example illustrates the use of this of this +method and how to determine whether to make an offer when the reservation is refused. + +A market will only exist if there are reservations for at least one buyer or one seller. If the market fails to achieve +the minimum participation the error callback will be called. If only buyers or only sellers make reservations any +offers will be rejected with the reason that the market has not formed. + The Offer Callback ------------------ @@ -67,12 +65,13 @@ The Offer Callback If the agent has made a reservation for the market and a callback has been registered this callback is called. If the agent wishes to make an offer at this time the market agent computes either a supply or a demand curve as appropriate and offers the curve to the market service by calling the -:py:meth:`make_offer ` -method. -The name of the market and the roll being played are provided so that a single callback can handle -several markets. +:py:meth:`make_offer ` method. + +The name of the market and the roll being played are provided so that a single callback can handle several markets. + For each market joined either an offer callback, an aggregate callback, or a cleared price callback is required. + The Aggregate Callback ---------------------- @@ -80,19 +79,21 @@ The Aggregate Callback aggregate_callback(self, timestamp, market_name, buyer_seller, aggregate_curve) -When a market has received all its buy offers it calculates an aggregate demand curve. -When the market receives all of its sell offers it calculates an aggregate supply curve. -This callback delivers the aggregate curve to the market agent whenever the appropriate curve becomes available. -If the market agent wants to use this opportunity to make an offer on this or another market -it would do that using the -:py:meth:`make_offer ` -method. -If the aggregate demand curve is received, obviously you could only make a supply offer on this market. -If the aggregate supply curve is received, obviously you could only make a demand offer on this market. -You can of course use this information to make an offer on another market. The example AHUAgent does this. -The name of the market and the roll being played are provided so that a single callback can handle -several markets. -For each market joined either an offer callback, an aggregate callback, or a cleared price callback is required. +When a market has received all its buy offers it calculates an aggregate demand curve. When the market receives all of +its sell offers it calculates an aggregate supply curve. This callback delivers the aggregate curve to the market agent +whenever the appropriate curve becomes available. + +If the market agent wants to use this opportunity to make an offer on this or another market it would do that using the +:py:meth:`make_offer ` method. + +* If the aggregate demand curve is received, only a supply offer may be submitted for this market +* If the aggregate supply curve is received, only make a demand offer will be accepted by this market. + +You may use this information to make an offer on another market; The example AHUAgent does this. The name of the +market and the roll being played are provided so that a single callback can handle several markets. + +For each market joined, either an offer callback, an aggregate callback, or a cleared price callback is required. + The Price Callback ------------------ @@ -101,17 +102,17 @@ The Price Callback price_callback(self, timestamp, market_name, buyer_seller, price, quantity) -This callback is called when the market clears. -If the market agent wants to use this opportunity to make an offer on this or another market -it would do that using the -:py:meth:`make_offer ` -method. -Once the market has cleared you can't make an offer on that market. -You can of course use this information to make an offer on another market. The example AHUAgent does this. -The name of the market and the roll being played are provided so that a single callback can handle -several markets. +This callback is called when the market clears. If the market agent wants to use this opportunity to make an offer on +this or another market it would do that using the +:py:meth:`make_offer ` method. + +Once the market has cleared you can not make an offer on that market. Again, you may use this information to make an +offer on another market as in the example AHUAgent. The name of the market and the roll being played are provided so +that a single callback can handle several markets. + For each market joined either an offer callback, an aggregate callback, or a cleared price callback is required. + The Error Callback ------------------ @@ -119,10 +120,9 @@ The Error Callback error_callback(self, timestamp, market_name, buyer_seller, error_code, error_message, aux) -This callback is called when an error occurs isn't in response to an RPC call. -The error codes are documented in +This callback is called when an error occurs isn't in response to an RPC call. The error codes are documented in: -:: +.. code-block:: python from volttron.platform.agent.base_market_agent.error_codes import NOT_FORMED, SHORT_OFFERS, BAD_STATE, NO_INTERSECT @@ -132,5 +132,3 @@ The error codes are documented in * NO_INTERSECT - If the market fails to clear this would be called while clearing the market and an auxillary array will be included. The auxillary array contains comparisons between the supply max, supply min, demand max and demand min. They allow the market client to make determinations about why the curves did not intersect that may be useful. The error callback is optional, but highly recommended. - - diff --git a/docs/source/developing-volttron/developing-agents/documenting-agents.rst b/docs/source/developing-volttron/developing-agents/documenting-agents.rst new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/source/developing-volttron/developing-agents/example-agents/c-agent.rst b/docs/source/developing-volttron/developing-agents/example-agents/c-agent.rst new file mode 100644 index 0000000000..c2108be229 --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/c-agent.rst @@ -0,0 +1,48 @@ +.. _C-Agent: + +======= +C Agent +======= + +The C Agent uses the `ctypes` module to load a shared object into memory so its functions can be called from Python. + +There are two versions of the C Agent: + +* A standard agent that can be installed with the agent installation process +* A driver which can can be controlled using the Master Driver Agent + + +Building the Shared Object +-------------------------- + +The shared object library must be built before installing C Agent examples. Running ``make`` in the C Agent source +directory will compile the provided C code using the position independent flag, a requirement for creating shared +objects. + +Files created by make can be removed by running + +.. code-block:: bash + + make clean + + +Agent Installation +------------------ + +After building the shared object library the standard agent can be installed with the ``scripts/install-agent.py`` +script: + +.. code-block:: bash + + python scripts/install-agent.py -s examples/CAgent + +The other is a driver interface for the Master Driver. To use the C driver, the driver code file must be moved into +the Master Driver's `interfaces` directory: + + :: + + examples/CAgent/c_agent/driver/cdriver -> services/core/MasterDriverAgent/master_driver/interfaces + + +The C Driver configuration tells the interface where to find the shared object. An example is available in the C +Agent's `driver` directory. diff --git a/docs/source/developing-volttron/developing-agents/example-agents/config-actuation.rst b/docs/source/developing-volttron/developing-agents/example-agents/config-actuation.rst new file mode 100644 index 0000000000..b8e2fd8cdc --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/config-actuation.rst @@ -0,0 +1,22 @@ +.. _Config-Actuation: + +======================== +Config Actuation Example +======================== + +The Config Actuation example attempts to set points on a device when files are added or updated in its +:ref:`configuration store `. + + +Configuration +------------- + +The name of a configuration file must match the name of the device to be actuated. The configuration file is a JSON +dictionary of point name and value pairs. Any number of points on the device can be listed in the config. + +.. code-block:: python + + { + "point0": value, + "point1": value + } diff --git a/docs/source/developing-volttron/developing-agents/example-agents/csv-historian.rst b/docs/source/developing-volttron/developing-agents/example-agents/csv-historian.rst new file mode 100644 index 0000000000..7aa02b9d29 --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/csv-historian.rst @@ -0,0 +1,123 @@ +.. _CSV-Historian: + +============= +CSV Historian +============= + +The CSV Historian Agent is an example historian agent that writes device data to the CSV file specified in the +configuration file. + + +Explanation of CSV Historian +============================ + +The Utils module of the VOLTTRON platform includes functions for setting up global logging for the platform: + +.. code-block:: python + + utils.setup_logging() + _log = logging.getLogger(__name__) + + +The ``historian`` method is called by ``utils.vip_main`` when the agents is started (see below). ``utils.vip_main`` +expects a callable object that returns an instance of an Agent. This method of dealing with a configuration file and +instantiating an Agent is common practice. + +.. code-block:: python + + def historian(config_path, **kwargs): + if isinstance(config_path, dict): + config_dict = config_path + else: + config_dict = utils.load_config(config_path) + + output_path = config_dict.get("output", "~/historian_output.csv") + + return CSVHistorian(output_path = output_path, **kwargs) + +All historians must inherit from `BaseHistorian`. The `BaseHistorian` class handles the capturing and caching of all +device, logging, analysis, and record data published to the message bus. + +.. code-block:: python + + class CSVHistorian(BaseHistorian): + +The Base Historian creates a separate thread to handle publishing data to the data store. In this thread the Base +Historian calls two methods on the created historian, ``historian_setup`` and ``publish_to_historian``. + +The Base Historian created the new thread in it's ``__init__`` method. This means that any instance variables +must assigned in ``__init__`` before calling the Base Historian's ``__init__`` method. + +.. code-block:: python + + def __init__(self, output_path="", **kwargs): + self.output_path = output_path + self.csv_dict = None + super(CSVHistorian, self).__init__(**kwargs) + +Historian setup is called shortly after the new thread starts. This is where a Historian sets up a connect the first +time. In our example we create the `Dictwriter` object that we will use to create and add lines to the CSV file. + +We keep a reference to the file object so that we may flush its contents to disk after writing the header and after we +have written new data to the file. + +The CSV file we create will have 4 columns: `timestamp`, `source`, `topic`, and `value`. + +.. code-block:: python + + def historian_setup(self): + self.f = open(self.output_path, "wb") + self.csv_dict = csv.DictWriter(self.f, ["timestamp", "source", "topic", "value"]) + self.csv_dict.writeheader() + self.f.flush() + +``publish_to_historian`` is called when data is ready to be published. It is passed a list of dictionaries. Each +dictionary contains a record of a single value that was published to the message bus. + +The dictionary takes the form: + +.. code-block:: python + + { + '_id': 1, + 'timestamp': timestamp1.replace(tzinfo=pytz.UTC), #Timestamp in UTC + 'source': 'scrape', #Source of the data point. + 'topic': "pnnl/isb1/hvac1/thermostat", #Topic that published to without prefix. + 'value': 73.0, #Value that was published + 'meta': {"units": "F", "tz": "UTC", "type": "float"} #Meta data published with the topic + } + +Once the data is written to the historian we call ``self.report_all_handled()`` to inform the `BaseHistorian` that all +data we received was successfully published and can be removed from the cache. Then we can flush the file to ensure +that the data is written to disk. + +.. code-block:: python + + def publish_to_historian(self, to_publish_list): + for record in to_publish_list: + row = {} + row["timestamp"] = record["timestamp"] + + row["source"] = record["source"] + row["topic"] = record["topic"] + row["value"] = record["value"] + + self.csv_dict.writerow(row) + + self.report_all_handled() + self.f.flush() + +This agent does not support the Historian Query interface. + + +Agent Testing +------------- + +The CSV Historian can be tested by running the included `launch_my_historian.sh` script. + + +Agent Installation +------------------ + +This Agent may be installed on the platform using the standard method. + diff --git a/docs/source/developing-volttron/developing-agents/example-agents/data-publisher.rst b/docs/source/developing-volttron/developing-agents/example-agents/data-publisher.rst new file mode 100644 index 0000000000..e98fcbd0e7 --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/data-publisher.rst @@ -0,0 +1,157 @@ +.. _Data-Publisher: + +============== +Data Publisher +============== + +This is a simple agent that plays back data either from the config store or a CSV to the configured topic. It can also +provide basic emulation of the Actuator Agent for testing agents that expect to be able to set points on a device in +response to device publishes. + + +Installation notes +------------------ + +In order to simulate the actuator you must install the agent with the VIP identity of `platform.actuator`. If an +an actuator is already installed on the platform, this will cause VIP identity conflicts. To install the agent, the +agent install script can be used: + +.. code-block:: bash + + python scripts/install-agent.py -s examples/DataPublisher -c + + +Configuration +------------- + +:: + + { + # basetopic can be devices, analysis, or custom base topic + "basepath": "devices/PNNL/ISB1", + + # use_timestamp uses the included in the input_data if present. + # Currently the column must be named `Timestamp`. + "use_timestamp": true, + + # Only publish data at most once every max_data_frequency seconds. + # Extra data is skipped. + # The time windows are normalized from midnight. + # ie 900 will publish one value for every 15 minute window starting from + # midnight of when the agent was started. + # Only used if timestamp in input file is used. + "max_data_frequency": 900, + + # The meta data published with the device data is generated + # by matching point names to the unittype_map. + "unittype_map": { + ".*Temperature": "Farenheit", + ".*SetPoint": "Farenheit", + "OutdoorDamperSignal": "On/Off", + "SupplyFanStatus": "On/Off", + "CoolingCall": "On/Off", + "SupplyFanSpeed": "RPM", + "Damper*.": "On/Off", + "Heating*.": "On/Off", + "DuctStatic*.": "On/Off" + }, + # Path to input CSV file. + # May also be a list of records or reference to a CSV file in the config store. + # Large CSV files should be referenced by file name and not + # stored in the config store. + "input_data": "econ_test2.csv", + # Publish interval in seconds + "publish_interval": 1, + + # Tell the playback to maintain the location a the file in the config store. + # Playback will be resumed from this point + # at agent startup even if this setting is changed to false before restarting. + # Saves the current line in line_marker in the DataPublishers's config store + # as plain text. + # default false + "remember_playback": true, + + # Start playback from 0 even if the line_marker configuration is set a non 0 value. + # default false + "reset_playback": false, + + # Repeat data from the start if this flag is true. + # Useful for data that does not include a timestamp and is played back in real time. + "replay_data": false + } + + +CSV File Format +--------------- + +The CSV file must have a single header line. The column names are appended to the `basepath` setting in the +configuration file and the resulting topic is normalized to remove extra` ``/`` characters. The values are all treated +as floating point values and converted accordingly. + +The corresponding device for each point is determined and the values are combined together to create an `all` topic +publish for each device. + +If a `Timestamp` column is in the input it may be used to set the timestamp in the header of the published data. + +.. csv-table:: Publisher Data + :header: Timestamp,centrifugal_chiller/OutsideAirTemperature,centrifugal_chiller/DischargeAirTemperatureSetPoint,fuel_cell/DischargeAirTemperature,fuel_cell/CompressorStatus,absorption_chiller/SupplyFanSpeed,absorption_chiller/SupplyFanStatus,boiler/DuctStaticPressureSetPoint,boiler/DuctStaticPressure + + 2012/05/19 05:07:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:08:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:09:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:10:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:11:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:12:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:13:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:14:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:15:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:16:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:17:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:18:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:19:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:20:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:21:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:22:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:23:00,0,56,0,0,75,1,1.4,1.38 + 2012/05/19 05:24:00,0,56,58.77,0,75,1,1.4,1.38 + 2012/05/19 05:25:00,48.78,56,58.87,0,75,1,1.4,1.38 + 2012/05/19 05:26:00,48.88,56,58.95,0,75,1,1.4,1.38 + 2012/05/19 05:27:00,48.93,56,58.91,0,75,1,1.4,1.38 + 2012/05/19 05:28:00,48.95,56,58.81,0,75,1,1.4,1.38 + 2012/05/19 05:29:00,48.92,56,58.73,0,75,1,1.4,1.38 + 2012/05/19 05:30:00,48.88,56,58.69,0,75,1,1.4,1.38 + 2012/05/19 05:31:00,48.88,56,58.81,0,75,1,1.4,1.38 + 2012/05/19 05:32:00,48.99,56,58.91,0,75,1,1.4,1.38 + 2012/05/19 05:33:00,49.09,56,58.85,0,75,1,1.4,1.38 + 2012/05/19 05:34:00,49.11,56,58.79,0,75,1,1.4,1.38 + 2012/05/19 05:35:00,49.07,56,58.71,0,75,1,1.4,1.38 + 2012/05/19 05:36:00,49.05,56,58.77,0,75,1,1.4,1.38 + 2012/05/19 05:37:00,49.09,56,58.87,0,75,1,1.4,1.38 + 2012/05/19 05:38:00,49.13,56,58.85,0,75,1,1.4,1.38 + 2012/05/19 05:39:00,49.09,56,58.81,0,75,1,1.4,1.38 + 2012/05/19 05:40:00,49.01,56,58.75,0,75,1,1.4,1.38 + 2012/05/19 05:41:00,48.92,56,58.71,0,75,1,1.4,1.38 + 2012/05/19 05:42:00,48.86,56,58.77,0,75,1,1.4,1.38 + 2012/05/19 05:43:00,48.92,56,58.87,0,75,1,1.4,1.38 + 2012/05/19 05:44:00,48.95,56,58.79,0,75,1,1.4,1.38 + 2012/05/19 05:45:00,48.92,56,58.69,0,75,1,1.4,1.38 + 2012/05/19 05:46:00,48.86,56,58.5,0,75,1,1.4,1.38 + 2012/05/19 05:47:00,48.78,56,58.34,0,75,1,1.4,1.38 + 2012/05/19 05:48:00,48.69,56,58.36,0,75,1,1.4,1.38 + 2012/05/19 05:49:00,48.65,56,58.46,0,75,1,1.4,1.38 + 2012/05/19 05:50:00,48.65,56,58.56,0,75,1,1.4,1.38 + 2012/05/19 05:51:00,48.65,56,58.48,0,75,1,1.4,1.38 + 2012/05/19 05:52:00,48.61,56,58.36,0,75,1,1.4,1.38 + 2012/05/19 05:53:00,48.59,56,58.21,0,75,1,1.4,1.38 + 2012/05/19 05:54:00,48.55,56,58.25,0,75,1,1.4,1.38 + 2012/05/19 05:55:00,48.63,56,58.42,0,75,1,1.4,1.38 + 2012/05/19 05:56:00,48.76,56,58.56,0,75,1,1.4,1.38 + 2012/05/19 05:57:00,48.95,56,58.71,0,75,1,1.4,1.38 + 2012/05/19 05:58:00,49.24,56,58.83,0,75,1,1.4,1.38 + 2012/05/19 05:59:00,49.54,56,58.93,0,75,1,1.4,1.38 + 2012/05/19 06:00:00,49.71,56,58.95,0,75,1,1.4,1.38 + 2012/05/19 06:01:00,49.79,56,59.07,0,75,1,1.4,1.38 + 2012/05/19 06:02:00,49.94,56,59.17,0,75,1,1.4,1.38 + 2012/05/19 06:03:00,50.13,56,59.25,0,75,1,1.4,1.38 + 2012/05/19 06:04:00,50.18,56,59.15,0,75,1,1.4,1.38 + 2012/05/19 06:05:00,50.15,56,59.04,0,75,1,1.4,1.38 diff --git a/docs/source/developing-volttron/developing-agents/example-agents/dds-agent.rst b/docs/source/developing-volttron/developing-agents/example-agents/dds-agent.rst new file mode 100644 index 0000000000..c25acfa7f9 --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/dds-agent.rst @@ -0,0 +1,49 @@ +.. _DDS-Agent: + +========= +DDS Agent +========= + +The DDS example agent demonstrates VOLTTRON's capacity to be extended with tools and libraries not used in the core +codebase. DDS is a messaging platform that implements a publish-subscribe system for well defined data types. + +This agent example is meant to be run the command line, as opposed to installing it like other agents. From the +`examples/DDSAgent` directory, the command to start it is: + +.. code-block:: shell + + $ AGENT_CONFIG=config python -m ddsagent.agent + +The `rticonnextdds-connector` library needs to be installed for this example to function properly. We'll retrieve it +from GitHub since it is not available through Pip. Download the source with: + +.. code-block:: shell + + $ wget https://github.com/rticommunity/rticonnextdds-connector/archive/master.zip + +and unpack it in `examples/DDSAgent/ddsagent` with: + +.. code-block:: shell + + $ unzip master.zip + +The ``demo_publish()`` output can be viewed with the `rtishapesdemo` available from RTI. + + +Configuration +------------- + +Each data type that this agent will have access to needs to have an XML document defining its structure. The XML will +include a participant name, publisher name, and a subscriber name. These are recorded in the configuration with the +location on disk of the XML file. + +.. code-block:: json + + { + "square": { + "participant_name": "MyParticipantLibrary::Zero", + "xml_config_path": "./ddsagent/rticonnextdds-connector-master/examples/python/ShapeExample.xml", + "publisher_name": "MyPublisher::MySquareWriter", + "subscriber_name": "MySubscriber::MySquareReader" + } + } diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/cmd-image.png b/docs/source/developing-volttron/developing-agents/example-agents/files/cmd-image.png new file mode 100644 index 0000000000..515b186fd0 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/cmd-image.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/cmd-image_2.png b/docs/source/developing-volttron/developing-agents/example-agents/files/cmd-image_2.png new file mode 100644 index 0000000000..a137623de0 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/cmd-image_2.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/env-vars-image_1.png b/docs/source/developing-volttron/developing-agents/example-agents/files/env-vars-image_1.png new file mode 100644 index 0000000000..182d6d2e7e Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/env-vars-image_1.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/env-vars-image_2.png b/docs/source/developing-volttron/developing-agents/example-agents/files/env-vars-image_2.png new file mode 100644 index 0000000000..30500ad2ee Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/env-vars-image_2.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/extract-image_1.png b/docs/source/developing-volttron/developing-agents/example-agents/files/extract-image_1.png new file mode 100644 index 0000000000..729fa18884 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/extract-image_1.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/extract-image_2.png b/docs/source/developing-volttron/developing-agents/example-agents/files/extract-image_2.png new file mode 100644 index 0000000000..219ac18645 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/extract-image_2.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/github-image.png b/docs/source/developing-volttron/developing-agents/example-agents/files/github-image.png new file mode 100644 index 0000000000..5739c4bba9 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/github-image.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/github-zip-image.png b/docs/source/developing-volttron/developing-agents/example-agents/files/github-zip-image.png new file mode 100644 index 0000000000..62d6eecafa Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/github-zip-image.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/matlab-agent-diagram.png b/docs/source/developing-volttron/developing-agents/example-agents/files/matlab-agent-diagram.png new file mode 100644 index 0000000000..42bfd3df1b Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/matlab-agent-diagram.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/node-red-flow.png b/docs/source/developing-volttron/developing-agents/example-agents/files/node-red-flow.png new file mode 100644 index 0000000000..49c3b2b6b8 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/node-red-flow.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/files/node-red.png b/docs/source/developing-volttron/developing-agents/example-agents/files/node-red.png new file mode 100644 index 0000000000..d83d079401 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/example-agents/files/node-red.png differ diff --git a/docs/source/developing-volttron/developing-agents/example-agents/index.rst b/docs/source/developing-volttron/developing-agents/example-agents/index.rst new file mode 100644 index 0000000000..f071612c7f --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/index.rst @@ -0,0 +1,27 @@ +.. _Example-Agents: + +============== +Example Agents +============== + +Some example agents are included with the platform to help explore its features. These agents represent concrete +implementations of important agent sub-types such as Historians or Weather Agents, or demonstrate a development pattern +for accomplishing common tasks. + +More complex agents contributed by other researchers can also be found in the examples directory. It is recommended +that developers new to VOLTTRON understand the example agents first before diving into the other agents. + + +.. toctree:: + :maxdepth: 1 + + c-agent + config-actuation + csv-historian + data-publisher + dds-agent + listener-agent + matlab-agent + node-red + scheduler-example-agent + simple-web-agent-walk-through diff --git a/docs/source/developing-volttron/developing-agents/example-agents/listener-agent.rst b/docs/source/developing-volttron/developing-agents/example-agents/listener-agent.rst new file mode 100644 index 0000000000..3b7673179b --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/listener-agent.rst @@ -0,0 +1,79 @@ +.. _Listener-Agent: + +============== +Listener Agent +============== + +The ListenerAgent subscribes to all topics and is useful for testing that agents being developed are publishing +correctly. It also provides a template for building other agents as it expresses the requirements of a platform agent. + + +Explanation of Listener Agent Code +================================== + +Use :code:`utils` to setup logging, which we’ll use later. + +.. code-block:: python + + utils.setup_logging() + _log = logging.getLogger(__name__) + + +The Listener agent extends (inherits from) the Agent class for its default functionality such as responding to platform +commands: + +.. code-block:: python + + class ListenerAgent(Agent): + ''' + Listens to everything and publishes a heartbeat according to the + heartbeat period specified in the settings module. + ''' + +After the class definition, the Listener agent reads the configuration file, extracts the configuration parameters, and +initializes any Listener agent instance variable. This is done through the agent's ``__init__`` method: + +.. code-block:: python + + def __init__(self, config_path, **kwargs): + super(ListenerAgent, self).__init__(**kwargs) + self.config = utils.load_config(config_path) + self._agent_id = self.config.get('agentid', DEFAULT_AGENTID) + log_level = self.config.get('log-level', 'INFO') + if log_level == 'ERROR': + self._logfn = _log.error + elif log_level == 'WARN': + self._logfn = _log.warn + elif log_level == 'DEBUG': + self._logfn = _log.debug + else: + self._logfn = _log.info + +Next, the Listener agent will run its setup method. This method is tagged to run after the agent is initialized by the +decorator ``@Core.receiver('onsetup')``. This method accesses the configuration parameters, logs a message to the +platform log, and sets the agent ID. + +.. code-block:: python + + @Core.receiver('onsetup') + def onsetup(self, sender, **kwargs): + # Demonstrate accessing a value from the config file + _log.info(self.config.get('message', DEFAULT_MESSAGE)) + self._agent_id = self.config.get('agentid') + +The Listener agent subscribes to all topics published on the message bus. Publish and subscribe interactions with the +message bus are handled by the `PubSub` module located at `~/volttron/volttron/platform/vip/agent/subsystems/pubsub.py`. + +The Listener agent uses an empty string to subscribe to all messages published. This is done in a +`decorator `_ for simplifying subscriptions. + +.. code-block:: python + + @PubSub.subscribe('pubsub', '') + def on_match(self, peer, sender, bus, topic, headers, message): + '''Use match_all to receive all messages and print them out.''' + if sender == 'pubsub.compat': + message = compat.unpack_legacy_message(headers, message) + self._logfn( + "Peer: %r, Sender: %r:, Bus: %r, Topic: %r, Headers: %r, " + "Message: %r", peer, sender, bus, topic, headers, message) diff --git a/docs/source/developing-volttron/developing-agents/example-agents/matlab-agent.rst b/docs/source/developing-volttron/developing-agents/example-agents/matlab-agent.rst new file mode 100644 index 0000000000..88ed164fab --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/matlab-agent.rst @@ -0,0 +1,606 @@ +.. _Matlab-Agent: + +============ +MatLab Agent +============ + +The MatLab agent and Matlab Standalone Agent together are example agents that allow for MatLab scripts to be run in a +Windows environment and interact with the VOLTTRON platform running in a Linux environment. + +The MatLab agent takes advantage of the config store to dynamically send scripts and commandline arguments across the +message bus to one or more Standalone Agents in Windows. The Standalone Agent then executes the requested script and +arguments, and sends back the results to the MatLab agent. + + +Overview of Matlab Agents +========================= + +There are multiple components that are used for the MatLab agent. This diagram is to represent the components that are +connected to the MatLab Agents. In this example, the scripts involved are based on the default settings in the MatLab +Agent. + +|matlab-agent-diagram| + + +MatLabAgentV2 +------------- + +MatLabAgentV2 publishes the name of a python script along with any command line arguments that are needed for the script +to the appropriate topic. The agent then listens on another topic, and whenever anything is published on this topic, it +stores the message in the log file chosen when the VOLTTRON instance is started. If there are multiple standalone +agents, the agent can send a a script to each of them, along with their own set of command line arguments. In this +case, each script name and set of command line arguments should be sent to separate subtopics. This is done so that no +matter how many standalone agents are in use, MatLabAgentV2 will record all of their responses. + +.. code:: + + class MatlabAgentV2(Agent): + + def __init__(self,script_names=[], script_args=[], topics_to_matlab=[], + topics_to_volttron=None,**kwargs): + + super(MatlabAgentV2, self).__init__(**kwargs) + _log.debug("vip_identity: " + self.core.identity) + + self.script_names = script_names + self.script_args = script_args + self.topics_to_matlab = topics_to_matlab + self.topics_to_volttron = topics_to_volttron + self.default_config = {"script_names": script_names, + "script_args": script_args, + "topics_to_matlab": topics_to_matlab, + "topics_to_volttron": topics_to_volttron} + + + #Set a default configuration to ensure that self.configure is called immediately to setup + #the agent. + self.vip.config.set_default("config", self.default_config) + #Hook self.configure up to changes to the configuration file "config". + self.vip.config.subscribe(self.configure, actions=["NEW", "UPDATE"], pattern="config") + + def configure(self, config_name, action, contents): + """ + Called after the Agent has connected to the message bus. + If a configuration exists at startup this will be + called before onstart. + Is called every time the configuration in the store changes. + """ + config = self.default_config.copy() + config.update(contents) + + _log.debug("Configuring Agent") + + try: + script_names = config["script_names"] + script_args = config["script_args"] + topics_to_matlab = config["topics_to_matlab"] + topics_to_volttron = config["topics_to_volttron"] + + except ValueError as e: + _log.error("ERROR PROCESSING CONFIGURATION: {}".format(e)) + return + + self.script_names = script_names + self.script_args = script_args + self.topics_to_matlab = topics_to_matlab + self.topics_to_volttron = topics_to_volttron + self._create_subscriptions(self.topics_to_volttron) + + for script in range(len(self.script_names)): + cmd_args = "" + for x in range(len(self.script_args[script])): + cmd_args += ",{}".format(self.script_args[script][x]) + _log.debug("Publishing on: {}".format(self.topics_to_matlab[script])) + self.vip.pubsub.publish('pubsub', topic=self.topics_to_matlab[script], + message="{}{}".format(self.script_names[script],cmd_args)) + _log.debug("Sending message: {}{}".format(self.script_names[script],cmd_args)) + + _log.debug("Agent Configured!") + +For this example, the agent is publishing to the `matlab/to_matlab/1` topic, and is listening to the +`matlab/to_volttron` topic. It is sending the script name `testScript.py` with the argument 20. These are the default +values found in the agent, if no configuration is loaded. + +.. code:: + + script_names = config.get('script_names', ["testScript.py"]) + script_args = config.get('script_args', [["20"]]) + topics_to_matlab = config.get('topics_to_matlab', ["matlab/to_matlab/1"]) + topics_to_volttron = config.get('topics_to_volttron', "matlab/to_volttron/") + + +StandAloneMatLab.py +------------------- + +The `StandAloneMatLab.py` script is a standalone agent designed to be able to run in a Windows environment. Its purpose +is to listen to a topic, and when something is published to this topic, it takes the message, and sends it to the +``script_runner`` function in `scriptwrapper.py`. This function processes the inputs, and then the output is published +to another topic. + +.. code:: + + class StandAloneMatLab(Agent): + '''The standalone version of the MatLab Agent''' + + @PubSub.subscribe('pubsub', _topics['volttron_to_matlab']) + def print_message(self, peer, sender, bus, topic, headers, message): + print('The Message is: ' + str(message)) + messageOut = script_runner(message) + self.vip.pubsub.publish('pubsub', _topics['matlab_to_volttron'], message=messageOut) + + +settings.py +----------- + +The topic to listen to and the topic to publish to are defined in `settings.py`, along with the information needed to +connect the Standalone Agent to the primary VOLTTRON instance. These should be the same topics that the MatLabAgentV2 +is publishing and listening to, so that the communication can be successful. To connect the Standalone Agent to the +primary VOLTTRON instance, the IP address and port of the instance are needed, along with the server key. + +.. code:: + + _topics = { + 'volttron_to_matlab': 'matlab/to_matlab/1', + 'matlab_to_volttron': 'matlab/to_volttron/1' + } + + # The parameters dictionary is used to populate the agent's + # remote vip address. + _params = { + # The root of the address. + # Note: + # 1. volttron instance should be configured to use tcp. use command vcfg + # to configure + 'vip_address': 'tcp://192.168.56.101', + 'port': 22916, + + # public and secret key for the standalone_matlab agent. + # These can be created using the command: volttron-ctl auth keypair + # public key should also be added to the volttron instance auth + # configuration to enable standalone agent access to volttron instance. Use + # command 'vctl auth add' Provide this agent's public key when prompted + # for credential. + + 'agent_public': 'dpu13XKPvGB3XJNVUusCNn2U0kIWcuyDIP5J8mAgBQ0', + 'agent_secret': 'Hlya-6BvfUot5USdeDHZ8eksDkWgEEHABs1SELmQhMs', + + # Public server key from the remote platform. This can be + # obtained using the command: + # volttron-ctl auth serverkey + 'server_key': 'QTIzrRGQ0-b-37AbEYDuMA0l2ETrythM2V1ac0v9CTA' + + } + + def remote_url(): + return "{vip_address}:{port}?serverkey={server_key}" \ + "&publickey={agent_public}&" \ + "secretkey={agent_secret}".format(**_params) + +The primary VOLTTRON instance will then need to add the public key from the Standalone Agent. In this example, the +topic that the Standalone Agent is listening to is `matlab/to_matlab/1`, and the topic it is publishing to is +`matlab/to_volttron/1`. + + +scriptwrapper.py +---------------- + +`Scriptwrapper.py` contains the script_runner function. The purpose of this function is to take in a string that +contains a Python script and command line arguments separated by commas. This string is parsed and passed to the system +arguments, which allows the script sent to the function to use the command line arguments. The function then redirects +standard output to a `StringIO` file object, and then attempts to execute the script. If there are any errors with the +script, the error that is generated is returned to the standalone agent. Otherwise, the file object stores the output +from the script, is converted to a string, and is sent to the standalone agent. In this example, the script that is to +be run is `testScript.py`. + +.. code:: + + #Script to take in a string, run the program, + #and output the results of the command as a string. + + import time + import sys + from io import StringIO + + + def script_runner(message): + original = sys.stdout + # print(message) + # print(sys.argv) + sys.argv = message.split(',') + # print(sys.argv) + + try: + out = StringIO() + sys.stdout = out + exec(open(sys.argv[0]).read()) + sys.stdout = original + return out.getvalue() + except Exception as ex: + out = str(ex) + sys.stdout = original + return out + +.. note:: + + The script that is to be run needs to be in the same folder as the agent and the `scriptwrapper.py` script. The + `script_runner` function needs to be edited if it is going to call a script at a different location. + + +testScript.py +------------- + +This is a very simple test script designed to demonstrate the calling of a MatLab function from within Python. First it +initializes the MatLab engine for Python. It then takes in a single command line argument, and passes it to the MatLab +function `testPy.m`. If no arguments are sent, it will send 0 to the `testPy.m` function. It then prints the result of +the `testPy.m` function. In this case, since standard output is being redirected to a file object, this is how the +result is passed from this function to the Standalone Agent. + +.. code-block:: python + + import matlab.engine + import sys + + + eng = matlab.engine.start_matlab() + + if len(sys.argv) == 2: + result = eng.testPy(float(sys.argv[1])) + else: + result = eng.testPy(0.0) + + print(result) + + +testPy.m +-------- + +This MatLab function is a very simple example, designed to show a function that takes an argument, and produces an array +as the output. The input argument is added to each element in the array, and the entire array is then returned. + +.. code:: + + function out = testPy(z) + x = 1:100 + out = x + z + end + + +Setup on Linux +-------------- + +1. Setup and run VOLTTRON from develop branch using instructions :ref:`here `. + +2. Configure volttron instance using the ``vcfg`` command. When prompted for the vip address use + ``tcp://``. This is necessary to enable volttron communication with external + processes. + + .. note:: + + If you are running VOLTTRON from within VirtualBox, jit would be good to set one of your adapters as a + `Host-only` adapter. This can be done within the VM's settings, under the `Network` section. Once this is + done, use this IP for the VIP address. + + +.. _Matlab-Agent-Config: + +3. Update the configuration for MatLabAgent_v2 at `/example/MatLabAgent_v2/config`. + + The configuration file for the MatLab agent has four variables. + + 1. script_names + + 2. script_args + + 3. topics_to_matlab + + 4. topics_to_volttron + + An example config file included with the folder. + + .. code:: + + { + # VOLTTRON config files are JSON with support for python style comments. + "script_names": ["testScript.py"], + "script_args": [["20"]], + "topics_to_matlab": ["matlab/to_matlab/1"], + "topics_to_volttron": "matlab/to_volttron/" + } + + To edit the configuration, the format should be as follows: + + .. code-block:: json + + { + "script_names": ["script1.py", "script2.py", "..."], + "script_args": [["arg1","arg2"], ["arg1"], ["..."]], + "topics_to_matlab": ["matlab/to_matlab/1", "matlab/to_matlab/2", "..."], + "topics_to_volttron": "matlab/to_volttron/" + } + + The config requires that each script name lines up with a set of commandline arguments and a topic. A + commandline argument must be included, even if it is not used. The placement of brackets are important, even when + only communicating with one standalone agent. + + For example, if only one standalone agent is used, and no command line arguments are in place, the config file may + look like this. + + .. code-block:: json + + { + "script_names": ["testScript.py"], + "script_args": [["0"]], + "topics_to_matlab": ["matlab/to_matlab/1"], + "topics_to_volttron": "matlab/to_volttron/" + } + + +4. Install MatLabAgent_v2 and start agent (from volttron root directory) + + .. code-block:: bash + + python ./scripts/install-agent.py -s examples/MatLabAgent_v2 --start + + .. note:: + + The MatLabAgent_v2 publishes the command to be run to the message bus only on start or on a configuration + update. Once we configure the `standalone_matlab` agent on the Windows machine, we will send a configuration + update to the running MatLabAgent_v2. The configuration would contain the topics to which the Standalone Agent + is listening to and will be publishing result to. + + .. seealso:: + + The MatLab agent uses the configuration store to dynamically change inputs. More information on the config + store and how it used can be found here. + + * :ref:`VOLTTRON Configuration Store ` + + * :ref:`Agent Configuration Store ` + + * :ref:`Agent Configuration Store Interface ` + +5. Run the below command and make a note of the server key. This is required for configuring the stand alone agent + on Windows. (This is run on the linux machine) + + .. code-block:: bash + + vctl auth serverkey + + +Setup on Windows +---------------- + +Install pre-requisites +^^^^^^^^^^^^^^^^^^^^^^ + +1. Install Python3.6 64-bit from the `Python website `_. + +2. Install the MatLab engine from + `MathWorks `_. + + .. warning:: + + The MatLab engine for Python only supports certain version of Python depending on the version of MatLab used. + Please check `here `__ to see if the current + version of MatLab supports your version of Python. + + +.. note:: + + At this time, you may want to verify that you are able to communicate with your Linux machine across your network. + The simplest method would be to open up the command terminal and use ``ping ``, and ``telnet + ``. Please make sure that the port is + opened for outside access. + + +Install Standalone MatLab Agent +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The standalone MatLab agent is designed to be usable in a Windows environment. + +.. warning:: + + VOLTTRON is not designed to run in a Windows environment. Outside of cases where it is stated to be usable in a + Windows environment, it should be assumed that it will **NOT** function as expected. + +#. Download VOLTTRON + + Download the VOLTTRON develop repository from Github. Download the zip from + `GitHub `_. + + |github-image| + + |github-zip-image| + + Once the zipped file has been downloaded, go to your `Downloads` folder, right-click on the file, and select + `Extract All...` + + |extract-image_1| + + Choose a location for the extracted folder, and select "Extract" + + |extract-image_2| + +#. Setup the `PYTHONPATH` + + Open the Windows explorer, and navigate to `Edit environment variables for your account`. + + |cmd-image| + + Select "New" + + |env-vars-image_1| + + For "Variable name" enter: ``PYTHONPATH`` + For "Variable value" either browse to your VOLTTRON installation, or enter in the path to your VOLTTRON + installation. + + |env-vars-image_2| + + Select `OK` twice. + +#. Set Python version in MatLab + + Open your MatLab application. Run the command: + + .. code-block:: bash + + pyversion + + This should print the path to Python2.7. If you have multiple versions of python on your machine and `pyversion` + points to a different version of Python, use: + + .. code-block:: bash + + pyversion /path/to/python.exe + + to set the appropriate version of python for your system. + + For example, to use python 3.6 with MatLab: + + .. code-block:: console + + pyversion C:\Python36\python.exe + +#. Set up the environment. + + Open up the command prompt + + |cmd-image_2| + + Navigate to your VOLTTRON installation + + ``cd \Your\directory\path\to\volttron-develop`` + + Use pip to install and setup dependencies. + + ``pip install -r examples\StandAloneMatLab\requirements.txt`` + + ``pip install -e .`` + + .. note:: + + If you get the error doing the second step because of an already installed volttron from a different directory, + manually delete the `volttron-egg.` link file from your `\\Lib\\site-packages` directory (for + example: + + .. code-block:: bash + + del C:\\Python27\\lib\\site-packages\\volttron-egg.link + + and re-run the second command + +#. Configure the agent + + The configuration settings for the standalone agent are in setting.py (located in + `volttron-develop\\examples\\StandAloneMatLab\\`) + + **settings.py** + + * `volttron_to_matlab` needs to be set to the topic that will send your script and command line arguments to your + stand alone agent. This was defined in the :ref:`config. ` + + * `matlab_to_volttron` needs to be set to the topic that will send your script's + output back to your volttron platform. This was defined in :ref:`config. ` + + * `vip_address` needs to be set to the address of your volttron instance + + * `port` needs to be set to the port of your volttron instance + + * `server_key` needs to be set to the public server key of your primary volttron platform. This can be obtained + from the primary volttron platform using ``vctl auth serverkey`` (VOLTTRON must be running to use this command.) + + It is possible to have multiple standalone agents running. In this case, copy the `StandAloneMatLab` folder, and + make the necessary changes to the new `settings.py` file. Unless it is connecting to a separate VOLTTRON instance, + you should only need to change the `volttron_to_matlab` setting. + + .. note:: + + It is recommended that you generate a new "agent_public" and "agent_private" key for your standalone agent. + This can be done using the ``vctl auth keypair`` command on your primary VOLTTRON platform on Linux. If you + plan to use multiple standalone agents, they will each need their own keypair. + +6. Add standalone agent key to VOLTTRON platform + + * Copy the public key from `settings.py` in the StandAloneMatLab folder. + + * While the primary VOLTTRON platform is running on the linux machine, add the agent public key using the ``vctl + auth`` command on the Linux machine. This will make VOLTTRON platform allow connections from the standalone agent + + .. code-block:: bash + + vctl auth add --credentials + +7. Run standalone agent + + + At this point, the agent is ready to run. To use the agent, navigate to the example folder and use python to start + the agent. The agent will then wait for a message to be published to the selected topic by the MatLab agent. + + .. code-block:: bash + + cd examples\StandAloneMatLab\ + + python standalone_matlab.py + + The output should be similar to this: + + .. code-block:: console + + 2019-08-01 10:42:47,592 volttron.platform.vip.agent.core DEBUG: identity: standalone_matlab + 2019-08-01 10:42:47,592 volttron.platform.vip.agent.core DEBUG: agent_uuid: None + 2019-08-01 10:42:47,594 volttron.platform.vip.agent.core DEBUG: serverkey: None + 2019-08-01 10:42:47,596 volttron.platform.vip.agent.core DEBUG: AGENT RUNNING on ZMQ Core standalone_matlab + 2019-08-01 10:42:47,598 volttron.platform.vip.zmq_connection DEBUG: ZMQ connection standalone_matlab + 2019-08-01 10:42:47,634 volttron.platform.vip.agent.core INFO: Connected to platform: router: ebae9efa-5e8f-49e3-95a0-2020ddff9e8a version: 1.0 identity: standalone_matlab + 2019-08-01 10:42:47,634 volttron.platform.vip.agent.core DEBUG: Running onstart methods. + + + .. note:: + + If you have Python3 as your default Python run the command ``python -2 standalone_matlab.py`` + +8. On the Linux machine configure the Matlab Agent to publish commands to the topic standalone agent is listening to. +To load a new configuration or to change the current configuration enter + + .. code-block:: bash + + vctl config store config + + Whenever there is a change in the configuration in the config store, or whenever the agent starts, the MatLab Agent + sends the configured command to the topic configured. As long as the standalone agent has been started and is + listening to the appropriate topic, the output in the log should look similar to this: + + .. code:: + + 2019-08-01 10:43:18,925 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Configuring Agent + 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Publishing on: matlab/to_matlab/1 + 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Sending message: testScript2.py,20 + 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Agent Configured! + 2019-08-01 10:43:18,979 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent INFO: Agent: matlab/to_volttron/1 + Message: + '20' + + Once the matlab agent publishes the message (in the above case, "testScript2.py,20") on the windows command prompt + running the standalone agent, you should see the message that was received by the standalone agent. + + .. code:: + + 2019-08-01 10:42:47,671 volttron.platform.vip.agent.subsystems.configstore DEBUG: Processing callbacks for affected files: {} + The Message is: testScript2.py,20 + + .. note:: + + If MatLabAgent_v2 has been installed and started, and you have not started the `standalone_matlab agent`, you + will need to either restart the matlab_agentV2, or make a change to the configuration in the config store to + send command to the topic standalone agent is actively listening to. + +.. |github-image| image:: files/github-image.png +.. |cmd-image| image:: files/cmd-image.png +.. |env-vars-image_1| image:: files/env-vars-image_1.png +.. |env-vars-image_2| image:: files/env-vars-image_2.png +.. |cmd-image_2| image:: files/cmd-image_2.png +.. |github-zip-image| image:: files/github-zip-image.png +.. |extract-image_1| image:: files/extract-image_1.png +.. |extract-image_2| image:: files/extract-image_2.png +.. |matlab-agent-diagram| image:: files/matlab-agent-diagram.png diff --git a/docs/source/developing-volttron/developing-agents/example-agents/node-red.rst b/docs/source/developing-volttron/developing-agents/example-agents/node-red.rst new file mode 100644 index 0000000000..523c5718f5 --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/node-red.rst @@ -0,0 +1,79 @@ +.. _Node-Red: + +================ +Node Red Example +================ + +Node Red is a visual programming language wherein users connect small units of functionality "nodes" to create "flows". + +There are two example nodes that allow communication between Node-Red and VOLTTRON. One node reads subscribes to +messages on the VOLTTRON message bus and the other publishes to it. + + +Dependencies +------------ + +The example nodes depend on `python-shell` to be installed and available to the Node Red environment. + + +Installation +------------ + +Copy all files from `volttron/examples/NodeRed` to your `~/.node-red/nodes` directory. `~/.node-red` is the default +directory for Node Red files. If you have set a different directory use that instead. + +Set the variables at the beginning of the `volttron.js` file to be a valid VOLTTRON environment, VOLTTRON home, and +Python PATH. + +Valid CURVE keys need to be added to the `settings.py` file. If they are generated with the `vctl auth keypair` command +then the public key should be added to VOLTTRON's authorization file with the following: + +.. code-block:: console + + $ vctl auth add + +The serverkey can be found with: + +.. code-block:: console + + $ vctl auth serverkey + + +Usage +----- + +Start VOLTTRON and Node Red. + +.. code-block:: console + + $ node-red + + + Welcome to Node-RED + =================== + + 11 Jan 15:26:49 - [info] Node-RED version: v0.14.4 + 11 Jan 15:26:49 - [info] Node.js version: v0.10.25 + 11 Jan 15:26:49 - [info] Linux 3.16.0-38-generic x64 LE + 11 Jan 15:26:49 - [info] Loading palette nodes + 11 Jan 15:26:49 - [warn] ------------------------------------------------------ + 11 Jan 15:26:49 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node + 11 Jan 15:26:49 - [warn] ------------------------------------------------------ + 11 Jan 15:26:49 - [info] Settings file : /home/volttron/.node-red/settings.js + 11 Jan 15:26:49 - [info] User directory : /home/volttron/.node-red + 11 Jan 15:26:49 - [info] Flows file : /home/volttron/.node-red/flows_volttron.json + 11 Jan 15:26:49 - [info] Server now running at http://127.0.0.1:1880/ + 11 Jan 15:26:49 - [info] Starting flows + 11 Jan 15:26:49 - [info] Started flows + +The output from the Node Red command indicates the address of its web interface. Nodes available for use are in the +left sidebar. + +|Node Red| + +We can now use the VOLTTRON nodes to read from and write to VOLTTRON. + +|Flow| + +.. |Node Red| image:: files/node-red.png +.. |Flow| image:: files/node-red-flow.png diff --git a/docs/source/developing-volttron/developing-agents/example-agents/scheduler-example-agent.rst b/docs/source/developing-volttron/developing-agents/example-agents/scheduler-example-agent.rst new file mode 100644 index 0000000000..83817ae8e8 --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/scheduler-example-agent.rst @@ -0,0 +1,57 @@ +.. _Scheduler-Example-Agent: + +======================= +Scheduler Example Agent +======================= + +The Scheduler Example Agent demonstrates how to use the scheduling feature of the :ref`Actuator Agent ` +as well as how to send a command. This agent publishes a request for a reservation on a (fake) device then takes an +action when it's scheduled time appears. The ActuatorAgent must be running to exercise this example. + +.. Note:: + + Since there is no actual device, an error is produced when the agent attempts to take its action. + +.. code-block:: python + + def publish_schedule(self): + '''Periodically publish a schedule request''' + headers = { + 'AgentID': agent_id, + 'type': 'NEW_SCHEDULE', + 'requesterID': agent_id, #The name of the requesting agent. + 'taskID': agent_id + "-ExampleTask", #The desired task ID for this task. It must be unique among all other scheduled tasks. + 'priority': 'LOW', #The desired task priority, must be 'HIGH', 'LOW', or 'LOW_PREEMPT' + } + + start = str(datetime.datetime.now()) + end = str(datetime.datetime.now() + datetime.timedelta(minutes=1)) + + + msg = [ + ['campus/building/unit',start,end] + ] + self.vip.pubsub.publish( + 'pubsub', topics.ACTUATOR_SCHEDULE_REQUEST, headers, msg) + +The agent listens to schedule announcements from the actuator and then issues a command: + +.. code-block:: python + + @PubSub.subscribe('pubsub', topics.ACTUATOR_SCHEDULE_ANNOUNCE(campus='campus', + building='building',unit='unit')) + def actuate(self, peer, sender, bus, topic, headers, message): + print ("response:",topic,headers,message) + if headers[headers_mod.REQUESTER_ID] != agent_id: + return + '''Match the announce for our fake device with our ID + Then take an action. Note, this command will fail since there is no + actual device''' + headers = { + 'requesterID': agent_id, + } + self.vip.pubsub.publish( + 'pubsub', topics.ACTUATOR_SET(campus='campus', + building='building',unit='unit', + point='point'), + headers, 0.0) diff --git a/docs/source/developing-volttron/developing-agents/example-agents/simple-web-agent-walk-through.rst b/docs/source/developing-volttron/developing-agents/example-agents/simple-web-agent-walk-through.rst new file mode 100644 index 0000000000..adcf642274 --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/example-agents/simple-web-agent-walk-through.rst @@ -0,0 +1,71 @@ +.. _Simple-Web-Agent-Walk-through: + +============================= +Simple Web Agent Walk-through +============================= + +A simple web enabled agent that will hook up with a VOLTTRON message bus and allow interaction between it via HTTP. +This example agent shows a simple file serving agent, a JSON-RPC based call, and a websocket based connection mechanism. + + +Starting VOLTTRON Platform +-------------------------- + +.. note:: + + Activate the environment first :ref:`active the environment ` + +In order to start the simple web agent, we need to bind the VOLTTRON instance to the a web server. We need to specify +the address and the port for the web server. For example, if we want to bind the `localhost:8080` as the web server +we start the VOLTTRON platform as follows: + +.. code-block:: bash + + ./start-volttron --bind-web-address http://127.0.0.1:8080 + +Once the platform is started, we are ready to run the Simple Web Agent. + + +Running Simple Web Agent +------------------------ + +.. note:: + + The following assumes the shell is located at the :ref:`VOLTTRON_ROOT`. + +Copy the following into your shell (save it to a file for executing it again later): + +.. code-block:: console + + python scripts/install-agent.py \ + --agent-source examples/SimpleWebAgent \ + --tag simpleWebAgent \ + --vip-identity webagent \ + --force \ + --start + +This will create a web server on ``http://localhost:8080``. The `index.html` file under `simpleweb/webroot/simpleweb/` +can be any HTML page which binds to the VOLTTRON message bus .This provides a simple example of providing a web endpoint +in VOLTTRON. + + +Path based registration examples +-------------------------------- + +- Files will need to be in `webroot/simpleweb` in order for them to be browsed from + ``http://localhost:8080/simpleweb/index.html`` + +- Filename is required as we don't currently auto-redirect to any default pages as shown in + ``self.vip.web.register_path("/simpleweb", os.path.join(WEBROOT))`` + +The following two examples show the way to call either a JSON-RPC (default) endpoint and one that returns a different +content-type. With the JSON-RPC example from volttron central we only allow post requests, however this is not +required. + +- Endpoint will be available at `http://localhost:8080/simple/text` + ``self.vip.web.register_endpoint("/simple/text", self.text)`` + +- Endpoint will be available at `http://localhost:8080/simple/jsonrpc` + ``self.vip.web.register_endpoint("/simpleweb/jsonrpc", self.rpcendpoint)`` +- ``text/html`` content type specified so the browser can act appropriately like ``[("Content-Type", "text/html")]`` +- The default response is ``application/json so our`` endpoint returns appropriately with a JSON based response. diff --git a/docs/source/devguides/agent_development/files/1-eclipse-desktop.jpg b/docs/source/developing-volttron/developing-agents/files/1-eclipse-desktop.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/1-eclipse-desktop.jpg rename to docs/source/developing-volttron/developing-agents/files/1-eclipse-desktop.jpg diff --git a/docs/source/devguides/agent_development/files/10-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/10-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/10-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/10-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/11-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/11-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/11-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/11-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/12-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/12-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/12-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/12-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/13-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/13-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/13-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/13-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/14-check-volttron-from-github.jpg b/docs/source/developing-volttron/developing-agents/files/14-check-volttron-from-github.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/14-check-volttron-from-github.jpg rename to docs/source/developing-volttron/developing-agents/files/14-check-volttron-from-github.jpg diff --git a/docs/source/devguides/agent_development/files/15-check-volttron-from-github.jpg b/docs/source/developing-volttron/developing-agents/files/15-check-volttron-from-github.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/15-check-volttron-from-github.jpg rename to docs/source/developing-volttron/developing-agents/files/15-check-volttron-from-github.jpg diff --git a/docs/source/devguides/agent_development/files/16-check-volttron-from-github.jpg b/docs/source/developing-volttron/developing-agents/files/16-check-volttron-from-github.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/16-check-volttron-from-github.jpg rename to docs/source/developing-volttron/developing-agents/files/16-check-volttron-from-github.jpg diff --git a/docs/source/devguides/agent_development/files/17-check-volttron-from-github.jpg b/docs/source/developing-volttron/developing-agents/files/17-check-volttron-from-github.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/17-check-volttron-from-github.jpg rename to docs/source/developing-volttron/developing-agents/files/17-check-volttron-from-github.jpg diff --git a/docs/source/devguides/agent_development/files/18-check-volttron-from-github.jpg b/docs/source/developing-volttron/developing-agents/files/18-check-volttron-from-github.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/18-check-volttron-from-github.jpg rename to docs/source/developing-volttron/developing-agents/files/18-check-volttron-from-github.jpg diff --git a/docs/source/devguides/agent_development/files/19-check-volttron-from-github.jpg b/docs/source/developing-volttron/developing-agents/files/19-check-volttron-from-github.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/19-check-volttron-from-github.jpg rename to docs/source/developing-volttron/developing-agents/files/19-check-volttron-from-github.jpg diff --git a/docs/source/devguides/agent_development/files/2-egit-plugin.jpg b/docs/source/developing-volttron/developing-agents/files/2-egit-plugin.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/2-egit-plugin.jpg rename to docs/source/developing-volttron/developing-agents/files/2-egit-plugin.jpg diff --git a/docs/source/devguides/agent_development/files/20-check-volttron-from-github.jpg b/docs/source/developing-volttron/developing-agents/files/20-check-volttron-from-github.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/20-check-volttron-from-github.jpg rename to docs/source/developing-volttron/developing-agents/files/20-check-volttron-from-github.jpg diff --git a/docs/source/devguides/agent_development/files/21-configuring-pydev.jpg b/docs/source/developing-volttron/developing-agents/files/21-configuring-pydev.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/21-configuring-pydev.jpg rename to docs/source/developing-volttron/developing-agents/files/21-configuring-pydev.jpg diff --git a/docs/source/devguides/agent_development/files/22-configuring-pydev.jpg b/docs/source/developing-volttron/developing-agents/files/22-configuring-pydev.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/22-configuring-pydev.jpg rename to docs/source/developing-volttron/developing-agents/files/22-configuring-pydev.jpg diff --git a/docs/source/devguides/agent_development/files/23-configuring-pydev.jpg b/docs/source/developing-volttron/developing-agents/files/23-configuring-pydev.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/23-configuring-pydev.jpg rename to docs/source/developing-volttron/developing-agents/files/23-configuring-pydev.jpg diff --git a/docs/source/devguides/agent_development/files/24-setting-pydev-project.jpg b/docs/source/developing-volttron/developing-agents/files/24-setting-pydev-project.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/24-setting-pydev-project.jpg rename to docs/source/developing-volttron/developing-agents/files/24-setting-pydev-project.jpg diff --git a/docs/source/devguides/agent_development/files/25-setting-pydev-perspective.jpg b/docs/source/developing-volttron/developing-agents/files/25-setting-pydev-perspective.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/25-setting-pydev-perspective.jpg rename to docs/source/developing-volttron/developing-agents/files/25-setting-pydev-perspective.jpg diff --git a/docs/source/devguides/agent_development/files/26-running-volttron.jpg b/docs/source/developing-volttron/developing-agents/files/26-running-volttron.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/26-running-volttron.jpg rename to docs/source/developing-volttron/developing-agents/files/26-running-volttron.jpg diff --git a/docs/source/devguides/agent_development/files/27-running-volttron.jpg b/docs/source/developing-volttron/developing-agents/files/27-running-volttron.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/27-running-volttron.jpg rename to docs/source/developing-volttron/developing-agents/files/27-running-volttron.jpg diff --git a/docs/source/devguides/agent_development/files/28-running-volttron.jpg b/docs/source/developing-volttron/developing-agents/files/28-running-volttron.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/28-running-volttron.jpg rename to docs/source/developing-volttron/developing-agents/files/28-running-volttron.jpg diff --git a/docs/source/devguides/agent_development/files/29-running-volttron.jpg b/docs/source/developing-volttron/developing-agents/files/29-running-volttron.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/29-running-volttron.jpg rename to docs/source/developing-volttron/developing-agents/files/29-running-volttron.jpg diff --git a/docs/source/devguides/agent_development/files/3-egit-plugin.jpg b/docs/source/developing-volttron/developing-agents/files/3-egit-plugin.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/3-egit-plugin.jpg rename to docs/source/developing-volttron/developing-agents/files/3-egit-plugin.jpg diff --git a/docs/source/devguides/agent_development/files/30-running-listener-agent.jpg b/docs/source/developing-volttron/developing-agents/files/30-running-listener-agent.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/30-running-listener-agent.jpg rename to docs/source/developing-volttron/developing-agents/files/30-running-listener-agent.jpg diff --git a/docs/source/devguides/agent_development/files/31-running-listener-agent.jpg b/docs/source/developing-volttron/developing-agents/files/31-running-listener-agent.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/31-running-listener-agent.jpg rename to docs/source/developing-volttron/developing-agents/files/31-running-listener-agent.jpg diff --git a/docs/source/devguides/agent_development/files/32-running-listener-agent.jpg b/docs/source/developing-volttron/developing-agents/files/32-running-listener-agent.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/32-running-listener-agent.jpg rename to docs/source/developing-volttron/developing-agents/files/32-running-listener-agent.jpg diff --git a/docs/source/devguides/agent_development/files/33-running-listener-agent.jpg b/docs/source/developing-volttron/developing-agents/files/33-running-listener-agent.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/33-running-listener-agent.jpg rename to docs/source/developing-volttron/developing-agents/files/33-running-listener-agent.jpg diff --git a/docs/source/devguides/agent_development/files/34-running-listener-agent.jpg b/docs/source/developing-volttron/developing-agents/files/34-running-listener-agent.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/34-running-listener-agent.jpg rename to docs/source/developing-volttron/developing-agents/files/34-running-listener-agent.jpg diff --git a/docs/source/devguides/agent_development/files/35-listening_agent_output.jpg b/docs/source/developing-volttron/developing-agents/files/35-listening_agent_output.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/35-listening_agent_output.jpg rename to docs/source/developing-volttron/developing-agents/files/35-listening_agent_output.jpg diff --git a/docs/source/devguides/agent_development/files/36-agent-test-folder.jpg b/docs/source/developing-volttron/developing-agents/files/36-agent-test-folder.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/36-agent-test-folder.jpg rename to docs/source/developing-volttron/developing-agents/files/36-agent-test-folder.jpg diff --git a/docs/source/devguides/agent_development/files/37-testagent-output.jpg b/docs/source/developing-volttron/developing-agents/files/37-testagent-output.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/37-testagent-output.jpg rename to docs/source/developing-volttron/developing-agents/files/37-testagent-output.jpg diff --git a/docs/source/devguides/agent_development/files/38-console-output.jpg b/docs/source/developing-volttron/developing-agents/files/38-console-output.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/38-console-output.jpg rename to docs/source/developing-volttron/developing-agents/files/38-console-output.jpg diff --git a/docs/source/devguides/agent_development/files/39-testagent-output-weather-subscribed.jpg b/docs/source/developing-volttron/developing-agents/files/39-testagent-output-weather-subscribed.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/39-testagent-output-weather-subscribed.jpg rename to docs/source/developing-volttron/developing-agents/files/39-testagent-output-weather-subscribed.jpg diff --git a/docs/source/devguides/agent_development/files/4-egit-plugin.jpg b/docs/source/developing-volttron/developing-agents/files/4-egit-plugin.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/4-egit-plugin.jpg rename to docs/source/developing-volttron/developing-agents/files/4-egit-plugin.jpg diff --git a/docs/source/developing-volttron/developing-agents/files/40-message-debugger.jpg b/docs/source/developing-volttron/developing-agents/files/40-message-debugger.jpg new file mode 100644 index 0000000000..99cc3c8a85 Binary files /dev/null and b/docs/source/developing-volttron/developing-agents/files/40-message-debugger.jpg differ diff --git a/docs/source/devguides/agent_development/files/5-install-eclipse-pydev-plugin.jpg b/docs/source/developing-volttron/developing-agents/files/5-install-eclipse-pydev-plugin.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/5-install-eclipse-pydev-plugin.jpg rename to docs/source/developing-volttron/developing-agents/files/5-install-eclipse-pydev-plugin.jpg diff --git a/docs/source/devguides/agent_development/files/6-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/6-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/6-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/6-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/7-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/7-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/7-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/7-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/8-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/8-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/8-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/8-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/9-check-volttron-with-eclipse.jpg b/docs/source/developing-volttron/developing-agents/files/9-check-volttron-with-eclipse.jpg similarity index 100% rename from docs/source/devguides/agent_development/files/9-check-volttron-with-eclipse.jpg rename to docs/source/developing-volttron/developing-agents/files/9-check-volttron-with-eclipse.jpg diff --git a/docs/source/devguides/agent_development/files/run_configuration.png b/docs/source/developing-volttron/developing-agents/files/run_configuration.png similarity index 100% rename from docs/source/devguides/agent_development/files/run_configuration.png rename to docs/source/developing-volttron/developing-agents/files/run_configuration.png diff --git a/docs/source/specifications/aggregate.rst b/docs/source/developing-volttron/developing-agents/specifications/aggregate.rst similarity index 99% rename from docs/source/specifications/aggregate.rst rename to docs/source/developing-volttron/developing-agents/specifications/aggregate.rst index c6198112e4..795bcff51a 100644 --- a/docs/source/specifications/aggregate.rst +++ b/docs/source/developing-volttron/developing-agents/specifications/aggregate.rst @@ -1,4 +1,4 @@ -.. _AggregateHistorianSpec: +.. _Aggregate-Historian-Spec: ======================================= Aggregate Historian Agent Specification diff --git a/docs/source/specifications/files/aggregate_historian.jpg b/docs/source/developing-volttron/developing-agents/specifications/files/aggregate_historian.jpg similarity index 100% rename from docs/source/specifications/files/aggregate_historian.jpg rename to docs/source/developing-volttron/developing-agents/specifications/files/aggregate_historian.jpg diff --git a/docs/source/developing-volttron/developing-agents/specifications/index.rst b/docs/source/developing-volttron/developing-agents/specifications/index.rst new file mode 100644 index 0000000000..c85089ba2b --- /dev/null +++ b/docs/source/developing-volttron/developing-agents/specifications/index.rst @@ -0,0 +1,14 @@ +.. _Agent-Specifications: + +==================== +Agent Specifications +==================== + +Documents included below are intended to provide a specification to classes of agents which include a base class in the +VOLTTRON repository and have a well defined set of functions and services. + +.. toctree:: + + aggregate + tagging-service + weather-service diff --git a/docs/source/specifications/tagging_service.rst b/docs/source/developing-volttron/developing-agents/specifications/tagging-service.rst similarity index 100% rename from docs/source/specifications/tagging_service.rst rename to docs/source/developing-volttron/developing-agents/specifications/tagging-service.rst diff --git a/docs/source/specifications/Weather_Service_Spec.rst b/docs/source/developing-volttron/developing-agents/specifications/weather-service.rst similarity index 100% rename from docs/source/specifications/Weather_Service_Spec.rst rename to docs/source/developing-volttron/developing-agents/specifications/weather-service.rst diff --git a/docs/source/devguides/agent_development/Testing-Agents.rst b/docs/source/developing-volttron/developing-agents/writing-agent-tests.rst similarity index 71% rename from docs/source/devguides/agent_development/Testing-Agents.rst rename to docs/source/developing-volttron/developing-agents/writing-agent-tests.rst index 98d0d779a3..6eefe10096 100644 --- a/docs/source/devguides/agent_development/Testing-Agents.rst +++ b/docs/source/developing-volttron/developing-agents/writing-agent-tests.rst @@ -1,35 +1,37 @@ .. _Writing-Agent-Tests: -******************* +=================== Writing Agent Tests -******************* +=================== -The VOLTTRON team strongly encourages developing agents with a set of unit and integration tests. Test-driven +The VOLTTRON team strongly encourages developing agents with a set of unit and integration tests. Test-driven development can save developers significant time and effort by clearly defining behavioral expectations for agent code. -We recommend developing agent tests using Pytest. Agent code contributed to VOLTTRON is expected to include a set of -tests using Pytest in the agent module directory. Following are instructions for setting up Pytest, structuring your +We recommend developing agent tests using Pytest. Agent code contributed to VOLTTRON is expected to include a set of +tests using Pytest in the agent module directory. Following are instructions for setting up Pytest, structuring your tests, how to write unit and integration tests (including some helpful tools using Pytest and Mock) and how to run your tests. + Installation -############ +============ To get started with Pytest, install it in an activated environment: -:: +.. code-block:: bash pip install pytest -Or when running VOLTTRON's bootstrap process, specify the `--testing` optional argument. +Or when running VOLTTRON's bootstrap process, specify the ``--testing`` optional argument. -:: +.. code-block:: bash python bootstrap.py --testing `Pytest on PyPI `_ + Module Structure -################ +================ We suggest the following structure for your agent module: @@ -50,12 +52,13 @@ We suggest the following structure for your agent module: │ └── setup.py The test suite should be in a `tests` directory in the root agent directory, and should contain one or more -test code files (with the `test_` convention). Conftest.py can be used to give all agent tests -access to some portion of the VOLTTRON code - in many cases agents use conftest to import VOLTTRON testing +test code files (with the `test_` convention). `conftest.py` can be used to give all agent tests +access to some portion of the VOLTTRON code. In many cases, agents use `conftest.py` to import VOLTTRON testing fixtures for integration tests. + Naming Conventions -################## +------------------ Pytest tests are discovered and run using some conventions: @@ -64,7 +67,7 @@ Pytest tests are discovered and run using some conventions: * Pytest will search in those directories for files called test_.py or _test.py * In those files, Pytest will test: * functions and methods prefixed by "test" outside of any class - * functions and methonds prefixed by "test" inside of any class prefixed by "test" + * functions and methods prefixed by "test" inside of any class prefixed by "test" :: @@ -101,20 +104,31 @@ Pytest tests are discovered and run using some conventions: assert False In the above example, Pytest will run the tests `test_success` from the file test1.py and `test_success` and test_fail -from test2.py. No tests will be run from file.txt, even though it contains test code, nor will it try to run +from test2.py. No tests will be run from file.txt, even though it contains test code, nor will it try to run `helper_method` from test1.py as a test. + Writing Unit Tests -################## +================== These tests should test the various methods of the code base, checking for success and fail conditions. These tests -should capture how the components of the system should function - tests should describe all the possible output +should capture how the components of the system should function; and describe all the possible output conditions given the possible range of inputs including how they should fail if given improper input. `Pytest guide to Unit Testing `_ -VOLTTRON agents include code for many platform features, these features can be mocked to allow unit tests to test only -the features of the agent, without having to account for the behaviors of the core platform: +Mocking Dependencies +-------------------- + +VOLTTRON agents include code for many platform features; these features can be mocked to allow unit tests to test only +the features of the agent without having to account for the behaviors of the core platform. While there are many tools +that can mock dependencies of an agent, we recommend Volttron's AgentMock or Python's Mock testing library. + +AgentMock +^^^^^^^^^ +AgentMock was specifically created to run unit tests on agents. AgentMock takes an Agent class and mocks the attributes +and methods of that Agent's dependencies. AgentMock also allows you to customize the behavior of dependencies within +each individual test. Below is an example: .. code-block:: python @@ -138,24 +152,77 @@ the features of the agent, without having to account for the behaviors of the co assert isinstance(result.get("test2"), str) # ... + def test_success_case_custom_mocks(): + agent.some_dependency.some_method.return_value = "foobar" + agent.some_attribute = "custom, dummy value" + result = agent.do_function_that_relies_on_custom_mocks("valid input") + # ... + def test_failure_case() # pytests.raises can be useful for testing exceptions, more information about usage below with pytest.raises(ValueError, match=r'Invalid input string for do_function') result = agent.do_function("invalid input") +Mock +^^^^ + +Simliar to AgentMock, Python's Mock testing library allows a user to replace the behavior of dependencies with a +user-specified behavior. This is useful for replacing VOLTTRON platform behavior, remote API behavior, modules, +etc. where using them in unit or integration tests is impractical or impossible. +Below is an example that uses the patch decorator to mock an Agent's web request. + +`Mock documentation `_ + +.. code-block:: python + + class UserAgent() + + def __init__(): + # Code here + + def get_remote_data() + response = self._get_data_from_remote() + return "Remote response: {}".format(response) + + # it can be useful to create private functions for use with mock for things like making web requests + def _get_data_from_remote(): + url = "test.com/test1" + headers = {} + return requests.get(url, headers) + + # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + import pytest + import mock + + def get_mock_response(): + return "test response" + + # here we're mocking the UserAgent's _get_data_from_remote method and replacing it with our get_mock_response method + # to feed our test some fake remote data + @mock.patch.object(UserAgent, '_get_data_from_remote', get_mock_response) + def test_get_remote_data(): + assert UserAgent.get_remote_Data() == "Remote response: test response" + + + + + + Pytest Tools -************ +------------ + +Pytest includes many helpful tools for developing your tests. We'll highlight a few that have been useful for +VOLTTRON core tests, but checkout `the Pytest documentation `_ for additional information on +each tool as well as tools not covered in this guide. -Pytest includes many helpful tools for developing your tests, here we'll highlight a few that have been useful for -VOLTTRON core tests, but checkout `the Pytest documentation `_ for additional information on each tool as well as -tools not covered in this guide. Pytest Fixtures ---------------- +^^^^^^^^^^^^^^^ Pytest fixtures can be used to create reusable code for tests that can be accessed by every test in a module based on -scope. There are several kinds of scopes, but commonly used are "module" (the fixture is run once per module for all -the tests of that module) or "function" (the fixture is run once per test). For fixtures to be used by tests, they +scope. There are several kinds of scopes, but commonly used are "module" (the fixture is run once per module for all +the tests of that module) or "function" (the fixture is run once per test). For fixtures to be used by tests, they should be passed as parameters. `Pytest Fixture documentation `_ @@ -177,8 +244,7 @@ Here is an example of a fixture, along with using it in a test: cursor.commit() sqlite.conn.close() - # now when we pass the cleanup function, we should expect that the table will be dropped and rebuilt before the test - # runs + # when we pass the cleanup function, we expect that the table will be dropped and rebuilt before the test runs def test_store_data(cleanup_database): sqlite_conn = sqlite.connect("test.sqlite") cursor = sqlite_conn.cursor() @@ -192,21 +258,21 @@ Here is an example of a fixture, along with using it in a test: assert count == 1 - Pytest.mark ------------ +^^^^^^^^^^^ Pytest marks are used to set metadata for test functions. Defining your own custom marks can allow you to run -subsections of your tests. Parametrize can be used to pass a series of parameters to a test, so that it can be run -many times to cover the space of potential inputs. Marks also exist to specify expected behavior for tests. +subsections of your tests. Parametrize can be used to pass a series of parameters to a test, so that it can be run +many times to cover the space of potential inputs. Marks also exist to specify expected behavior for tests. `Mark documentation `_ + Custom Marks -~~~~~~~~~~~~ +"""""""""""" To add a custom mark, add the name of the mark followed by a colon then a description string to the 'markers' section -of Pytest.ini (an example of this exists in the core VOLTTRON repository). Then add the appropriate decorator: +of Pytest.ini (an example of this exists in the core VOLTTRON repository). Then add the appropriate decorator: .. code-block:: python @@ -225,8 +291,9 @@ The VOLTTRON team also has a `dev` mark for running individual (or a few) one-of # TODO unit test here pass + Parametrize -~~~~~~~~~~~ +""""""""""" Parametrize will allow tests to be run with a variety of parameters. Add the parametrize decorator, and for parameters include a list of parameter names matching the test parameter names as a comma-delimited string followed by a list of @@ -236,15 +303,16 @@ tuples containing parameters for each test. .. code-block:: python - @pytest.mark.parametrize("param1,param2,param3", [(1, 2, 3), (-1, 0, "")]) + @pytest.mark.parametrize("test_input1, test_input2, expected", [(1, 2, 3), (-1, 0, "")]) def test_user_agent(param1, param2, param3): # TODO unit test here pass + Skip, skipif, and xfail -~~~~~~~~~~~~~~~~~~~~~~~ +""""""""""""""""""""""" -The skip mark can be used to skip a test for any reason every time the test suite is run: +The `skip` mark can be used to skip a test for any reason every time the test suite is run: .. code-block:: python @@ -254,7 +322,7 @@ The skip mark can be used to skip a test for any reason every time the test suit # TODO unit test here pass -The skipif mark can be used to skip a test based on some condition: +The `skipif` mark can be used to skip a test based on some condition: .. code-block:: python @@ -264,7 +332,7 @@ The skipif mark can be used to skip a test based on some condition: # TODO unit test here pass -The xfail mark can be used to run a test, but to show that the test is currently expected to fail +The `xfail` mark can be used to run a test, but to show that the test is currently expected to fail .. code-block:: python @@ -276,58 +344,22 @@ The xfail mark can be used to run a test, but to show that the test is currently `Skip, skipif, and xfail docs `_ -Mock -**** - -Mock allows a user to replace the behavior of dependencies with a user specified behavior. This is useful for replacing -VOLTTRON platform behavior, remote API behavior, modules, etc. where using them in unit or integration tests is -impractical or impossible. - -`Mock documentation `_ - -.. code-block:: python - - class UserAgent() - - def __init__(): - # Code here - - def get_remote_data() - response = self._get_data_from_remote() - return "Remote response: {}".format(response) - - # it can be useful to create private functions for use with mock for things like making web requests - def _get_data_from_remote(): - url = "test.com/test1" - headers = {} - return requests.get(url, headers) - - # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - import pytest - import mock - - def get_mock_response(): - return "test response" - - # here we're mocking the UserAgent's _get_data_from_remote method and replacing it with our get_mock_response method - # to feed our test some fake remote data - @mock.patch.object(UserAgent, '_get_data_from_remote', get_mock_response) - def test_get_remote_data(): - assert UserAgent.get_remote_Data() == "Remote response: test response" Writing Integration Tests -######################### +========================= -Integration tests are useful for testing the faults that occur between integrated units. In the context of VOLTTRON +Integration tests are useful for testing the faults that occur between integrated units. In the context of VOLTTRON agents, integration tests should test the interactions between the agent, the platform, and other agents installed on -the platform that would interface with the agent. It is typical for integration tests to test configuration, behavior +the platform that would interface with the agent. It is typical for integration tests to test configuration, behavior and content of RPC calls and agent Pub/Sub, the agent subsystems, etc. `Pytest best practices for Integration Testing `_ -The Volttrontesting directory includes several helpful fixtures for your tests. Including the following line at the top -of your tests, or in conftest.py, will allow you to utilize the platform wrapper fixtures, and more. +Volttrontesting Directory +------------------------- + +The `Volttrontesting` directory includes several helpful fixtures for your tests. Including the following line at the +top of your tests, or in `conftest.py`, will allow you to utilize the platform wrapper fixtures, and more. .. code-block:: python @@ -391,22 +423,45 @@ Here is an example success case integration test: For more integration test examples, it is recommended to take a look at some of the VOLTTRON core agents, such as historian agents and weather service agents. +Using Docker for Limited-Integration Testing +-------------------------------------------- + +If you want to run limited-integration tests which do not require the setup of a volttron system, you can use Docker +containers to mimic dependencies of an agent. The `volttrontesting/fixtures/docker_wrapper.py` module provides a +convenient function to create docker containers for use in limited-integration tests. For example, suppose that you +had an agent with a dependency on a MySQL database. If you want to test the connection between the Agent and the MySQL +dependency, you can create a Docker container to act as a real MySQL database. Below is an example: + +.. code-block:: python + + from volttrontesting.fixtures.docker_wrapper import create_container + from UserAgent import UserAgentClass + + def test_docker_wrapper_example(): + ports_config = {'3306/tcp': 3306} + with create_container("mysql:5.7", ports=ports_config) as container: + init_database(container) + agent = UserAgent(ports_config) + + results = agent.some_method_that_talks_to_container() + + Running your Tests and Debugging -################################ +================================ Pytest can be run from the command line to run a test module. -:: +.. code-block:: bash pytest -If using marks, you can add "-m " to specify your testing subset, and -s can be used to surpress standard output. -For more information about optional arguments you can type `pytest --help` into your command line interface to see the -full list of options. +If using marks, you can add ``-m `` to specify your testing subset, and -s can be used to suppress standard +output. For more information about optional arguments you can type `pytest --help` into your command line interface to +see the full list of options. Testing output should look something like this: -:: +.. code-block:: console (volttron) @:~/volttron$ pytest services/core/SQLHistorian/ ======================================================== test session starts ========================================================= @@ -453,12 +508,12 @@ Testing output should look something like this: Running Tests Via PyCharm -************************* +------------------------- -To run our Pytests using PyCharm, we'll need to create a run configuration. To do so, select "edit configurations" from +To run our Pytests using PyCharm, we'll need to create a run configuration. To do so, select "edit configurations" from the "Run" menu (or if using the toolbar UI element you can click on the run configurations dropdown to select "edit -configurations"). Use the plus symbol at the top right of the pop-up menu, scroll to "Python Tests" and expand this -menu and select "pytest". This will create a run configuration, which will then need to be filled out. We recommend the +configurations"). Use the plus symbol at the top right of the pop-up menu, scroll to "Python Tests" and expand this +menu and select "pytest". This will create a run configuration, which will then need to be filled out. We recommend the following in general: * Set the "Script Path" radio and fill the form with the path to your module. Pytest will run any tests in that diff --git a/docs/source/devguides/walkthroughs/Driver-Creation-Walkthrough.rst b/docs/source/developing-volttron/developing-drivers/driver-development-walk-through.rst similarity index 71% rename from docs/source/devguides/walkthroughs/Driver-Creation-Walkthrough.rst rename to docs/source/developing-volttron/developing-drivers/driver-development-walk-through.rst index 862d2fc494..d3e1669527 100644 --- a/docs/source/devguides/walkthroughs/Driver-Creation-Walkthrough.rst +++ b/docs/source/developing-volttron/developing-drivers/driver-development-walk-through.rst @@ -1,52 +1,56 @@ -.. _DriverCreationWalkthrough: +.. _Driver-Development-Walk-through: -Volttron Drivers Overview -========================= +=============================== +Driver Development Walk-through +=============================== -In order for Volttron agents to gather data from a device or to set device values, agents send requests to the Master -Driver Agent to read or set points. The Master Driver Agent then sends these requests on to the appropriate driver for +In order for VOLTTRON agents to gather data from a device or to set device values, agents send requests to the Master +Driver Agent to read or set points. The Master Driver Agent then sends these requests on to the appropriate driver for interfacing with that device based on the topic specified in the request and the configuration of the Master Driver. Drivers provide an interface between the device and the master driver by implementing portions of the devices' protocols needed to serve the functions of setting and reading points. -As a demonstration of developing a driver, a driver can be made to read and set points in a CSV file. This driver will +As a demonstration of developing a driver a driver can be made to read and set points in a CSV file. This driver will only differ from a real device driver in terms of the specifics of the protocol. + Create a Driver and Register class -================================== +********************************** When a new driver configuration is added to the Master Driver, the Master Driver will look for a file in its interfaces directory (services/core/MasterDriverAgent/master_driver/interfaces) that shares the name of the value specified by -"driver_type" in the configuration file. For the CSV Driver, create a file named csvdriver.py in that directory. +"driver_type" in the configuration file. For the CSV Driver, create a file named csvdriver.py in that directory. :: ├── master_driver - │   ├── agent.py - │   ├── driver.py - │   ├── __init__.py - │   ├── interfaces - │   │   ├── __init__.py - │   │   ├── bacnet.py - | | ├── csvdriver.py - │   │   └── modbus.py - │   └── socket_lock.py + │ ├── agent.py + │ ├── driver.py + │ ├── __init__.py + │ ├── interfaces + │ │ ├── __init__.py + │ │ ├── bacnet.py + | | ├── csvdriver.py + │ │ └── modbus.py + │ └── socket_lock.py ├── master-driver.agent └── setup.py + Interface Basics ----------------- +================ + A complete interface consists of two parts: the interface class and one or more register classes. Interface Class Skeleton -~~~~~~~~~~~~~~~~~~~~~~~~ -When the Master Driver processes a driver configuration file, it creates an instance of the interface class found in the -interface file (such as the one we've just created). The interface class is responsible for managing the communication -between the Volttron Platform, and the device. Each device has many registers which hold the values Volttron agents are -interested in, so generally the interface manages reading and writing to and from a device's registers. At a minimum, +------------------------ +When the Master Driver processes a driver configuration file it creates an instance of the interface class found in the +interface file (such as the one we've just created). The interface class is responsible for managing the communication +between the Volttron Platform, and the device. Each device has many registers which hold the values Volttron agents are +interested in so generally the interface manages reading and writing to and from a device's registers. At a minimum, the interface class should be configurable, be able to read and write registers, as well as read all registers with a -single request. First create the csv interface class boilerplate. +single request. First create the csv interface class boilerplate. .. code-block:: python @@ -66,18 +70,24 @@ single request. First create the csv interface class boilerplate. def _scrape_all(self): pass -This class should inherit from the BaseInterface, and at a minimum implement the configure, get_point, set_point, and +This class should inherit from the BaseInterface and at a minimum implement the configure, get_point, set_point, and scrape_all methods. -.. Note:: In some sense, drivers are sub-agents running under the same process as the Master Driver. They should be instantiated following the agent pattern, so a function to handle configuration and create the Driver object has been included. +.. Note:: + + In some sense, drivers are sub-agents running under the same process as the Master Driver. They should be + instantiated following the agent pattern, so a function to handle configuration and create the Driver object has + been included. + Register Class Skeleton -~~~~~~~~~~~~~~~~~~~~~~~ -The interface needs some information specifying the communication for each register on the device. For each different -type of register, a register class should be defined, which will help identify individual registers, and determine how -to communicate with them. Our CSV driver will be fairly basic, with one kind of "register", which will be a column in -a CSV file, however other drivers may require many kinds of registers; For instance, the Modbus protocol driver has -registers which store data in byte sized chunks, and registers which store individual bits, therefore the Modbus driver +----------------------- + +The interface needs some information specifying the communication for each register on the device. For each different +type of register a register class should be defined which will help identify individual registers and determine how +to communicate with them. Our CSV driver will be fairly basic, with one kind of "register", which will be a column in +a CSV file. Other drivers may require many kinds of registers; for instance, the Modbus protocol driver has +registers which store data in byte sized chunks and registers which store individual bits, therefore the Modbus driver has bit and byte registers. For the CSV driver, create the register class boilerplate: @@ -89,24 +99,25 @@ For the CSV driver, create the register class boilerplate: default_value=None, description=''): super(CsvRegister, self).__init__("byte", read_only, pointName, units, description=description) -This class should inherit from the BaseRegister. The class should keep register metadata, and depending upon the +This class should inherit from the BaseRegister. The class should keep register metadata, and depending upon the requirements of the protocol/device, may perform the communication. -The BACNet and Modbus drivers may be used as examples of more specific implementations. For the purpose of this -demonstration, writing and reading points will be done in the register, however, this may not always be the case (as in +The BACNet and Modbus drivers may be used as examples of more specific implementations. For the purpose of this +demonstration writing and reading points will be done in the register, however, this may not always be the case (as in the case of the BACNet driver). + Filling out the Interface class -------------------------------- +=============================== + The CSV interface will be writing to and reading from a CSV file, so the device configuration should include a path -specifying a CSV file to use as the "device". The CSV "device: path value is set at the beginning of the agent loop -which runs the configure method when the Master Driver starts. Since this Driver is for demonstration, we'll create the -CSV with some default values if the configured path doesn't exist. The CSV device will consist of 2 columns, "Point +specifying a CSV file to use as the "device". The CSV "device: path value is set at the beginning of the agent loop +which runs the configure method when the Master Driver starts. Since this Driver is for demonstration, we'll create the +CSV with some default values if the configured path doesn't exist. The CSV device will consist of 2 columns: "Point Name" specifying the name of the register, and "Point Value", the current value of the register. .. code-block:: python - _log = logging.getLogger(__name__) CSV_FIELDNAMES = ["Point Name", "Point Value"] @@ -146,11 +157,11 @@ Name" specifying the name of the register, and "Point Value", the current value writer.writerows(CSV_DEFAULT) self.parse_config(registry_config_str) -At the end of the configuration method, the Driver parses the registry configuration. The registry configuration is -a csv which is used to tell the Driver which register the user wishes to communicate with, and includes a few meta-data +At the end of the configuration method, the Driver parses the registry configuration. The registry configuration is +a csv which is used to tell the Driver which register the user wishes to communicate with and includes a few meta-data values about each register, such as whether the register can be written to, if the register value uses a specific -measurement unit, etc. After each register entry is parsed from the registry config, a register is added to the driver's -list of active registers. +measurement unit, etc. After each register entry is parsed from the registry config a register is added to the +driver's list of active registers. .. code-block:: python @@ -192,8 +203,8 @@ list of active registers. self.insert_register(register) -Since the driver's registers will be doing the work of parsing the registers, the interface only needs to select the -correct register to read from or write to, and instruct the register to perform the corresponding unit of work. +Since the driver's registers will be doing the work of parsing the registers the interface only needs to select the +correct register to read from or write to and instruct the register to perform the corresponding unit of work. .. code-block:: python @@ -216,11 +227,13 @@ correct register to read from or write to, and instruct the register to perform result[register.point_name] = register.get_state() return result + Writing the Register class --------------------------- +========================== + The CSV driver's register class is responsible for parsing the CSV, reading the corresponding rows to return the -register's current value and writing updated values into the CSV for the register. On a device which communicates via -a protocol such as Modbus, the same units of work would be done, but using pymodbus to perform the reads and writes. +register's current value and writing updated values into the CSV for the register. On a device which communicates via +a protocol such as Modbus the same units of work would be done, but using pymodbus to perform the reads and writes. Here, Python's CSV library will be used as our "protocol implementation". The Register class determines which file to read based on values passed from the Interface class. @@ -234,8 +247,8 @@ The Register class determines which file to read based on values passed from the description=description) self.csv_path = csv_path -To find its value, the register will read the CSV file, iterate over each row until a row with the point name the same -as the register name, at which point it extracts the point value, and returns it. The register should be written to +To find its value the register will read the CSV file, iterate over each row until a row with the point name the same +as the register name at which point it extracts the point value, and returns it. The register should be written to handle problems which may occur, such as no correspondingly named row being present in the CSV file. .. code-block:: python @@ -256,8 +269,8 @@ handle problems which may occur, such as no correspondingly named row being pres raise RuntimeError("CSV device at {} does not exist".format(self.csv_path)) Likewise to overwrite an existing value, the register will iterate over each row until the point name matches the -register name, saving the output as it goes. When it finds the correct row, it instead saves the output updated with the -new value, then continues on. Finally it writes the output back to the csv. +register name, saving the output as it goes. When it finds the correct row it instead saves the output updated with the +new value then continues on. Finally it writes the output back to the csv. .. code-block:: python @@ -287,13 +300,14 @@ new value, then continues on. Finally it writes the output back to the csv. writer.writerows([dict(row) for row in points]) return self.get_state() -At this point, we should be able to scrape the CSV device using the Master Driver, and set points using the actuator. +At this point we should be able to scrape the CSV device using the Master Driver and set points using the actuator. Creating Driver Configurations ------------------------------- +============================== + The configuration files for the CSV driver are very simple, but in general, the device configuration should specify -the parameters which the interface requires to communicate with the device, and the registry configuration contains -rows which correspond to registers, and specifies their usage. +the parameters which the interface requires to communicate with the device and the registry configuration contains +rows which correspond to registers and specifies their usage. Here's the driver configuration for the CSV driver: @@ -307,7 +321,10 @@ Here's the driver configuration for the CSV driver: "timezone": "UTC" } -.. Note:: the "driver_type" value must match the name of the driver's python file, as this is what the Master Driver will look for when searching for the correct interface. +.. Note:: + + The "driver_type" value must match the name of the driver's python file as this is what the Master Driver + will look for when searching for the correct interface. And here's the registry configuration: @@ -321,38 +338,47 @@ And here's the registry configuration: The BACNet and Modbus driver docs and example configurations can be used to compare these configurations to more complex configurations. + Testing your driver -=================== +******************* To test the driver's scrape all functionality, one can install a ListenerAgent and Master Driver with the driver's -configurations, and run them. To do so for the CSV driver using the configurations above: activate the Volttron +configurations, and run them. To do so for the CSV driver using the configurations above: activate the Volttron environment start the platform, tail the platform's log file, then try the following: - | python scripts/install-agent.py -s examples/ListenerAgent - | python scripts/install-agent.py -s services/core/MasterDriverAgent -c - services/core/MasterDriverAgent/master-driver.agent - | vctl config store platform.driver devices///csv_driver - | vctl config store platform.driver +.. code-block:: bash -.. Note:: "vctl config list platform.driver" will list device and registry configurations stored for the master driver and "vctl config delete platform.driver " can be used to remove a configuration entry - these commands are very useful for debugging + python scripts/install-agent.py -s examples/ListenerAgent + python scripts/install-agent.py -s services/core/MasterDriverAgent -c services/core/MasterDriverAgent/master-driver.agent + vctl config store platform.driver devices///csv_driver + vctl config store platform.driver -After the Master Driver starts, the driver's output should appear in the logs at regular intervals based on the Master +.. Note:: + + `vctl config list platform.driver` will list device and registry configurations stored for the master driver and + `vctl config delete platform.driver ` can be used to remove a configuration entry - + these commands are very useful for debugging + +After the Master Driver starts the driver's output should appear in the logs at regular intervals based on the Master Driver's configuration. + Here is some sample CSV driver output: - | 2019-11-15 10:32:00,010 (listeneragent-3.3 22996) listener.agent INFO: Peer: pubsub, Sender: platform.driver:, Bus: - | , Topic: devices/pnnl/isb1/csv_driver/all, Headers: {'Date': '2019-11-15T18:32:00.001360+00:00', 'TimeStamp': - | '2019-11-15T18:32:00.001360+00:00', 'SynchronizedTimeStamp': '2019-11-15T18:32:00.000000+00:00', - | 'min_compatible_version': '3.0', 'max_compatible_version': ''}, Message: - | [{'test1': '0', 'test2': '1', 'test3': 'testpoint'}, - | {'test1': {'type': 'integer', 'tz': 'UTC', 'units': None}, - | 'test2': {'type': 'integer', 'tz': 'UTC', 'units': None}, - | 'test3': {'type': 'integer', 'tz': 'UTC', 'units': None}}] +.. code-block:: console + + 2019-11-15 10:32:00,010 (listeneragent-3.3 22996) listener.agent INFO: Peer: pubsub, Sender: platform.driver:, Bus: + , Topic: devices/pnnl/isb1/csv_driver/all, Headers: {'Date': '2019-11-15T18:32:00.001360+00:00', 'TimeStamp': + '2019-11-15T18:32:00.001360+00:00', 'SynchronizedTimeStamp': '2019-11-15T18:32:00.000000+00:00', + 'min_compatible_version': '3.0', 'max_compatible_version': ''}, Message: + [{'test1': '0', 'test2': '1', 'test3': 'testpoint'}, + {'test1': {'type': 'integer', 'tz': 'UTC', 'units': None}, + 'test2': {'type': 'integer', 'tz': 'UTC', 'units': None}, + 'test3': {'type': 'integer', 'tz': 'UTC', 'units': None}}] This output is an indication of the basic scrape all functionality working in the Interface class - in our implementation this is also an indication of the basic functionality of the Interface class "get_point" method and Register class "get_state" methods working (although edge cases should still be tested!). -To test the Interface's "set_point" method and Register's "set_state" method, we'll need to use the Actuator agent. +To test the Interface's "set_point" method and Register's "set_state" method we'll need to use the Actuator agent. The following agent code can be used to alternate a point's value on a schedule using the actuator, as well as perform an action based on a pubsub subscription to a single point: diff --git a/docs/source/developing-volttron/development-environment/index.rst b/docs/source/developing-volttron/development-environment/index.rst new file mode 100644 index 0000000000..e76a64dc5c --- /dev/null +++ b/docs/source/developing-volttron/development-environment/index.rst @@ -0,0 +1,13 @@ +.. _Development-Environment-Setup: + +==================================== +Setting Up a Development Environment +==================================== + +An example development environment used by the VOLTTRON team would consist of a Linux VM running on the host development +machine on which an IDE would be running. The guides can be used to set up a development environment. + +.. toctree:: + + virtual_machine/install-vm + pycharm/index diff --git a/docs/source/devguides/files/00_open_pycharm.png b/docs/source/developing-volttron/development-environment/pycharm/files/00_open_pycharm.png similarity index 100% rename from docs/source/devguides/files/00_open_pycharm.png rename to docs/source/developing-volttron/development-environment/pycharm/files/00_open_pycharm.png diff --git a/docs/source/devguides/files/01_load_volttron.png b/docs/source/developing-volttron/development-environment/pycharm/files/01_load_volttron.png similarity index 100% rename from docs/source/devguides/files/01_load_volttron.png rename to docs/source/developing-volttron/development-environment/pycharm/files/01_load_volttron.png diff --git a/docs/source/devguides/files/02_set_project_interpreter.png b/docs/source/developing-volttron/development-environment/pycharm/files/02_set_project_interpreter.png similarity index 100% rename from docs/source/devguides/files/02_set_project_interpreter.png rename to docs/source/developing-volttron/development-environment/pycharm/files/02_set_project_interpreter.png diff --git a/docs/source/devguides/files/03_run_settings.png b/docs/source/developing-volttron/development-environment/pycharm/files/03_run_settings.png similarity index 100% rename from docs/source/devguides/files/03_run_settings.png rename to docs/source/developing-volttron/development-environment/pycharm/files/03_run_settings.png diff --git a/docs/source/devguides/files/04_listener_settings.png b/docs/source/developing-volttron/development-environment/pycharm/files/04_listener_settings.png similarity index 100% rename from docs/source/devguides/files/04_listener_settings.png rename to docs/source/developing-volttron/development-environment/pycharm/files/04_listener_settings.png diff --git a/docs/source/devguides/files/05_run_listener.png b/docs/source/developing-volttron/development-environment/pycharm/files/05_run_listener.png similarity index 100% rename from docs/source/devguides/files/05_run_listener.png rename to docs/source/developing-volttron/development-environment/pycharm/files/05_run_listener.png diff --git a/docs/source/devguides/files/06_run_tests.png b/docs/source/developing-volttron/development-environment/pycharm/files/06_run_tests.png similarity index 100% rename from docs/source/devguides/files/06_run_tests.png rename to docs/source/developing-volttron/development-environment/pycharm/files/06_run_tests.png diff --git a/docs/source/devguides/pycharm/files/07_run_forwarder.png b/docs/source/developing-volttron/development-environment/pycharm/files/07_run_forwarder.png similarity index 100% rename from docs/source/devguides/pycharm/files/07_run_forwarder.png rename to docs/source/developing-volttron/development-environment/pycharm/files/07_run_forwarder.png diff --git a/docs/source/devguides/pycharm/files/08_run_historian.png b/docs/source/developing-volttron/development-environment/pycharm/files/08_run_historian.png similarity index 100% rename from docs/source/devguides/pycharm/files/08_run_historian.png rename to docs/source/developing-volttron/development-environment/pycharm/files/08_run_historian.png diff --git a/docs/source/devguides/pycharm/index.rst b/docs/source/developing-volttron/development-environment/pycharm/index.rst similarity index 99% rename from docs/source/devguides/pycharm/index.rst rename to docs/source/developing-volttron/development-environment/pycharm/index.rst index 1cae39d848..c2044d04d0 100644 --- a/docs/source/devguides/pycharm/index.rst +++ b/docs/source/developing-volttron/development-environment/pycharm/index.rst @@ -1,5 +1,6 @@ .. _Pycharm-Dev-Environment: +=============================== Pycharm Development Environment =============================== diff --git a/docs/source/images/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png similarity index 100% rename from docs/source/images/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png diff --git a/docs/source/files/VOLTTRON_User_Guide.pdf b/docs/source/developing-volttron/development-environment/virtual_machine/files/VOLTTRON_User_Guide.pdf similarity index 100% rename from docs/source/files/VOLTTRON_User_Guide.pdf rename to docs/source/developing-volttron/development-environment/virtual_machine/files/VOLTTRON_User_Guide.pdf diff --git a/docs/source/images/add-chart.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/add-chart.png similarity index 100% rename from docs/source/images/add-chart.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/add-chart.png diff --git a/docs/source/devguides/eclipse/files/eclipse-marketplace.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/eclipse-marketplace.png similarity index 100% rename from docs/source/devguides/eclipse/files/eclipse-marketplace.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/eclipse-marketplace.png diff --git a/docs/source/images/edit-chart.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/edit-chart.png similarity index 100% rename from docs/source/images/edit-chart.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/edit-chart.png diff --git a/docs/source/images/example_market.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/example_market.png similarity index 100% rename from docs/source/images/example_market.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/example_market.png diff --git a/docs/source/devguides/eclipse/files/finish-import.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/finish-import.png similarity index 100% rename from docs/source/devguides/eclipse/files/finish-import.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/finish-import.png diff --git a/docs/source/devguides/eclipse/files/import-project.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/import-project.png similarity index 100% rename from docs/source/devguides/eclipse/files/import-project.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/import-project.png diff --git a/docs/source/images/install-volttron-restricted.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/install-volttron-restricted.png similarity index 100% rename from docs/source/images/install-volttron-restricted.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/install-volttron-restricted.png diff --git a/docs/source/images/linux-mint.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/linux-mint.png similarity index 100% rename from docs/source/images/linux-mint.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/linux-mint.png diff --git a/docs/source/images/login-screen.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/login-screen.png similarity index 100% rename from docs/source/images/login-screen.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/login-screen.png diff --git a/docs/source/devguides/eclipse/files/new-python-run.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/new-python-run.png similarity index 100% rename from docs/source/devguides/eclipse/files/new-python-run.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/new-python-run.png diff --git a/docs/source/images/overview.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/overview.png similarity index 100% rename from docs/source/images/overview.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/overview.png diff --git a/docs/source/files/pin-to-dashboard.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/pin-to-dashboard.png similarity index 100% rename from docs/source/files/pin-to-dashboard.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/pin-to-dashboard.png diff --git a/docs/source/images/platform-default.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/platform-default.png similarity index 100% rename from docs/source/images/platform-default.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/platform-default.png diff --git a/docs/source/devguides/eclipse/files/platform-run-config.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/platform-run-config.png similarity index 100% rename from docs/source/devguides/eclipse/files/platform-run-config.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/platform-run-config.png diff --git a/docs/source/images/platforms.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/platforms.png similarity index 100% rename from docs/source/images/platforms.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/platforms.png diff --git a/docs/source/devguides/eclipse/files/pydev-python.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/pydev-python.png similarity index 100% rename from docs/source/devguides/eclipse/files/pydev-python.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/pydev-python.png diff --git a/docs/source/files/register-new-platform-authorization.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/register-new-platform-authorization.png similarity index 100% rename from docs/source/files/register-new-platform-authorization.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/register-new-platform-authorization.png diff --git a/docs/source/images/register-new-platform.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/register-new-platform.png similarity index 100% rename from docs/source/images/register-new-platform.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/register-new-platform.png diff --git a/docs/source/images/select-branch.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/select-branch.png similarity index 100% rename from docs/source/images/select-branch.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/select-branch.png diff --git a/docs/source/devguides/eclipse/files/select-path.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/select-path.png similarity index 100% rename from docs/source/devguides/eclipse/files/select-path.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/select-path.png diff --git a/docs/source/devguides/eclipse/files/select-repo.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/select-repo.png similarity index 100% rename from docs/source/devguides/eclipse/files/select-repo.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/select-repo.png diff --git a/docs/source/devguides/eclipse/files/set-as-pydev.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/set-as-pydev.png similarity index 100% rename from docs/source/devguides/eclipse/files/set-as-pydev.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/set-as-pydev.png diff --git a/docs/source/devguides/eclipse/files/setup-python.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/setup-python.png similarity index 100% rename from docs/source/devguides/eclipse/files/setup-python.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/setup-python.png diff --git a/docs/source/images/transport-payload.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/transport-payload.png similarity index 100% rename from docs/source/images/transport-payload.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/transport-payload.png diff --git a/docs/source/images/vbox-bidirectional.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-bidirectional.png similarity index 100% rename from docs/source/images/vbox-bidirectional.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-bidirectional.png diff --git a/docs/source/images/vbox-controller.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-controller.png similarity index 100% rename from docs/source/images/vbox-controller.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-controller.png diff --git a/docs/source/images/vbox-credentials.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-credentials.png similarity index 100% rename from docs/source/images/vbox-credentials.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-credentials.png diff --git a/docs/source/images/vbox-download.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-download.png similarity index 100% rename from docs/source/images/vbox-download.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-download.png diff --git a/docs/source/images/vbox-hard-disk-xfce.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-hard-disk-xfce.png similarity index 100% rename from docs/source/images/vbox-hard-disk-xfce.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-hard-disk-xfce.png diff --git a/docs/source/images/vbox-memory-size.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-memory-size.png similarity index 100% rename from docs/source/images/vbox-memory-size.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-memory-size.png diff --git a/docs/source/images/vbox-naming.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-naming.png similarity index 100% rename from docs/source/images/vbox-naming.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-naming.png diff --git a/docs/source/images/vbox-proc-settings.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-proc-settings.png similarity index 100% rename from docs/source/images/vbox-proc-settings.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vbox-proc-settings.png diff --git a/docs/source/images/vc-run-demo.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/vc-run-demo.png similarity index 100% rename from docs/source/images/vc-run-demo.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/vc-run-demo.png diff --git a/docs/source/devguides/eclipse/files/volttron-console.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-console.png similarity index 100% rename from docs/source/devguides/eclipse/files/volttron-console.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-console.png diff --git a/docs/source/images/volttron-main-args.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-main-args.png similarity index 100% rename from docs/source/images/volttron-main-args.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-main-args.png diff --git a/docs/source/devguides/eclipse/files/volttron-main.png b/docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-main.png similarity index 100% rename from docs/source/devguides/eclipse/files/volttron-main.png rename to docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-main.png diff --git a/docs/source/images/volttron-webimage.jpg b/docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-webimage.jpg similarity index 100% rename from docs/source/images/volttron-webimage.jpg rename to docs/source/developing-volttron/development-environment/virtual_machine/files/volttron-webimage.jpg diff --git a/docs/source/setup/install-vm.rst b/docs/source/developing-volttron/development-environment/virtual_machine/install-vm.rst similarity index 76% rename from docs/source/setup/install-vm.rst rename to docs/source/developing-volttron/development-environment/virtual_machine/install-vm.rst index b8c2c6d002..23f844ea21 100644 --- a/docs/source/setup/install-vm.rst +++ b/docs/source/developing-volttron/development-environment/virtual_machine/install-vm.rst @@ -1,8 +1,8 @@ -.. _install-vm: +.. _Install-VM: -================================ -Installing Linux Virtual Machine -================================ +================================== +Installing a Linux Virtual Machine +================================== VOLTTRON requires a Linux system to run. For Windows users this will require a virtual machine (VM). @@ -12,7 +12,7 @@ https://www.virtualbox.org/wiki/Downloads. |VirtualBox Download| -.. |VirtualBox Download| image:: images/vbox-download.png +.. |VirtualBox Download| image:: files/vbox-download.png After installing VirtualBox download a virtual box appliance from https://www.osboxes.org/linux-mint/ extract the VDI from the downlaoded archive, **or** download a system installation disk. VOLTTRON version 7.0.x has been tested @@ -27,37 +27,37 @@ as an example, however platform setup in Ubuntu should be identical. Adding a VDI Image to VirtualBox Environment --------------------------------------------- +******************************************** |Linux Mint| -.. |Linux Mint| image:: images/linux-mint.png +.. |Linux Mint| image:: files/linux-mint.png The below info holds the VM's preset username and password. |Linux Mint Credentials| -.. |Linux Mint Credentials| image:: images/vbox-credentials.png +.. |Linux Mint Credentials| image:: files/vbox-credentials.png Create a new VirtualBox Image. |VirtualBox VM Naming| -.. |VirtualBox VM Naming| image:: images/vbox-naming.png +.. |VirtualBox VM Naming| image:: files/vbox-naming.png Select the amount of RAM for the VM. The recommended minimum is shown in the image below: |VirtualBox Memory Size Selection| -.. |VirtualBox Memory Size Selection| image:: images/vbox-memory-size.png +.. |VirtualBox Memory Size Selection| image:: files/vbox-memory-size.png Specify the hard drive image using the extracted VDI file. |VirtualBox Hard Disk| -.. |VirtualBox Hard Disk| image:: images/vbox-hard-disk-xfce.png +.. |VirtualBox Hard Disk| image:: files/vbox-hard-disk-xfce.png With the newly created VM selected, choose Machine from the VirtualBox menu in the top left corner of the VirtualBox window; from the drop down menu, choose Settings. @@ -67,7 +67,7 @@ Drag’n’Drop as Bidirectional. |VirtualBox Bidirectional| -.. |VirtualBox Bidirectional| image:: images/vbox-bidirectional.png +.. |VirtualBox Bidirectional| image:: files/vbox-bidirectional.png .. note:: Currently, this feature only works under certain circumstances (e.g. copying / pasting text). @@ -76,7 +76,7 @@ Go to System Settings. In the processor tab, set the number of processors to two |VirtualBox Processors| -.. |VirtualBox Processors| image:: images/vbox-proc-settings.png +.. |VirtualBox Processors| image:: files/vbox-proc-settings.png Go to Storage Settings. Confirm that the Linux Mint VDI is attached to Controller: SATA. @@ -87,9 +87,7 @@ Go to Storage Settings. Confirm that the Linux Mint VDI is attached to Controlle |VirtualBox Controller| -.. |VirtualBox Controller| image:: images/vbox-controller.png +.. |VirtualBox Controller| image:: files/vbox-controller.png Start the machine by saving these changes and clicking the “Start” arrow located on the upper left hand corner of the main VirtualBox window. - - diff --git a/docs/source/devguides/supporting/files/jupyter_notebooks.jpg b/docs/source/developing-volttron/jupyter/files/jupyter_notebooks.jpg similarity index 100% rename from docs/source/devguides/supporting/files/jupyter_notebooks.jpg rename to docs/source/developing-volttron/jupyter/files/jupyter_notebooks.jpg diff --git a/docs/source/developing-volttron/jupyter/jupyter-notebooks.rst b/docs/source/developing-volttron/jupyter/jupyter-notebooks.rst new file mode 100644 index 0000000000..5dae9152d3 --- /dev/null +++ b/docs/source/developing-volttron/jupyter/jupyter-notebooks.rst @@ -0,0 +1,234 @@ +.. _Jupyter-Notebooks: + +================= +Jupyter Notebooks +================= + +Jupyter is an open-source web application that lets you create and share “notebook” documents. A notebook displays +formatted text along with live code that can be executed from the browser, displaying the execution output and +preserving it in the document. Notebooks that execute Python code used to be called `iPython Notebooks`. The iPython +Notebook project has now merged into Project Jupyter. + + +Using Jupyter to Manage a Set of VOLTTRON Servers +------------------------------------------------- + +The following Jupyter notebooks for VOLTTRON have been provided as examples: + + - **Collector notebooks**. Each Collector notebook sets up a particular type of device driver + and forwards device data to another VOLTTRON instance, the Aggregator. + + - **SimulationCollector notebook**. This notebook sets up a group of Simulation device drivers + and forwards device data to another VOLTTRON instance, the Aggregator. + - **BacnetCollector notebook**. This notebook sets up a Bacnet (or Bacnet gateway) device driver + and forwards device data to another VOLTTRON instance, the Aggregator. + - **ChargePointCollector notebook**. This notebook sets up a ChargePoint device driver + and forwards device data to another VOLTTRON instance, the Aggregator. + - **SEP2Collector notebook**. This notebook sets up a SEP2.0 (IEEE 2030.5) device driver + and forwards device data to another VOLTTRON instance, the Aggregator. + The Smart Energy Profile 2.0 ("SEP2") protocol implements IEEE 2030.5, and is capable + of connecting a wide array of smart energy devices to the Smart Grid. The standard is + designed to run over TCP/IP and is physical layer agnostic. + + - **Aggregator notebook**. This notebook sets up and executes aggregation of forwarded data + from other VOLTTRON instances, using a historian to record the data. + - **Observer notebook**. This notebook sets up and executes a DataPuller that captures data from + another VOLTTRON instance, using a Historian to record the data. It also uses the + Message Debugger agent to monitor messages flowing across the VOLTTRON bus. + +Each notebook configures and runs a set of VOLTTRON Agents. When used as a set they implement a +multiple-VOLTTRON-instance architecture that captures remote device data, aggregates it, and reports on it, routing the +data as follows: + +.. image:: files/jupyter_notebooks.jpg + + +Install VOLTTRON and Jupyter on a Server +---------------------------------------- + +The remainder of this guide describes how to set up a host for VOLTTRON and Jupyter. Use this setup process on a server +in order to prepare it to run Jupyter notebook for VOLTTRON. + + +Set Up the Server and Install VOLTTRON +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following is a complete, but terse, description of the steps for installing and running VOLTTRON on a server. For +more detailed, general instructions, see :ref:`Installing Volttron `. + +The VOLTTRON server should run on the same host as the Jupyter server. + +* Load third-party software: + +.. code-block:: bash + + $ sudo apt-get update + $ sudo apt-get install build-essential python-dev openssl libssl-dev libevent-dev git + $ sudo apt-get install sqlite3 + +* Clone the VOLTTRON repository from github: + +.. code-block:: bash + + $ cd ~ + $ mkdir repos + $ cd repos + $ git clone https://github.com/VOLTTRON/volttron/ + +* Check out the develop (or master) branch and bootstrap the development environment: + +.. code-block:: bash + + $ cd volttron + $ git checkout develop + $ python bootstrap.py + +* Activate and initialize the VOLTTRON virtual environment: + + Run the following each time you open a new command-line shell on the server: + + .. code-block:: bash + + $ export VOLTTRON_ROOT=~/repos/volttron + $ export VOLTTRON_HOME=~/.volttron + $ cd $VOLTTRON_ROOT + $ source env/bin/activate + + +Install Extra Libraries +^^^^^^^^^^^^^^^^^^^^^^^ + +* Add Python libraries to the VOLTTRON virtual environment: + +These notebooks use third-party software that's not included in VOLTTRON's standard distribution that was loaded by +`bootstrap.py`. The following additional packages are required: + +- Jupyter +- SQLAlchemy (for the Message Debugger) +- Suds (for the ChargePoint driver, if applicable) +- Numpy and MatPlotLib (for plotted output) + +.. Note:: + + A Jupyter installation also installs and/or upgrades many dependent libraries. Doing so could disrupt other work on + the OS, so it’s safest to load Jupyter (and any other library code) in a virtual environment. VOLTTRON runs in a + virtual environment during normal operation, so if you're using Jupyter in conjunction with VOLTTRON, it should be + installed in your VOLTTRON virtual environment (In other words, be sure to use `cd $VOLTTRON_ROOT` and + `source env/bin/activate` to activate the virtual environment before running ``pip install``.) + +* Install the third-party software: + +.. code-block:: bash + + $ pip install SQLAlchemy==1.1.4 + $ pip install suds-jurko==0.6 + $ pip install numpy + $ pip install matplotlib + $ pip install jupyter + +.. Note:: + + If `pip install` fails due to an untrusted cert, try using this command instead: + + .. code-block:: bash + + $ pip install --trusted-host pypi.python.org + + An InsecurePlatformWarning may be displayed, but it typically won't stop the installation from proceeding. + + +Configure VOLTTRON +------------------ + +Use the `vcfg` wizard to configure the VOLTTRON instance. By default, the wizard configures a VOLTTRON instance that +communicates with agents only on the local host (ip 127.0.0.1). This set of notebooks manages communications among +multiple VOLTTRON instances on different hosts. To enable this cross-host communication on VOLTTRON's web server, +replace 127.0.0.1 with the host's IP address, as follows: + +.. code-block:: bash + + $ vcfg + +Accept all defaults, except as follows: + +- If a prompt defaults to 127.0.0.1 as an IP address, substitute the `host's IP address` (this may happen multiple + times). +- When asked whether this is a volttron central, answer `Y`. +- When prompted for a username and password, use `admin` and `admin`. + + +Start VOLTTRON +-------------- + +Start the main VOLTTRON process, logging to $VOLTTRON_ROOT/volttron.log: + +.. code-block:: bash + + $ volttron -vv -l volttron.log --msgdebug + +This runs VOLTTRON as a foreground process. To run it in the background, use: + +.. code-block:: + + $ ./start-volttron --msgdebug + +This also enables the Message Debugger, a non-production VOLTTRON debugging aid that's used by some notebooks. To run +with the Message Debugger disabled (VOLTTRON's normal state), omit the ``--msgdebug`` flag. + +Now that VOLTTRON is running, it's ready for agent configuration and execution. Each Jupyter notebook contains detailed +instructions and executable code for doing that. + + +Configure Jupyter +----------------- + +More detailed information about installing, configuring and using Jupyter Notebooks is available on the Project Jupyter +site, http://jupyter.org/. + +* Create a Jupyter configuration file: + +.. code-block:: bash + + $ jupyter notebook --generate-config + +* Revise the Jupyter configuration: + +Open `~/.jupyter/jupyter_notebook_config.py` in your favorite text editor. Change the configuration to accept +connections from any IP address (not just from localhost) and use a specific, non-default port number: + +- Un-comment ``c.NotebookApp.ip`` and set it to: ``*`` instead of ``localhost`` +- Un-comment ``c.NotebookApp.port`` and set it to: ``8891`` instead of ``8888`` + +Save the config file. + +* Open ports for TCP connections: + +Make sure that your Jupyter server host's security rules allow inbound TCP connections on port `8891`. + +If the VOLTTRON instance needs to receive TCP requests, for example ForwardHistorian or DataPuller messages from other +VOLTTRON instances, make sure that the host's security rules also allow inbound TCP communications on VOLTTRON's port, +which is usually `22916`. + + +Launch Jupyter +-------------- + +* Start the Jupyter server: + +In a separate command-line shell, set up VOLTTRON's environment variables and virtual environment, and then launch the +Jupyter server: + +.. code-block:: bash + + $ export VOLTTRON_HOME=(your volttron home directory, e.g. ~/.volttron) + $ export VOLTTRON_ROOT=(where volttron was installed; e.g. ~/repos/volttron) + $ cd $VOLTTRON_ROOT + $ source env/bin/activate + $ cd examples/JupyterNotebooks + $ jupyter notebook --no-browser + +* Open a Jupyter client in a web browser: + +Look up the host's IP address (e.g., using ifconfig). Open a web browser and navigate to the URL that was displayed when +you started jupyter, replacing `localhost` with that IP address. A Jupyter web page should display, listing your +notebooks. diff --git a/docs/source/developing-volttron/python-for-matlab-users.rst b/docs/source/developing-volttron/python-for-matlab-users.rst new file mode 100644 index 0000000000..289c874be2 --- /dev/null +++ b/docs/source/developing-volttron/python-for-matlab-users.rst @@ -0,0 +1,194 @@ +.. _Python-for-Matlab-Users: + +======================= +Python for Matlab Users +======================= + +Matlab is a popular proprietary programming language and tool suite with built in support for matrix operations and +graphically plotting computation results. The purpose of this document is to introduce Python to those already familiar +Matlab so it will be easier for them to develop tools and agents in VOLTTRON. + + +A Simple Function +----------------- + +Python and Matlab are similar in many respects, syntactically and semantically. With the addition of the NumPy library +in Python, almost all numerical operations in Matlab can be emulated or directly translated. Here are functions in each +language that perform the same operation: + +.. code-block:: matlab + + % Matlab + function [result] = times_two(number) + result = number * 2; + end + +.. code-block:: python + + # Python + def times_two(number): + result = number * 2 + return result + +Some notes about the previous functions: + +#. Values are explicitly returned with the `return` statement. It is possible to return multiple values, as in Matlab, + but doing this without a good reason can lead to overcomplicated functions. + +#. Semicolons are not used to end statements in python, and white space is significant. After a block is started (if, + for, while, functions, classes) subsequent lines should be indented with four spaces. The block ends when the + programmer stops adding the extra level of indentation. + + +Translating +----------- + +The following may be helpful if you already have a Matlab file or function that will be translated into Python. Many of +the syntax differences between Matlab and Python can be rectified with your text editor's find and replace feature. + +Start by copying all of your Matlab code into a new file with a `.py` extension. It is recommended to start by +commenting everything out and uncommenting the Matlab code in chunks. This way it is possible to write valid Python and +verify it as you translate, instead of waiting till the whole file is "translated". Editors designed to work with +Python should be able to highlight syntax errors as well. + +#. Comments are created with a `%`. Find and replace these with `#`. + +.. code-block:: python + + def test_function(): + # single line Python comment + """ + Multi-line Python comment + """ + pass # inline Python comment + +#. Change `elseif` blocks to `elif` blocks. + +.. code-block:: python + + if thing == 0: + do_thing1() + elif thing ==1: + do_thing2() + else: + do_the_last_thing() + +#. Python indexes start at zero instead of one. Array slices and range operations don't include the upper bound, so + only the lower bound should decrease by one. The following examples are of Python code in the console: + +.. code-block:: console + + >>> test_array = [0, 1, 2, 3, 4] + >>> test_array[0] + 0 + >>> test_array[1] + 1 + >>> test_array[0:2] + [0, 1] + >>>>>> test_array[:2] + [0, 1] + >>> test_array[2:] + [2, 3, 4] + >>> + +#. Semicolons in Matlab are used to suppress output at the end of lines and for organizing array literals. After + arranging the arrays into nested lists, all semicolons can be removed. + +#. The `end` keyword in Matlab is used both to access the last element in an array and to close blocks. The array use + case can be replaced with `-1` and the others can be removed entirely. + +.. code-block:: console + + >>> test_array = [0, 1, 2, 3, 4] + >>> test_array[-1] + 4 + >>> + + +A More Concrete Example +^^^^^^^^^^^^^^^^^^^^^^^ + +In the `Building Economic Dispatch `_ project, a sibling project to VOLTTRON, +a number of components written in Matlab would create a matrix out of some collection of columns and perform least +squares regression using the `matrix division` operator. This is straightforward and very similar in both languages +assuming that all of the columns are defined and are the same length. + +.. code-block:: matlab + + % Matlab + XX = [U, xbp, xbp2, xbp3, xbp4, xbp5]; + AA = XX \ ybp; + +.. code-block:: python + + # Python + import numpy as np + + XX = np.column_stack((U, xbp, xbp2, xbp3, xbp4, xbp5)) + AA, resid, rank, s = np.linalg.lstsq(XX, ybp) + +This pattern also included the creation of the `U` column, a column of ones used as the bias term in the linear equation +. In order to make the Python version more readable and more robust, the pattern was removed from each component and +replaced with a single function call to `least_squares_regression`. + +This function does some validation on the input parameters, automatically creates the bias column, and returns the least +squares solution to the system. Now if we want to change how the solution is calculated we only have to change the one +function, instead of each instance where the pattern was written originally. + +.. code-block:: python + + def least_squares_regression(inputs=None, output=None): + if inputs is None: + raise ValueError("At least one input column is required") + if output is None: + raise ValueError("Output column is required") + + if type(inputs) != tuple: + inputs = (inputs,) + + ones = np.ones(len(inputs[0])) + x_columns = np.column_stack((ones,) + inputs) + + solution, resid, rank, s = np.linalg.lstsq(x_columns, output) + return solution + +Lessons Learned (sometimes the hard way) +---------------------------------------- + + +Variable Names +^^^^^^^^^^^^^^ + +Use descriptive function and variable names whenever possible. The most important things to consider here are reader +comprehension and searching. Consider a variable called `hdr`. Is it `header` without any vowels, or is it short for +`high-dynamic-range`? Spelling out full words in variable names can save someone else a lot of guesswork. + +Searching comes in when we're looking for instances of a string or variable. Single letter variable names are +impossible to search for. Variables names describing the value being stored in a concise but descriptive manner are +preferred. + + +Matlab load/save +^^^^^^^^^^^^^^^^ + +Matlab has built-in functions to automatically save and load variables from your programs to disk. Using these +functions can lead to poor program design and should be avoided if possible. It would be best to refactor as you +translate if they are being used. Few operations are so expensive that that cannot be redone every time the program is +run. For part of the program that saves variables, consider making a function that simply returns them instead. + +If your Matlab program is loading csv files then use the Pandas library when working in python. Pandas works well with +NumPy and is the go-to library when using csv files that contain numeric data. + + +More Resources +-------------- + +`NumPy for Matlab Users +`_ +Has a nice list of common operations in Matlab and NumPy. + +`NumPy Homepage +`_ + +`Pandas Homepage +`_ diff --git a/docs/source/devguides/agent_development/Develop-Driver-Agent.rst b/docs/source/devguides/agent_development/Develop-Driver-Agent.rst deleted file mode 100644 index 24da1aa0b5..0000000000 --- a/docs/source/devguides/agent_development/Develop-Driver-Agent.rst +++ /dev/null @@ -1,260 +0,0 @@ -.. _Develop-Driver-Agent: - -Driver Development -================== - -Introduction ------------- - -All Voltton drivers are implemented through the :doc:`Master Driver -Agent <../../core_services/drivers/Driver-Configuration>` and are technically sub-agents running in -the same process as the :doc:`Master Driver -Agent <../../core_services/drivers/Driver-Configuration>`. -Each of these driver sub-agents is responsible for creating an interface -to a single device. Creating that interface is facilitated by an -instance of an interface class. Currently there are two interface -classes included: :ref:`Modbus ` and -:ref:`BACnet `. - -Existing Drivers ----------------- - -In the directory for the Master Driver Agent you'll see a directory -called interfaces: - -:: - - ├── master_driver - │   ├── agent.py - │   ├── driver.py - │   ├── __init__.py - │   ├── interfaces - │   │   ├── __init__.py - │   │   ├── bacnet.py - │   │   └── modbus.py - │   └── socket_lock.py - ├── master-driver.agent - └── setup.py - -The files bacnet.py and modbus.py implement the interface class for each -respective protocol. (The BACnet interface is mostly just a pass-though -to the :ref:`BACnet Proxy Agent `, but the Modbus -interface is self contained.) - -Looking at those two files is a good introduction into how they work. - -The file name is used when configuring a driver to determine which -interface to use. The name of the interface class in the file must be -called Interface. - -.. note:: - - Developing a new driver does not require that your code live with the - MasterDriverAgent code. You may create the interface file anywhere that - you would like and then create a symbolic link to the interface file - in the interfaces directory. When the MasterDriverAgent is packed for - distribution the a copy of the file represented by the symbolic link - is packed into the agent wheel. - See :ref:`Using Third Party Drivers ` - -Interface Basics ----------------- - -A complete interface consists of two parts: One or more register classes -and the interface class. - -Register Class -~~~~~~~~~~~~~~ - -The Base Interface class uses a Register class to describe the registers -of a device to the driver sub-agent. This class is commonly sub-classed -to store protocol specific information for the interface class to use. -For example, the BACnet interface uses a sub-classed base register to -store the instance number, object type, and property name of the point -on the device represented by the register class. The Modbus interface -uses several different Register classes to deal with the different types -of registers on Modbus devices and their different needs. - -The register class contains the following attributes: - -- **read\_only** - True or False -- **register\_type** - "bit" or "byte", used by the driver sub-agent to - help deduce some meta data about the point. -- **point\_name** - Name of the point on the device. Used by the base - interface for reference. -- **units** - units of the value, meta data for the driver -- **description** - meta data for the driver -- **python\_type** - python type of the point, used to produce meta - data. This must be set explicitly otherwise it default to int. - -Here is an example of a Registry Class for the BACnet driver: - -:: - - class Register(BaseRegister): - def __init__(self, instance_number, object_type, property_name, read_only, pointName, units, description = ''): - super(Register, self).__init__("byte", read_only, pointName, units, description = '') - self.instance_number = int(instance_number) - self.object_type = object_type - self.property = property_name - -Note that this implementation is incomplete. It does not properly set -the register\_type or python\_type. - -Interface Class -~~~~~~~~~~~~~~~ - -The Interface Class is what is instantiated by the driver sub-agent to -do it's work. - -configure(self, config\_dict, registry\_config\_str) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This method must be implemented by an Interface implementation. - -- **config\_dict** is a dictionary of key values pairs from the - configuration file's "driver\_config" section. -- **registry\_config\_str** is the contents of the "registry\_config" - entry in the driver configuration file. It is up to the Interface - class to parse this file according to the needs of the driver. - -Here is an example taken from the :ref:`BACnet ` driver: - -:: - - def configure(self, config_dict, registry_config_str): - self.parse_config(registry_config_str) #Parse the configuration string. - self.target_address = config_dict["device_address"] - self.proxy_address = config_dict.get("proxy_address", "platform.bacnet_proxy") - self.ping_target(self.target_address) #Establish routing to the device if needed. - -And here is the parse\_config method (See :ref:`BACnet Registry -Configuration `: - -:: - - def parse_config(self, config_string): - if config_string is None: - return - - f = StringIO(config_string) #Python's CSV file parser wants a file like object. - - configDict = DictReader(f) #Parse the CVS file contents. - - for regDef in configDict: - #Skip lines that have no address yet. - if not regDef['Point Name']: - continue - - io_type = regDef['BACnet Object Type'] - read_only = regDef['Writable'].lower() != 'true' - point_name = regDef['Volttron Point Name'] - index = int(regDef['Index']) - description = regDef['Notes'] - units = regDef['Units'] - property_name = regDef['Property'] - - register = Register(index, - io_type, - property_name, - read_only, - point_name, - units, - description = description) - - self.insert_register(register) - -Once a register is created it must be added with the insert\_register -method. - -get\_point(self, point\_name) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This method must be implemented by an Interface implementation. - -Gets the value of a point from a device and returns it. - -Here is a simple example from the BACnet driver. In this case it only -has to pass the work on to the BACnet Proxy Agent for handling. - -:: - - def get_point(self, point_name): - register = self.get_register_by_name(point_name) - point_map = {point_name:[register.object_type, - register.instance_number, - register.property]} - result = self.vip.rpc.call(self.proxy_address, 'read_properties', - self.target_address, point_map).get() - return result[point_name] - -Failure should be indicated by a useful exception being raised. (In this -case the we just leave the Exception raised by the BACnet proxy -un-handled. This could be improved with better handling when register -that does not exist is requested.) - -The Register instance for the point can be retrieved with -self.get\_register\_by\_name(point\_name) - -set\_point(self, point\_name, value) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This method must be implemented by an Interface implementation. - -Sets the value of a point on a device and ideally returns the actual -value set if different. - -Here is a simple example from the BACnet driver. In this case it only -has to pass the work on to the BACnet Proxy Agent for handling. - -:: - - def set_point(self, point_name, value): - register = self.get_register_by_name(point_name) - if register.read_only: - raise IOError("Trying to write to a point configured read only: "+point_name) - args = [self.target_address, value, - register.object_type, - register.instance_number, - register.property] - result = self.vip.rpc.call(self.proxy_address, 'write_property', *args).get() - return result - -Failure to raise a useful exception being raised. (In this case the we -just leave the Exception raised by the BACnet proxy un-handled unless -the point is read only.) - -scrape\_all(self) -^^^^^^^^^^^^^^^^^ - -This method must be implemented by an Interface implementation. - -This must return a dictionary mapping point names to values for ALL -registers. - -Here is a simple example from the BACnet driver. In this case it only -has to pass the work on to the BACnet Proxy Agent for handling. - -:: - - def scrape_all(self): - point_map = {} - read_registers = self.get_registers_by_type("byte", True) - write_registers = self.get_registers_by_type("byte", False) - for register in read_registers + write_registers: - point_map[register.point_name] = [register.object_type, - register.instance_number, - register.property] - - result = self.vip.rpc.call(self.proxy_address, 'read_properties', - self.target_address, point_map).get() - return result - -self.get\_registers\_by\_type allows you to get lists of registers by -their type and if they are read only. (As BACnet currently only uses -"byte", "bit" is ignored.) As the procedure for handling all the -different types in BACnet is the same we can bundle them all up into a -single request from the proxy. - -In the Modbus protocol the distinction is important and so each category -must be handled differently. diff --git a/docs/source/devguides/agent_development/Developing-Historian-Agents.rst b/docs/source/devguides/agent_development/Developing-Historian-Agents.rst deleted file mode 100644 index 178c645eff..0000000000 --- a/docs/source/devguides/agent_development/Developing-Historian-Agents.rst +++ /dev/null @@ -1,112 +0,0 @@ -.. _Developing-Historian-Agents: - -Developing Historian Agents -=========================== - -VOLTTRON provides a convenient base class for developing new historian -agents. The base class automatically subscribes to all pertinent topics, -cache published data to disk until it is successfully recorded to a -historian, create the public facing interface for querying results, and -spells out a simple interface for concrete implementation to meet to -make a working Historian Agent. The VOLTTRON provides support for -several :ref:`historians ` without modification. -Please use one of these if it fits your project criteria, otherwise -continue reading. - -The base class also breaks data to publish into reasonably sized chunks -before handing it off to the concrete implementation for publication. -The size of the chunk is configurable. - -The base class sets up a separate thread for publication. This way if -publication code needs to block for a long period of time (up to 10s of -seconds) this will no disrupt the collection of data from the bus or the -functioning of the agent itself. - -BaseHistorian -------------- - -All Historians must inherit from the BaseHistorian class in -volttron.platform.agent.base\_historian and implement the following -methods: - -publish\_to\_historian(self, to\_publish\_list) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This method is called by the BaseHistorian class when it has received -data from the message bus to be published. to\_publish\_list is a list -of records to publish in the form - -:: - - [ - { - '_id': 1, - 'timestamp': timstamp, - 'source': 'scrape', - 'topic': 'campus/building/unit/point', - 'value': 90, - 'meta': {'units':'F'} - } - { - ... - } - ] - -- \_id - ID of record. All IDs in the list are unique. This is used for - internal record tracking. -- timestamp - Python datetime object of the time data was published at - timezone UTC -- source - Source of the data. Can be scrape, analysis, log, or - actuator. -- topic - Topic data was published on. Prefix's such as "device" are - dropped. -- value - Value of the data. Can be any type. -- meta - Metadata for the value. Some sources will omit this entirely. - -For each item in the list the concrete implementation should attempt to -publish (or discard if non-publishable) every item in the list. -Publication should be batched if possible. For every successfully -published record and every record that is to be discarded because it is -non-publishable the agent must call report\_handled on those records. -Records that should be published but were not for whatever reason -require no action. Future calls to publish\_to\_historian will include -these unpublished records. publish\_to\_historian is always called with -the oldest unhandled records. This allows the historian to no lose data -due to lost connections or other problems. - -As a convenience report\_all\_handled can be called if all of the items -in published\_list were successfully handled. - -query\_topic\_list(self) -~~~~~~~~~~~~~~~~~~~~~~~~ - -Must return a list of all unique topics published. - -query\_historian(self, topic, start=None, end=None, skip=0, count=None, order=None) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This function must return the results of a query in the form: - -:: - - {"values": [(timestamp1: value1), (timestamp2: value2), ...], - "metadata": {"key1": value1, "key2": value2, ...}} - -metadata is not required (The caller will normalize this to {} for you -if you leave it out) - -- topic - the topic the user is querying for. -- start - datetime of the start of the query. None for the beginning of - time. -- end - datetime of the end of of the query. None for the end of time. -- skip - skip this number of results (for pagination) -- count - return at maximum this number of results (for pagination) -- order - "FIRST\_TO\_LAST" for ascending time stamps, - "LAST\_TO\_FIRST" for descending time stamps. - -historian\_setup(self) -~~~~~~~~~~~~~~~~~~~~~~ - -Implementing this is optional. This function is run on the same thread -as the rest of the concrete implementation at startup. It is meant for -connection setup. diff --git a/docs/source/devguides/agent_development/Eclipse.rst b/docs/source/devguides/agent_development/Eclipse.rst deleted file mode 100644 index cb383bdda4..0000000000 --- a/docs/source/devguides/agent_development/Eclipse.rst +++ /dev/null @@ -1,701 +0,0 @@ -.. _Eclipse: - -Agent Development in Eclipse -============================ - -The Eclipse IDE (integrated development environment), while not required for agent development, can be a powerful developmental tool. Download the IDE from the following links. Choose a download mirror closest to your location. - -`For 32-bit machines `_ - -`For 64-bit machines `_ - -To go to the main Eclipse webpage, go to `http://eclipse.org/ `_ - -Installing Eclipse ------------------- - -To install Eclipse, enter the following commands in a terminal: - -1. Install Eclipse dependency: - -:: - - # apt-get install openjdk-7-jdk - -2. After downloading the eclipse archive file, move the package to the opt directory (enter this command from a terminal in the directory where eclipse was downloaded): - -:: - - $ tar -xvf eclipse-java-mars-R-linux-gtk-x86_64.tar.gz - # mv eclipse /opt/ - -- For 32-bit machines, replace “gtk-x86_64” with “linux-gtk” in the previous command. - -3. Create desktop shortcut: - -:: - - # touch /usr/share/applications/eclipse.desktop - # nano /usr/share/applications/eclipse.desktop - -Enter the following text, as shown in Figure 1, and save the file. To avoid typos, copy and paste the following: - -:: - - [Desktop Entry] - Name=Eclipse - Type=Application - Exec=/opt/eclipse/eclipse - Terminal=false - Icon=/opt/eclipse/icon.xpm - Comment=Integrated Development Environment - NoDisplay=false - Categories=Development;IDE - Name[en]=eclipse - -.. image:: files/1-eclipse-desktop.jpg - -Figure 1. Eclipse Desktop File - -4. Copy the shortcut to the desktop: - -:: - - $ cp /usr/share/applications/eclipse.desktop ~/Desktop/ - -Eclipse is now installed and ready to use. - -Installing Pydev and EGit Eclipse Plug-ins ------------------------------------------- -The transactional network code is stored in a Git repository. A plug-in is available for Eclipse that makes development more convenient (note: you must have Git installed on the system and have built the project). - -1. Select Help. Select Install New Software (Figure 2). - -.. image:: files/2-egit-plugin.jpg - -Figure 2. Installing Eclipse EGit Plugin - -2. Click the Add button (Figure 3). - -.. image:: files/3-egit-plugin.jpg - -Figure 3. Installing Eclipse EGit Plugin (continued) - -3. As shown in Figure 4, enter the following information: - - For name use: EGit - - For location: http://download.eclipse.org/egit/updates - -.. image:: files/4-egit-plugin.jpg - -Figure 4. Installing Eclipse Egit Plugin (continued) - -4. After clicking OK, check the Select All button. -5. Click through Next > Agree to Terms > Finish. Allow Eclipse to restart. - -6. After installing Eclipse, you must add the PyDev plug-in to the environment. - -In Eclipse: - -- Select Help and select Install New Software. -- Click the Add button. -- As shown in Figure 5, enter the following information: - - For name use: PyDev - - For location: http://pydev.org/updates - - Click OK. - -.. image:: files/5-install-eclipse-pydev-plugin.jpg - -Figure 5. Installing Eclipse PyDev Plugin - -7. Check the box for PyDev. -8. Click through Next > Agree to Terms > Finish. Allow Eclipse to restart. - -Checkout VOLTTRON Project -------------------------- -VOLTTRON can be imported into Eclipse from an existing VOLTTRON project (VOLTTRON was previously checked out from GitHub) or a new download from GitHub. - -Import VOLTTRON into Eclipse from an Existing Local Repository (Previously Downloaded VOLTTRON Project) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To import an existing VOLTTRON project into Eclipse, complete the following steps: - -1. Select File and select Import (Figure 6). - -.. image:: files/6-check-volttron-with-eclipse.jpg - -Figure 6. Checking VOLTTRON with Eclipse from Local Source - -2. Select Git. Select Projects from Git. Click the Next button (Figure 7). - -.. image:: files/7-check-volttron-with-eclipse.jpg - -Figure 7. Checking VOLTTRON with Eclipse from Local Source (continued) - -3. Select Existing local repository and click the Next button (Figure 8). - -.. image:: files/8-check-volttron-with-eclipse.jpg - -Figure 8. Checking VOLTTRON with Eclipse from Local Source (continued) - -4. Select Add (Figure 9). - -.. image:: files/9-check-volttron-with-eclipse.jpg - -Figure 9. Checking VOLTTRON with Eclipse from Local Source (continued) - -5. Select Browse. Navigate to the top-level base VOLTTRON directory. Select OK (Figure 10). - -.. image:: files/10-check-volttron-with-eclipse.jpg - -Figure 10. Checking Out VOLTTRON with Eclipse from Local Source (continued) - -6. Click Finish (Figure 11). - -.. image:: files/11-check-volttron-with-eclipse.jpg - -Figure 11. Checking Out VOLTTRON with Eclipse from Local Source (continued) - -7. Click Next (Figure 12). - -.. image:: files/12-check-volttron-with-eclipse.jpg - -Figure 12. Checking Out VOLTTRON with Eclipse from Local Source (continued) - -8. Select Import as general project. Click Next. Click Finish (Figure 13). The project will be imported into the workspace. - -.. image:: files/13-check-volttron-with-eclipse.jpg - -Figure 13. Checking Out VOLTTRON with Eclipse from Local Source (continued) - -Import New VOLTTRON Project from GitHub -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To import a new VOLTTRON project directly from GitHub into Eclipse, complete the following steps: - -1. Select File and select Import (Figure 14). - -.. image:: files/14-check-volttron-from-github.jpg - -Figure 14. Checking Out VOLTTRON with Eclipse from GitHub - -2. Select Git. Select Projects from Git. Click the Next button (Figure 15). - -.. image:: files/15-check-volttron-from-github.jpg - -Figure 15. Checking Out VOLTTRON with Eclipse from GitHub (continued) - -3. Select Clone URI and select Next (Figure 16). - -.. image:: files/16-check-volttron-from-github.jpg - -Figure 16. Checking Out VOLTTRON with Eclipse GitHub (continued) - -4. Fill in https://github.com/VOLTTRON/volttron.git for the URI. If you have a GitHub account, enter a username and password in the User and Password sections. This is not required but will allow you to receive notifications from GitHub for VOLTTRON related news. (Figure 17) - -.. image:: files/17-check-volttron-from-github.jpg - -Figure 17. Checking Out VOLTTRON with Eclipse from GitHub (continued) - -5. Select the master branch (Figure 18). - -.. image:: files/18-check-volttron-from-github.jpg - -Figure 18. Checking Out VOLTTRON with Eclipse from GitHub (continued) - -6. Select a location to save the local repository (Figure 19). - -.. image:: files/19-check-volttron-from-github.jpg - -Figure 19. Checking Out VOLTTRON with Eclipse from GitHub (continued) - -7. Select Import as general project. Select Next. Select Finish (Figure 20). The project will now be imported into the workspace. - -.. image:: files/20-check-volttron-from-github.jpg - -Figure 20. Checking Out VOLTTRON with Eclipse from GitHub (continued) - -If the VOLTTRON project has not been built (/bootstrap.py file has not been run), proceed to ##Section 2.4 Building the VOLTTRON Platform## and follow the instruction for running the bootstrap.py script before proceeding to the following sections. - -Linking Eclipses ----------------- -PyDev must now be configured to use the Python interpreter packaged with VOLTTRON. - -1. Select Window and select Preferences. -2. Expand the PyDev tree. -3. Select Interpreters and select Python interpreter. -4. Select New (Figure 21). - -.. image:: files/21-configuring-pydev.jpg - -Figure 21. Configuring PyDev - -5. Select Browse and navigate to the pydev-python file located at (``/scripts/pydev-python``) (Figure 22). - -6. Select OK (Figure 22). - -.. image:: files/22-configuring-pydev.jpg - -Figure 22. Configuring PyDev (continued) - -7. Select All and uncheck the VOLTTRON base directory (Figure 23). - -.. image:: files/23-configuring-pydev.jpg - -Figure 23. Configuring PyDev (continued) - -8. In the Project/PackageExplorer view on the left, right-click on the project, PyDev, and set as PyDev Project (Figure 24). - -.. image:: files/24-setting-pydev-project.jpg - -Figure 24. Setting as PyDev Project - -9. Switch to the PyDev perspective: Select Window. Select Perspective. Select Open Perspective. Select Other. Select PyDev (Figure 25). Eclipse should now be configured to use the project's environment. - -.. image:: files/25-setting-pydev-perspective.jpg - -Figure 25. Setting PyDev Perspective in Eclipse - -Running the VOLTTRON Platform and Agents ----------------------------------------- - -VOLTTRON and agents within VOLTTRON can now be run within Eclipse. This section will describe the process to run VOLTTRON and an agent within Eclipse. - -Setup a Run Configuration for the Platform -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The following steps describe the process for running VOLTTRON within Eclipse: - -1. Select Run and select Run Configurations (Figure 26). - -.. image:: files/26-running-volttron.jpg - -Figure 26. Running VOLTTRON Platform, Setting Up a Run Configuration - -2. Select Python Run from the menu on left. Click the New launch configuration button (Figure 27). - -.. image:: files/27-running-volttron.jpg - -Figure 27. Running VOLTTRON Platform, Setting Up a Run Configuration (continued) - -3. Change the name (any name may be used but for this example the name VOLTTRON was chosen) and select the main module (``/volttron/platform/main.py``). - -4. Select the Arguments tab and enter '-vv' in the Program arguments field (Figure 28) then select the Run button. - -.. image:: files/28-running-volttron.jpg - -Figure 28. Running VOLTTRON Platform, Setting Up a Run Configuration (continued) - -5. If the run is successful, the console should appear similar to Figure 29. If the run does not succeed (red text describing why the run failed will populate the console), click the all stop icon (two red boxes overlaid) on the console and then retry. - -.. image:: files/29-running-volttron.jpg - -Figure 29. Running VOLTTRON Platform, Console View on Successful Run - -Configure a Run Configuration for the Listener Agent -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The following steps describe the process for configuring an agent within Eclipse: - -1. Select Run and select Run Configurations (Figure 30). - -.. image:: files/30-running-listener-agent.jpg - -Figure 30. Running the Listener Agent, Setting Up a Run Configuration - -2. Select Python Run from the menu on left and click the New launch configuration button (Figure 31). - -.. image:: files/31-running-listener-agent.jpg - -Figure 31. Running the Listener Agent, Setting Up a Run Configuration (continued) - -3. Change the name (for this example Listener is used) and select the main module (``/examples/ListenerAgent/listener/agent.py``) (Figure 32). - -.. image:: files/32-running-listener-agent.jpg - -Figure 32. Running the Listener Agent, Setting Up a Run Configuration (continued) - -4. Click the Arguments tab and change Working directory to Default (Figure 33). - -.. image:: files/33-running-listener-agent.jpg - -Figure 33. Running the Listener Agent, Setting Up a Run Configuration (continued) - -5. In the Environment tab, select New and add the following environment variables (bulleted list below), as shown in Figure 34: - -- AGENT_CONFIG = /home//examples /ListenerAgent/config - -AGENT_CONFIG is the absolute path the agent’s configuration file. To access a remote message bus, use the VIP address as described in ##Section 3.5 Platform Management:VOLTTRON Management Central.## - -.. image:: files/34-running-listener-agent.jpg - -Figure 34. Running the Listener Agent, Setting Up a Run Configuration - -6. Click Run. This launches the agent. You should see the agent start to publish and receive its own heartbeat message (Figure 35). - -.. image:: files/35-listening_agent_output.jpg - -Figure 35. Listener Agent Output on Eclipse Console - -The process for running other agents in Eclipse is identical to that of the Listener agent. Several useful development tools are available within Eclipse and PyDev that make development, debugging, and testing of agents much simpler. - -Agent Creation Walkthrough --------------------------- -Developers should look at the Listener agent before developing their own agent. The Listener agent illustrates the basic functionality of an agent. The following example demonstrates the steps for creating an agent. - -Agent Folder Setup -^^^^^^^^^^^^^^^^^^ - -Create a folder within the workspace to help consolidate the code your agent will utilize. - -1. In the VOLTTRON base directory, create a new folder TestAgent. - -2. In TestAgent, create a new folder tester. This is the package where the Python code will be created (Figure 36). - -.. image:: files/36-agent-test-folder.jpg - -Figure 36. Creating an Agent Test Folder - -Create Agent Code -^^^^^^^^^^^^^^^^^ - -The following steps describe the necessary agent files and modules. - -1. In tester, create a file called *__init__.py*, which tells Python to treat this folder as a package. - -2. In the tester package folder, create the file *testagent.py* - -3. Create a class called TestAgent. - -4. Import the packages and classes needed: - -:: - - from __future__ import absolute_import - - from datetime import datetime - import logging - import sys - - from volttron.platform.vip.agent import Agent, Core - from volttron.platform.agent import utils - -5. Set up a logger. The ``utils`` module from ``volttron.platform.agent`` builds on Python’s already robust logging module and is easy to use. Add the following lines after the import statements: - -:: - - utils.setup_logging() - _log = logging.getLogger(__name__) - -This agent will inherit features from the Agent class (base class) extending the agent’s default functionality. The class definition for the TestAgent will be configured as shown below (with ``__init__``). - -:: - - class TestAgent(Agent): - def __init__(self, config_path, **kwargs): - super(TestAgent, self).__init__(**kwargs) - -Setting up a Subscription -^^^^^^^^^^^^^^^^^^^^^^^^^ -1. Create a startup method. This method is tagged with the decorator ``@Core.receiver("onstart")``. The startup method will run after the agent is initialized. The TestAgent’s startup method will contain a subscription to the Listener agent’s heartbeat (heartbeat/listeneragent). The TestAgent will detect when a message with this topic is published on the message bus and will run the method specified with the callback keyword argument passed to ``self.vip.pubsub.subscribe``. - -:: - - @Core.receiver("onstart") - def starting(self, sender, **kwargs): - ''' - Subscribes to the platform message bus on - the heatbeat/listeneragent topic - ''' - print('TestAgent example agent start-up function') - self.vip.pubsub.subscribe('pubsub', 'heartbeat/listeneragent', - callback=self.on_heartbeat) - -2. Create the callback method. Typically, the callback is the response to a message (or event). In this simple example, the TestAgent will do a print statement and publish a message to the bus: - -:: - - def on_heartbeat(self, peer, sender, bus, topic, headers, message): - '''TestAgent callback method''' - print('Matched topic: {}, for bus: {}'.format(topic, bus)) - self.vip.pubsub.publish('pubsub', - 'testagent/publish', - headers=headers, - message='test publishing').get(timeout=30) - -Argument Parsing Main Method -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The test agent will need to be able to parse arguments being passed on the command line by the agent launcher. Use the **utils.default_main** method to handle argument parsing and other default behavior. - -1. Create a main method that can be called by the launcher: - -:: - - def main(argv=sys.argv): - '''Main method called by the eggsecutable.''' - try: - utils.vip_main(TestAgent) - except Exception as e: - _log.exception(e) - - if __name__ == '__main__': - # Entry point for script - sys.exit(main()) - -Create Support Files for Test Agent -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -VOLTTRON agents need configuration files for packaging, configuration, and launching. The “setup.py” file details the naming and Python package information. The launch configuration file is a JSON-formatted text file used by the platform to launch instances of the agent. - -Packaging Configuration -^^^^^^^^^^^^^^^^^^^^^^^ -In the TestAgent folder, create a file called "setup.py". This file sets up the name, version, required packages, method to execute, etc. for the agent. The packaging process will also use this information to name the resulting file. - -:: - - from setuptools import setup, find_packages - - #get environ for agent name/identifier - packages = find_packages('.') - package = packages[0] - - setup( - name = package + 'agent', - version = "0.1", - install_requires = ['volttron'], - packages = packages, - entry_points = { - 'setuptools.installation': [ - 'eggsecutable = ' + package + '.testagent:main', - ] - } - ) - -Launch Configuration -^^^^^^^^^^^^^^^^^^^^ -In TestAgent, create a file called "testagent.launch.json". This is the file the platform will use to launch the agent. It can also contain configuration parameters for the agent: - -:: - - { - "agentid": "Test1" - } - -Testing the Agent -^^^^^^^^^^^^^^^^^ -From a terminal, in the base VOLTTRON directory, enter the following commands (with the platform activated and VOLTTRON running): - -1. Run `pack_install` script on TestAgent: - -:: - - $ ./scripts/core/pack_install.sh TestAgent TestAgent/config test-agent - -- Upon successful completion of this command, the terminal output will show the install directory, the agent UUID (unique identifier for an agent; the UUID shown in red is only an example and each instance of an agent will have a different UUID) and the agent name (blue text): - -:: - - Installed /home/volttron-user/.volttron/packaged/testeragent-0.1-py2-none-any.whl - as d4ca557a-496c-4f02-8ad9-42f5d435868a testeragent-0.1 - -2. Start the agent: - -:: - - $ vctl start --tag test-agent - -3. Verify that the agent is running: - -:: - - $ vctl status - $ tail -f volttron.log - -If changes are made to the Passive AFDD agent’s configuration file after the agent is launched, stop and reload the agent. In a terminal, enter the following commands: - -:: - - $ vctl stop --tag test-agent - $ vctl remove --tag test-agent - -Re-build and start the updated agent (Figure 37). - -.. image:: files/37-testagent-output.jpg - -Figure 37. TestAgent Output In VOLTTRON Log - -Running the TestAgent in Eclipse -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. warning:: - Before attempting to run an agent in Eclipse, please see the note in: :ref:`AgentDevelopment ` - -If you are working in Eclipse, create a run configuration for the TestAgent based on the Listener agent configuration in the Eclipse development environment ##(Section 5.5.5 Running the VOLTTRON Platform and Agents)##. - -1. Launch the platform (##Section 5.5.5.1 Setup a Run Configuration for the Platform##) - -2. Launch the TestAgent by following the steps outlined in `Launching the Listener ` for launching the Listener agent. - -3. Launch the Listener agent. TestAgent should start receiving the heartbeats from Listener agent and the following should be displayed in the console (Figure 38). - -.. image:: files/38-console-output.jpg - -Figure 38. Console Output for TestAgent - -Adding Additional Features to the TestAgent -------------------------------------------- -Additional code can be added to the TestAgent to utilize additional services in the platform. The following sections show how to use the weather and device scheduling service within the TestAgent. - -Subscribing to Weather Data -^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This agent can be modified to listen to weather data from the Weather agent by adding the following line at the end of the TestAgent startup method. This will subscribe the agent to the temperature subtopic. For the full list of topics available, please see: - -https://github.com/VOLTTRON/volttron/wiki/WeatherAgentTopics - -:: - - self.vip.pubsub.subscribe('pubsub', 'weather/temperature/temp_f', - callback=self.on_weather) - -Add the callback method ``on_weather``: - -:: - - def on_weather(self, peer, sender, bus, topic, headers, message): - print("TestAgent got weather\nTopic: {}, Message: {}".format(topic, message)) - -The platform log file should appear similar to Figure 39. - -.. image:: files/39-testagent-output-weather-subscribed.jpg - -Figure 39. TestAgent Output when Subscribing to Weather Topic - -Utilizing the Scheduler Agent -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The TestAgent can be modified to publish a schedule to the Actuator agent by reserving time on virtual devices. Modify the following code to include current time ranges and include a call to the publish schedule method in setup. The following example posts a simple schedule. For more detailed information on device scheduling, please see: - -https://github.com/VOLTTRON/volttron/wiki/ActuatorAgent - -Ensure the Actuator agent is running as per ##Section 3.3 Device Control: Configuring and Launching the Actuator Agent##. Add the following line to the TestAgent’s import statements: - -:: - - from volttron.platform.messaging import topics - -Add the following lines to the TestAgent’s starting method. This sets up a subscription to the **ACTUATOR_RESPONSE** topic and calls the **publish_schedule** method. - -:: - - self.vip.pubsub.subscribe('pubsub', topics.ACTUATOR_RESPONSE, - callback=self.on_schedule_result) - self.publish_schedule() - -The **publish_schedule** method sends a schedule request message to the Actuator agent (Update the schedule with appropriate times): - -:: - - def publish_schedule(self): - headers = { - 'AgentID': self._agent_id, - 'type': 'NEW_SCHEDULE', - 'requesterID': self._agent_id, # Name of requesting agent - 'taskID': self._agent_id + "-TASK", # Unique task ID - 'priority': 'LOW' # Task Priority (HIGH, LOW, LOW_PREEMPT) - } - msg = [ - ["campus/building/device1", # First time slot. - "2014-1-31 12:27:00", # Start of time slot. - "2014-1-31 12:29:00"], # End of time slot. - ["campus/building/device1", # Second time slot. - "2014-1-31 12:26:00", # Start of time slot. - "2014-1-31 12:30:00"], # End of time slot. - ["campus/building/device2", # Third time slot. - "2014-1-31 12:30:00", # Start of time slot. - "2014-1-31 12:32:00"], # End of time slot. - #etc... - ] - self.vip.rpc.call('platform.actuator', # Target agent - 'request_new_schedule', # Method to call - agent_id, # Requestor - "some task", # TaskID - "LOW", # Priority - msg).get(timeout=10) # Request message - -Add the call back method for the schedule request: - -:: - - def on_schedule_result(self, topic, headers, message, match): - print (("TestAgent schedule result \nTopic: {topic}, " - "{headers}, Message: {message}") - .format(topic=topic, headers=headers, message=message)) - -Full TestAgent Code -^^^^^^^^^^^^^^^^^^^ -The following is the full TestAgent code built in the previous steps: - -:: - - from __future__ import absolute_import - - from datetime import datetime - import logging - import sys - - from volttron.platform.vip.agent import Agent, Core - from volttron.platform.agent import utils - from volttron.platform.messaging import headers as headers_mod - - utils.setup_logging() - _log = logging.getLogger(__name__) - - class TestAgent(Agent): - def __init__(self, config_path, **kwargs): - super(TestAgent, self).__init__(**kwargs) - - @Core.receiver("onstart") - def starting(self, sender, **kwargs): - ''' - Subscribes to the platform message bus on - the heatbeat/listeneragent topic - ''' - _log.info('TestAgent example agent start-up function') - self.vip.pubsub.subscribe(peer='pubsub', topic='heartbeat/listeneragent', - callback=self.on_heartbeat) - self.vip.pubsub.subscribe('pubsub', topics.ACTUATOR_RESPONSE, - callback=self.on_schedule_result) - self.vip.pubsub.subscribe('pubsub', 'weather/temperature/temp_f', - callback=self.on_weather) - - self.publish_schedule() - - def on_heartbeat(self, peer, sender, bus, topic, headers, message): - '''TestAgent callback method''' - _log.info('Matched topic: {}, for bus: {}'.format(topic, bus)) - self.vip.pubsub.publish(peer='pubsub', - topic='testagent/publish', - headers=headers, - message='test publishing').get(timeout=30) - - def on_weather(self, peer, sender, bus, topic, headers, message): - _log.info( - "TestAgent got weather\nTopic: {}, Message: {}".format(topic, message)) - - def on_schedule_result(self, topic, headers, message, match): - print (("TestAgent schedule result \nTopic: {topic}, " - "{headers}, Message: {message}") - .format(topic=topic, headers=headers, message=message)) - - def main(argv=sys.argv): - '''Main method called by the eggsecutable.''' - try: - utils.vip_main(TestAgent) - except Exception as e: - _log.info(e) - - if __name__ == '__main__': - # Entry point for script - sys.exit(main()) - - - - - - - - diff --git a/docs/source/devguides/agent_development/TestAgent.rst b/docs/source/devguides/agent_development/TestAgent.rst deleted file mode 100644 index f0d9fee37e..0000000000 --- a/docs/source/devguides/agent_development/TestAgent.rst +++ /dev/null @@ -1,204 +0,0 @@ -.. _Test-Agent: - -TestAgent Source Code -===================== - -Full code of agent detailed in AgentDevelopment: - -:: - - """ - Agent documentation goes here. - """ - - __docformat__ = 'reStructuredText' - - import logging - import sys - from volttron.platform.agent import utils - from volttron.platform.vip.agent import Agent, Core, RPC - - _log = logging.getLogger(__name__) - utils.setup_logging() - __version__ = "0.5" - - - def tester(config_path, **kwargs): - """Parses the Agent configuration and returns an instance of - the agent created using that configuration. - - :param config_path: Path to a configuration file. - - :type config_path: str - :returns: Tester - :rtype: Tester - """ - try: - config = utils.load_config(config_path) - except StandardError: - config = {} - - if not config: - _log.info("Using Agent defaults for starting configuration.") - - setting1 = int(config.get('setting1', 1)) - setting2 = config.get('setting2', "some/random/topic") - - return Tester(setting1, - setting2, - **kwargs) - - - class Tester(Agent): - """ - Document agent constructor here. - """ - - def __init__(self, setting1=1, setting2="some/random/topic", - **kwargs): - super(Tester, self).__init__(**kwargs) - _log.debug("vip_identity: " + self.core.identity) - - self.setting1 = setting1 - self.setting2 = setting2 - - self.default_config = {"setting1": setting1, - "setting2": setting2} - - - #Set a default configuration to ensure that self.configure is called immediately to setup - #the agent. - self.vip.config.set_default("config", self.default_config) - #Hook self.configure up to changes to the configuration file "config". - self.vip.config.subscribe(self.configure, actions=["NEW", "UPDATE"], pattern="config") - - def configure(self, config_name, action, contents): - """ - Called after the Agent has connected to the message bus. If a configuration exists at startup - this will be called before onstart. - - Is called every time the configuration in the store changes. - """ - config = self.default_config.copy() - config.update(contents) - - _log.debug("Configuring Agent") - - try: - setting1 = int(config["setting1"]) - setting2 = str(config["setting2"]) - except ValueError as e: - _log.error("ERROR PROCESSING CONFIGURATION: {}".format(e)) - return - - self.setting1 = setting1 - self.setting2 = setting2 - - self._create_subscriptions(self.setting2) - - def _create_subscriptions(self, topic): - #Unsubscribe from everything. - self.vip.pubsub.unsubscribe("pubsub", None, None) - - self.vip.pubsub.subscribe(peer='pubsub', - prefix=topic, - callback=self._handle_publish) - - def _handle_publish(self, peer, sender, bus, topic, headers, - message): - pass - - @Core.receiver("onstart") - def onstart(self, sender, **kwargs): - """ - This is method is called once the Agent has successfully connected to the platform. - This is a good place to setup subscriptions if they are not dynamic or - do any other startup activities that require a connection to the message bus. - Called after any configurations methods that are called at startup. - - Usually not needed if using the configuration store. - """ - #Example publish to pubsub - #self.vip.pubsub.publish('pubsub', "some/random/topic", message="HI!") - - #Exmaple RPC call - #self.vip.rpc.call("some_agent", "some_method", arg1, arg2) - - @Core.receiver("onstop") - def onstop(self, sender, **kwargs): - """ - This method is called when the Agent is about to shutdown, but before it disconnects from - the message bus. - """ - pass - - @RPC.export - def rpc_method(self, arg1, arg2, kwarg1=None, kwarg2=None): - """ - RPC method - - May be called from another agent via self.core.rpc.call """ - return self.setting1 + arg1 - arg2 - - def main(): - """Main method called to start the agent.""" - utils.vip_main(tester, - version=__version__) - - - if __name__ == '__main__': - # Entry point for script - try: - sys.exit(main()) - except KeyboardInterrupt: - pass - - -Contents of setup.py for TestAgent: - -:: - - from setuptools import setup, find_packages - - MAIN_MODULE = 'agent' - - # Find the agent package that contains the main module - packages = find_packages('.') - agent_package = 'tester' - - # Find the version number from the main module - agent_module = agent_package + '.' + MAIN_MODULE - _temp = __import__(agent_module, globals(), locals(), ['__version__'], -1) - __version__ = _temp.__version__ - - # Setup - setup( - name=agent_package + 'agent', - version=__version__, - author_email="volttron@pnnl.gov", - url="https://volttron.org/", - description="Agent development tutorial.", - author="VOLTTRON Team", - install_requires=['volttron'], - packages=packages, - entry_points={ - 'setuptools.installation': [ - 'eggsecutable = ' + agent_module + ':main', - ] - } - ) - -Contents of config: - -:: - - { - # VOLTTRON config files are JSON with support for python style comments. - "setting1": 2, #Integers - "setting2": "some/random/topic2", #Strings - "setting3": true, #Booleans: remember that in JSON true and false are not capitalized. - "setting4": false, - "setting5": 5.1, #Floating point numbers. - "setting6": [1,2,3,4], #Lists - "setting7": {"setting7a": "a", "setting7b": "b"} #Objects - } diff --git a/docs/source/devguides/agent_development/index.rst b/docs/source/devguides/agent_development/index.rst deleted file mode 100644 index 623a6f81de..0000000000 --- a/docs/source/devguides/agent_development/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. _Agent_Development: - -================= -Agent Development -================= - -.. toctree:: - :glob: - :maxdepth: 1 - - * diff --git a/docs/source/devguides/deployment/Deployment-Options.rst b/docs/source/devguides/deployment/Deployment-Options.rst deleted file mode 100644 index 770038dd42..0000000000 --- a/docs/source/devguides/deployment/Deployment-Options.rst +++ /dev/null @@ -1,172 +0,0 @@ -================== -Deployment Options -================== - -There are several ways to deploy the VOLTTRON platform in a Linux environment. It is up to the user to determine which -is right for them. The following assumes that the platform has already been bootstrapped and is ready to run. - -Simple Command Line -******************* - -With the VOLTTRON environment activated the platform can be started simply by running VOLTTRON on the command -line. - -:: - - $volttron -vv - -This will start the platform in the current terminal with very verbose logging turned on. This -is most appropriate for testing Agents or testing a deployment for problems before switching to a -more long term solution. This will print all log messages to the console in real time. - -This should not be used for long term deployment. As soon as an SSH session is terminated for whatever reason -the processes attached to that session will be killed. This also will not capture log message to a file. - -Running VOLTTRON as a Background Process -**************************************** - -A simple, more long term solution, is to run volttron in the background and disown it from the current terminal. - -.. warning:: - If you plan on running VOLTTRON in the background and detaching it from the - terminal with the ``disown`` command be sure to redirect stderr and stdout to ``/dev/null``. - Even if logging to a file is used some libraries which VOLTTRON relies on output - directly to stdout and stderr. This will cause problems if those file descriptors - are not redirected to ``/dev/null``. - -:: - - $volttron -vv -l volttron.log > /dev/null 2>&1& - -Alternatively: - -:: - - ``./start-volttron`` - -.. note:: If you are not in an activated environment, this script will start - the platform running in the background in the correct environment, however - the environment will not be activated for you, you must activate it yourself. - -**If there are other jobs running in your terminal be sure to disown the correct one.** - -:: - - $jobs - [1]+ Running something else - [2]+ Running ./start-volttron - - #Disown VOLTTRON - $disown %2 - -This will run the VOLTTRON platform in the background and turn it into a daemon. The log output will be directed -to a file called ``volttron.log`` in the current directory. - -To keep the size of the log under control for more longer term deployments us the rotating log configuration file -``examples/rotatinglog.py``. - -:: - - $volttron -vv --log-config examples/rotatinglog.py > /dev/null 2>&1& - -This will start a rotate the log file at midnight and limit the total log data to seven days worth. - -The main downside to this approach is that the VOLTTRON platform will not automatically -resume if the system is restarted. It will need to be restarted manually after reboot. - -Setting up VOLTTRON as a System Service -*************************************** - -Systemd -------- - -An example service file ``scripts/admin/volttron.service`` for systemd cas be used as a starting point -for setting up VOLTTRON as a service. Note that as this will redirect all the output that would -be going to stdout - to the syslog. This can be accessed using journalctl. For systems that run -all the time or have a high level of debugging turned on, we recommend checking the system's -logrotate settings. - -:: - - [Unit] - Description=VOLTTRON Platform Service - After=network.target - - [Service] - Type=simple - - #Change this to the user that VOLTTRON will run as. - User=volttron - Group=volttron - - #Uncomment and change this to specify a different VOLTTRON_HOME - #Environment="VOLTTRON_HOME=/home/volttron/.volttron" - - #Change these to settings to reflect the install location of VOLTTRON - WorkingDirectory=/var/lib/volttron - ExecStart=/var/lib/volttron/env/bin/volttron -vv - ExecStop=/var/lib/volttron/env/bin/volttron-ctl shutdown --platform - - - [Install] - WantedBy=multi-user.target - -After the file has been modified to reflect the setup of the platform you can install it with the -following commands. These need to be run as root or with sudo as appropriate. - -:: - - #Copy the service file into place - cp scripts/admin/volttron.service /etc/systemd/system/ - - #Set the correct permissions if needed - chmod 644 /etc/systemd/system/volttron.service - - #Notify systemd that a new service file exists (this is crucial!) - systemctl daemon-reload - - #Start the service - systemctl start volttron.service - -Init.d ------- - -An example init script ``scripts/admin/volttron`` can be used as a starting point for -setting up VOLTTRON as a service on init.d based systems. - -Minor changes may be needed for the file to work on the target system. Specifically -the ``USER``, ``VLHOME``, and ``VOLTTRON_HOME`` variables may need to be changed. - -:: - - ... - #Change this to the user VOLTTRON will run as. - USER=volttron - #Change this to the install location of VOLTTRON - VLHOME=/var/lib/volttron - - ... - - #Uncomment and change this to specify a different VOLTTRON_HOME - #export VOLTTRON_HOME=/home/volttron/.volttron - - -The script can be installed with the following commands. These need to be run as root or -with sudo as appropriate. - -:: - - #Copy the script into place - cp scripts/admin/volttron /etc/init.d/ - - #Make the file executable - chmod 755 /etc/init.d/volttron - - #Change the owner to root - chown root:root /etc/init.d/volttron - - #These will set it to startup automatically at boot - update-rc.d volttron defaults - - #Start the service - /etc/init.d/volttron start diff --git a/docs/source/devguides/deployment/Multiple-Address-Configuration.rst b/docs/source/devguides/deployment/Multiple-Address-Configuration.rst deleted file mode 100644 index 1a38cb0062..0000000000 --- a/docs/source/devguides/deployment/Multiple-Address-Configuration.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. _External-Address-Configuration: - -Platform External Address Configuration -======================================= - -In the configuration file located in $VOLTTRON\_HOME/config add -vip-address=tcp://ip:port for each address you want to listen on - -:: - - Example - vip-address=tcp://127.0.0.102:8182 - vip-address=tcp://127.0.0.103:8083 - vip-address=tcp://127.0.0.103:8183 - -.. note:: The config file is generated after running the vcfg command. The vip-address is for the local platform, NOT the remote platform. diff --git a/docs/source/devguides/deployment/index.rst b/docs/source/devguides/deployment/index.rst deleted file mode 100644 index bdb203ba06..0000000000 --- a/docs/source/devguides/deployment/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -================= -Deployment Advice -================= - -.. toctree:: - :glob: - :maxdepth: 2 - - Deployment-Options - Linux-Platform-Hardening-Recommendations-for-VOLTTRON-users - Multiple-Address-Configuration diff --git a/docs/source/devguides/eclipse/Manual-Plugin-Install.rst b/docs/source/devguides/eclipse/Manual-Plugin-Install.rst deleted file mode 100644 index d9361ab872..0000000000 --- a/docs/source/devguides/eclipse/Manual-Plugin-Install.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. _Manual-Plugin-Install: - -Manual Python Plugin Installation -================================= - -If Eclipse doesn't have the Marketplace, follow these steps to get the -plugins: - -- Help -> Install New Software -- Click on the "Add" button -- For name use: EGit -- For location: \\ -- After hitting OK, check the box for: Check Select All -- Click through Next, Agree to Terms, then Finish -- Allow Eclipse to restart - -After installing Eclipse, you must add the PyDev plugin to the -environment. In Eclipse: - -- Help -> Install New Software -- Click on the "Add" button -- For name use: PyDev -- For location: ``__ -- Check the box for PyDev -- Click through Next, Agree to Terms, Finish -- Allow Eclipse to restart - diff --git a/docs/source/devguides/eclipse/files/clone-existing.png b/docs/source/devguides/eclipse/files/clone-existing.png deleted file mode 100755 index 465f351bf0..0000000000 Binary files a/docs/source/devguides/eclipse/files/clone-existing.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/eclipse-marketplace2.png b/docs/source/devguides/eclipse/files/eclipse-marketplace2.png deleted file mode 100755 index 91b4b5f706..0000000000 Binary files a/docs/source/devguides/eclipse/files/eclipse-marketplace2.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/general-project.png b/docs/source/devguides/eclipse/files/general-project.png deleted file mode 100755 index 106228a41d..0000000000 Binary files a/docs/source/devguides/eclipse/files/general-project.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/git-view.png b/docs/source/devguides/eclipse/files/git-view.png deleted file mode 100755 index 0b59e8fcc3..0000000000 Binary files a/docs/source/devguides/eclipse/files/git-view.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/listener-all-vars.png b/docs/source/devguides/eclipse/files/listener-all-vars.png deleted file mode 100755 index f4fdde5296..0000000000 Binary files a/docs/source/devguides/eclipse/files/listener-all-vars.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/pick-python.png b/docs/source/devguides/eclipse/files/pick-python.png deleted file mode 100755 index 2d3e8eafaf..0000000000 Binary files a/docs/source/devguides/eclipse/files/pick-python.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/run-results.png b/docs/source/devguides/eclipse/files/run-results.png deleted file mode 100755 index 5568a59585..0000000000 Binary files a/docs/source/devguides/eclipse/files/run-results.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/select-branch.png b/docs/source/devguides/eclipse/files/select-branch.png deleted file mode 100755 index 4333c0df90..0000000000 Binary files a/docs/source/devguides/eclipse/files/select-branch.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/volttron-main-args.png b/docs/source/devguides/eclipse/files/volttron-main-args.png deleted file mode 100755 index 241e1c35c7..0000000000 Binary files a/docs/source/devguides/eclipse/files/volttron-main-args.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/files/volttron-pick-main.png b/docs/source/devguides/eclipse/files/volttron-pick-main.png deleted file mode 100755 index 13168ba066..0000000000 Binary files a/docs/source/devguides/eclipse/files/volttron-pick-main.png and /dev/null differ diff --git a/docs/source/devguides/eclipse/index.rst b/docs/source/devguides/eclipse/index.rst deleted file mode 100644 index 4cef99041e..0000000000 --- a/docs/source/devguides/eclipse/index.rst +++ /dev/null @@ -1,210 +0,0 @@ -================= -Eclipse IDE Setup -================= - -The only thing that is necessary to create a VOLTTRON agent is a text -editor and the shell. However, we have found the Eclipse Development -Environment (IDE) to be a valuable tool for helping to develop VOLTTRON -agents. You can obtain the latest (MARS as fo 10/7/15) from -http://www.eclipse.org/. Once downloaded the `PyDev Plugin <#pydev-plugin>`__ -is a valuable tool for executing the platform as well as debugging agent code. - -- `Install PyDev Plugin <#pydev-plugin>`__ -- `Clone the VOLTTRON Source <#cloning-the-source-code>`__ -- `Build VOLTTRON <#build-volttron>`__ -- `Link Eclipse to VOLTTRON Python Environment <#linking-eclipse-and-the-volttron-python-environment>`__ -- `Make Project a PyDev Project <#make-project-a-pydev-project>`__ -- `Testing the Installation <#testing-the-installation>`__ -- `Execute VOLTTRON Through Shell <#execute-volttron-through-shell>`__ -- `Execute VOLTTRON Through Eclipse <#execute-volttron-through-eclipse>`__ -- `Start a ListenerAgent <#start-a-listeneragent>`__ - -PyDev Plugin ------------- - -Installing the PyDev plugin from the Eclipse Market place There is a -python plugin for eclipse that makes development much easier. Install it -from the eclipse marketplace. - -|Help -> Eclipse Marketplace...| - -|Click Install| - -Cloning the Source Code ------------------------ - -The VOLTTRON code is stored in a git repository. Eclipse (Luna and Mars) -come with a git plugin out of the box. For other versions the plugin is -available for Eclipse that makes development more convenient (note: you -must have Git :ref:`already installed ` on the -system and have :ref:`built VOLTTRON `): - -If your version of Eclipse does not have the marketplace follow these -:ref:`instructions `. - -The project can now be checked out from the repository into Eclipse. - -#. Open the Git view - - |Select Git view| - -#. Clone a Git Repository - - |Clone existing repo| - -#. Fill out the URI: https://github.com/VOLTTRON/volttron - - |Select repo| - -#. Select master for latest stable version - - |Select branch repo| - -#. Import the cloned repository as a general project - - |Import project| - -#. Pick a project name (default volttron) and hit Finish - - |Finish import| - -#. Switch to the PyDev perspective - -Build VOLTTRON --------------- - -Continue the setup process by opening a command shell. Make the current -directory the root of your cloned VOLTTRON directory. Follow the -instructions in our `Building VOLTTRON `__ section of -the wiki and then continue below. - -Linking Eclipse and the VOLTTRON Python Environment ---------------------------------------------------- - -From the Eclipse IDE right click on the project name and select Refresh -so eclipse will be aware of the file system changes. The next step will -define the python version that PyDev will use for VOLTTRON - -#. Choose Window - > Preferences -#. Expand the PyDev tree -#. Select Interpreters - > Python Interpreter -#. Click New -#. Click Browse and browse to the pydev-python file located in scripts - directory off of the volttron source -#. Click Ok - - |Pick Python| - -#. Select All, then uncheck the VOLTTRON root like the picture below - - |Select path| - -#. Click Ok - -.. note:: - - You may need redo this stage after platform updates - -Make Project a PyDev Project ----------------------------- - -#. In the Project/PackageExplorer view on the left, right-click on the - project, PyDev-> Set as PyDev Project -#. Switch to the PyDev perspective (if it has not already switched), - Window -> Open Perspective -> PyDev - |Set as Pydev| - -Eclipse should now be configured to use the project's environment. - -Testing the Installation ------------------------- - -In order to test the installation the VOLTTRON platform must be running. -You can do this either through `the shell <#execute-volttron-through-shell>`__ or -`through Eclipse <#execute-volttron-through-eclipse>`__. - -.. _Execute-Volttron-From-Shell: - -Execute VOLTTRON Through Shell -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -#. Open a console and cd into the root of the volttron repository. -#. Execute `source env/bin/activate` -#. Execute `volttron -vv` - - |Execute VOLTTRON in Shell| - -You now have a running VOLTTRON logging to standard out. The next step -to verifying the installation is to `start a listeneragent <#start-a-listeneragent>`__. - -Execute VOLTTRON Through Eclipse -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -#. Click Run -> Run Configuration from the Eclipse Main Menu -#. Click the New Launch Configuration button - - |New Launch Configuration| - -#. Change the name and select the main module `volttron/platform/main.py` - - |Main Module| - -#. Click the Arguments Tab add '-vv' to the arguments and change the working directory to default - - |Arguments| - -#. Click Run. The following image displays the output of a successfully started platform - - |Successful Start| - -:ref: _Start-Listener-Eclipse: - -Start a ListenerAgent -~~~~~~~~~~~~~~~~~~~~~ - -.. warning:: - Before attempting to run an agent in Eclipse, please see the note in: :ref:`AgentDevelopment ` - - - -The listener agent will listen to the message bus for any published -messages. It will also publish a heartbeat message ever 10 seconds (by -default). - -Create a new run configuration entry for the listener agent. - -#. In the Package Explorer view, open examples -> ListenerAgent --> - listener -#. Righ-click on agent.py and select Run As -> Python Run (this will - create a run configuration but fail) -#. On the menu bar, pick Run -> Run Configurations... -#. Under Python Run pick "volttron agent.py" -#. Click on the Arguments tab and Change Working Directory to Default - -#. In the Environment tab, click new set the variable to AGENT\_CONFIG - with the value of /home/git/volttron/examples/ListenerAgent/config - - |Listener Vars| - -#. Click Run, this launches the agent - -You should see the agent start to publish and receive its own heartbeat -message in the console. - -.. |Help -> Eclipse Marketplace...| image:: files/eclipse-marketplace.png -.. |Click Install| image:: files/eclipse-marketplace2.png -.. |Select Git view| image:: files/git-view.png -.. |Clone existing repo| image:: files/clone-existing.png -.. |Select repo| image:: files/select-repo.png -.. |Select branch repo| image:: files/select-branch.png -.. |Import project| image:: files/import-project.png -.. |Finish import| image:: files/finish-import.png -.. |Pick Python| image:: files/pick-python.png -.. |Select path| image:: files/select-path.png -.. |Set as Pydev| image:: files/set-as-pydev.png -.. |Execute VOLTTRON in Shell| image:: files/volttron-console.png -.. |New Launch Configuration| image:: files/new-python-run.png -.. |Main Module| image:: files/volttron-pick-main.png -.. |Arguments| image:: files/volttron-main-args.png -.. |Successful Start| image:: files/run-results.png -.. |Listener Vars| image:: files/listener-all-vars.png diff --git a/docs/source/devguides/files/clone-existing.png b/docs/source/devguides/files/clone-existing.png deleted file mode 100755 index 465f351bf0..0000000000 Binary files a/docs/source/devguides/files/clone-existing.png and /dev/null differ diff --git a/docs/source/devguides/files/eclipse-marketplace.png b/docs/source/devguides/files/eclipse-marketplace.png deleted file mode 100755 index cf6a0136b2..0000000000 Binary files a/docs/source/devguides/files/eclipse-marketplace.png and /dev/null differ diff --git a/docs/source/devguides/files/eclipse-marketplace2.png b/docs/source/devguides/files/eclipse-marketplace2.png deleted file mode 100755 index 91b4b5f706..0000000000 Binary files a/docs/source/devguides/files/eclipse-marketplace2.png and /dev/null differ diff --git a/docs/source/devguides/files/finish-import.png b/docs/source/devguides/files/finish-import.png deleted file mode 100755 index 5a32e44151..0000000000 Binary files a/docs/source/devguides/files/finish-import.png and /dev/null differ diff --git a/docs/source/devguides/files/general-project.png b/docs/source/devguides/files/general-project.png deleted file mode 100755 index 106228a41d..0000000000 Binary files a/docs/source/devguides/files/general-project.png and /dev/null differ diff --git a/docs/source/devguides/files/git-view.png b/docs/source/devguides/files/git-view.png deleted file mode 100755 index 0b59e8fcc3..0000000000 Binary files a/docs/source/devguides/files/git-view.png and /dev/null differ diff --git a/docs/source/devguides/files/import-project.png b/docs/source/devguides/files/import-project.png deleted file mode 100755 index 5ac524f41d..0000000000 Binary files a/docs/source/devguides/files/import-project.png and /dev/null differ diff --git a/docs/source/devguides/files/listener-all-vars.png b/docs/source/devguides/files/listener-all-vars.png deleted file mode 100755 index f4fdde5296..0000000000 Binary files a/docs/source/devguides/files/listener-all-vars.png and /dev/null differ diff --git a/docs/source/devguides/files/new-python-run.png b/docs/source/devguides/files/new-python-run.png deleted file mode 100755 index 580b462c8f..0000000000 Binary files a/docs/source/devguides/files/new-python-run.png and /dev/null differ diff --git a/docs/source/devguides/files/pick-python.png b/docs/source/devguides/files/pick-python.png deleted file mode 100755 index 2d3e8eafaf..0000000000 Binary files a/docs/source/devguides/files/pick-python.png and /dev/null differ diff --git a/docs/source/devguides/files/platform-run-config.png b/docs/source/devguides/files/platform-run-config.png deleted file mode 100755 index 083e157d83..0000000000 Binary files a/docs/source/devguides/files/platform-run-config.png and /dev/null differ diff --git a/docs/source/devguides/files/pydev-python.png b/docs/source/devguides/files/pydev-python.png deleted file mode 100755 index e20f45bdb2..0000000000 Binary files a/docs/source/devguides/files/pydev-python.png and /dev/null differ diff --git a/docs/source/devguides/files/run-results.png b/docs/source/devguides/files/run-results.png deleted file mode 100755 index 5568a59585..0000000000 Binary files a/docs/source/devguides/files/run-results.png and /dev/null differ diff --git a/docs/source/devguides/files/select-branch.png b/docs/source/devguides/files/select-branch.png deleted file mode 100755 index 4333c0df90..0000000000 Binary files a/docs/source/devguides/files/select-branch.png and /dev/null differ diff --git a/docs/source/devguides/files/select-path.png b/docs/source/devguides/files/select-path.png deleted file mode 100755 index 7f690f5696..0000000000 Binary files a/docs/source/devguides/files/select-path.png and /dev/null differ diff --git a/docs/source/devguides/files/select-repo.png b/docs/source/devguides/files/select-repo.png deleted file mode 100755 index d74fd20b45..0000000000 Binary files a/docs/source/devguides/files/select-repo.png and /dev/null differ diff --git a/docs/source/devguides/files/set-as-pydev.png b/docs/source/devguides/files/set-as-pydev.png deleted file mode 100755 index 681eeefd60..0000000000 Binary files a/docs/source/devguides/files/set-as-pydev.png and /dev/null differ diff --git a/docs/source/devguides/files/setup-python.png b/docs/source/devguides/files/setup-python.png deleted file mode 100755 index 6e90e4537c..0000000000 Binary files a/docs/source/devguides/files/setup-python.png and /dev/null differ diff --git a/docs/source/devguides/files/volttron-console.png b/docs/source/devguides/files/volttron-console.png deleted file mode 100755 index 1a63411b9a..0000000000 Binary files a/docs/source/devguides/files/volttron-console.png and /dev/null differ diff --git a/docs/source/devguides/files/volttron-main-args.png b/docs/source/devguides/files/volttron-main-args.png deleted file mode 100755 index 241e1c35c7..0000000000 Binary files a/docs/source/devguides/files/volttron-main-args.png and /dev/null differ diff --git a/docs/source/devguides/files/volttron-main.png b/docs/source/devguides/files/volttron-main.png deleted file mode 100755 index 2443671f96..0000000000 Binary files a/docs/source/devguides/files/volttron-main.png and /dev/null differ diff --git a/docs/source/devguides/files/volttron-pick-main.png b/docs/source/devguides/files/volttron-pick-main.png deleted file mode 100755 index 13168ba066..0000000000 Binary files a/docs/source/devguides/files/volttron-pick-main.png and /dev/null differ diff --git a/docs/source/devguides/index.rst b/docs/source/devguides/index.rst deleted file mode 100644 index 0d736ceb29..0000000000 --- a/docs/source/devguides/index.rst +++ /dev/null @@ -1,21 +0,0 @@ -.. _devguides_index: - -=================== -Developing VOLTTRON -=================== - -.. toctree:: - :glob: - :maxdepth: 1 - - agent_development/index - deployment/index - walkthroughs/index - eclipse/index - pycharm/index - scalability/index - supporting/index - roadmap/index - - - * diff --git a/docs/source/devguides/pycharm/files/00_open_pycharm.png b/docs/source/devguides/pycharm/files/00_open_pycharm.png deleted file mode 100644 index e785320dd7..0000000000 Binary files a/docs/source/devguides/pycharm/files/00_open_pycharm.png and /dev/null differ diff --git a/docs/source/devguides/pycharm/files/01_load_volttron.png b/docs/source/devguides/pycharm/files/01_load_volttron.png deleted file mode 100644 index f46d9ab762..0000000000 Binary files a/docs/source/devguides/pycharm/files/01_load_volttron.png and /dev/null differ diff --git a/docs/source/devguides/pycharm/files/02_set_project_interpreter.png b/docs/source/devguides/pycharm/files/02_set_project_interpreter.png deleted file mode 100644 index d37151ef7d..0000000000 Binary files a/docs/source/devguides/pycharm/files/02_set_project_interpreter.png and /dev/null differ diff --git a/docs/source/devguides/pycharm/files/03_run_settings.png b/docs/source/devguides/pycharm/files/03_run_settings.png deleted file mode 100644 index ad87c45697..0000000000 Binary files a/docs/source/devguides/pycharm/files/03_run_settings.png and /dev/null differ diff --git a/docs/source/devguides/pycharm/files/04_listener_settings.png b/docs/source/devguides/pycharm/files/04_listener_settings.png deleted file mode 100644 index 9e0659ae68..0000000000 Binary files a/docs/source/devguides/pycharm/files/04_listener_settings.png and /dev/null differ diff --git a/docs/source/devguides/pycharm/files/05_run_listener.png b/docs/source/devguides/pycharm/files/05_run_listener.png deleted file mode 100644 index dc007e0ef2..0000000000 Binary files a/docs/source/devguides/pycharm/files/05_run_listener.png and /dev/null differ diff --git a/docs/source/devguides/pycharm/files/06_run_tests.png b/docs/source/devguides/pycharm/files/06_run_tests.png deleted file mode 100644 index 63f31d1106..0000000000 Binary files a/docs/source/devguides/pycharm/files/06_run_tests.png and /dev/null differ diff --git a/docs/source/devguides/roadmap/3.0-Drivers.rst b/docs/source/devguides/roadmap/3.0-Drivers.rst deleted file mode 100644 index 6384778a21..0000000000 --- a/docs/source/devguides/roadmap/3.0-Drivers.rst +++ /dev/null @@ -1,59 +0,0 @@ -3.X drivers -=========== - -Changes from v2.X ------------------ - -- PNNL Point Name is now: Volttron Point Name -- Drivers are now agents -- No more smap config file, now it is an Agent config file. -- MODBUS, add port argument to driver\_config dictionary -- BACnet Change of Value services are supported by the Master - Driver Agent starting with version 3.2. -- Agent config file has links to driver config files which have links - to driver register file. - -Edit the master driver config. This points to the configuration files -for specific drivers. Each of these drivers uses a CSV file to specify -their points (registry file). - -Master Driver Config --------------------- - -- agentid - name of agent -- driver\_config\_list - list of configuration files for drivers under - this master - - | { - | "agentid": "master\_driver", - | "driver\_config\_list": [ - | "/home/user/git/volttron/services/core/MasterDriverAgent/master\_driver/test\_modbus1.config" - | ] - | } - -Device Driver Config --------------------- - -- driver\_config - driver specific information, modbus just needs the - ip for the device being controlled -- campus/building/unit - path to the device -- driver\_type - specify the type of driver (modbus, bacnet, custom) -- registry\_config - the registry file specifying points to collect -- interval - how often to grab/publish data -- timezone - TZ of data being collected -- heart\_beat\_point - registry point to use as a hearbeat to indicate - that VOLTTRON is still controlling device - - | { - | "driver\_config": {"device\_address": "", - | "proxy\_address": "9f18c8d7-ec4b-4674-ad49-e7d0d3328f99"}, - | "campus": "campus", - | "building": "building", - | "unit": "bacnet1", - | "driver\_type": "bacnet", - | "registry\_config":"/home/user/git/volttron/volttron/drivers/bacnet\_lab.csv", - | "interval": 5, - | "timezone": "UTC" - | } - - diff --git a/docs/source/devguides/roadmap/3.0-to-3.5-Migration.rst b/docs/source/devguides/roadmap/3.0-to-3.5-Migration.rst deleted file mode 100644 index 0a918ac940..0000000000 --- a/docs/source/devguides/roadmap/3.0-to-3.5-Migration.rst +++ /dev/null @@ -1,48 +0,0 @@ -Migration from 3.0 to 3.5 -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Drivers -======= - -The BACnet driver configurations now require device ids. - -3.0 configurations had the line: - -:: - - "driver_config": {"device_address": address}, - -3.5 configs needs the following addition to the the *driver_config* dictionary: - -:: - - "driver_config": {"device_address": address, - "device_id": id}, - - -Historian -========= - -The 3.5 MySQL historian will try adding rows to a metadata table but will -not create the table automatically. It can be added to the database with - -.. code-block:: sql - - CREATE TABLE meta(topic_id INTEGER NOT NULL, - metadata TEXT NOT NULL, - PRIMARY KEY(topic_id)); - - -ActuatorAgent -============= - -The Heartbeat agent has been removed in version 3.5, its job now being -done from within the actuator. The period of the heartbeat toggle function -can be set by adding - -:: - - "heartbeat_period": 20 - -to the actuator's config file. This period defaults to 60 seconds if it -is not specified. diff --git a/docs/source/devguides/roadmap/4.1-5.0_Migration.rst b/docs/source/devguides/roadmap/4.1-5.0_Migration.rst deleted file mode 100644 index fa085119d6..0000000000 --- a/docs/source/devguides/roadmap/4.1-5.0_Migration.rst +++ /dev/null @@ -1,114 +0,0 @@ -.. _4.1_to_5.0: - -Migration from 4.1 to 5.0 -========================= - -5.0 includes numerous changes (Tagging Service, Message Bus performance increase, Multi-platform pub/sub, etc.), but -the majority of these should be invisible to most users. - -Key issues to note are: - - -Operations Agents ------------------ - -Several agents have been moved from "services/core" to "services/ops" to highlight their use in monitoring a -deployment. They are not necessary when developing against a single instance, but are essential for VOLTTRON(tm) in a -deployed environment. - -Agents affected: - -- services/ops/AgentWatcher -- services/ops/AlertAgent,0.4 -- services/ops/AlertMonitor -- services/ops/Alerter -- services/ops/EmailerAgent -- services/ops/FailoverAgent -- services/ops/FileWatchPublisher -- services/ops/LogStatisticsAgent -- services/ops/MessageDebuggerAgent -- services/ops/SysMonAgent -- services/ops/ThresholdDetectionAgent - - -Rebuild Agents --------------- - -Rebuilding agents is :underline:`required` when upgrading to a new VOLTTRON(tm) version to ensure that agents are -operating with the latest code. Errors will occur if agents built in a previous version attempt to run with the -latest version of the platform. - -ForwardHistorian ----------------- -The ForwardHistorian configuration has been changed. -Please see: https://github.com/VOLTTRON/volttron/blob/develop/services/core/ForwardHistorian/README.rst for the -new options. - -.. note:: NOTE If you have no entry for service_topic_list in your configuration, the new default will cause - ALL data to be forwarded. Please update your configuration if you are forwarding a subset of data. - - -VOLTTRON Central Management UI ------------------------------- - -The url for VOLTTRON Central Management is now http://IP:port/vc/index.html - -Agent Versions --------------- - -To get the versions of agents in the VOLTTRON project, run "python scripts/get_versions.py". - - -========================== ====== ======= -Agent Name 4.1 5.0 -========================== ====== ======= -CAgent 1.0 1.0 -CSVHistorian N/A 1.0.1 -ConfigActuation 0.1 0.1 -DataPublisher 3.0.1 3.0.1 -DataPuller N/A 3.5 -ExampleDrivenControlAgent 0.1 0.1 -ExampleSubscriber 3.0 3.0 -ListenerAgent 3.2 3.2 -ProcessAgent 0.1 0.1 -SchedulerExample 0.1 0.1 -SimpleForwarder 3.0 3.0 -SimpleWebAgent 0.1 0.1 -WeatherForecastCSV_UW -WebRPC -WebSocketAgent 0.0.1 0.0.1 -PrometheusScrapeAgent N/A 0.0.1 -WeatherAgent -ActuatorAgent 1.0 1.0 -BACnetProxy 0.2 0.3 -CrateHistorian 1.0.1 1.0.2 -DataMover 0.1 0.1 -ExternalData 1.0 1.0 -ForwardHistorian 3.7 4.0 -MQTTHistorian 0.1 0.2 -MasterDriverAgent 3.1.1 3.1.1 -MongodbAggregateHistorian 1.0 1.0 -MongodbHistorian 2.1 2.1 -MongodbTaggingService N/A 1.0 -OpenEISHistorian 3.1 3.1 -SEP2Agent N/A 1.0 -SEP2DriverTestAgent N/A 1.0 -SQLAggregateHistorian 1.0 1.0 -SQLHistorian 3.6.1 3.6.1 -SQLiteTaggingService N/A 1.0 -VolttronCentral 4.0.3 4.2 -VolttronCentralPlatform 4.0 4.5.2 -WeatherAgent 3.0 3.0 -AgentWatcher 0.1 0.1 -AlertAgent 0.4 0.4 -AlertMonitor 0.1 0.1 -Alerter 0.1 0.1 -EmailerAgent 1.3 1.3.1 -FailoverAgent 0.2 0.2 -FileWatchPublisher 3.6 3.6 -LogStatisticsAgent 1.0 1.0 -MessageDebuggerAgent N/A 1.0 -SysMonAgent 3.6 3.6 -ThresholdDetectionAgent 3.7 3.7 -========================== ====== ======= - diff --git a/docs/source/devguides/roadmap/index.rst b/docs/source/devguides/roadmap/index.rst deleted file mode 100644 index 7215a50ad3..0000000000 --- a/docs/source/devguides/roadmap/index.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. _Roadmap: - -=============================== -Development History and Roadmap -=============================== - -For information on updating to the latest version of the platform see :ref:`4.0 to 5.0 migration <4.1_to_5.0>`. - - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/devguides/scalability/index.rst b/docs/source/devguides/scalability/index.rst deleted file mode 100644 index ed09005d88..0000000000 --- a/docs/source/devguides/scalability/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -======================= -Scalability Experiments -======================= - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/devguides/supporting/JupyterNotebooks.rst b/docs/source/devguides/supporting/JupyterNotebooks.rst deleted file mode 100644 index f3005cc120..0000000000 --- a/docs/source/devguides/supporting/JupyterNotebooks.rst +++ /dev/null @@ -1,210 +0,0 @@ -.. _Jupyter-Notebooks: - -Jupyter Notebooks -================= - -Jupyter is an open-source web application that lets you create and share “notebook” documents. -A notebook displays formatted text along with live code that can be executed from the browser, -displaying the execution output and preserving it in the document. -Notebooks that execute Python code used to be called `iPython Notebooks`. -The iPython Notebook project has now merged into Project Jupyter. - -Using Jupyter to Manage a Set of VOLTTRON Servers -------------------------------------------------- - -The following Jupyter notebooks for VOLTTRON have been provided as examples: - - - **Collector notebooks**. Each Collector notebook sets up a particular type of device driver - and forwards device data to another VOLTTRON instance, the Aggregator. - - - **SimulationCollector notebook**. This notebook sets up a group of Simulation device drivers - and forwards device data to another VOLTTRON instance, the Aggregator. - - **BacnetCollector notebook**. This notebook sets up a Bacnet (or Bacnet gateway) device driver - and forwards device data to another VOLTTRON instance, the Aggregator. - - **ChargePointCollector notebook**. This notebook sets up a ChargePoint device driver - and forwards device data to another VOLTTRON instance, the Aggregator. - - **SEP2Collector notebook**. This notebook sets up a SEP2.0 (IEEE 2030.5) device driver - and forwards device data to another VOLTTRON instance, the Aggregator. - The Smart Energy Profile 2.0 ("SEP2") protocol implements IEEE 2030.5, and is capable - of connecting a wide array of smart energy devices to the Smart Grid. The standard is - designed to run over TCP/IP and is physical layer agnostic. - - - **Aggregator notebook**. This notebook sets up and executes aggregation of forwarded data - from other VOLTTRON instances, using a historian to record the data. - - **Observer notebook**. This notebook sets up and executes a DataPuller that captures data from - another VOLTTRON instance, using a Historian to record the data. It also uses the - Message Debugger agent to monitor messages flowing across the VOLTTRON bus. - -Each notebook configures and runs a set of VOLTTRON Agents. When used as a set, they -implement a multiple-VOLTTRON-instance architecture that catures remote device data, aggregates it, -and reports on it, routing the data as follows: - -.. image:: files/jupyter_notebooks.jpg - - -Install VOLTTRON and Jupyter on a Server ----------------------------------------- - -The remainder of this guide describes how to set up a host for VOLTTRON and Jupyter. -Use this setup process on a server in order to prepare it to run Jupyter notebook for VOLLTTRON. - -**Set Up the Server and Install VOLTTRON** - -The following is a complete, but terse, description of the steps for installing -and running VOLTTRON on a server. For more detailed, general instructions, -see :ref:`Installing Volttron `. - -The VOLTTRON server should run on the same host as the Jupyter server. - -*Load third-party software:* -:: - - $ sudo apt-get update - $ sudo apt-get install build-essential python-dev openssl libssl-dev libevent-dev git - $ sudo apt-get install sqlite3 - -*Clone the VOLTTRON repository from github:* -:: - - $ cd ~ - $ mkdir repos - $ cd repos - $ git clone https://github.com/VOLTTRON/volttron/ - -*Check out the develop (or master) branch and bootstrap the development environment:* -:: - - $ cd volttron - $ git checkout develop - $ python bootstrap.py - -*Activate and initialize the VOLTTRON virtual environment:* - -Run the following each time you open a new command-line shell on the server: -:: - - $ export VOLTTRON_ROOT=~/repos/volttron - $ export VOLTTRON_HOME=~/.volttron - $ cd $VOLTTRON_ROOT - $ source env/bin/activate - -**Install Extra Libraries** - -*Add Python libraries to the VOLTTRON virtual environment:* - -These notebooks use third-party software that's not included in VOLTTRON's standard distribution -that was loaded by ``bootstrap.py``. The following additional packages are required: - -- Jupyter -- SQLAlchemy (for the Message Debugger) -- Suds (for the ChargePoint driver) -- Numpy and MatPlotLib (for plotted output) - -*Note*: A Jupyter installation also installs and/or upgrades many dependent libraries. -Doing so could disrupt other work on the OS, so it’s safest to load Jupyter (and any other -library code) in a virtual environment. VOLTTRON runs in a virtual environment anyway, -so if you're using Jupyter in conjunction with VOLTTRON, it should be installed in your -VOLTTRON virtual environment. -(In other words, be sure to use ``cd $VOLTTRON_ROOT`` and ``source env/bin/activate`` -to activate the virtual environment before running ``pip install``.) - -*Install the third-party software:* -:: - - $ pip install SQLAlchemy==1.1.4 - $ pip install suds-jurko==0.6 - $ pip install numpy - $ pip install matplotlib - $ pip install jupyter - -Note: If ``pip install`` fails due to an untrusted cert, try using this command instead: -:: - - $ pip install --trusted-host pypi.python.org - -(An InsecurePlatformWarning may be displayed, but it typically won't stop the installation from proceeding.) - -**Configure VOLTTRON** - -Use the ``vcfg`` wizard to configure the VOLTTRON instance. By default, the wizard -configures a VOLTTRON instance that communicates with agents only on the local host (ip 127.0.0.1). -This set of notebooks manages communications among multiple VOLTTRON instances on different hosts. -To enable this cross-host communication on VOLTTRON's web server, replace 127.0.0.1 with the -host's IP address, as follows: -:: - - $ vcfg - -- Accept all defaults, except as follows. -- If a prompt defaults to 127.0.0.1 as an IP address, substitute the ``host's IP address`` (this may happen multiple times). -- When asked whether this is a volttron central, answer ``Y``. -- When prompted for a username and password, use ``admin`` and ``admin``. - -**Start VOLTTRON** - -Start the main VOLTTRON process, logging to $VOLTTRON_ROOT/volttron.log: -:: - - $ volttron -vv -l volttron.log --msgdebug - -This runs VOLTTRON as a foreground process. To run it in the background, use: -:: - - $ ./start-volttron --msgdebug - -This also enables the Message Debugger, a non-production VOLTTRON debugging aid -that's used by some notebooks. To run with the Message Debugger disabled (VOLTTRON's normal state), -omit the ``--msgdebug`` flag. - -Now that VOLTTRON is running, it's ready for agent configuration and execution. -Each Jupyter notebook contains detailed instructions and executable code for doing that. - -**Configure Jupyter** - -More detailed information about installing, configuring and using Jupyter Notebooks is available -on the Project Jupyter site, http://jupyter.org/. - -*Create a Jupyter configuration file:* -:: - - $ jupyter notebook --generate-config - -*Revise the Jupyter configuration:* - -Open ``~/.jupyter/jupyter_notebook_config.py`` in your favorite text editor. -Change the configuration to accept connections from any IP address (not just from localhost) -and use a specific, non-default port number: - -- Un-comment ``c.NotebookApp.ip`` and set it to: ``'*'`` instead of ``'localhost'`` -- Un-comment ``c.NotebookApp.port`` and set it to: ``'8891'`` instead of ``'8888'`` - -Save the config file. - -*Open ports for TCP connections:* - -Make sure that your Jupyter server host's security rules allow inbound TCP connections on port ``8891``. - -If the VOLTTRON instance needs to receive TCP requests, for example ForwardHistorian or DataPuller -messages from other VOLTTRON instances, make sure that the host's security rules also allow inbound TCP -communications on VOLTTRON's port, which is usually ``22916``. - -**Launch Jupyter** - -*Start the Jupyter server:* - -In a separate command-line shell, set up VOLTTRON's environment variables and virtual environment, -and then launch the Jupyter server: -:: - - $ export VOLTTRON_HOME=(your volttron home directory, e.g. ~/.volttron) - $ export VOLTTRON_ROOT=(where volttron was installed; e.g. ~/repos/volttron) - $ cd $VOLTTRON_ROOT - $ source env/bin/activate - $ cd examples/JupyterNotebooks - $ jupyter notebook --no-browser - -*Open a Jupyter client in a web browser:* - -Look up the host's IP address (e.g., using ifconfig). Open a web browser and navigate to -the URL that was displayed when you started jupyter, replacing ``localhost`` with that -IP address. A Jupyter web page should display, listing your notebooks. diff --git a/docs/source/devguides/supporting/Python-for-Matlab-Users.rst b/docs/source/devguides/supporting/Python-for-Matlab-Users.rst deleted file mode 100644 index 78af2f0c41..0000000000 --- a/docs/source/devguides/supporting/Python-for-Matlab-Users.rst +++ /dev/null @@ -1,170 +0,0 @@ -.. _Python-for-Matlab-Users: - -Python for Matlab Users -======================= - -Matlab is a popular, proprietary programming language and tool suite with built -in support for matrix operations and graphically plotting computation results. -The purpose of this document is to introduce Python to those already familiar -Matlab so it will be easier for them to develop tools and agents in VOLTTRON. - -A Simple Function ------------------ - -Python and Matlab are similar in many respects, syntactically and semantically. -With the addition of the NumPy library in Python, almost all numerical -operations in Matlab can be emulated or directly translated. Here are functions -in each language that perform the same operation: - -.. code-block:: matlab - - % Matlab - function [result] = times_two(number) - result = number * 2; - end - -.. code-block:: python - - # Python - def times_two(number): - result = number * 2 - return result - -Some notes about the previous functions: - -#. Values are explicitly returned with the `return` statement. It is possible - to return multiple values, as in Matlab, but doing this without a good reason - can lead to overcomplicated functions. - -#. Semicolons are not used to end statements in python, and white space is - significant. After a block is started (if, for, while, functions, classes) - subsequent lines should be indented with four spaces. The block ends when the - programmer stops adding the extra level of indentation. - -Translating ------------ - -The following may be helpful if you already have a Matlab file or function -that will be translated into Python. Many of the syntactic differences -between Matlab and Python can be rectified with your text editor's find and -replace feature. - -Start by copying all of your Matlab code into a new file with a `.py` -extension. I recommend commenting everything out and uncommenting the -Matlab code in chunks. This way you can write valid Python and verify -it as you translate, instead of waiting till the whole file is "translated". -Editors designed to work with Python should be able to highlight syntax errors -for you as well. - -#. Comments are created with a `%`. Find and replace these with `#`. - -#. Change `elseif` blocks to `elif` blocks. - -#. Python indexes start at zero instead of one. Array slices - and range operations, however, don't include the upper bound, so only the - lower bound should decrease by one. - -#. Semicolons in Matlab are used to suppress output at the - end of lines and for organizing array literals. After arranging the arrays - into nested lists, all semicolons can be removed. - -#. The `end` keyword in Matlab is used both to access the last element - in an array and to close blocks. The array use case can be replaced with `-1` - and the others can be removed entirely. - - -A More Concrete Example ------------------------ - -In the `Building Economic Dispatch `_ -project, a sibling project to VOLTTRON, a number of components written in Matlab -would create a matrix out of some collection of columns and perform least -squares regression using the `matrix division` operator. This is straightforward -and very similar in both languages so long as all of the columns are defined and -are the same length. - -.. code-block:: matlab - - % Matlab - XX = [U, xbp, xbp2, xbp3, xbp4, xbp5]; - AA = XX \ ybp; - -.. code-block:: python - - # Python - import numpy as np - - XX = np.column_stack((U, xbp, xbp2, xbp3, xbp4, xbp5)) - AA, resid, rank, s = np.linalg.lstsq(XX, ybp) - -This pattern also included the creation of the `U` column, a column of -ones used as the bias term in the linear equation. In order to make the Python -version more readable and more robust, the pattern was removed from each -component and replaced with a single function call to -`least_squares_regression`. - -This function does some validation on the input -parameters, automatically creates the bias column, and returns the least squares -solution to the system. Now if we want to change how the solution is calculated -we only have to change the one function, instead of each instance where the -pattern was written originally. - -.. code-block:: python - - def least_squares_regression(inputs=None, output=None): - if inputs is None: - raise ValueError("At least one input column is required") - if output is None: - raise ValueError("Output column is required") - - if type(inputs) != tuple: - inputs = (inputs,) - - ones = np.ones(len(inputs[0])) - x_columns = np.column_stack((ones,) + inputs) - - solution, resid, rank, s = np.linalg.lstsq(x_columns, output) - return solution - -Lessons Learned (sometimes the hard way) ----------------------------------------- - -Variable Names -~~~~~~~~~~~~~~ - -Use descriptive function and variable names whenever possible. The most -important things to consider here are reader comprehension and searching. -Consider a variable called `hdr`. Is it `header` without any vowels, or is it -short for `high-dynamic-range`? Spelling out full words in variable names can -save someone else a lot of guesswork. - -Searching comes in when we're looking for instances of a string or variable. -Single letter variable names are impossible to search for. Variables with two -or three characters are often not much better. - -Matlab load/save -~~~~~~~~~~~~~~~~ - -Matlab has built-in functions to automatically save and load variables from your -programs to disk. Using these functions can lead to poor program design and -should be avoided if possible. It would be best to refactor as you translate if -they are being used. Few operations are so expensive that that cannot be -redone every time the program is run. For part of the program that saves -variables, consider making a function that simply returns them instead. - -If your Matlab program is loading csv files then use the Pandas library when -working in python. Pandas works well with NumPy and is the go-to library when -using csv files that contain numeric data. - -More Resources --------------- - -`NumPy for Matlab Users -`_ -Has a nice list of common operations in Matlab and NumPy. - -`NumPy Homepage -`_ - -`Pandas Homepage -`_ diff --git a/docs/source/devguides/supporting/applications/index.rst b/docs/source/devguides/supporting/applications/index.rst deleted file mode 100755 index 83e4d7f302..0000000000 --- a/docs/source/devguides/supporting/applications/index.rst +++ /dev/null @@ -1,14 +0,0 @@ -============ -Applications -============ -These resources summarize the use of the sample applications that are pre-packaged with VOLTTRON. For detailed -information on these applications, refer to the report Transactional Network Platform: Applications available -at http://www.pnl.gov/main/publications/external/technical_reports/PNNL-22941.pdf. - -Note, as of VOLTTRON 4.0, applications are now in their own repository at: https://github.com/VOLTTRON/volttron-applications - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/devguides/supporting/examples/CAgent.rst b/docs/source/devguides/supporting/examples/CAgent.rst deleted file mode 100644 index 578716a16b..0000000000 --- a/docs/source/devguides/supporting/examples/CAgent.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. _CAgent: - -CAgent -====== - -The C Agent uses the ctypes module to load a shared object into memory -so its functions can be called from python. - -There are two versions of the C Agent. The first is a standard agent that can -be installed with the make agent script. The other is a driver interface for -the master driver. - -Building the Shared Object --------------------------- - -The shared object library must be built before installing C Agent examples. -Running *make* in the C Agent source directory will compile the provided C code -using the position independent flag; a requirement for creating shared objects. - -Files created by make can be removed by running *make clean*. - -Agent Installation ------------------- - -After building the shared object library the standard agent can be installed -with the scripts/install-agent.py script. - -The driver interface example must be copied or moved to the master driver's -interface directory. The C Driver configuration tells the interface where to -find the shared object. An example is available in the C Agent's *driver* -directory. diff --git a/docs/source/devguides/supporting/examples/Example-Agents.rst b/docs/source/devguides/supporting/examples/Example-Agents.rst deleted file mode 100644 index 5f330ee570..0000000000 --- a/docs/source/devguides/supporting/examples/Example-Agents.rst +++ /dev/null @@ -1,41 +0,0 @@ -.. _Example-Agents: - -Example Agents Overview -======================= - -Some example agents are included with the platform to help explore its -features. - -- :ref:`DataPublisher` -- :ref:`ListenerAgent` -- :ref:`ProcessAgent` -- :ref:`SchedulerExampleAgent` -- :ref:`CAgent` -- :ref:`DDSAgent` -- :ref:`CSVHistorian` - -More complex agents contributed by other researchers can also be found -in the examples directory. It is recommended that developers new to -VOLTTRON understand the example agents first before diving into the -other agents. - -Example Agent Conventions -------------------------- - -Some of the example agent classes are defined inside a method, for -instance: - -:: - - def ScheduleExampleAgent(config_path, **kwargs): - config = utils.load_config(config_path) - campus= config['campus'] - -This allows configuration information to be extracted from an agent -config file for use in topics. - -:: - - @Pubsub.subscribe('pubsub', DEVICES_VALUE(campus=campus)) - def actuate(self, peer, sender, bus, topic, headers, message): - diff --git a/docs/source/devguides/supporting/examples/FakeDriver.rst b/docs/source/devguides/supporting/examples/FakeDriver.rst deleted file mode 100644 index 1777ce244b..0000000000 --- a/docs/source/devguides/supporting/examples/FakeDriver.rst +++ /dev/null @@ -1,67 +0,0 @@ -.. _FakeDriver: - -.. role:: bash(code) - :language: bash - -Fake Driver -=========== - -The FakeDriver is included as a way to quickly see data published to the message bus in a format -that mimics what a true Driver would produce. This is an extremely simple implementation of the -:ref:`VOLTTRON driver framework`. - -Here, we make a script to build and deploy the fake driver. - - -- Create a config directory (if one doesn't already exist) inside your Volttron repository: :code:`mkdir config`. All local config files will be worked on here. -- Copy over the example config file and registry config file: - -.. code-block:: bash - - `cp examples/configurations/drivers/fake.config config/` - `cp examples/configurations/drivers/fake.csv config/` - -- Edit :code:`registry_config` for the paths on your system: - -fake.config:: - - { - "driver_config": {}, - "registry_config": "config://fake.csv", - "interval": 5, - "timezone": "US/Pacific", - "heart_beat_point": "Heartbeat", - "driver_type": "fakedriver", - "publish_breadth_first_all": false, - "publish_depth_first": false, - "publish_breadth_first": false - } - -- Create a copy of the Master Driver config: - -.. code-block:: bash - - cp examples/configurations/drivers/master-driver.agent config/fake-master-driver.config - -- Add fake.csv and fake.config to the :ref:`configuration store`: - -.. code-block:: bash - - vctl config store platform.driver devices/campus/building/fake config/fake.config - vcfl config store platform.driver fake.csv config/fake.csv --csv - -- Edit fake-master-driver.config to reflect paths on your system - -fake-master-driver.config:: - - { - "driver_scrape_interval": 0.05 - } - -- Use the scripts/install-agent.py script to install the Master Driver agent: - -.. code-block:: bash - - python scripts/install-agent.py -s services/core/MasterDriverAgent -c config/fake-master-driver.config - -- If you have a :ref:`Listener Agent` already installed, you should start seeing data being published to the bus. diff --git a/docs/source/devguides/supporting/examples/MatLabAgent.rst b/docs/source/devguides/supporting/examples/MatLabAgent.rst deleted file mode 100644 index cb6a7e2b77..0000000000 --- a/docs/source/devguides/supporting/examples/MatLabAgent.rst +++ /dev/null @@ -1,610 +0,0 @@ -.. _MatlabAgent: - -MatLab Agent -============ - -The MatLab agent and Matlab standalone agent together are -example agents that allow for matlab scripts to be run in a -Windows environment and interact with the VOLTTRON platform running in a Linux environment. -The MatLab agent takes advantage of the config store to -dynamically send scripts and commandline arguments across -the message bus to one or more standalone agents in -Windows. The standalone agent then executes the requested script -and arguments, and sends back the results to the MatLab agent. - - -Overview of Matlab Agents -------------------------- - -There are multiple components that are used for the matlab agent. -This diagram is to represent the components that are connected to -the matlab agents. In this example, the scripts involved are -based on the default settings in the matlab agent. - -|matlab-agent-diagram| - -MatLabAgentV2 -~~~~~~~~~~~~~ - -MatLabAgentV2 publishes the name of a python script along with any command -line arguments that are needed for the script to the appropriate topic. -The agent then listens on another topic, and whenever anything is published -on this topic, it stores the message in the log file chosen when the volttron -instance is started. If there are multiple standalone agents, the agent can -send a a script to each of them, along with their own set of command line -arguments. In this case, each script name and set of command line arguments -should be sent to separate subtopics. This is done so that no matter how many -standalone agents are in use, MatLabAgentV2 will record all of their responses. - -.. code:: - - class MatlabAgentV2(Agent): - - def __init__(self,script_names=[], script_args=[], topics_to_matlab=[], - topics_to_volttron=None,**kwargs): - - super(MatlabAgentV2, self).__init__(**kwargs) - _log.debug("vip_identity: " + self.core.identity) - - self.script_names = script_names - self.script_args = script_args - self.topics_to_matlab = topics_to_matlab - self.topics_to_volttron = topics_to_volttron - self.default_config = {"script_names": script_names, - "script_args": script_args, - "topics_to_matlab": topics_to_matlab, - "topics_to_volttron": topics_to_volttron} - - - #Set a default configuration to ensure that self.configure is called immediately to setup - #the agent. - self.vip.config.set_default("config", self.default_config) - #Hook self.configure up to changes to the configuration file "config". - self.vip.config.subscribe(self.configure, actions=["NEW", "UPDATE"], pattern="config") - - def configure(self, config_name, action, contents): - """ - Called after the Agent has connected to the message bus. - If a configuration exists at startup this will be - called before onstart. - Is called every time the configuration in the store changes. - """ - config = self.default_config.copy() - config.update(contents) - - _log.debug("Configuring Agent") - - try: - script_names = config["script_names"] - script_args = config["script_args"] - topics_to_matlab = config["topics_to_matlab"] - topics_to_volttron = config["topics_to_volttron"] - - except ValueError as e: - _log.error("ERROR PROCESSING CONFIGURATION: {}".format(e)) - return - - self.script_names = script_names - self.script_args = script_args - self.topics_to_matlab = topics_to_matlab - self.topics_to_volttron = topics_to_volttron - self._create_subscriptions(self.topics_to_volttron) - - for script in range(len(self.script_names)): - cmd_args = "" - for x in range(len(self.script_args[script])): - cmd_args += ",{}".format(self.script_args[script][x]) - _log.debug("Publishing on: {}".format(self.topics_to_matlab[script])) - self.vip.pubsub.publish('pubsub', topic=self.topics_to_matlab[script], - message="{}{}".format(self.script_names[script],cmd_args)) - _log.debug("Sending message: {}{}".format(self.script_names[script],cmd_args)) - - _log.debug("Agent Configured!") - -For this example, the agent is publishing to the matlab/to_matlab/1 topic, -and is listening to the matlab/to_volttron topic. It is sending the script -name testScript.py with the argument 20. These are the default values found -in the agent, if no configuration is loaded. - -.. code:: - - script_names = config.get('script_names', ["testScript.py"]) - script_args = config.get('script_args', [["20"]]) - topics_to_matlab = config.get('topics_to_matlab', ["matlab/to_matlab/1"]) - topics_to_volttron = config.get('topics_to_volttron', "matlab/to_volttron/") - -StandAloneMatLab.py -~~~~~~~~~~~~~~~~~~~ - -The StandAloneMatLab.py script is a standalone agent designed to be able to -run in a windows environment. Its purpose is to listen to a topic, and when -something is published to this topic, it takes the message, and sends it to -the script_runner function in scriptwrapper.py. This function processes the -inputs, and then the output is published to another topic. - -.. code:: - - class StandAloneMatLab(Agent): - '''The standalone version of the MatLab Agent''' - - @PubSub.subscribe('pubsub', _topics['volttron_to_matlab']) - def print_message(self, peer, sender, bus, topic, headers, message): - print('The Message is: ' + str(message)) - messageOut = script_runner(message) - self.vip.pubsub.publish('pubsub', _topics['matlab_to_volttron'], message=messageOut) - -settings.py -~~~~~~~~~~~ - -The topic to listen to and the topic to publish to are defined in settings.py, -along with the information needed to connect the standalone agent to the primary -volttron instance. These should be the same topics that the MatLabAgentV2 is -publishing and listening to, so that the communication can be successful. To -connect the standalone agent to the primary volttron instance, the ip address -and port of the instance are needed, along with the server key. - -.. code:: - - _topics = { - 'volttron_to_matlab': 'matlab/to_matlab/1', - 'matlab_to_volttron': 'matlab/to_volttron/1' - } - - # The parameters dictionary is used to populate the agent's - # remote vip address. - _params = { - # The root of the address. - # Note: - # 1. volttron instance should be configured to use tcp. use command vcfg - # to configure - 'vip_address': 'tcp://192.168.56.101', - 'port': 22916, - - # public and secret key for the standalone_matlab agent. - # These can be created using the command: volttron-ctl auth keypair - # public key should also be added to the volttron instance auth - # configuration to enable standalone agent access to volttron instance. Use - # command 'vctl auth add' Provide this agent's public key when prompted - # for credential. - - 'agent_public': 'dpu13XKPvGB3XJNVUusCNn2U0kIWcuyDIP5J8mAgBQ0', - 'agent_secret': 'Hlya-6BvfUot5USdeDHZ8eksDkWgEEHABs1SELmQhMs', - - # Public server key from the remote platform. This can be - # obtained using the command: - # volttron-ctl auth serverkey - 'server_key': 'QTIzrRGQ0-b-37AbEYDuMA0l2ETrythM2V1ac0v9CTA' - - } - - def remote_url(): - return "{vip_address}:{port}?serverkey={server_key}" \ - "&publickey={agent_public}&" \ - "secretkey={agent_secret}".format(**_params) - -The primary volttron instance will then need to add the public key from the -standalone agent. In this example, the topic that the standalone agent is -listening to is matlab/to_matlab/1, and the topic it is publishing to is matlab/to_volttron/1. - -scriptwrapper.py -~~~~~~~~~~~~~~~~ - -Scriptwrapper.py contains the script_runner function. The purpose of -this function is to take in a string that contains a python script -and command line arguments separated by commas. This string is parsed -and passed to the system arguments, which allows the script sent to -the function to use the command line arguments. The function then -redirects standard output to a StringIO file object, and then attempts -to execute the script. If there are any errors with the script, the -error that is generated is returned to the standalone agent. Otherwise, -the file object stores the output from the script, is converted to a string, -and is sent to the standalone agent. -In this example, the script that is to be run is testScript.py. - -.. code:: - - #Script to take in a string, run the program, - #and output the results of the command as a string. - - import time - import sys - from io import StringIO - - - def script_runner(message): - original = sys.stdout - # print(message) - # print(sys.argv) - sys.argv = message.split(',') - # print(sys.argv) - - try: - out = StringIO() - sys.stdout = out - exec(open(sys.argv[0]).read()) - sys.stdout = original - return out.getvalue() - except Exception as ex: - out = str(ex) - sys.stdout = original - return out - -.. note:: - - The script that is to be run needs to be in the same folder as the agent - and the scriptwrapper.py script. The script_runner function needs to be edited - if it is going to call a script at a different location. - - -testScript.py -~~~~~~~~~~~~~ - -This is a very simple test script designed to demonstrate the -calling of a matlab function from within python. First it initializes -the matlab engine for python. It then takes in a single command line -argument, and passes it to the matlab function testPy.m. If no -arguments are sent, it will send 0 to the testPy.m function. It then -prints the result of the testPy.m function. In this case, since -standard output is being redirected to a file object, this is -how the result is passed from this function to the standalone agent. - -.. code:: - - import matlab.engine - import sys - - - eng = matlab.engine.start_matlab() - - if len(sys.argv) == 2: - result = eng.testPy(float(sys.argv[1])) - else: - result = eng.testPy(0.0) - - print(result) - -testPy.m -~~~~~~~~ - -This matlab function is a very simple example, designed to show a -function that takes an argument, and produces an array as the output. -The input argument is added to each element in the array, and the -entire array is then returned. - -.. code:: - - function out = testPy(z) - x = 1:100 - out = x + z - end - -Setup on Linux --------------- - -1. Setup and run Volttron from develop branch using instructions :ref:`here `. - -2. Configure volttron instance using the ``vcfg`` command. - When prompted for the vip address use tcp://. - This is necessary to enable volttron communication with external processes. - - .. note:: - - If you are running VOLTTRON from within VirtualBox, It would be good to set - one of your adapters as a Host-only adapter. This can be done within the - VM's settings, under the Network section. Once this is done, use this IP - for the vip address. - - -.. _MatlabAgent_config: - -3. Update the configuration for MatLabAgent_v2 at /example/MatLabAgent_v2/config. - - The configuration file for the MatLab agent has four variables. - - 1. script_names - - 2. script_args - - 3. topics_to_matlab - - 4. topics_to_volttron - - An example config file included with the folder. - - .. code:: - - { - # VOLTTRON config files are JSON with support for python style comments. - "script_names": ["testScript.py"], - "script_args": [["20"]], - "topics_to_matlab": ["matlab/to_matlab/1"], - "topics_to_volttron": "matlab/to_volttron/" - } - - To edit the configuration, the format should be as follows: - - .. code:: - - { - "script_names": ["script1.py", "script2.py", ...], - "script_args": [["arg1","arg2"], ["arg1"], ...], - "topics_to_matlab": ["matlab/to_matlab/1", "matlab/to_matlab/2", ...], - "topics_to_volttron": "matlab/to_volttron/" - } - - The config requires that each script name lines up with a set of - commandline arguments and a topic. So a commandline argument - must be included, even if it is not used. The placement of - brackets are important, even when only communicating with one - standalone agent. - - For example, if only one standalone agent is used, and no command line - arguments are in place, the config file may look like this. - - .. code:: - - { - "script_names": ["testScript.py"], - "script_args": [["0"]], - "topics_to_matlab": ["matlab/to_matlab/1"], - "topics_to_volttron": "matlab/to_volttron/" - } - - -4. Install MatLabAgent_v2 and start agent (from volttron root directory) - - ``python ./scripts/install-agent.py -s examples/MatLabAgent_v2 --start`` - - .. note:: - - The MatLabAgent_v2 pulishes the command to be run to the message bus only on start or on a - configuration update. Once we configure the standalone_matlab agent on the windows machine, we will - send a configuration update to the running MatLabAgent_v2. The configuration would contain the topics to - which the standalone agent is listening to and will be publishing result to. - - .. seealso:: - - The MatLab agent uses the configuration store to dynamically change inputs. - More information on the config store and how it used can be found here. - - * :ref:`VOLTTRON Configuration Store ` - - * :ref:`Agent Configuration Store ` - - * :ref:`Agent Configuration Store Interface ` - -5. Run the below command and make a note of the server key. This is required for configuring the stand alone agent - on windows. (This is run on the linux machine) - - ``vctl auth serverkey`` - - -Setup on Windows ----------------- - -Install pre-requisites -~~~~~~~~~~~~~~~~~~~~~~~ - -1. Install python 3.6 64-bit from `here `__. - -2. Install MatLab engine from `here `_. - - .. warning:: - - The MatLab engine for Python only supports certain version of Python - depending on the version of MatLab used. - Please check `here `__ to see - if the current version of MatLab supports your version of Python. - - -.. note:: - - At this time, you may want to verify that you are able to communicate with - your Linux machine across your network. The simplest method would be to open - up the command terminal and use ``ping ``, and ``telnet - `` Please make sure that the port is opened for outside access. - -Install StandAloneMatLab Agent -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The standalone MatLab agent is designed to be usable in a -Windows environment. - -.. warning:: - - VOLTTRON is not designed to run in a Windows environment. - Outside of cases where it is stated to be usable in a - Windows environment, it should be assumed that it will - NOT function as expected. - - -1. Download VOLTTRON - - Download the VOLTTRON develop repository from github. Download the zip - from `GitHub `_. - - |github-image| - - |github-zip-image| - - Once the zipped file has been downloaded, go to your Downloads folder, - right-click on the file, and select "Extract All..." - - |extract-image_1| - - Choose a location for the extracted folder, and select "Extract" - - |extract-image_2| - - -2. Setup the PYTHONPATH - - Open the Windows explorer, and navigate to "Edit environment variables for your account." - - |cmd-image| - - Select "New" - - |env-vars-image_1| - - For "Variable name" enter: "PYTHONPATH" - For "Variable value" either browse to your volttron installation, or enter in the path to your volttron installation. - - |env-vars-image_2| - - Select "OK" twice. - -3. Set python version in MatLab - - Open your MatLab application. Run the command **pyversion** This should print the path to python2.7. If you have - multiple versions of python on your machine and pyversion points to a different version of python, - use **pyversion /path/to/python.exe** to set the appropriate version of python for your system. - For example, to use python 3.6 with MatLab: - - .. code:: - - pyversion C:\Python36\python.exe - -4. Set up the environment. - - Open up the command prompt. - - |cmd-image_2| - - Naviage to your volttron installation. - - ``cd \Your\directory\path\to\volttron-develop`` - - Use pip to install and setup dependencies. - - ``pip install -r examples\StandAloneMatLab\requirements.txt`` - - ``pip install -e .`` - - .. note:: - - If you get the error doing the second step because of an already installed volttron - from a different directory, manually delete the volttron-egg.link file from your - \\Lib\\site-pacakages directory (for example, - del C:\\Python27\\lib\\site-packages\\volttron-egg.link ) - and re-run the second command - -5. Configure the agent - - The configuration settings for the standalone agent are in setting.py (located in volttron-develop\\examples\\StandAloneMatLab\\) - - **settings.py** - - * 'volttron_to_matlab' needs to be set to the topic that will send your script - and command line arguments to your stand alone agent. This was defined in :ref:`config. ` - - * 'matlab_to_volttron' needs to be set to the topic that will send your script's - output back to your volttron platform. This was defined in :ref:`config. ` - - * 'vip_address' needs to be set to the address of your volttron instance - - * 'port' needs to be set to the port of your volttron instance - - * 'server_key' needs to be set to the public server key of your primary volttron platform. - This can be obtained from the primary volttron platform using ``vctl auth serverkey``. - (volttron must be running to use this command) - - - It is possible to have multiple standalone agents running. In this case, - copy the StandAloneMatLab folder, and make the necessary changes to the - new settings.py file. Unless it is connecting to a separate volttron instance, - you should only need to change the volttron_to_matlab. - - .. note:: - - It is recommended that you generate a new agent_public and agent_private - key for your standalone agent. This can be done using the ``vctl auth keypair`` - command on your primary volttron platform on Linux. If you plan to use multiple standalone agents, - they will each need their own keypair. - -6. Add standalone agent key to volttron platform - - * Copy the public key from settings.py in the StandAloneMatLab folder. - - * While the primary volttron platform is running on the linux machine, - add the agent public key using the vctl auth command on the linux machine. This will make volttron platform - allow connections from the standalone agent - - .. code:: - - vctl auth add --credentials - -7. Run standalone agent - - - At this point, the agent is ready to run. To use the agent, navigate to the - example folder and use python to start the agent. The agent will then wait for - a message to be published to the selected topic by the MatLab agent. - - ``cd examples\StandAloneMatLab\`` - - ``python standalone_matlab.py`` - - Your output should be similar to this: - - .. code:: - - 2019-08-01 10:42:47,592 volttron.platform.vip.agent.core DEBUG: identity: standalone_matlab - 2019-08-01 10:42:47,592 volttron.platform.vip.agent.core DEBUG: agent_uuid: None - 2019-08-01 10:42:47,594 volttron.platform.vip.agent.core DEBUG: serverkey: None - 2019-08-01 10:42:47,596 volttron.platform.vip.agent.core DEBUG: AGENT RUNNING on ZMQ Core standalone_matlab - 2019-08-01 10:42:47,598 volttron.platform.vip.zmq_connection DEBUG: ZMQ connection standalone_matlab - 2019-08-01 10:42:47,634 volttron.platform.vip.agent.core INFO: Connected to platform: router: ebae9efa-5e8f-49e3-95a0-2020ddff9e8a version: 1.0 identity: standalone_matlab - 2019-08-01 10:42:47,634 volttron.platform.vip.agent.core DEBUG: Running onstart methods. - - - .. note:: - - If you have python3 as your default python run the command ``python -2 standalone_matlab.py`` - -8. On the Linux machine configure the MatlabAgent to publish commands to the topic standalone agent is listening to. -To load a new configuration or to change the current configuration enter - - .. code:: - - vctl config store config - - Whenever there is a change in the configuration in the config store, or whenever - the agent starts, the matlab agent sends the configgured command to the topic configured. As long as the standalone - agent has been started and is listening to the appropriate topic, the output in the log - should look similar to this: - - .. code:: - - 2019-08-01 10:43:18,925 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Configuring Agent - 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Publishing on: matlab/to_matlab/1 - 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Sending message: testScript2.py,20 - 2019-08-01 10:43:18,926 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent DEBUG: Agent Configured! - 2019-08-01 10:43:18,979 (matlab_agentV2agent-0.3 3539) matlab_agentV2.agent INFO: Agent: matlab/to_volttron/1 - Message: - '20' - - Once the matlab agent publishes the message (in the above case, "testScript2.py,20") on the windows command prompt - running the standalone agent, you should see the message that was received by the standalone agent. - - .. code:: - - 2019-08-01 10:42:47,671 volttron.platform.vip.agent.subsystems.configstore DEBUG: Processing callbacks for affected files: {} - The Message is: testScript2.py,20 - - .. note:: - - If MatLabAgent_v2 has been installed and started, and you have not started the - standalone_matlab agent, you will need to either restart the matlab_agentV2, or make - a change to the configuration in the config store to send command to the topic standalone agent is - actively listening to. - -.. |github-image| image:: files/github-image.png -.. |cmd-image| image:: files/cmd-image.png -.. |env-vars-image_1| image:: files/env-vars-image_1.png -.. |env-vars-image_2| image:: files/env-vars-image_2.png -.. |cmd-image_2| image:: files/cmd-image_2.png -.. |github-zip-image| image:: files/github-zip-image.png -.. |extract-image_1| image:: files/extract-image_1.png -.. |extract-image_2| image:: files/extract-image_2.png -.. |matlab-agent-diagram| image:: files/matlab-agent-diagram.png diff --git a/docs/source/devguides/supporting/examples/SchedulerExampleAgent.rst b/docs/source/devguides/supporting/examples/SchedulerExampleAgent.rst deleted file mode 100644 index 385f146b99..0000000000 --- a/docs/source/devguides/supporting/examples/SchedulerExampleAgent.rst +++ /dev/null @@ -1,59 +0,0 @@ -.. _SchedulerExampleAgent: - -SchedulerExampleAgent -===================== - -The SchedulerExampleAgent demonstrates how to use the scheduling feature -of the [[ActuatorAgent]] as well as how to send a command. This agent -publishes a request for a reservation on a (fake) device then takes an -action when it's scheduled time appears. The ActuatorAgent must be -running to exercise this example. - -Note: Since there is no actual device, an error is produced when the -agent attempts to take its action. - -:: - - def publish_schedule(self): - '''Periodically publish a schedule request''' - headers = { - 'AgentID': agent_id, - 'type': 'NEW_SCHEDULE', - 'requesterID': agent_id, #The name of the requesting agent. - 'taskID': agent_id + "-ExampleTask", #The desired task ID for this task. It must be unique among all other scheduled tasks. - 'priority': 'LOW', #The desired task priority, must be 'HIGH', 'LOW', or 'LOW_PREEMPT' - } - - start = str(datetime.datetime.now()) - end = str(datetime.datetime.now() + datetime.timedelta(minutes=1)) - - - msg = [ - ['campus/building/unit',start,end] - ] - self.vip.pubsub.publish( - 'pubsub', topics.ACTUATOR_SCHEDULE_REQUEST, headers, msg) - -The agent listens to schedule announcements from the actuator and then -issues a command - -:: - - @PubSub.subscribe('pubsub', topics.ACTUATOR_SCHEDULE_ANNOUNCE(campus='campus', - building='building',unit='unit')) - def actuate(self, peer, sender, bus, topic, headers, message): - print ("response:",topic,headers,message) - if headers[headers_mod.REQUESTER_ID] != agent_id: - return - '''Match the announce for our fake device with our ID - Then take an action. Note, this command will fail since there is no - actual device''' - headers = { - 'requesterID': agent_id, - } - self.vip.pubsub.publish( - 'pubsub', topics.ACTUATOR_SET(campus='campus', - building='building',unit='unit', - point='point'), - headers, 0.0) - diff --git a/docs/source/devguides/supporting/examples/index.rst b/docs/source/devguides/supporting/examples/index.rst deleted file mode 100644 index b9b734e67e..0000000000 --- a/docs/source/devguides/supporting/examples/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -====== -Agents -====== - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/devguides/supporting/index.rst b/docs/source/devguides/supporting/index.rst deleted file mode 100644 index da273cdd11..0000000000 --- a/docs/source/devguides/supporting/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -============================ -Examples/Samples -============================ - -.. toctree:: - :glob: - :maxdepth: 2 - - examples/index - utilities/index - applications/index - JupyterNotebooks - * diff --git a/docs/source/devguides/supporting/utilities/Driven-Applications.rst b/docs/source/devguides/supporting/utilities/Driven-Applications.rst deleted file mode 100644 index 946aec2d00..0000000000 --- a/docs/source/devguides/supporting/utilities/Driven-Applications.rst +++ /dev/null @@ -1,39 +0,0 @@ -.. _Driven-Applications: - -Driven Agents -============= - -Configuration for running OpenEIS applications within VOLTTRON. ---------------------------------------------------------------- - -The configuration of an agent within VOLTTRON requires a small -modification to the imports of the OpenEIS application and a couple of -configuration parameters. - -Import and Extend -~~~~~~~~~~~~~~~~~ - -:: - - from volttron.platform.agent import (AbstractDrivenAgent, Results) - ... - class OpeneisApp(AbstractDrivenAgent): - -Configuration -~~~~~~~~~~~~~ - -The two parameters that are necessary in the json configuration file are -"application" and "device". An optional but recommended argument should -also be added "agentid". - -:: - - { - "agentid": "drivenlogger1", - "application": "drivenlogger.logdevice.LogDevice", - "device": "pnnl/isb1/oat", - ... - } - -Any other keys will be passed on to the openeis application when it is -run. diff --git a/docs/source/devguides/supporting/utilities/ProcessAgent.rst b/docs/source/devguides/supporting/utilities/ProcessAgent.rst deleted file mode 100644 index 21ac5da147..0000000000 --- a/docs/source/devguides/supporting/utilities/ProcessAgent.rst +++ /dev/null @@ -1,15 +0,0 @@ -.. _ProcessAgent: - -Process Agent -============= - -This agent can be used to launch non-Python agents in the VOLTTRON -platform. The agent handles keeping track of the process so that it can -be started and stopped with platform commands. Edit the configuration -file to specify how to launch your process. - -This agent was originally created for launching sMAP along with the -platform, but can be used for any process. - -Note: Currently this agent does not respond to a blanket "shutdown" -request and must be stopped with the "stop" command. diff --git a/docs/source/devguides/supporting/utilities/Scripts.rst b/docs/source/devguides/supporting/utilities/Scripts.rst deleted file mode 100644 index 8975ef4a72..0000000000 --- a/docs/source/devguides/supporting/utilities/Scripts.rst +++ /dev/null @@ -1,39 +0,0 @@ -.. _Scripts: - -Scripts -======= - -In order to make repetitive tasks less repetitive the VOLTTRON team has -create several scripts in order to help. These tasks are available in -the scripts directory. Before using these scripts you should become -familiar with the :ref:`Agent Development ` process. - -In addition to the scripts directory, the VOLTTRON team has added the -config directory to the .gitignore file. By convention this is where we -store customized scripts and configuration that will not be made public. -Please feel free to use this convention in your own processes. - -The scripts/core directory is laid out in such a way that we can build -scripts on top of a base core. For example the scripts in sub-folders -such as the historian-scripts and demo-comms use the scripts that are -present in the core directory. - -The most widely used script is scripts/install-agent.py. The -install_agent.py script will remove an agent if the tag is already -present, create a new agent package, and install the agent to -VOLTTRON\_HOME. This script has three required arguments and has the -following signature: - -:: - - # Agent to Package must have a setup.py in the root of the directory. - scripts/install_agent.py - -The install_agent.py script will respect the VOLTTRON\_HOME specified on -the command line or set in the global environment. An example of setting -VOLTTRON\_HOME is as follows. - -:: - - # Sets VOLTTRON_HOME to /tmp/v1home - VOLTTRON_HOME=/tmp/v1home scripts/core/pack_install.sh diff --git a/docs/source/devguides/supporting/utilities/index.rst b/docs/source/devguides/supporting/utilities/index.rst deleted file mode 100644 index dde196e865..0000000000 --- a/docs/source/devguides/supporting/utilities/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -========= -Utilities -========= - -.. toctree:: - :glob: - :maxdepth: 2 - - * diff --git a/docs/source/devguides/walkthroughs/Forward-Historian-Deployment.rst b/docs/source/devguides/walkthroughs/Forward-Historian-Deployment.rst deleted file mode 100644 index 4fa1bd92be..0000000000 --- a/docs/source/devguides/walkthroughs/Forward-Historian-Deployment.rst +++ /dev/null @@ -1,83 +0,0 @@ -.. _Forward-Historian-Deployment: - -Forward Historian Deployment -============================= - -This guide describes a simple setup where one Volttron instance collects -data from a fake devices and sends to another instance . Lets consider the -following example. - -We are going to create two VOLTTRON instances and send data from one VOLTTRON -instance running a fake driver(subscribing values from a fake device)and sending -the values to the second VOLTTRON instance. - -VOLTTRON instance 1 forwards data to VOLTTRON instance 2 --------------------------------------------------------- - -VOLTTRON instance 1 -~~~~~~~~~~~~~~~~~~~ - -- vctl shutdown –platform (if the platform is already working) -- vcfg (this helps in configuring the volttron instance - http://volttron.readthedocs.io/en/releases-4.1/core_services/control/VOLTTRON-Config.html - - - Specify the IP of the machine : tcp://130.20.*.*:22916" - - Specify the port you want to use - - Specify if you want to run VC(Volttron Central) here or this this instance would be controlled - by a VC and the IP and port of the VC - - - Then install agents like Master driver Agent with fake driver agent for the instance. - - Install a listener agent so see the topics that are coming from the diver agent - - Then run the volttron instance by : ./start-volttron -- Volttron authentication: We need to add the IP of the instance 2 in the auth.config file of the VOLTTRON agent. - This is done as follows: - - - vctl auth-add - - We specify the IP of the instance 2 and the credentials of the agent - (http://volttron.readthedocs.io/en/releases-4.1/devguides/walkthroughs/Agent-Authentication-Walkthrough.html?highlight=auth-add) - - For specifying authentication for all the agents , we specify /.*/ for credentials as shown in - http://volttron.readthedocs.io/en/releases-4.1/devguides/agent_development/index.html - - This should enable authentication for all the volttron-instance based on the IP you specify here - -For this documentation, the topics from the driver agent will be send to the instance 2 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- We use the existing agent called the Forward Historian for this purpose which is available in service/core in the VOLTTRON directory. -- In the config file under the ForwardHistorian directory , we modify the following field: - - - Destination-vip : the IP of the volttron instance to which we have to forward the data to along with the port number . - Example : "tcp://130.20.*.*:22916" - - Destination-serverkye: The server key of the VOLTTRON instance to which we need to forward the data to. - This can be obtained at the VOLTTRON instance by typing vctl auth serverkey - -- Service_topic_list: specify the topics you want to forward specifically instead of all the values. -- Once the above values are set, your forwarder is all set . -- You can create a script file for the same and execute the agent. - -VOLTTRON instance 2 -~~~~~~~~~~~~~~~~~~~ - -- Vctl shutdown –platform (if the platform is already working ) -- volttron-cfg (this helps in configuring the volttron instance ) - http://volttron.readthedocs.io/en/releases-4.1/core_services/control/VOLTTRON-Config.html - - - Specify the IP of the machine : tcp://130.20.*.*:22916 - - Specify the port you want to use. - - Install the listener agent (this will show the connection from instance 1 if its successful - and then show all the topics from instance 1. - -- Volttron authentication: We need to add the IP of the instance 1 in the auth.config file of the VOLTTRON agent .This is done as follows: - - - vctl auth-add - - We specify the IP of the instance 1 and the credentials of the agent - http://volttron.readthedocs.io/en/releases-4.1/devguides/walkthroughs/Agent-Authentication-Walkthrough.html?highlight=auth-add - - For specifying authentication for all the agents , we specify /.*/ for credentials as shown in - http://volttron.readthedocs.io/en/releases-4.1/devguides/agent_development/index.html - - This should enable authentication for all the volttron-instance based on the IP you specify here - -Listener Agent -~~~~~~~~~~~~~~ -- Run the listener agent on this instance to see the values being forwarded from instance 1. - -Once the above setup is done, you should be able to see the values from instance 1 on the listener agent of instance 2. - - diff --git a/docs/source/devguides/walkthroughs/Forward-Historian-Walkthrough.rst b/docs/source/devguides/walkthroughs/Forward-Historian-Walkthrough.rst deleted file mode 100644 index 4eb7534168..0000000000 --- a/docs/source/devguides/walkthroughs/Forward-Historian-Walkthrough.rst +++ /dev/null @@ -1,68 +0,0 @@ -.. _Forward-Historian-Walkthrough: - -Forward Historian Walkthrough -============================= - -This guide describes a simple setup where one VOLTTRON instance collects -data from a fake devices and sends to another instance . Lets consider the -following example. - -We are going to create two VOLTTRON instances and send data from one VOLTTRON -instance running a fake driver(subscribing values from a fake device)and sending -the values to the second VOLTTRON instance. - -VOLTTRON instance 1 forwards data to VOLTTRON instance 2 --------------------------------------------------------- - -VOLTTRON INSTANCE 1 -~~~~~~~~~~~~~~~~~~~ -- ``vctl shutdown --platform`` (If VOLTTRON is already running it must be shut down before running ``volttron-cfg``). -- ``vcfg`` - this helps in configuring the VOLTTRON instance(:ref:`VOLTTRON Config `). - - - Specify the IP of the machine : ``tcp://127.0.0.1:22916``. - - Specify the port you want to use. - - Specify if you want to run VC ( VOLTTRON Central) here or this this instance would be controlled by a VC and the IP and port of the VC. -- Then start the VOLTTRON instance by : ``volttron -vv & > volttron.log&``. -- Then install agents like Master driver Agent with fake driver agent for the instance. -- Install a listener agent so see the topics that are coming from the diver agent. -- VOLTTRON authentication : We need to add the IP of the instance 1 in the auth.config file of the VOLTTRON agent .This is done as follow : - - - ``vctl auth-add`` - - We specify the IP of the instance 1 and the credentials of the agent.(:ref:`Agent authentication walkthrough `) - - For specifying authentication for all the agents , we specify ``/.*/`` for credentials as shown in :ref:`Agent Development`. - - This should enable authentication for all the VOLTTRON instances based on the IP you specify here . - -For this documentation, the topics from the driver agent will be sent to the instance 2 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- We use the existing agent called the Forward Historian for this purpose which is available in ``service/core`` in the VOLTTRON directory. -- In the config file under the ForwardHistorian directory , we modify the following field: - - - destination-vip : the IP of the VOLTTRON instance to which we have to forward the data to along with the port number . Example : ``tcp://130.20.*.*:22916``. - - destination-serverkey: The server key of the VOLTTRON instance to which we need to forward the data to. This can be obtained at the VOLTTRON instance by typing ``volttron-ctl auth serverkey``. - - service_topic_list: specify the topics you want to forward specifically instead of all the values. -- Once the above values are set, your forwarder is all set . -- You can create a script file for the same and execute the agent. - -VOLTTRON INSTANCE 2 -~~~~~~~~~~~~~~~~~~~ - -- ``vctl shutdown --platform`` (If VOLTTRON is already running it must be shut down before running ``volttron-cfg``). -- ``vcfg`` - this helps in configuring the VOLTTRON instance.(:ref:`VOLTTRON Config `) - - Specify the IP of the machine : ``tcp://127.0.0.1:22916``. - - Specify the port you want to use. - - Install the listener agent (this will show the connection from instance 1 if its successful and then show all the topics from instance 1. -- Then start the VOLTTRON instance by : ``volttron -vv & > volttron.log&``. -- VOLTTRON authentication : We need to add the IP of the instance 1 in the auth.config file of the VOLTTRON agent .This is done as follow : - - - ``vctl auth-add`` - - We specify the IP of the instance 1 and the credentials of the agent.(:ref:`Agent authentication walkthrough `) - - For specifying authentication for all the agents , we specify ``/.*/`` for credentials as shown in :ref:`Agent Development`. - - This should enable authentication for all the VOLTTRON instances based on the IP you specify here . - -LISTENER AGENT - -- Run the listener agent on this instance to see the values being forwarded from instance 1. - -Once the above setup is done, you should be able to see the values from instance 1 on the listener agent of instance 2. - - diff --git a/docs/source/devguides/walkthroughs/Simple-WebAgent-Walkthrough.rst b/docs/source/devguides/walkthroughs/Simple-WebAgent-Walkthrough.rst deleted file mode 100644 index eca93f8a0f..0000000000 --- a/docs/source/devguides/walkthroughs/Simple-WebAgent-Walkthrough.rst +++ /dev/null @@ -1,65 +0,0 @@ -.. _Simple-WebAgent-Walkthrough: - -Simple Web Agent Walkthrough -============================ - -A simple web enabled agent that will hook up with a volttron message bus and -allow interaction between it via http. This example agent shows a simple file -serving agent, a json-rpc based call, and a websocket based connection -mechanism. - -Starting VOLTTRON Platform --------------------------- - -.. note:: Activate the environment first :ref:`active the environment ` - -In order to start the simple web agent, we need to bind the VOLTTRON instance -to the a web server. We need to specify the address and the port for the -web server. For example, if we want to bind the localhost:8080 as the web server -we start the VOLTTRON platform as follows: - -``./start-volttron --bind-web-address http://127.0.0.1:8080`` - -Once the platform is started, we are ready to run the Simple Web Agent. - -Running Simple Web Agent ------------------------- - -.. note:: The following assumes the shell is located at the :ref:`VOLTTRON_ROOT`. - -Copy the following into your shell (save it to a file for executing it again -later). - -.. code-block:: console - - python scripts/install-agent.py \ - --agent-source examples/SimpleWebAgent \ - --tag simpleWebAgent \ - --vip-identity webagent \ - --force \ - --start - -This will create a web server on http://localhost:8080. The index.html file -under ``simpleweb/webroot/simpleweb/`` can be any html page which binds to the -VOLTTRON message bus .This provides a simple example of providing a web endpoint -in VOLTTRON. - -Path based registration examples --------------------------------- - -- Files will need to be in webroot/simpleweb in order for them to be browsed - from http://localhost:8080/simpleweb/index.html - -- Filename is required as we don't currently autoredirect to any default pages - as shown in ``self.vip.web.register_path("/simpleweb", os.path.join(WEBROOT))`` - -The following two examples show the way to call either a jsonrpc (default) -endpoint and one that returns a different content-type. With the JSON-RPC -example from volttron central we only allow post requests, however this is not -required. - -- Endpoint will be available at http://localhost:8080/simple/text ``self.vip.web.register_endpoint("/simple/text", self.text)`` - -- Endpoint will be available at http://localhost:8080/simple/jsonrpc ``self.vip.web.register_endpoint("/simpleweb/jsonrpc", self.rpcendpoint)`` -- Text/html content type specified so the browser can act appropriately like ``[("Content-Type", "text/html")]`` -- The default response is application/json so our endpoint returns appropriately with a json based response. diff --git a/docs/source/devguides/walkthroughs/VC-Device-Configuration-Demo.rst b/docs/source/devguides/walkthroughs/VC-Device-Configuration-Demo.rst deleted file mode 100644 index 5ddd86e0fa..0000000000 --- a/docs/source/devguides/walkthroughs/VC-Device-Configuration-Demo.rst +++ /dev/null @@ -1,332 +0,0 @@ -.. _Device-Conifiguration-in-VOLTTRON-Central: - -======================================== -Device Configuration in VOLTTRON Central -======================================== - -Devices in your network can be detected and configured through the VOLTTRON Central UI. The current version of VOLTTRON enables device detection and configuration for BACnet devices. The following sections describe the processes involved with performing scans to detect physical devices and get their points, and configuring them as virtual devices installed on VOLTTRON instances. - -- `Launching Device Configuration <#launching-device-configuration>`__ -- `Scanning for Devices <#scanning-for-devices>`__ -- `Scanning for Points <#scanning-for-points>`__ -- `Registry Configuration File <#registry-configuration-file>`__ -- `Additional Attributes <#additional-attributes>`__ -- `Quick Edit Features <#quick-edit-features>`__ -- `Keyboard Commands <#keyboard-commands>`__ -- `Registry Preview <#registry-preview>`__ -- `Registry Configuration Options <#registry-configuration-options>`__ -- `Reloading Device Points <#reloading-device-points>`__ -- `Device Configuration Form <#device-configuration-form>`__ -- `Configuring Subdevices <#configuring-subdevices>`__ -- `Reconfiguring Devices <#reconfiguring-devices>`__ -- `Exporting Registry Configuration Files <#exporting-registry-configuration-files>`__ - -Launching Device Configuration ------------------------------- - -To begin device configuration in VOLTTRON Central, extend the side panel on the left and find the cogs button next to the platform instance you want to add a device to. Click the cogs button to launch the device configuration feature. - -|Add Devices| - -|Install Devices| - -Currently the only method of adding devices is to conduct a scan to detect BACnet devices. A BACnet Proxy Agent must be running in order to do the scan. If more than one BACnet Proxy is installed on the platform, choose the one that will be used for the scan. - -The scan can be conducted using default settings that will search for all physical devices on the network. However, optional settings can be used to focus on specific devices or change the duration of the scan. Entering a range of device IDs will limit the scan to return only devices with IDs in that range. Advanced options include the ability to specify the IP address of a device to detect as well as the ability to change the duration of the scan from the default of five seconds. - -Scanning for Devices --------------------- - -To start the scan, click the large cog button to the right of the scan settings. - -|Start Scan| - -Devices that are detected will appear in the space below the scan settings. Scanning can be repeated at any time by clicking the large cog button again. - -|Devices Found| - -Scanning for Points -------------------- - -Another scan can be performed on each physical device to retrieve its available points. This scan is initiated by clicking the triangle next to the device in the list. The first time the arrow is clicked, it initiates the scan. After the points are retrieved, the arrow becomes a hide-and-show toggle button and won't reinitiate scanning the device. - -|Get Device Points| - -After the points have been retrieved once, the only way to scan the same device -for points again is to relaunch the device configuration process from the start -by clicking on the small cogs button next to the platform instance in the panel tree. - -Registry Configuration File -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The registry configuration determines which points on the physical device will be associated with the virtual device that uses that particular registry configuration. The registry configuration determines which points' data will be published to the message bus and recorded by the historian, and it determines how the data will be presented. - -When all the points on the device have been retrieved, the points are loaded into the registry configuration editor. There, the points can be modified and selected to go into the registry configuration file for a device. - -Each row in the registry configuration editor represents a point, and each cell in the row represents an attribute of the point. - -Only points that have been selected will be included in the registry configuration file. To select a point, check the box next to the point in the editor. - -|Select Point Before| - -|Select Point During| - -|Select Point After| - -Type directly in a cell to change an attribute value for a point. - -|Edit Points| - -Additional Attributes ---------------------- - -The editor's default view shows the attributes that are most likely to be changed during configuration: the VOLTTRON point name, the writable setting, and the units. Other attributes are present but not shown in the default view. To see the entire set of attributes for a point, click the Edit Point button (the three dots) at the end of the point row. - -|Edit Point Button| - -In the window that opens, point attributes can be changed by typing in the fields and clicking the Apply button. - -|Edit Point Dialog| - -Checking or unchecking the "Show in Table" box for an attribute will add or remove it as a column in the registry configuration editor. - -Quick Edit Features -------------------- - -Several quick-edit features are available in the registry configuration editor. - -The list of points can be filtered based on values in the first column by clicking the filter button in the first column's header and entering a filter term. - -|Filter Points Button| - -|Filter Set| - -The filter feature allows points to be edited, selected, or deselected more quickly by narrowing down potentially large lists of points. However, the filter doesn't select points, and if the registry configuration is saved while a filter is applied, any selected points not included in the filter will still be included in the registry file. - -To clear the filter, click on the Clear Filter button in the filter popup. - -|Clear Filter| - -To add a new point to the points listed in the registry configuration editor, click on the Add Point button in the header of the first column. - -|Add New Point| - -|Add Point Dialog| - -Provide attribute values, and click the Apply button to add the new point, which will be appended to the bottom of the list. - -To remove points from the list, select the points and click the Remove Points button in the header of the first column. - -|Remove Points| - -|Confirm Remove Points| - -Each column has an Edit Column button in its header. - -|Edit Columns| - -Click on the button to display a popup menu of operations to perform on the column. The options include inserting a blank new column, duplicating an existing column, removing a column, or searching for a value within a column. - -|Edit Column Menu| - -A duplicate or new column has to be given a unique name. - -|Name Column| - -|Duplicated Column| - -To search for values in a column, choose the Find and Replace option in the popup menu. - -|Find in Column| - -Type the term to search for, and click the Find Next button to highlight all the matched fields in the column. - -|Find Next| - -Click the Find Next button again to advance the focus down the list of matched terms. - -To quickly replace the matched term in the cell with focus, type a replacement term, and click on the Replace button. - -|Replace in Column| - -To replace all the matched terms in the column, click on the Replace All button. Click the Clear Search button to end the search. - -Keyboard Commands ------------------ - -Some keyboard commands are available to expedite the selection or de-selection of points. To initiate use of the keyboard commands, strike the Control key on the keyboard. For keyboard commands to be activated, the registry configuration editor has to have focus, which comes from interacting with it. But the commands won't be activated if the cursor is in a typable field. - -If the keyboard commands have been successfully activated, a faint highlight will appear over the first row in the registry configuration editor. - -|Start Keyboard Commands| - -Keyboard commands are deactivated when the mouse cursor moves over the configuration editor. If unintentional deactivation occurs, strike the Control key again to reactivate the commands. - -With keyboard commands activated, the highlighted row can be advanced up or down by striking the up or down arrow on the keyboard. A group of rows can be highlighted by striking the up or down arrow while holding down the Shift key. - -|Keyboard Highlight| - -To select the highlighted rows, strike the Enter key. - -|Keyboard Select| - -Striking the Enter key with rows highlighted will also deselect any rows that were already selected. - -Click on the Keyboard Shortcuts button to show a popup list of the available keyboard commands. - -|Keyboard Shortcuts Button| - -|Keyboard Shortcuts| - -Registry Preview ----------------- - -To save the registry configuration, click the Save button at the bottom of the registry configuration editor. - -|Save Registry Button| - -A preview will appear to let you confirm that the configuration is what you intended. - -|Registry Preview Table| - -The configuration also can be inspected in the comma-separated format of the actual registry configuration file. - -|Registry Preview CSV| - -Provide a name for the registry configuration file, and click the Save button to save the file. - -|Name Registry File| - -|Registry Saved| - -Registry Configuration Options ------------------------------- - -Different subsets of configured points can be saved from the same physical device and used to create separate registry files for multiple virtual devices and subdevices. Likewise, a single registry file can be reused by multiple virtual devices and subdevices. - -To reuse a previously saved registry file, click on the Select Registry File (CSV) button at the end of the physical device's listing. - -|Select Saved Registry File| - -The Previously Configured Registry Files window will appear, and a file can be selected to load it into the registry configuration editor. - -|Saved Registry Selector| - -Another option is to import a registry configuration file from the computer running the VOLTTRON Central web application, if one has been saved to local storage connected to the computer. To import a registry configuration file from local storage, click on the Import Registry File (CSV) button at the end of the physical device's listing, and use the file selector window to locate and load the file. - -|File Import Button| - -Reloading Device Points ------------------------ - -Once a physical device has been scanned, the original points from the scan can be reloaded at any point during device configuration by clicking on the Reload Points From Device button at the end of the device's listing. - -|Reload Points| - -Device Configuration Form -~~~~~~~~~~~~~~~~~~~~~~~~~ - -After the registry configuration file has been saved, the device configuration form appears. Creating the device configuration results in the virtual device being installed in the platform and determines the device's position in the side panel tree. It also contains some settings that determine how data is collected from the device. - -|Configure Device Dialog| - -After the device configuration settings have been entered, click the Save button to save the configuration and add the device to the platform. - -|Save Device Config| - -|Device Added| - -Configuring Subdevices ----------------------- - -After a device has been configured, subdevices can be configured by pointing to their position in the Path attribute of the device configuration form. But a subdevice can't be configured until its parent device has been configured first. - -|Subdevice Path| - -|Subdevice 2| - -As devices are configured, they're inserted into position in the side panel tree, along with their configured points. - -|Device Added to Tree| - - -Reconfiguring Devices -~~~~~~~~~~~~~~~~~~~~~ - -A device that's been added to a VOLTTRON instance can be reconfigured by changing its registry configuration or its device configuration. To launch reconfiguration, click on the wrench button next to the device in the side panel tree. - -|Reconfigure Device Button| - -Reconfiguration reloads the registry configuration editor and the device configuration form for the virtual device. The editor and the form work the same way in reconfiguration as during initial device configuration. - -|Reconfiguring Device| - -The reconfiguration view shows the name, address, and ID of the physical device that the virtual device was configured from. It also shows the name of the registry configuration file associated with the virtual device as well as its configured path. - -A different registry configuration file can be associated with the device by clicking on the Select Registry File (CSV) button or the Import Registry File (CSV) button. - -The registry configuration can be edited by making changes to the configuration in the editor and clicking the Save button. - -To make changes to the device configuration form, click on the File to Edit selector and choose Device Config. - -|Reconfigure Option Selector| - -|Reconfigure Device Config| - -Exporting Registry Configuration Files --------------------------------------- - -The registry configuration file associated with a virtual device can be exported from the web browser to the computer's local storage by clicking on the File Export Button in the device reconfiguration view. - -|File Export Button| - -.. |Add Devices| image:: files/01-add-devices.png -.. |Install Devices| image:: files/02-install-devices.png -.. |Start Scan| image:: files/03-start-scan.png -.. |Devices Found| image:: files/04-devices-found.png -.. |Get Device Points| image:: files/05-get-device-points.png -.. |Select Point Before| image:: files/07-select-point-a.png -.. |Select Point During| image:: files/07-select-point-b.png -.. |Select Point After| image:: files/07-select-point-c.png -.. |Edit Points| image:: files/07-edit-points.png -.. |Edit Point Button| image:: files/21-edit-point-button.png -.. |Edit Point Dialog| image:: files/22-edit-point-dialog.png -.. |Filter Points Button| image:: files/08-filter-points-button.png -.. |Filter Set| image:: files/09-filter-set.png -.. |Clear Filter| image:: files/10-clear-filter.png -.. |Add New Point| image:: files/11-add-new-point.png -.. |Add Point Dialog| image:: files/12-add-point-dialog.png -.. |Remove Points| image:: files/13-remove-points-button.png -.. |Confirm Remove Points| image:: files/14-confirm-remove-points.png -.. |Edit Columns| image:: files/15-edit-column-button.png -.. |Edit Column Menu| image:: files/16-edit-column-menu.png -.. |Name Column| image:: files/17-name-column.png -.. |Duplicated Column| image:: files/18-duplicated-column.png -.. |Find in Column| image:: files/19-find-in-column.png -.. |Find Next| image:: files/19-find-in-column-b.png -.. |Replace in Column| image:: files/20-replace-in-column.png -.. |Start Keyboard Commands| image:: files/23-start-keyboard-commands.png -.. |Keyboard Highlight| image:: files/24-keyboard-highlight.png -.. |Keyboard Select| image:: files/25-keyboard-select.png -.. |Keyboard Shortcuts Button| image:: files/26-keyboard-shortcuts-button.png -.. |Keyboard Shortcuts| image:: files/27-keyboard-shortcuts.png -.. |Save Registry Button| image:: files/28-save-registry-button.png -.. |Registry Preview Table| image:: files/29-registry-preview-table.png -.. |Registry Preview CSV| image:: files/30-preview-registry-csv.png -.. |Name Registry File| image:: files/31-name-registry-file.png -.. |Registry Saved| image:: files/32-registry-saved.png -.. |Select Saved Registry File| image:: files/38-select-saved-registry-file.png -.. |Saved Registry Selector| image:: files/39-saved-registry-selector.png -.. |File Import Button| image:: files/40-file-import-button.png -.. |Reload Points| image:: files/41-reload-points-from-device.png -.. |Configure Device Dialog| image:: files/33-configure-device-dialog.png -.. |Save Device Config| image:: files/34-save-device-config.png -.. |Device Added| image:: files/37-device-added.png -.. |Subdevice Path| image:: files/35-subdevice-path.png -.. |Subdevice 2| image:: files/36-subdevice2.png -.. |Device Added to Tree| image:: files/37-device-added-b.png -.. |Reconfigure Device Button| image:: files/43-reconfigure-device-button.png -.. |Reconfiguring Device| image:: files/44-reconfiguring-device.png -.. |Reconfigure Option Selector| image:: files/45-reconfigure-option-selector.png -.. |Reconfigure Device Config| image:: files/46-reconfigure-device-config.png -.. |File Export Button| image:: files/47-file-export-button.png diff --git a/docs/source/devguides/walkthroughs/files/platform-config.png b/docs/source/devguides/walkthroughs/files/platform-config.png deleted file mode 100644 index 3dac2abeff..0000000000 Binary files a/docs/source/devguides/walkthroughs/files/platform-config.png and /dev/null differ diff --git a/docs/source/devguides/walkthroughs/files/register-new-platform.png b/docs/source/devguides/walkthroughs/files/register-new-platform.png deleted file mode 100644 index 6c4d27d1a0..0000000000 Binary files a/docs/source/devguides/walkthroughs/files/register-new-platform.png and /dev/null differ diff --git a/docs/source/devguides/walkthroughs/files/vc-config.png b/docs/source/devguides/walkthroughs/files/vc-config.png deleted file mode 100644 index 5c1793810b..0000000000 Binary files a/docs/source/devguides/walkthroughs/files/vc-config.png and /dev/null differ diff --git a/docs/source/devguides/walkthroughs/index.rst b/docs/source/devguides/walkthroughs/index.rst deleted file mode 100644 index d7e26d6612..0000000000 --- a/docs/source/devguides/walkthroughs/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -============ -Walkthroughs -============ - -.. toctree:: - :glob: - :maxdepth: 1 - - * diff --git a/docs/source/driver-framework/actuator/actuator-agent.rst b/docs/source/driver-framework/actuator/actuator-agent.rst new file mode 100644 index 0000000000..452d86fed7 --- /dev/null +++ b/docs/source/driver-framework/actuator/actuator-agent.rst @@ -0,0 +1,457 @@ +.. _Actuator-Agent: + +============== +Actuator Agent +============== + +This agent is used to manage write access to devices. Agents may request scheduled times, called Tasks, to interact with +one or more devices. + + +.. _Actuator-Communication: + +Actuator Agent Communication +============================ + + +Scheduling a Task +----------------- + +An agent can request a task schedule by publishing to the `devices/actuators/schedule/request` topic with the following +header: + +.. code-block:: python + + { + 'type': 'NEW_SCHEDULE', + 'requesterID': + 'taskID': , #The desired task ID for this task. It must be unique among all other scheduled tasks. + 'priority': , #The desired task priority, must be 'HIGH', 'LOW', or 'LOW_PREEMPT' + } + +with the following message: + +.. code-block:: python + + [ + ["campus/building/device1", #First time slot. + "2013-12-06 16:00:00", #Start of time slot. + "2013-12-06 16:20:00"], #End of time slot. + ["campus/building/device1", #Second time slot. + "2013-12-06 18:00:00", #Start of time slot. + "2013-12-06 18:20:00"], #End of time slot. + ["campus/building/device2", #Third time slot. + "2013-12-06 16:00:00", #Start of time slot. + "2013-12-06 16:20:00"], #End of time slot. + #etc... + ] + +.. warning:: + + If time zones are not included in schedule requests then the Actuator will interpret them as being in local time. + This may cause remote interaction with the actuator to malfunction. + + +Points on Task Scheduling +^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Everything in the header is required +- Task id and requester id (agentid) should be a non empty value of type string +- A Task schedule must have at least one time slot. +- The start and end times are parsed with `dateutil's date/time + parser `__. + **The default string representation of a python datetime object will parse without issue.** +- Two Tasks are considered conflicted if at least one time slot on a device from one task overlaps the time slot of the + other on the same device. +- The end time of one time slot can be the same as the start time of another time slot for the same device. This will + not be considered a conflict. For example, ``time_slot1(device0, time1, **time2**)`` and + ``time_slot2(device0, **time2**, time3)`` are not considered a conflict +- A request must not conflict with itself +- If something goes wrong see :ref:`this failure string list ` for an + explanation of the error. + + +Task Priorities +^^^^^^^^^^^^^^^ + +* `HIGH`: This Task cannot be preempted under any circumstance. This task may preempt other conflicting preemptable + Tasks. + +* `LOW`: This Task cannot be preempted **once it has started**. A Task is considered started once the earliest time slot + on any device has been reached. This Task may **not** preempt other Tasks. + +* `LOW_PREEMPT`: This Task may be preempted at any time. If the Task is preempted once it has begun running any + current time slots will be given a grace period (configurable in the ActuatorAgent configuration file, defaults to 60 + seconds) before being revoked. This Task may **not** preempt other Tasks. + + +Canceling a Task +---------------- + +A task may be canceled by publishing to the `devices/actuators/schedule/request` topic with the following header: + +.. code-block:: python + + { + 'type': 'CANCEL_SCHEDULE', + 'requesterID': + 'taskID': , #The desired task ID for this task. It must be unique among all other scheduled tasks. + } + + +Points on Task Canceling +^^^^^^^^^^^^^^^^^^^^^^^^ + +- The requesterID and taskID must match the original values from the original request header. +- After a Tasks time has passed there is no need to cancel it. Doing so will result in a `TASK_ID_DOES_NOT_EXIST` + error. +- If something goes wrong see :ref:`this failure string list ` for an explanation + of the error. + + +Actuator Agent Schedule Response +-------------------------------- + +In response to a Task schedule request the ActuatorAgent will respond on the topic `devices/actuators/schedule/result` +with the header: + +.. code-block:: python + + { + 'type': <'NEW_SCHEDULE', 'CANCEL_SCHEDULE'> + 'requesterID': , + 'taskID': + } + +And the message (after parsing the json): + +.. code-block:: python + + { + 'result': <'SUCCESS', 'FAILURE', 'PREEMPTED'>, + 'info': , + 'data': + } + +The Actuator Agent may publish cancellation notices for preempted Tasks using the `PREEMPTED` result. + + +Preemption Data +^^^^^^^^^^^^^^^ + +Preemption data takes the form: + +.. code-block:: python + + { + 'agentID': , + 'taskID': + } + + +.. _Actuator-Failure-Reasons: + +Failure Reasons +^^^^^^^^^^^^^^^ + +In many cases the Actuator Agent will try to give good feedback as to why a request failed. + + +General Failures +"""""""""""""""" + +* `INVALID_REQUEST_TYPE`: Request type was not `NEW_SCHEDULE` or `CANCEL_SCHEDULE`. +* `MISSING_TASK_ID`: Failed to supply a taskID. +* `MISSING_AGENT_ID`: AgentID not supplied. + + +Task Schedule Failures +"""""""""""""""""""""" + +* `TASK_ID_ALREADY_EXISTS`: The supplied taskID already belongs to an existing task. +* `MISSING_PRIORITY`: Failed to supply a priority for a Task schedule request. +* `INVALID_PRIORITY`: Priority not one of `HIGH`, `LOW`, or `LOW_PREEMPT`. +* `MALFORMED_REQUEST_EMPTY`: Request list is missing or empty. +* `REQUEST_CONFLICTS_WITH_SELF`: Requested time slots on the same device overlap. +* `MALFORMED_REQUEST`: Reported when the request parser raises an unhandled exception. The exception name and info are + appended to this info string. +* `CONFLICTS_WITH_EXISTING_SCHEDULES`: This schedule conflict with an existing schedules that it cannot preempt. The + data item for the results will contain info about the conflicts in this form (after parsing json) + +.. code-block:: python + + { + '': + { + '': + [ + ["campus/building/device1", + "2013-12-06 16:00:00", + "2013-12-06 16:20:00"], + ["campus/building/device1", + "2013-12-06 18:00:00", + "2013-12-06 18:20:00"] + ] + '':[...] + } + '': {...} + } + + +Task Cancel Failures +"""""""""""""""""""" + +* `TASK_ID_DOES_NOT_EXIST`: Trying to cancel a Task which does not exist. This error can also occur when trying to + cancel a finished Task. +* `AGENT_ID_TASK_ID_MISMATCH`: A different agent ID is being used when trying to cancel a Task. + + +.. _Actuator-Value-Request: + +Actuator Agent Value Request +---------------------------- + +Once an Task has been scheduled and the time slot for one or more of the devices has started an agent may interact with +the device using the **get** and **set** topics. + +Both **get** and **set** are responded to the same way. See :ref:`Actuator Reply ` below. + +Getting values +^^^^^^^^^^^^^^ + +While a driver for a device should always be setup to periodically broadcast the state of a device you may want an +up-to-the-moment value for an actuation point on a device. + +To request a value publish a message to the following topic: + +.. code-block:: python + + 'devices/actuators/get//' + + +Setting Values +^^^^^^^^^^^^^^ + +Value are set in a similar manner: + +To set a value publish a message to the following topic: + +.. code-block:: python + + 'devices/actuators/set//' + +With this header: + +.. code-block:: python + + #python + { + 'requesterID': + } + +And the message contents being the new value of the actuator. + +.. warning:: + + The actuator agent expects all messages to be JSON and will parse them accordingly. Use `publish_json` to send + messages where possible. This is significant for Boolean values especially + +.. _Actuator-Reply: + +Actuator Reply +^^^^^^^^^^^^^^ + +The ActuatorAgent will reply to both `get` and `set` on the `value` topic for an actuator: + +.. code-block:: python + + 'devices/actuators/value//' + +With this header: + +.. code-block:: python + + { + 'requesterID': + } + +With the message containing the value encoded in JSON. + +Actuator Error Reply +^^^^^^^^^^^^^^^^^^^^ + +If something goes wrong the Actuator Agent will reply to both `get` and `set` on the `error` topic for an actuator: + +.. code-block:: python + + 'devices/actuators/error//' + +With this header: + +.. code-block:: python + + { + 'requesterID': + } + +The message will be in the following form: + +.. code-block:: python + + { + 'type': + 'value': + } + +Common Error Types +^^^^^^^^^^^^^^^^^^ + +* `LockError`: Returned when a request is made when we do not have permission to use a device. (Forgot to schedule, + preempted and we did not handle the preemption message correctly, ran out of time in time slot, etc...) +* `ValueError`: Message missing or could not be parsed as JSON + + +.. _Actuator-Schedule-State: + +Schedule State Broadcast +------------------------ + +Periodically the ActuatorAgent will publish the state of all currently scheduled devices. For each device the +ActuatorAgent will publish to an associated topic: + +.. code-block:: python + + 'devices/actuators/schedule/announce/' + +With the following header: + +.. code-block:: python + + { + 'requesterID': , + 'taskID': + 'window': + } + +The frequency of the updates is configurable with the `schedule_publish_interval` setting. + + +Task Preemption +--------------- + +Both `LOW` and `LOW_PREEMPT` priority Tasks can be preempted. `LOW` priority Tasks may be preempted by a conflicting +`HIGH` priority Task before it starts. `LOW_PREEMPT` priority Tasks can be preempted by `HIGH` priority Tasks even +after they start. + +When a Task is preempted the ActuatorAgent will publish to `devices/actuators/schedule/response` with the following +header: + +.. code-block:: python + + { + 'type': 'CANCEL_SCHEDULE', + 'requesterID': , + 'taskID': + } + +And the message (after parsing the json): + +.. code-block:: python + + { + 'result': 'PREEMPTED', + 'info': '', + 'data': + { + 'agentID': , + 'taskID': + } + } + + +Preemption Grace Time +^^^^^^^^^^^^^^^^^^^^^ + +If a `LOW_PREEMPT` priority Task is preempted while it is running the Task will be given a grace period to clean up +before ending. For every device which has a current time slot the window of remaining time will be reduced to the grace +time. At the end of the grace time the Task will finish. If the Task has no currently open time slots on any devices +it will end immediately. + + +.. _Actuator-Config: + +ActuatorAgent Configuration +--------------------------- + +* `schedule_publish_interval`: Interval between current schedules being published to the message bus for all devices +* `preempt_grace_time`: Minimum time given to Tasks which have been preempted to clean up in seconds. Defaults to 60 +* `schedule_state_file`: File used to save and restore Task states if the ActuatorAgent restarts for any reason. File + will be created if it does not exist when it is needed + +Sample configuration file +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: json + + { + "schedule_publish_interval": 30, + "schedule_state_file": "actuator_state.pickle" + } + + +Heartbeat Signal +---------------- + +The ActuatorAgent can be configured to send a heartbeat message to the device to indicate the platform is running. +Ideally, if the heartbeat signal is not sent the device should take over and resume normal operation. + +The configuration has two parts, the interval (in seconds) for sending the heartbeat and the specific point that should +be modified each iteration. + +The heart beat interval is specified with a global `heartbeat_interval` setting. The ActuatorAgent will automatically +set the heartbeat point to alternating "1" and "0" values. Changes to the heartbeat point will be published like any +other value change on a device. + +The heartbeat points are specified in the driver configuration file of individual devices. + + +.. _Actuator-Notes: + +Notes on Working With the ActuatorAgent +--------------------------------------- + +- An agent can watch the window value from :ref:`device state updates ` to perform scheduled + actions within a timeslot + + - If an Agent's Task is `LOW_PREEMPT` priority it can watch for device state updates where the window is less than + or equal to the grace period (default 60.0) + +- When considering if to schedule long or multiple short time slots on a single device: + + - Do we need to ensure the device state for the duration between slots? + + - Yes: Schedule one long time slot instead + - No: Is it all part of the same Task or can we break it up in case there is a conflict with one of our time + slots? + +- When considering time slots on multiple devices for a single Task: + + - Is the Task really dependent on all devices or is it actually multiple Tasks? + +- When considering priority: + + - Does the Task have to happen **on an exact day**? + + - Yes: Use `HIGH` + - No: Consider `LOW` and reschedule if preempted + + - Is it problematic to prematurely stop a Task once started? + + - Yes: Consider `LOW` or `HIGH` + - No: Consider `LOW_PREEMPT` and watch the device state updates for a small window value + +- If an agent is only observing but needs to assure that no another Task is going on while taking readings it can + schedule the time to prevent other agents from messing with a devices state. The schedule updates can be used as a + reminder as to when to start watching +- **Any** device, existing or not, can be scheduled. This allows for agents to schedule fake devices to create + reminders to start working later rather then setting up their own internal timers and schedules diff --git a/docs/source/driver-framework/bacnet/bacnet-auto-configuration.rst b/docs/source/driver-framework/bacnet/bacnet-auto-configuration.rst new file mode 100644 index 0000000000..9733954ce1 --- /dev/null +++ b/docs/source/driver-framework/bacnet/bacnet-auto-configuration.rst @@ -0,0 +1,288 @@ +.. _BACnet-Auto-Configuration: + +========================= +BACnet Auto-Configuration +========================= + +Included with the platform are two scripts for finding and configuring BACnet devices. These scripts are located in +`scripts/bacnet`. `bacnet_scan.py` will scan the network for devices. `grab_bacnet_config.py` creates a CSV file +for the BACnet driver that can be used as a starting point for creating your own register configuration. + +Both scripts are configured with the file `BACpypes.ini`. + + +Configuring the Utilities +------------------------- + +While running both scripts create a temporary virtual BACnet device using the `bacpypes` library. The virtual +device must be configured properly in order to work. This configuration is stored in `scripts/bacnet/BACpypes.ini` +and will be read automatically when the utility is run. + +.. note:: + + The only value that (usually) needs to be changed is the **address** field. + +.. warning:: + + This is the address bound to the port on the machine you are running the script from, **NOT A TARGET DEVICE** + +This value should be set to the IP address of the network interface used to communicate with the remote device. If +there is more than one network interface you must use the address of the interface connected to the network that can +reach the device. + +In Linux you can usually get the addresses bound to all interfaces by running ``ifconfig`` from the command line. + +If a different outgoing port other than the default 47808 must be used, it can be specified as part of the address in +the form: + +:: + +
: + +In some cases, the netmask of the network will be needed for proper configuration. This can be done following this +format: + +:: + +
/: + +where ```` is the netmask length. The most common value is 24. See +http://www.computerhope.com/jargon/n/netmask.htm + +In some cases, you may also need to specify a different device ID by changing the value of `objectIdentifier` so the +virtual BACnet device does not conflict with any devices on the network. `objectIdentifier` defaults to 599. + + +Sample BACpypes.ini +^^^^^^^^^^^^^^^^^^^ + +.. code-block:: console + + [BACpypes] + objectName: Betelgeuse + address: 10.0.2.15/24 + objectIdentifier: 599 + maxApduLengthAccepted: 1024 + segmentationSupported: segmentedBoth + vendorIdentifier: 15 + + +Scanning for BACnet Devices +--------------------------- + +If the addresses for BACnet devices are unknown they can be discovered using the `bacnet_scan.py` utility. + +To run the utility simply execute the following command: + +.. code-block:: bash + + python bacnet_scan.py + +and expect output similar to this: + +.. code-block:: console + + Device Address =
+ Device Id = 699 + maxAPDULengthAccepted = 1024 + segmentationSupported = segmentedBoth + vendorID = 15 + + Device Address = + Device Id = 540011 + maxAPDULengthAccepted = 480 + segmentationSupported = segmentedBoth + vendorID = 5 + + +Reading Output +^^^^^^^^^^^^^^ + +The address where the device can be reached is listed on the `Device Address` line. The BACnet device ID is listed on +the `Device Id` line. The remaining lines are informational and not needed to configure the BACnet driver. + +For the first example, the IP address ``192.168.1.42`` can be used to reach the device. The second device is behind a +BACnet router and can be reached at ``1002:11``. See :ref:`BACnet router addressing `. + + +BACNet Scan Options +^^^^^^^^^^^^^^^^^^^ + + - ``--address ADDRESS``: Send the WhoIs request only to a specific address. Useful as a way to ping devices on a + network that blocks broadcast traffic. + - ``--range LOW/HIGH``: Specify the device ID range for the results. Useful for filtering. + - ``--timeout SECONDS``: Specify how long to wait for responses to the original broadcast. This defaults to 5 which + should be sufficient for most networks. + - ``--csv-out CSV_OUT``: Write the discovered devices to a CSV file. This can be used as inout for + ``grab_multiple_configs.py``. See :ref:`Scraping Multiple Devices `. + + +Automatically Generating a BACnet Registry Configuration File +------------------------------------------------------------- + +A CSV registry configuration file for the BACnet driver can be generated with the ``grab_bacnet_config.py`` script. + +.. warning:: + + This configuration will need to be edited before it can be used! + +The utility is invoked with the command: + +.. code-block:: bash + + python grab_bacnet_config.py + +This will query the device with the matching device ID for configuration information and print the resulting CSV file to +the console. + +In order to save the configuration to a file use the ``--out-file`` option to specify the output file name. + +Optionally the ``--address`` option can be used to specify the address of the target. In some cases, this is needed to +help establish a route to the device. + + +Output and Assumptions +^^^^^^^^^^^^^^^^^^^^^^ + +* Attempts at determining if a point is writable proved too unreliable. Therefore all points are considered to be + read-only in the output. + +* The only property for which a point is setup for an object is `presentValue`. + +* By default, the `Volttron Point Name` is set to the value of the `name` property of the BACnet object on the + device. In most cases this name is vague. No attempt is made at choosing a better name. A duplicate of + `Volttron Point Name` column called `Reference Point Name` is created to so that once `Volttron Point Name` is + changed a reference remains to the actual BACnet device object name. + +* Meta data from the objects on the device is used to attempt to put useful info in the `Units`, `Unit Details`, + and ``Notes`` columns. Information such as the range of valid values, defaults, the resolution or sensor input, and + enumeration or state names are scraped from the device. + +With a few exceptions "Units" is pulled from the object's "units" property and given the name used by the `bacpypes` +library to describe it. If a value in the **Units** column takes the form + +.. code-block:: python + + UNKNOWN UNIT ENUM VALUE: + +then the device is using a nonstandard value for the units on that object. + + +.. _Scraping-Multiple-BACnet-Devices: + +Scraping Multiple Devices +------------------------- + +The `grab_multiple_configs.py` script will use the CSV output of `bacnet_scan.py` to automatically run +`grab_bacnet_config.py` on every device listed in the CSV file. + +The output is put in two directories. `devices/` contains basic driver configurations for the scrapped devices. +`registry_configs/` contains the registry file generated by grab_bacnet_config.py. + +`grab_multiple_configs.py` makes no assumptions about device names or topics, however the output is appropriate for +the `install_master_driver_configs.py` script. + + +Grab Multiple Configs Options +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + - ``--out-directory OUT_DIRECTORY`` Specify the output directory. + - ``--use-proxy`` Use `proxy_grab_bacnet_config.py` to gather configuration data. + + +BACnet Proxy Alternative Scripts +-------------------------------- + +Both `grab_bacnet_config.py` and `bacnet_scan.py` have alternative versions called +`proxy_grab_bacnet_config.py` and `proxy_bacnet_scan.py` respectively. These versions require that the +VOLTTRON platform is running and BACnet Proxy agent is running. Both of these agents use the same command line +arguments as their independent counterparts. + +.. warning:: + + These versions of the BACnet scripts are intended as a proof of concept and have not been optimized for performance. + `proxy_grab_bacnet_config.py` takes about 10 times longer to grab a configuration than `grab_bacnet_config.py` + + +Problems and Debugging +---------------------- + +* Both `grab_bacnet_config.py` and `bacnet_scan.py` creates a virtual device that open up a port for communication + with devices. If the BACnet Proxy is running on the VOLTTRON platform it will cause both of these scripts to fail at + startup. Stopping the BACnet Proxy will resolve the problem. + +* Typically the utility should run quickly and finish in 30 seconds or less. In our testing, we have never seen a + successful scrape take more than 15 seconds on a very slow device with many points. Many devices will scrape in less + than 3 seconds. + +* If the utility has not finished after about 60 seconds it is probably having trouble communicating with the device and + should be stopped. Rerunning with debug output can help diagnose the problem. + +To output debug messages to the console add the ``--debug`` switch to the **end** of the command line arguments. + +.. code-block:: bash + + python grab_bacnet_config.py --out-file test.csv --debug + +On a successful run you will see output similar to this: + +.. code-block:: console + + DEBUG:main:initialization + DEBUG:main: - args: Namespace(address='10.0.2.20', buggers=False, debug=[], ini=, max_range_report=1e+20, out_file=) + DEBUG:main.SynchronousApplication:init (, '10.0.2.15') + DEBUG:main:starting build + DEBUG:main:pduSource =
+ DEBUG:main:iAmDeviceIdentifier = ('device', 500) + DEBUG:main:maxAPDULengthAccepted = 1024 + DEBUG:main:segmentationSupported = segmentedBoth + DEBUG:main:vendorID = 5 + DEBUG:main:device_name = MS-NCE2560-0 + DEBUG:main:description = + DEBUG:main:objectCount = 32 + DEBUG:main:object name = Building/FCB.Local Application.Room Real Temp 2 + DEBUG:main: object type = analogInput + DEBUG:main: object index = 3000274 + DEBUG:main: object units = degreesFahrenheit + DEBUG:main: object units details = -50.00 to 250.00 + DEBUG:main: object notes = Resolution: 0.1 + DEBUG:main:object name = Building/FCB.Local Application.Room Real Temp 1 + DEBUG:main: object type = analogInput + DEBUG:main: object index = 3000275 + DEBUG:main: object units = degreesFahrenheit + DEBUG:main: object units details = -50.00 to 250.00 + DEBUG:main: object notes = Resolution: 0.1 + DEBUG:main:object name = Building/FCB.Local Application.OSA + DEBUG:main: object type = analogInput + DEBUG:main: object index = 3000276 + DEBUG:main: object units = degreesFahrenheit + DEBUG:main: object units details = -50.00 to 250.00 + DEBUG:main: object notes = Resolution: 0.1 + ... + +and will finish something like this: + +.. code-block:: console + + ... + DEBUG:main:object name = Building/FCB.Local Application.MOTOR1-C + DEBUG:main: object type = binaryOutput + DEBUG:main: object index = 3000263 + DEBUG:main: object units = Enum + DEBUG:main: object units details = 0-1 (default 0) + DEBUG:main: object notes = BinaryPV: 0=inactive, 1=active + DEBUG:main:finally + +Typically if the BACnet device is unreachable for any reason (wrong IP, network down/unreachable, wrong interface +specified, device failure, etc) the scraper will stall at this message: + +.. code-block:: console + + DEBUG:main:starting build + +If you have not specified a valid interface in BACpypes.ini you will see the following error with a stack trace: + +.. code-block:: console + + ERROR:main:an error has occurred: [Errno 99] Cannot assign requested address + diff --git a/docs/source/driver-framework/bacnet/bacnet-driver.rst b/docs/source/driver-framework/bacnet/bacnet-driver.rst new file mode 100644 index 0000000000..9dd2505490 --- /dev/null +++ b/docs/source/driver-framework/bacnet/bacnet-driver.rst @@ -0,0 +1,157 @@ +.. _BACnet-Driver: + +============= +BACnet Driver +============= + +BACnet Driver Configuration +=========================== + +Communicating with BACnet devices requires that the :ref:`BACnet Proxy Agent ` is configured and +running. All device communication happens through this agent. + + +Requirements +------------ +The BACnet driver requires the BACPypes package. This package can be installed in an activated environment with: + +.. code-block:: bash + + pip install bacpypes + +Alternatively, running :ref:`bootstrap.py ` with the ``--drivers`` option will install all +requirements for drivers included in the repository including BACnet. + +.. code-block:: bash + + python3 bootstrap.py --drivers + +.. warning:: + + Current versions of VOLTTRON support **only** BACPypes version 0.16.7 + + +.. _BACnet-Configuration-File: + +Driver Config +------------- + +There are nine arguments for the `driver_config` section of the device configuration file: + + - **device_address** - Address of the device. If the target device is behind an IP to MS/TP router then Remote + Station addressing will probably be needed for the driver to find the device + - **device_id** - BACnet ID of the device. Used to establish a route to the device at startup + - **min_priority** - (Optional) Minimum priority value allowed for this device whether specifying the priority + manually or via the registry config. Violating this parameter either in the configuration or when writing to the + point will result in an error. Defaults to 8 + - **max_per_request** - (Optional) Configure driver to manually segment read requests. The driver will only grab up + to the number of objects specified in this setting at most per request. This setting is primarily for scraping + many points off of low resource devices that do not support segmentation. Defaults to 10000 + - **proxy_address** - (Optional) VIP address of the BACnet proxy. Defaults to "platform.bacnet_proxy". See + :ref:`bacnet-proxy-multiple-networks` for details. Unless your BACnet network has special needs you should not + change this value + - **ping_retry_interval** - (Optional) The driver will ping the device to establish a route at startup. If the + BACnet proxy is not available the driver will retry the ping at this interval until it succeeds. Defaults to 5 + - **use_read_multiple** - (Optional) During a scrape the driver will tell the proxy to use a + ReadPropertyMultipleRequest to get data from the device. Otherwise the proxy will use multiple ReadPropertyRequest + calls. If the BACnet proxy is reporting a device is rejecting requests try changing this to false for that device. + Be aware that setting this to false will cause scrapes for that device to take much longer. Only change if needed. + Defaults to true + - **cov_lifetime** - (Optional) When a device establishes a change of value subscription for a point, this argument + will be used to determine the lifetime and renewal period for the subscription, in seconds. Defaults to 180 + (Added to Master Driver version 3.2) + +Here is an example device configuration file: + +.. code-block:: json + + { + "driver_config": {"device_address": "10.1.1.3", + "device_id": 500, + "min_priority": 10, + "max_per_request": 24 + }, + "driver_type": "bacnet", + "registry_config":"config://registry_configs/vav.csv", + "interval": 5, + "timezone": "UTC", + "heart_beat_point": "heartbeat" + } + +A `sample BACnet configuration file `_ +can be found in the VOLTTRON repository at `examples/configurations/drivers/bacnet1.config` + + +.. _BACnet-Registry-Configuration-File: + +BACnet Registry Configuration File +---------------------------------- + +The registry configuration file is a `CSV `_ file. Each row +configures a point on the device. + +Most of the configuration file can be generated with the `grab_bacnet_config.py` utility in `scripts/bacnet`. See +:ref:`BACnet Auto-Configuration `. + +Currently, the driver provides no method to access array type properties even if the members of the array are of a +supported type. + +The following columns are required for each row: + + - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this + point. For instance, if the Volttron Point Name is `HeatCall1` (and using the example device configuration above) + then an agent would use `pnnl/isb2/hvac1/HeatCall1` to refer to the point when using the RPC interface of the + Actuator agent + - **Units** - Used for meta data when creating point information on the historian. + - **BACnet Object Type** - A string representing what kind of BACnet standard object the point belongs to. Examples + include: + + * analogInput + * analogOutput + * analogValue + * binaryInput + * binaryOutput + * binaryValue + * multiStateValue + + - **Property** - A string representing the name of the property belonging to the object. Usually, this will be + `presentValue` + - **Writable** - Either `TRUE` or `FALSE`. Determines if the point can be written to. Only points labeled `TRUE` + can be written to through the Actuator Agent. Points labeled `TRUE` incorrectly will cause an error to be + returned when an agent attempts to write to the point + - **Index** - Object ID of the BACnet object + +The following columns are optional: + + - **Write Priority** - BACnet priority for writing to this point. Valid values are 1-16. Missing this column or + leaving the column blank will use the default priority of 16 + - **COV Flag** - Either `True` or `False`. Determines if a BACnet Change-of-Value subscription should be + established for this point. Missing this column or leaving the column blank will result in no change of value + subscriptions being established. (Added to Master Driver version 3.2) + +Any additional columns will be ignored. It is common practice to include a `Point Name` or `Reference Point Name` +column to include the device documentation's name for the point and `Notes` and `Unit Details` columns for additional +information about a point. + +.. csv-table:: BACnet + :header: Point Name,Volttron Point Name,Units,Unit Details,BACnet Object Type,Property,Writable,Index,Notes + + Building/FCB.Local Application.PH-T,PreheatTemperature,degreesFahrenheit,-50.00 to 250.00,analogInput,presentValue,FALSE,3000119,Resolution: 0.1 + Building/FCB.Local Application.RA-T,ReturnAirTemperature,degreesFahrenheit,-50.00 to 250.00,analogInput,presentValue,FALSE,3000120,Resolution: 0.1 + Building/FCB.Local Application.RA-H,ReturnAirHumidity,percentRelativeHumidity,0.00 to 100.00,analogInput,presentValue,FALSE,3000124,Resolution: 0.1 + Building/FCB.Local Application.CLG-O,CoolingValveOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000107,Resolution: 0.1 + Building/FCB.Local Application.MAD-O,MixedAirDamperOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000110,Resolution: 0.1 + Building/FCB.Local Application.PH-O,PreheatValveOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000111,Resolution: 0.1 + Building/FCB.Local Application.RH-O,ReheatValveOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000112,Resolution: 0.1 + Building/FCB.Local Application.SF-O,SupplyFanSpeedOutputCommand,percent,0.00 to 100.00 (default 0.0),analogOutput,presentValue,TRUE,3000113,Resolution: 0.1 + + +A sample BACnet registry file can be found `here `_ or +in the VOLTTRON repository in `examples/configurations/drivers/bacnet.csv` + + +.. toctree:: + + bacnet-proxy-agent + bacnet-auto-configuration + bacnet-router-addressing diff --git a/docs/source/driver-framework/bacnet/bacnet-proxy-agent.rst b/docs/source/driver-framework/bacnet/bacnet-proxy-agent.rst new file mode 100644 index 0000000000..a68ead9a15 --- /dev/null +++ b/docs/source/driver-framework/bacnet/bacnet-proxy-agent.rst @@ -0,0 +1,206 @@ +.. _BACnet-Proxy-Agent: + +================== +BACnet Proxy Agent +================== + +Introduction +------------ + +Communication with BACnet device on a network happens via a single virtual BACnet device. In VOLTTRON driver framework, +we use a separate agent specifically for communicating with BACnet devices and managing the virtual BACnet device. + +Requirements +------------ +The BACnet Proxy agent requires the BACPypes package. This package can be installed in an activated environment with: + +.. code-block:: bash + + pip install bacpypes + +Alternatively, running :ref:`bootstrap.py ` with the `--drivers` option will install all +requirements for drivers included in the repository including BACnet. + +.. code-block:: bash + + python3 bootstrap.py --drivers + +.. warning:: + + Current versions of VOLTTRON support **only** BACPypes version 0.16.7 + + +Configuration +------------- + +The agent configuration sets up the virtual BACnet device. + +.. code-block:: json + + { + "device_address": "10.0.2.15", + "max_apdu_length": 1024, + "object_id": 599, + "object_name": "Volttron BACnet driver", + "vendor_id": 15, + "segmentation_supported": "segmentedBoth" + } + + +BACnet device settings +********************** + +- **device_address** - Address bound to the network port over which BACnet communication will happen on the computer + running VOLTTRON. This is **NOT** the address of any target device. See + :ref:`BACnet Router Addressing `. +- **object_id** - ID of the Device object of the virtual BACnet device. Defaults to 599. Only needs to be changed if + there is a conflicting BACnet device ID on your network. + +These settings determine the capabilities of the virtual BACnet device. BACnet communication happens at the lowest +common denominator between two devices. For instance, if the BACnet proxy supports segmentation and the target device +does not communication will happen without segmentation support and will be subject to those limitations. Consequently, +there is little reason to change the default settings outside of the `max_apdu_length` (the default is not the largest +possible value). + +- **max_apdu_length** - (From bacpypes documentation) BACnet works on lots of different types of networks, from + high-speed Ethernet to “slower” and “cheaper” ARCNET or MS/TP (a serial bus protocol used for a field bus defined by + BACnet). For devices to exchange messages they have to know the maximum size message the device can handle. + (End BACpypes docs) + + This setting determines the largest APDU (Application Protocol Data Unit) accepted by the BACnet virtual + device. Valid options are 50, 128, 206, 480, 1024, and 1476. Defaults to 1024.(Optional) + +- **object_name** - Name of the object. Defaults to "Volttron BACnet driver". (Optional) +- **vendor_id** - Vendor ID of the virtual BACnet device. Defaults to 15. (Optional) +- **segmentation_supported** - (From bacpypes documentation) A vast majority of BACnet communications traffic fits into + one message, but there can be times when larger messages are convenient and more efficient. Segmentation allows + larger messages to be broken up into segments and spliced back together. It is not unusual for “low power” field + equipment to not support segmentation. (End BACpypes docs) + + Possible setting are "segmentedBoth" (default), "segmentedTransmit", "segmentedReceive", or "noSegmentation" + (Optional) + + +Device Addressing +----------------- + +In some cases, it will be needed to specify the subnet mask of the virtual device or a different port number to listen +on. The full format of the BACnet device address is: + +:: + +
/: + +where ```` is the port to use and ```` is the netmask length. The most common value is 24. See +http://www.computerhope.com/jargon/n/netmask.htm + +For instance, if you need to specify a subnet mask of ``255.255.255.0`` and the IP address bound to the network port is +``192.168.1.2`` you would use the address: + +:: + + 192.168.1.2/24 + +If your BACnet network is on a different port (47809) besides the default (47808) you would use the address: + +:: + + 192.168.1.2:47809 + +If you need to do both: + +:: + + 192.168.1.2/24:47809 + + +.. _bacnet-proxy-multiple-networks: + +Communicating With Multiple BACnet Networks +------------------------------------------- + +If two BACnet devices are connected to different ports they are considered to be on different BACnet networks. In order +to communicate with both devices, you will need to run one BACnet Proxy Agent per network. + +Each proxy will need to be bound to different ports appropriate for each BACnet network and will need a different VIP +identity specified. When configuring drivers you will need to specify which proxy to use by +:ref:`specifying the VIP identity `. + +For example, a proxy connected to the default BACnet network: + +.. code-block:: json + + { + "device_address": "192.168.1.2/24" + } + +and another on port 47809: + +.. code-block:: json + + { + "device_address": "192.168.1.2/24:47809" + } + +a device on the first network: + +.. code-block:: json + + { + "driver_config": {"device_address": "1002:12", + "proxy_address": "platform.bacnet_proxy_47808", + "timeout": 10}, + "driver_type": "bacnet", + "registry_config":"config://registry_configs/bacnet.csv", + "interval": 60, + "timezone": "UTC", + "heart_beat_point": "Heartbeat" + } + +and a device on the second network: + +.. code-block:: json + + { + "driver_config": {"device_address": "12000:5", + "proxy_address": "platform.bacnet_proxy_47809", + "timeout": 10}, + "driver_type": "bacnet", + "registry_config":"config://registry_configs/bacnet.csv", + "interval": 60, + "timezone": "UTC", + "heart_beat_point": "Heartbeat" + } + +Notice that both configs use the same registry configuration (config://registry_configs/bacnet.csv). This is perfectly +fine as long as the registry configuration is appropriate for both devices. For scraping large numbers of points from +a single BACnet device, there is an optional timeout parameter provided, to prevent the master driver timing out while +the BACnet Proxy Agent is collecting points. + + +BACnet Change of Value Services +------------------------------- + +|BACnet Change of Value Communications| + +Change of Value Services added in version 0.5 of the BACnet Proxy and version 3.2 of the Master Driver. + +There are a variety of scenarios in which a user may desire data from some BACnet device point values to be published +independently of the regular scrape interval. Bacpypes provides a "ChangeOfValueServices" (hereby referred to as 'COV') +module, which enables a device to push updates to the platform. + +The BACnet COV requires that points on the device be properly configured for COV. A point on the BACnet device can be +configured with the 'covIncrement' property, which determines the threshold for a COV notification (note: this property +must be configured by the device operator - VOLTTRON does not provide the ability to set or modify this property). + +Based on configuration options for BACnet drivers, the driver will instruct the BACnet Proxy to establish a COV +subscription with the device. The subscription will last for an amount of time specified in the driver configuration, +and will auto-renew the subscription. If the proxy loses communication with the device or the device driver is stopped +the subscription will be removed when the lifetime expires. + +While the subscription exists, the device will send (confirmed) notifications to which will be published, with the topic +based on the driver's configured publish topics. + +https://bacpypes.readthedocs.io/en/latest/modules/service/cov.html + +.. |BACnet Change of Value Communications| image:: files/bacnet_cov.png diff --git a/docs/source/driver-framework/bacnet/bacnet-router-addressing.rst b/docs/source/driver-framework/bacnet/bacnet-router-addressing.rst new file mode 100644 index 0000000000..dffb1ccc1b --- /dev/null +++ b/docs/source/driver-framework/bacnet/bacnet-router-addressing.rst @@ -0,0 +1,34 @@ +.. _BACnet-Router-Addressing: + +======================== +BACnet Router Addressing +======================== + +The underlying library that Volttron uses for BACnet supports IP to MS/TP routers. Devices behind the router use a +Remote Station address in the form: + +:: + + :
+ +where ```` is the configured network ID of the router and ``
`` is the address of the device behind the +router. + +For example to access the device at ``
`` 12 for a router configured for ```` 1002 can be accessed with +this address: + +:: + + 1002:12 + +```` must be number from **0 to 65534** and ``
`` must be a number from **0 to 255**. + +This type of address can be used anywhere an address is required in configuration of the :ref:`Volttron BACnet driver +`. + +Caveats +------- + +VOLTTRON uses a UDP broadcast mechanism to establish the route to the device. If the route cannot be established it +will fall back to a UDP broadcast for all communication with the device. If the IP network where the router is +connected blocks UDP broadcast traffic then these addresses will not work. diff --git a/docs/source/core_services/drivers/files/bacnet_cov.png b/docs/source/driver-framework/bacnet/files/bacnet_cov.png similarity index 100% rename from docs/source/core_services/drivers/files/bacnet_cov.png rename to docs/source/driver-framework/bacnet/files/bacnet_cov.png diff --git a/docs/source/core_services/drivers/driver_configuration/chargepoint-driver.rst b/docs/source/driver-framework/chargepoint/chargepoint-driver.rst similarity index 63% rename from docs/source/core_services/drivers/driver_configuration/chargepoint-driver.rst rename to docs/source/driver-framework/chargepoint/chargepoint-driver.rst index 70c2aebe8a..40e4edb970 100644 --- a/docs/source/core_services/drivers/driver_configuration/chargepoint-driver.rst +++ b/docs/source/driver-framework/chargepoint/chargepoint-driver.rst @@ -1,29 +1,36 @@ -.. _Chargepoint-config: +.. _Chargepoint-Driver: + +================== +Chargepoint Driver +================== + + +.. _Chargepoint-Config: Chargepoint Driver Configuration --------------------------------- +================================ -The chargepoint driver requires at least one additional python library and has its own ``requirements.txt``. -Make sure to run +The chargepoint driver requires at least one additional python library and has its own `requirements.txt`. +Make sure to run: -:: +.. code-block:: bash pip install -r /requirements.txt before using this driver. + driver_config -************* +------------- There are three arguments for the **driver_config** section of the device configuration file: - - **stationID** - Chargepoint ID of the station. This format is ususally '1:00001' + - **stationID** - Chargepoint ID of the station. This format is usually '1:00001' - **username** - Login credentials for the Chargepoint API - - **password** - Login credentials for the Chargepoint API + - **password**- Login credentials for the Chargepoint API -The Chargepoint login credentials are generated in the Chargepoint web portal and require -a chargepoint account with sufficient privileges. Station IDs are also available on -the web portal. +The Chargepoint login credentials are generated in the Chargepoint web portal and require a Chargepoint account with +sufficient privileges. Station IDs are also available on the web portal. Here is an example device configuration file: @@ -40,24 +47,30 @@ Here is an example device configuration file: "heart_beat_point": "heartbeat" } -A sample Chargepoint configuration file can be found in the VOLTTRON repository -in ``examples/configurations/drivers/chargepoint1.config`` +A sample Chargepoint configuration file can be found in the VOLTTRON repository in +`examples/configurations/drivers/chargepoint1.config` -.. _Chargepoint-Driver: + +.. _Chargepoint-Registry-Config: Chargepoint Registry Configuration File -*************************************** +--------------------------------------- -The registry configuration file is a `CSV `_ file. Each row configures a point on the device. +The registry configuration file is a `CSV `_ file. Each row +configures a point on the device. The following columns are required for each row: - - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this point. - - **Attribute Name** - Chargepoint API attribute name. This determines the field that will be read from the API response and must be one of the allowed values. - - **Port #** - If the point describes a specific port on the Chargestation, it is defined here. (Note 0 and an empty value are equivalent.) + - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this + point. + - **Attribute Name** - Chargepoint API attribute name. This determines the field that will be read from the API + response and must be one of the allowed values. + - **Port #** - If the point describes a specific port on the Chargestation, it is defined here. (Note 0 and an empty + value are equivalent.) - **Type** - Python type of the point value. - **Units** - Used for meta data when creating point information on the historian. - - **Writable** - Either "TRUE" or "FALSE". Determines if the point can be written to. Only points labeled TRUE can be written. + - **Writable** - Either "TRUE" or "FALSE". Determines if the point can be written to. Only points labeled TRUE can + be written. - **Notes** - Miscellaneous notes field. - **Register Name** - A string representing how to interpret the data register. Acceptable values are: * StationRegister @@ -67,7 +80,8 @@ The following columns are required for each row: * StationRightsRegister - **Starting Value** - Default value for writeable points. Read-only points should not have a value in this column. -Detailed descriptions for all available chargepoint registers may be found in the ``README.rst`` in the -chargepoint driver directory. +Detailed descriptions for all available Chargepoint registers may be found in the ``README.rst`` in the Chargepoint +driver directory. -A sample Chargepoint registry file can be found in the VOLTTRON repository in ``examples/configurations/drivers/chargepoint.csv`` +A sample Chargepoint registry file can be found in the VOLTTRON repository in +``examples/configurations/drivers/chargepoint.csv`` diff --git a/docs/source/core_services/drivers/driver_configuration/dnp3-driver.rst b/docs/source/driver-framework/dnp3-driver/dnp3-driver.rst similarity index 85% rename from docs/source/core_services/drivers/driver_configuration/dnp3-driver.rst rename to docs/source/driver-framework/dnp3-driver/dnp3-driver.rst index 38e68b6b4c..627de2b39d 100644 --- a/docs/source/core_services/drivers/driver_configuration/dnp3-driver.rst +++ b/docs/source/driver-framework/dnp3-driver/dnp3-driver.rst @@ -1,28 +1,32 @@ .. _DNP3-Driver-Config: +========================= DNP3 Driver Configuration -------------------------- +========================= VOLTTRON's DNP3 driver enables the use of `DNP3 `_ (Distributed Network Protocol) communications, reading and writing points via a DNP3 Outstation. -In order to use a DNP3 driver to read and write point data, VOLTTRON's DNP3Agent must also +In order to use a DNP3 driver to read and write point data, VOLTTRON's DNP3 Agent must also be configured and running. All communication between the VOLTTRON Outstation and a -DNP3 Master happens through this DNP3Agent. -For information about the DNP3Agent, please see the :ref:`DNP3 Platform Specification `. +DNP3 Master happens through the DNP3 Agent. + +For information about the DNP3 Agent, please see the :ref:`DNP3 Platform Specification `. + Requirements ------------- -The DNP3 driver requires the PyDNP3 package. This package can be installed in an -activated environment with: +============ + +The DNP3 driver requires the PyDNP3 package. This package can be installed in an activated environment with: -:: +.. code-block:: bash pip install pydnp3 -driver_config -************* + +Driver Configuration +==================== There is one argument for the "driver_config" section of the DNP3 driver configuration file: @@ -49,10 +53,11 @@ Here is a sample DNP3 driver configuration file: A sample DNP3 driver configuration file can be found in the VOLTTRON repository in ``services/core/MasterDriverAgent/example_configurations/test_dnp3.config``. + .. _DNP3-Driver: DNP3 Registry Configuration File -******************************** +================================ The driver's registry configuration file, a `CSV `_ file, specifies which DNP3 points the driver will read and/or write. Each row configures a single DNP3 point. @@ -75,9 +80,9 @@ A sample data dictionary is available in ``services/core/DNP3Agent/dnp3/mesa_poi Point definitions in the DNP3 driver's registry should look something like this: -.. code-block:: csv +.. csv-table:: DNP3 + :header: Volttron Point Name,Group,Index,Scaling,Units,Writable - Volttron Point Name,Group,Index,Scaling,Units,Writable DCHD.WTgt,41,65,1.0,NA,FALSE DCHD.WTgt-In,30,90,1.0,NA,TRUE DCHD.WinTms,41,66,1.0,NA,FALSE diff --git a/docs/source/driver-framework/drivers-overview.rst b/docs/source/driver-framework/drivers-overview.rst new file mode 100644 index 0000000000..d2fb6bdc0f --- /dev/null +++ b/docs/source/driver-framework/drivers-overview.rst @@ -0,0 +1,135 @@ +.. _Driver-Framework: + +========================= +Driver Framework Overview +========================= + +VOLTTRON drivers act as an interface between agents on the platform and a device. While running on the platform, +drivers are special purpose agents which instead of being run as a separate process, are run as a greenlet in the +Master Driver process. + +Driver instances are created by the Master Driver when a new driver configuration is added to the configuration store. +Drivers use the following topic pattern `devices///`. When a configuration file is added +to the Master Driver's store using this pattern, the Master Driver creates a Driver Agent. The Driver agent is in turn +instantiated with a instance of the Interface class corresponding to the `driver_type` parameter in the configuration +file. The Interface class is responsible for implementing the communication paradigms of a device or protocol. Once +configured, the Master Driver periodically polls the Driver Agent for data which is collected from the interface class. +Additionally, points can be requested ad-hoc via the Master Driver's JSON-RPC method "get_point". Points may be set +by using JSON-RPC with the Actuator agent to set up a schedule and calling the "set_point" method. + + +Driver Conventions +****************** + +- Drivers are polled by the Master Driver agent and values can be set using the `Actuator Agent` +- Drivers should have a 1-to-1 relationship with a device +- Driver modules should be written in Python files in the `services/core/MasterDriverAgent/master_driver/interfaces` + directory in the VOLTTRON repository. The master driver will search for a Python file in this directory matching the + name provided by the `driver_type` value from the driver configuration when creating the Driver agent. +- Driver code consists of an Interface class (exactly named), supported in most cases by one or more Register classes + + +.. _Driver_Communication: + +Agent-Driver Communication Patterns +*********************************** + +The VOLTTRON message bus has been developed to allow agents on the platform to interact with each other, as well as with +ICS (Industrial Control Systems) and IOT (Internet of Things) devices via the VOLTTRON driver framework. Agents and +drivers have the ability to publish data to the message bus and to subscribe to message bus topics to read in data as it +is published. Additionally, agents may implement JSONRPC calls and expose JSONRPC endpoints to communicate more directly +with other agents. The following diagram demonstrates typical platform communication patterns for a single platform +deployment. + + +Typical Single Platform Behavior +================================ + +.. |Single Platform Communication Pattern| image:: files/driver_data_flow.png + +The diagram features several entities that comprise the platform and its connected components: + +* The VOLTTRON message bus - The message bus is the means of transmission of information in VOLTTRON. The VOLTTRON + message bus is built around existing message bus software; currently VOLTTRON supports RabbitMQ and ZeroMQ. The + VOLTTRON integration includes Pub/Sub and JSON RPC interfaces for agent and driver communication. +* VOLTTRON Platform Agents and Subsystems - These agents and subsystems are installed on the platform to manage the + platform. They provide many user facing functions, aid in communication and manage other agents and drivers. +* User's Agents - These agents are either agents included in the core repository but installed by a user, or user built + agent modules. They may perform a huge variety of user specified tasks, including data collection, device control, + simulation, etc. +* Master Driver Agent - This agent is installed by a user to facilitate communication with drivers. Drivers should not + communicated with directly - the master driver implements several features for communicating with drivers to ensure + smooth operation and consistent driver behavior. +* Actuator agent - This agent is installed by user to provide scheduling capability for controlling drivers. The master + driver does not include protections for race conditions, etc. It is always recommended to use the Actuator agent to + set values on a device. +* Device Driver - Drivers are special purpose agents which provide an interface between the master driver and devices + such as Modbus, and BACnet devices. Drivers implement a specific set of features for protecting device communication + ensuring uniform behaviors across different devices. +* Device - Devices may be low level physical computers for controlling various systems such as PLCs (Programmable Logic + Controller), devices which communicate on the local network (such as a Smart T.V.), or devices which are accessed via + a remote web API (other smart devices). + + +Lines of Communication +---------------------- + +Connectivity of the platform follows the following paradigm: + +* Platform agents (including the Master Driver and Actuator), subsystems, and user agents communicate with the message + bus via a publish/subscribe system. +* Agents can communicate "directly" to each other via JSONRPC calls - JSONRPC calls use the VOLTTRON message bus router + to "direct" messages to an intended recipient. RPC calls from an agent specify a function for the recipient to + perform including input parameters, and the response to the sender should contain the value output by the specified + function. +* The Master Driver will periodically poll device drivers. This functionality is intentionally not user-facing. The + Master Driver iterates over the configured drivers and calls their respective "scrape_all" methods. This will trigger + the drivers to collect point values. +* The Driver will communicate with its configured end devices to collect data points which it then returns to the + driver. The driver then publishes the point data to the bus under the `///all` topic. +* To get an individual device point, the user agent should send an RPC call to the Master Driver for "get_point", + providing the point's corresponding topic. After the Master Driver processes the request, communication happens very + similarly to polling, but rather than an "all" publish, the data is returned via the Master Driver to the user agent. +* To set a point on a device, it is recommended to use an Actuator Agent. The user agent sends an RPC request to the + Actuator to schedule time for the agent to control the device. During that scheduled time the user agent may send it + a set point request. If the schedule has been created, the actuator will then forward that request to the Master + Driver, at which point the communication happens similarly to a "get_point" request. + +The general paradigm for the device-driver relationship as specified by the VOLTTRON driver framework is a 1-to-1 +relationship. Each end device should be interacted with via a single device driver configured on one platform. To +distribute device data, the DataPuller and forwarder agents can be used at the platform level. Multiple platforms are +not intended to collect data or share control of a single device. + + +Special Case Drivers +==================== + +Some drivers require a different communication paradigm. One common alternative is shown in the diagram below: + +.. |Driver Proxy Pattern| image:: files/proxy_driver_data_flow.png + +This example describes an alternative pattern wherein BACnet drivers communicate via a BACnet proxy agent to communicate +with end devices. This behavior is derived from the networking requirements of the BACnet specification. BACnet +specifies a star topology for a given network; "slave" devices in a BACnet network communicate with a single "master". +In this case, the BACnet proxy acts as a virtual BACnet master, and device drivers forward their requests to this agent +which then performs the BACnet communication (whereas the typical pattern would have devices communicate directly with +the corresponding device). There are many other situations which may require this paradigm to be adopted (such as +working with remote APIs with request limits), and it is up to the party implementing the driver to determine if this +pattern or another pattern may be the most appropriate implementation pattern for their respective use case. + + +Installing the Fake Driver +************************** + +The Fake Driver is included as a way to quickly see data published to the message bus in a format that mimics what a +true driver would produce. This is a simple implementation of the VOLTTRON driver framework. + +See :ref:`instructions for installing the fake driver ` + +To view data being published from the fake driver on the message bus, one can +:ref:`install the Listener Agent ` and read the VOLTTRON log file: + +.. code-block:: bash + + cd + tail -f volttron.log diff --git a/docs/source/core_services/drivers/driver_configuration/ecobee_web_driver.rst b/docs/source/driver-framework/ecobee/ecobee-web-driver.rst similarity index 92% rename from docs/source/core_services/drivers/driver_configuration/ecobee_web_driver.rst rename to docs/source/driver-framework/ecobee/ecobee-web-driver.rst index 7cd84fb189..dc454569df 100644 --- a/docs/source/core_services/drivers/driver_configuration/ecobee_web_driver.rst +++ b/docs/source/driver-framework/ecobee/ecobee-web-driver.rst @@ -4,7 +4,7 @@ Ecobee Driver ************* -The Ecobee driver is an implementation of a :ref:`VOLTTRON driver framework ` Interface. +The Ecobee driver is an implementation of a :ref:`VOLTTRON driver framework ` Interface. In this case, the Master Driver issues commands to the Ecobee driver to collect data from and send control signals to `Ecobee's remote web API `_ @@ -110,7 +110,7 @@ The driver configuration works as follows: **Getting API Key** Ecobee API keys require configuring an application using the Ecobee web UI. For more information on configuring an - application and obtaining the API key, please refer to the `Ecobee Application `_ heading in + application and obtaining the API key, please refer to the :ref:`Ecobee Application ` heading in this documentation. **Finding Device Identifier** @@ -150,11 +150,6 @@ This is an example registry configuration: | actualTemperature | actualTemperature | degF | hold | False | True | | | +-------------------+---------------------+-------------------+----------+----------+---------------+-------+ -.. note:: - - An example registry configuration containing all points from the development device is available in the - `examples/configurations/drivers/ecobee.csv` file in the VOLTTRON repository. - This configuration works as follows: @@ -185,6 +180,9 @@ This configuration works as follows: | Notes | Any user specified notes, this is optional | +---------------------+------------------------------------------------------------------------------------------------+ +An example registry configuration containing all points from the development device is available in the +`examples/configurations/drivers/ecobee.csv` file in the VOLTTRON repository. + For additional explanation on the quirks of Ecobee's readable/writable points, visit: https://www.ecobee.com/home/developer/api/documentation/v1/functions/SetHold.shtml @@ -343,6 +341,32 @@ In an agent: self.vip.rpc.call("platform.driver", "get_point", , ) + +Set_point Conventions +##################### + +To set points using the Ecobee driver, it is recommended to use the actuator +agent. Explanations of the actuation can be found in the VOLTTRON readthedocs +and example agent code can be found in the CsvDriverAgent ( +examples/CSVDriver/CsvDriverAgent/agent.py in the VOLTTRON repository) + +Setting values for Vacations and Programs requires understanding Vacation and +Program object structure for Ecobee. + +Documentation for Vacation structure can be found here: +https://www.ecobee.com/home/developer/api/documentation/v1/functions/CreateVacation.shtml + +Documentation for Program structure can be found here: +https://www.ecobee.com/home/developer/api/examples/ex11.shtml + +When using set_point for program, specifying a program structure will create a +new program. Otherwise, if the user has not specified resume_all, Ecobee will +resume the next program on the program stack. If resume_all, Ecobee will resume +all programs on the program stack. + +For all other points, the corresponding integer, string, boolean, etc. value may +be sent. + Versioning ---------- diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_add_app.png b/docs/source/driver-framework/ecobee/files/ecobee_add_app.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_add_app.png rename to docs/source/driver-framework/ecobee/files/ecobee_add_app.png diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_api_key.png b/docs/source/driver-framework/ecobee/files/ecobee_api_key.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_api_key.png rename to docs/source/driver-framework/ecobee/files/ecobee_api_key.png diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_apps.png b/docs/source/driver-framework/ecobee/files/ecobee_apps.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_apps.png rename to docs/source/driver-framework/ecobee/files/ecobee_apps.png diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_console.png b/docs/source/driver-framework/ecobee/files/ecobee_console.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_console.png rename to docs/source/driver-framework/ecobee/files/ecobee_console.png diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_create_app.png b/docs/source/driver-framework/ecobee/files/ecobee_create_app.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_create_app.png rename to docs/source/driver-framework/ecobee/files/ecobee_create_app.png diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_developer_menu.png b/docs/source/driver-framework/ecobee/files/ecobee_developer_menu.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_developer_menu.png rename to docs/source/driver-framework/ecobee/files/ecobee_developer_menu.png diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_pin.png b/docs/source/driver-framework/ecobee/files/ecobee_pin.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_pin.png rename to docs/source/driver-framework/ecobee/files/ecobee_pin.png diff --git a/docs/source/core_services/drivers/driver_configuration/files/ecobee_verify_pin.png b/docs/source/driver-framework/ecobee/files/ecobee_verify_pin.png similarity index 100% rename from docs/source/core_services/drivers/driver_configuration/files/ecobee_verify_pin.png rename to docs/source/driver-framework/ecobee/files/ecobee_verify_pin.png diff --git a/docs/source/driver-framework/fake-driver/fake-driver.rst b/docs/source/driver-framework/fake-driver/fake-driver.rst new file mode 100644 index 0000000000..dfd9f0c345 --- /dev/null +++ b/docs/source/driver-framework/fake-driver/fake-driver.rst @@ -0,0 +1,157 @@ +.. _Fake-Driver: + +=========== +Fake Driver +=========== + +The FakeDriver is included as a way to quickly see data published to the message bus in a format +that mimics what a true Driver would produce. This is an extremely simple implementation of the +:ref:`VOLTTRON driver framework `. + + +Fake Device Driver Configuration +================================ + +This driver does not connect to any actual device and instead produces random and or pre-configured values. + + +Driver Config +------------- + +There are no arguments for the `driver_config` section of the device configuration file. The `driver_config` entry must +still be present and should be left blank. + +Here is an example device configuration file: + +.. code-block:: json + + { + "driver_config": {}, + "driver_type": "bacnet", + "registry_config":"config://registry_configs/vav.csv", + "interval": 5, + "timezone": "UTC", + "heart_beat_point": "heartbeat" + } + +A sample fake device configuration file can be found in the VOLTTRON repository in +`examples/configurations/drivers/fake.config` + +Fake Device Registry Configuration File +--------------------------------------- + +The registry configuration file is a `CSV `_ file. Each row +configures a point on the device. + +The following columns are required for each row: + + - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this + point. For instance, if the `Volttron Point Name` is `HeatCall1` (and using the example device configuration + above) then an agent would use `pnnl/isb2/hvac1/HeatCall1` to refer to the point when using the RPC interface of + the actuator agent. + - **Units** - Used for meta data when creating point information on the historian. + - **Writable** - Either `TRUE` or `FALSE`. Determines if the point can be written to. Only points labeled `TRUE` + can be written to through the ActuatorAgent. Points labeled `TRUE` incorrectly will cause an error to be returned + when an agent attempts to write to the point. + + +The following columns are optional: + + - **Starting Value** - Initial value for the point. If the point is reverted it will change back to this value. By + default, points will start with a random value (1-100). + - **Type** - Value type for the point. Defaults to "string". Valid types are: + + * string + * integer + * float + * boolean + +Any additional columns will be ignored. It is common practice to include a `Point Name` or `Reference Point Name` to +include the device documentation's name for the point and `Notes` and `Unit Details` for additional information +about a point. Please note that there is nothing in the driver that will enforce anything specified in the +`Unit Details` column. + +.. csv-table:: BACnet + :header: Volttron Point Name,Units,Units Details,Writable,Starting Value,Type,Notes + + Heartbeat,On/Off,On/Off,TRUE,0,boolean,Point for heartbeat toggle + OutsideAirTemperature1,F,-100 to 300,FALSE,50,float,CO2 Reading 0.00-2000.0 ppm + SampleWritableFloat1,PPM,10.00 (default),TRUE,10,float,Setpoint to enable demand control ventilation + SampleLong1,Enumeration,1 through 13,FALSE,50,int,Status indicator of service switch + SampleWritableShort1,%,0.00 to 100.00 (20 default),TRUE,20,int,Minimum damper position during the standard mode + SampleBool1,On / Off,on/off,FALSE,TRUE,boolean,Status indicator of cooling stage 1 + SampleWritableBool1,On / Off,on/off,TRUE,TRUE,boolean,Status indicator + +A sample fake registry configuration file can be found +`here `_ +or in the VOLTTRON repository in ``examples/configurations/drivers/fake.csv`` + + +.. _Fake-Driver-Install: + +Installation +============ + +Installing a Fake driver in the :ref:`Master Driver Agent ` requires adding copies of the device +configuration and registry configuration files to the Master Driver's :ref:`configuration store ` + +- Create a config directory (if one doesn't already exist) inside your Volttron repository: + +.. code-block:: bash + + mkdir config + +All local config files will be worked on here. + +- Copy over the example config file and registry config file from the VOLTTRON repository: + +.. code-block:: bash + + cp examples/configurations/drivers/fake.config config/ + cp examples/configurations/drivers/fake.csv config/ + +- Edit the driver config `fake.config` for the paths on your system: + +.. code-block:: json + + { + "driver_config": {}, + "registry_config": "config://fake.csv", + "interval": 5, + "timezone": "US/Pacific", + "heart_beat_point": "Heartbeat", + "driver_type": "fakedriver", + "publish_breadth_first_all": false, + "publish_depth_first": false, + "publish_breadth_first": false + } + +- Create a copy of the Master Driver config from the VOLTTRON repository: + +.. code-block:: bash + + cp examples/configurations/drivers/master-driver.agent config/fake-master-driver.config + +- Add fake.csv and fake.config to the :ref:`configuration store `: + +.. code-block:: bash + + vctl config store platform.driver devices/campus/building/fake config/fake.config + vcfl config store platform.driver fake.csv config/fake.csv --csv + +- Edit `fake-master-driver.config` to reflect paths on your system + +.. code-block:: json + + { + "driver_scrape_interval": 0.05 + } + +- Use the scripts/install-agent.py script to install the Master Driver agent: + +.. code-block:: bash + + python scripts/install-agent.py -s services/core/MasterDriverAgent -c config/fake-master-driver.config + +- If you have a :ref:`Listener Agent` already installed, you should start seeing data being published to + the bus. diff --git a/docs/source/core_services/drivers/driver_configuration/ieee-2030-driver.rst b/docs/source/driver-framework/ieee-2030_5/ieee-2030-driver.rst similarity index 92% rename from docs/source/core_services/drivers/driver_configuration/ieee-2030-driver.rst rename to docs/source/driver-framework/ieee-2030_5/ieee-2030-driver.rst index 5346545b1e..d9500ef8cb 100644 --- a/docs/source/core_services/drivers/driver_configuration/ieee-2030-driver.rst +++ b/docs/source/driver-framework/ieee-2030_5/ieee-2030-driver.rst @@ -1,11 +1,12 @@ .. _IEEE2030_5-Driver-Config: +================================ IEEE 2030.5 Driver Configuration --------------------------------- +================================ Communicating with IEEE 2030.5 devices requires that the IEEE 2030.5 Agent is configured and running. All device communication happens through this agent. For information about the IEEE 2030.5 Agent, -please see :ref:`IEEE2030_5`. +please see :ref:`IEEE2030_5-Agent`. driver_config ************* @@ -42,7 +43,7 @@ in ``services/core/MasterDriverAgent/example_configurations/test_ieee2030_5_1.co IEEE 2030.5 Registry Configuration File *************************************** -For a description of IEEE 2030.5 registry values, see :ref:`IEEE2030_5`. +For a description of IEEE 2030.5 registry values, see :ref:`IEEE2030_5-Agent`. A sample IEEE 2030.5 registry configuration file can be found in the VOLTTRON repository in ``services/core/MasterDriverAgent/example_configurations/ieee2030_5.csv``. diff --git a/docs/source/driver-framework/master-driver/master-driver.rst b/docs/source/driver-framework/master-driver/master-driver.rst new file mode 100644 index 0000000000..cad9cc2658 --- /dev/null +++ b/docs/source/driver-framework/master-driver/master-driver.rst @@ -0,0 +1,13 @@ +.. _Master-Driver: + +============= +Master Driver +============= + +Documentation coming soon! + +Contents: + + what does it do + how to use it + links to configuration diff --git a/docs/source/driver-framework/modbus/modbus-driver.rst b/docs/source/driver-framework/modbus/modbus-driver.rst new file mode 100644 index 0000000000..66c89099a0 --- /dev/null +++ b/docs/source/driver-framework/modbus/modbus-driver.rst @@ -0,0 +1,132 @@ +.. _Modbus-Driver: + +============= +Modbus Driver +============= + + + +VOLTTRON's modbus driver supports the Modbus over TCP/IP protocol only. For Modbus RTU support, see VOLTTRON's +`Modbus-TK driver `. + +`About Modbus protocol `_ + + +.. _Modbus-Config: + +Modbus Driver Configuration +=========================== + +Requirements +------------ +The Modbus driver requires the pymodbus package. This package can be installed in an activated environment with: + +.. code-block:: bash + + pip install pymodbus + +Alternatively this requirement can be installed using :ref:`bootstrap.py ` with the ``--drivers`` +option: + +.. code-block:: bash + + python3 bootstrap.py --drivers + + +Driver Configuration +-------------------- + +There are three arguments for the `driver_config` section of the device configuration file: + + - **device_address** - IP Address of the device. + - **port** - Port the device is listening on. Defaults to 502 which is the standard port for Modbus devices. + - **slave_id** - Slave ID of the device. Defaults to 0. Use 0 for no slave. + +The remaining values are as follows: + + + +Here is an example device configuration file: + +.. code-block:: json + + { + "driver_config": {"device_address": "10.1.1.2", + "port": 502, + "slave_id": 5}, + "driver_type": "modbus", + "registry_config":"config://registry_configs/hvac.csv", + "interval": 60, + "timezone": "UTC", + "heart_beat_point": "heartbeat" + } + +A sample MODBUS configuration file can be found in the VOLTTRON repository in +`examples/configurations/drivers/modbus.config` + + +.. _Modbus-Registry-Configuration: + +Modbus Registry Configuration File +---------------------------------- + +The registry configuration file is a `CSV `_ file. Each row +configures a point on the device. + +The following columns are required for each row: + + - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this + point. For instance, if the Volttron Point Name is HeatCall1 (and using the example device configuration above) + then an agent would use `pnnl/isb2/hvac1/HeatCall1` to refer to the point when using the RPC interface of the + actuator agent. + - **Units** - Used for meta data when creating point information on the historian. + - **Modbus Register** - A string representing how to interpret the data register and how to read it from the device. + The string takes two forms: + + + "BOOL" for coils and discrete inputs. + + A format string for the Python struct module. See + `the Python3 Struct docs `_ for full documentation. The + supplied format string must only represent one value. See the documentation of your device to determine how to + interpret the registers. Some Examples: + + * ">f" - A big endian 32-bit floating point number. + * "l" - A big endian 32-bit integer. + + - **Writable** - Either `TRUE` or `FALSE`. Determines if the point can be written to. Only points labeled + **TRUE** can be written to through the ActuatorAgent. + - **Point Address** - Modbus address of the point. Cannot include any offset value, it must be the exact value of + the address. + - **Mixed Endian** - (Optional) Either `TRUE` or `FALSE`. For mixed endian values. This will reverse the order + of the Modbus registers that make up this point before parsing the value or writing it out to the device. Has no + effect on bit values. + +The following column is optional: + + - **Default Value** - The default value for the point. When the point is reverted by an agent it will change back + to this value. If this value is missing it will revert to the last known value not set by an agent. + +Any additional columns will be ignored. It is common practice to include a `Point Name` or `Reference Point Name` to +include the device documentation's name for the point and `Notes` and `Unit Details` for additional information +about a point. + +The following is an example of a Modbus registry configuration file: + +.. csv-table:: Catalyst 371 + :header: Reference Point Name,Volttron Point Name,Units,Units Details,Modbus Register,Writable,Point Address,Default Value,Notes + + CO2Sensor,ReturnAirCO2,PPM,0.00-2000.00,>f,FALSE,1001,,CO2 Reading 0.00-2000.0 ppm + CO2Stpt,ReturnAirCO2Stpt,PPM,1000.00 (default),>f,TRUE,1011,1000,Setpoint to enable demand control ventilation + Cool1Spd,CoolSupplyFanSpeed1,%,0.00 to 100.00 (75 default),>f,TRUE,1005,75,Fan speed on cool 1 call + Cool2Spd,CoolSupplyFanSpeed2,%,0.00 to 100.00 (90 default),>f,TRUE,1007,90,Fan speed on Cool2 Call + Damper,DamperSignal,%,0.00 - 100.00,>f,FALSE,1023,,Output to the economizer damper + DaTemp,DischargeAirTemperature,F,(-)39.99 to 248.00,>f,FALSE,1009,,Discharge air reading + ESMEconMin,ESMDamperMinPosition,%,0.00 to 100.00 (5 default),>f,TRUE,1013,5,Minimum damper position during the energy savings mode + FanPower,SupplyFanPower, kW,0.00 to 100.00,>f,FALSE,1015,,Fan power from drive + FanSpeed,SupplyFanSpeed,%,0.00 to 100.00,>f,FALSE,1003,,Fan speed from drive + HeatCall1,HeatCall1,On / Off,on/off,BOOL,FALSE,1113,,Status indicator of heating stage 1 need + HeartBeat,heartbeat,On / Off,on/off,BOOL,FALSE,1114,,Status indicator of heating stage 2 need + +A sample Modbus registry file can be found +`here `_ +or in the VOLTTRON repository in `examples/configurations/drivers/catalyst371.csv` diff --git a/docs/source/core_services/drivers/driver_configuration/modbus-tk-driver.rst b/docs/source/driver-framework/modbus/modbus-tk-driver.rst similarity index 60% rename from docs/source/core_services/drivers/driver_configuration/modbus-tk-driver.rst rename to docs/source/driver-framework/modbus/modbus-tk-driver.rst index d6b134bb0c..a9ce969394 100644 --- a/docs/source/core_services/drivers/driver_configuration/modbus-tk-driver.rst +++ b/docs/source/driver-framework/modbus/modbus-tk-driver.rst @@ -1,71 +1,92 @@ -.. _Modbus-TK-config: +.. _Modbus-TK-Driver: -Modbus-TK Driver Configuration ------------------------------- +================ +Modbus TK Driver +================ + +VOLTTRON's Modbus-TK driver, built on the Python Modbus-TK library, is an alternative to the original VOLTTRON modbus +driver. Unlike the original modbus driver, the Modbus-TK driver supports Modbus RTU as well as Modbus over TCP/IP. + +`About Modbus protocol `_ + +The Modbus-TK driver introduces a map library and configuration builder, intended as a way to streamline configuration +file creation and maintenance. .. warning:: Currently the modbus_tk library is not able to make connections from 2 masters on one host to 2 slaves on one host - this will will prevent a single platform from being able to communicate to 2 slaves on IP as each instance of a Modbus_Tk driver creates a new Modbus master. `Issue on Modbus_Tk Github `_. -VOLTTRON's Modbus-TK driver, built on the Python Modbus-TK library, is an alternative to the -original VOLTTRON modbus driver. Unlike the original modbus driver, the Modbus-TK driver -supports Modbus RTU as well as Modbus over TCP/IP. -The Modbus-TK driver introduces a map library and configuration builder, intended as a way -to streamline configuration file creation and maintenance. +.. _Modbus-TK-Config: + +Modbus-TK Driver Configuration +============================== + +The Modbus-TK driver is mostly backward-compatible with the parameter definitions in the original Modbus driver's +configuration (.config and .csv files). If the config file's parameter names use the Modbus driver's name conventions, +they are translated to the Modbus-TK name conventions, e.g. a Modbus CSV file's ``Point Address`` is interpreted as a +Modbus-TK "Address". Backward-compatibility exceptions are: -The Modbus-TK driver is mostly backward-compatible with the parameter definitions in the original -Modbus driver's configuration (.config and .csv files). -If the config file's parameter names use the Modbus driver's name conventions, they are -translated to the Modbus-TK name conventions, e.g. a Modbus CSV file's "Point Address" is -interpreted as a Modbus-TK "Address". Backward-compatibility exceptions are: + - If the config file has no ``port``, the default is 0, not 502. + - If the config file has no ``slave_id``, the default is 1, not 0. - - If the config file has no **port**, the default is 0, not 502. - - If the config file has no **slave_id**, the default is 1, not 0. Requirements ------------ The Modbus-TK driver requires the modbus-tk package. This package can be installed in an activated environment with: -:: +.. code-block:: bash pip install modbus-tk -driver_config -************* +Alternatively this requirement can be installed using :ref:`bootstrap.py ` with the ``--drivers`` +option: -The **driver_config** section of a Modbus-TK device configuration file supports a variety of parameter definitions, +.. code-block:: bash + + python3 bootstrap.py --drivers + + +Driver Configuration +-------------------- + +The ``driver_config`` section of a Modbus-TK device configuration file supports a variety of parameter definitions, but only **device_address** is required: - - **name** (Optional) - Name of the device. Defaults to "UNKNOWN". - - **device_type** (Optional) - Name of the device type. Defaults to "UNKNOWN". - - **device_address** (Required) - IP Address of the device. - - **port** (Optional) - Port the device is listening on. Defaults to 0 (no port). Use port 0 for RTU transport. - - **slave_id** (Optional) - Slave ID of the device. Defaults to 1. Use ID 0 for no slave. - - **baudrate** (Optional) - Serial (RTU) baud rate. Defaults to 9600. - - **bytesize** (Optional) - Serial (RTU) byte size: 5, 6, 7, or 8. Defaults to 8. - - **parity** (Optional) - Serial (RTU) parity: none, even, odd, mark, or space. Defaults to none. - - **stopbits** (Optional) - Serial (RTU) stop bits: 1, 1.5, or 2. Defaults to 1. - - **xonxoff** (Optional) - Serial (RTU) flow control: 0 or 1. Defaults to 0. - - **addressing** (Optional) - Data address table: offset, offset_plus, or address. Defaults to offset. + - ``name`` (Optional) - Name of the device. Defaults to "UNKNOWN". + - ``device_type`` (Optional) - Name of the device type. Defaults to "UNKNOWN". + - ``device_address`` (Required) - IP Address of the device. + - ``port`` (Optional) - Port the device is listening on. Defaults to 0 (no port). Use port 0 for RTU transport. + - ``slave_id`` (Optional) - Slave ID of the device. Defaults to 1. Use ID 0 for no slave. + - ``baudrate`` (Optional) - Serial (RTU) baud rate. Defaults to 9600. + - ``bytesize`` (Optional) - Serial (RTU) byte size: 5, 6, 7, or 8. Defaults to 8. + - ``parity`` (Optional) - Serial (RTU) parity: none, even, odd, mark, or space. Defaults to none. + - ``stopbits`` (Optional) - Serial (RTU) stop bits: 1, 1.5, or 2. Defaults to 1. + - ``xonxoff`` (Optional) - Serial (RTU) flow control: 0 or 1. Defaults to 0. + - ``addressing`` (Optional) - Data address table: offset, offset_plus, or address. Defaults to offset. + - address: The exact value of the address without any offset value. - offset: The value of the address plus the offset value. - offset_plus: The value of the address plus the offset value plus one. - : If an offset value is to be added, it is determined based on a point's properties in the CSV file: + - Type=bool, Writable=TRUE: 0 - Type=bool, Writable=FALSE: 10000 - Type!=bool, Writable=TRUE: 30000 - Type!=bool, Writable=FALSE: 40000 - - **endian** (Optional) - Byte order: big or little. Defaults to big. - - **write_multiple_registers** (Optional) - Write multiple coils or registers at a time. Defaults to true. - - : If write_multiple_registers is set to false, only register types unsigned short (uint16) and boolean (bool) + + - ``endian`` (Optional) - Byte order: big or little. Defaults to big. + - ``write_multiple_registers`` (Optional) - Write multiple coils or registers at a time. Defaults to true. + + - If write_multiple_registers is set to false, only register types unsigned short (uint16) and boolean (bool) are supported. The exception raised during the configure process. - - **register_map** (Optional) - Register map csv of unchanged register variables. Defaults to registry_config csv. -Sample Modbus-TK configuration files are checked into the VOLTTRON repository -in ``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps``. + - ``register_map`` (Optional) - Register map csv of unchanged register variables. Defaults to registry_config csv. + +Sample Modbus-TK configuration files are checked into the VOLTTRON repository in +``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps``. Here is a sample TCP/IP Modbus-TK device configuration: @@ -126,29 +147,35 @@ Here is a sample RTU Modbus-TK device configuration, with completely-specified s "timezone": "UTC" } -.. _Modbus-TK-Driver: + +.. _Modbus-TK-Register-Map: Modbus-TK Register Map CSV File -******************************* +=============================== -The registry configuration file is a `CSV `_ file. +Modbus TK requires an additional registry configuration file compared to the paradigm of most other drivers. The +registry map file is an analogue to the typical registry configuration file. The +:ref:`registry configuration file ` is a simple file which maps device point names to user +specified point names. + +The registry map file is a `CSV `_ file. Each row configures a register definition on the device. - - **Register Name** (Required) - The field name in the modbus client. This field is distinct and unchangeable. - - **Address** (Required) - The point's modbus address. The **addressing** option in the driver configuration + - ``Register Name`` (Required) - The field name in the modbus client. This field is distinct and unchangeable. + - ``Address`` (Required) - The point's modbus address. The ``addressing`` option in the driver configuration controls whether this is interpreted as an exact address or an offset. - - **Type** (Required) - The point's data type: bool, string[length], float, int16, int32, int64, uint16, + - ``Type`` (Required) - The point's data type: bool, string[length], float, int16, int32, int64, uint16, uint32, or uint64. - - **Units** (Optional) - Used for metadata when creating point information on a historian. Default is an + - ``Units`` (Optional) - Used for metadata when creating point information on a historian. Default is an empty string. - - **Writable** (Optional) - TRUE/FALSE. Only points for which Writable=TRUE can be updated by a VOLTTRON agent. + - ``Writable`` (Optional) - TRUE/FALSE. Only points for which Writable=TRUE can be updated by a VOLTTRON agent. Default is FALSE. - - **Default Value** (Optional) - The point's default value. If it is reverted by an agent, it changes back + - ``Default Value`` (Optional) - The point's default value. If it is reverted by an agent, it changes back to this value. If this value is missing, it will revert to the last known value not set by an agent. - - **Transform** (Optional) - Scaling algorithm: scale(multiplier), scale_int(multiplier), scale_reg(register_name), + - ``Transform`` (Optional) - Scaling algorithm: scale(multiplier), scale_int(multiplier), scale_reg(register_name), scale_reg_power10(register_name), scale_decimal_int_signed(multiplier), mod10k(reverse), mod10k64(reverse), mod10k48(reveres) or none. Default is an empty string. - - **Table** (Optional) - Standard modbus table name defining how information is stored in slave device. + - ``Table`` (Optional) - Standard modbus table name defining how information is stored in slave device. There are 4 different tables: - discrete_output_coils: read/write coil numbers 1-9999 @@ -159,18 +186,18 @@ Each row configures a register definition on the device. If this field is empty, the modbus table will be defined by **type** and **writable** fields. By that, when user sets read only for read/write coils/registers or sets read/write for read only coils/registers, it will select wrong table, and therefore raise exception. - - **Mixed Endian** (Optional) - TRUE/FALSE. If Mixed Endian is set to TRUE, the order of the MODBUS registers will + - ``Mixed Endian`` (Optional) - TRUE/FALSE. If Mixed Endian is set to TRUE, the order of the Modbus registers will be reversed before parsing the value or writing it out to the device. By setting mixed endian, transform must be None (no op). Defaults to FALSE. - - **Description** (Optional) - Additional information about the point. Default is an empty string. + - ``Description`` (Optional) - Additional information about the point. Default is an empty string. Any additional columns are ignored. -Sample Modbus-TK registry CSV files are checked into the VOLTTRON repository -in ``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps``. +Sample Modbus-TK registry map CSV files are checked into the VOLTTRON repository in +``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps``. -Here is a sample Modbus-TK registry configuration: +Here is a sample Modbus-TK registry map: .. csv-table:: :header: Register Name,Address,Type,Units,Writable,Default Value,Transform,Table @@ -185,24 +212,27 @@ Here is a sample Modbus-TK registry configuration: sample_bool,16,bool,None,TRUE,False,,analog_output_holding_registers sample_str,17,string[12],None,TRUE,hello world!,,analog_output_holding_registers -Modbus-TK Registry Configuration CSV File -***************************************** + +.. _Modbus-TK-Registry-Config: + +Modbus-TK Registry Configuration +================================ The registry configuration file is a `CSV `_ file. Each row configures a point on the device. - - **Volttron Point Name** (Required) - The name by which the platform and agents refer to the point. - For instance, if the Volttron Point Name is HeatCall1, then an agent would use ``my_campus/building2/hvac1/HeatCall1`` - to refer to the point when using the RPC interface of the actuator agent. - - **Register Name** (Required) - The field name in the modbus client. - It must be matched with the field name from **register_map**. + - ``Volttron Point Name`` (Required) - The name by which the platform and agents refer to the point. For instance, + if the Volttron Point Name is HeatCall1, then an agent would use ``my_campus/building2/hvac1/HeatCall1`` to refer + to the point when using the RPC interface of the actuator agent. + - ``Register Name`` (Required) - The field name in the modbus client. It must be matched with the field name from + ``register_map``. -Any additional columns will override the existed fields from **register_map**. +Any additional columns will override the existed fields from ``register_map``. Sample Modbus-TK registry CSV files are checked into the VOLTTRON repository in ``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps``. -Here is a sample Modbus-TK registry configuration with defined **register_map**: +Here is a sample Modbus-TK registry configuration with defined ``register_map``: .. csv-table:: :header: Volttron Point Name,Register Name @@ -217,32 +247,32 @@ Here is a sample Modbus-TK registry configuration with defined **register_map**: sample bool,sample_bool sample str,sample_str + .. _Modbus-TK-Maps: -Modbus-TK Driver Maps -********************* +Modbus-TK Driver Maps Repository +================================ -To help facilitate the creation of VOLTTRON device configuration entries (.config files) for Modbus-TK -devices, a library of device type definitions is now maintained -in ``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps/maps.yaml``. A -command-line tool (described below under **MODBUS TK Config Command Tool**) uses the contents -of ``maps.yaml`` while generating .config files. +To help facilitate the creation of VOLTTRON device configuration entries (.config files) for Modbus-TK devices, a +library of device type definitions is now maintained in +``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps/maps.yaml``. A command-line tool (described +below under ``MODBUS TK Config Command Tool``) uses the contents of ``maps.yaml`` while generating ``.config`` files. Each device type definition in ``maps.yaml`` consists of the following properties: - - **name** (Required) - Name of the device type (see the driver_config parameters). - - **file** (Required) - The name of the CSV file that defines all of the device type's supported points, + - ``name`` (Required) - Name of the device type (see the driver_config parameters). + - ``file`` (Required) - The name of the CSV file that defines all of the device type's supported points, e.g. watts_on.csv. - - **description** (Optional) - A description of the device type. - - **addressing** (Optional) - Data address type: offset, offset_plus, or address (see the driver_config parameters). - - **endian** (Optional) - Byte order: big or little (see the driver_config parameters). - - **write_multiple_registers** (Optional) - Write multiple registers at a time. Defaults to true. + - ``description`` (Optional) - A description of the device type. + - ``addressing`` (Optional) - Data address type: offset, offset_plus, or address (see the driver_config parameters). + - ``endian`` (Optional) - Byte order: big or little (see the driver_config parameters). + - ``write_multiple_registers`` (Optional) - Write multiple registers at a time. Defaults to true. -A device type definition is a template for a device configuration. Some additional data must -be supplied when a specific device's configuration is generated. In particular, the device_address must be supplied. +A device type definition is a template for a device configuration. Some additional data must be supplied when a specific +device's configuration is generated. In particular, the device_address must be supplied. -A sample ``maps.yml`` file is checked into the VOLTTRON repository -in ``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps/maps.yaml``. +A sample ``maps.yml`` file is checked into the VOLTTRON repository in +``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps/maps.yaml``. Here is a sample ``maps.yaml`` file: @@ -265,10 +295,11 @@ Here is a sample ``maps.yaml`` file: description: ION 8600 meter file: ion8600_map.csv + .. _Modbus-TK-Config-Cmd: Modbus-TK Config Command Tool -***************************** +============================= ``config_cmd.py`` is a command-line tool for creating and maintaining VOLTTRON driver configurations. The tool runs from the command line: @@ -280,9 +311,10 @@ runs from the command line: ``config_cmd.py`` supports the following commands: - - **help** - List all commands. - - **quit** - Quit the command-line tool. - - **list_directories** - List all setup directories, with an option to edit their paths. + - ``help`` - List all commands. + - ``quit`` - Quit the command-line tool. + - ``list_directories`` - List all setup directories, with an option to edit their paths. + + By default, all directories are in the VOLTTRON repository in ``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/maps``. + It is important to use the correct directories when adding/editing device types and driver configs, @@ -291,29 +323,30 @@ runs from the command line: * map_dir: directory in which ``maps.yaml`` is stored. * config_dir: directory in which driver config files are stored. * csv_dir: directory in which registry config CSV files are stored. - - **edit_directories** - Add/Edit map directory, driver config directory, and/or CSV config directory. + + - ``edit_directories`` - Add/Edit map directory, driver config directory, and/or CSV config directory. Press if no change is needed. Exits if the directory does not exist. - - **list_device_type_description** - List all device type descriptions in ``maps.yaml``. + - ``list_device_type_description`` - List all device type descriptions in ``maps.yaml``. Option to edit device type descriptions. - - **list_all_device_types** - List all device type information in ``maps.yaml``. Option to add more device types. - - **device_type** - List information for a selected device type. Option to select another device type. - - **add_device_type** - Add a device type to ``maps.yaml``. Option to add more than one device type. + - ``list_all_device_types`` - List all device type information in ``maps.yaml``. Option to add more device types. + - ``device_type`` - List information for a selected device type. Option to select another device type. + - ``add_device_type`` - Add a device type to ``maps.yaml``. Option to add more than one device type. Each device type includes its name, CSV file, description, addressing, and endian, as explained - in **MODBUS-TK Driver Maps**. If an invalid value is entered for addressing or endian, + in ``MODBUS-TK Driver Maps``. If an invalid value is entered for addressing or endian, the default value is used instead. - - **edit_device_type** - Edit an existing device type. If an invalid value is entered for addressing or endian, + - ``edit_device_type`` - Edit an existing device type. If an invalid value is entered for addressing or endian, the previous value is left unchanged. - - **list_drivers** - List all driver config names in ``config_dir``. - - **driver_config ** - Get a driver config from ``config_dir``. + - ``list_drivers`` - List all driver config names in ``config_dir``. + - ``driver_config `` - Get a driver config from ``config_dir``. Option to select the driver if no driver is found with that name. - - **add_driver_config ** - Add/Edit ``/.config``. + - ``add_driver_config `` - Add/Edit ``/.config``. Option to select the driver if no driver is found with that name. Press to exit. - - **load_volttron** - Load a driver config and CSV into VOLTTRON. Option to add the config or CSV file + - ``load_volttron`` - Load a driver config and CSV into VOLTTRON. Option to add the config or CSV file to config_dir or to csv_dir. VOLTTRON must be running when this command is used. - - **delete_volttron_config** - Delete a driver config from VOLTTRON. VOLTTRON must be running + - ``delete_volttron_config`` - Delete a driver config from VOLTTRON. VOLTTRON must be running when this command is used. - - **delete_volttron_csv** - Delete a registry csv config from VOLTTRON. VOLTTRON must be running + - ``delete_volttron_csv`` - Delete a registry csv config from VOLTTRON. VOLTTRON must be running when this command is used. -The ``config_cmd.py`` module is checked into the VOLTTRON repository -as ``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/config_cmd.py``. +The ``config_cmd.py`` module is checked into the VOLTTRON repository as +``services/core/MasterDriverAgent/master_driver/interfaces/modbus_tk/config_cmd.py``. diff --git a/docs/source/core_services/drivers/driver_configuration/obix.rst b/docs/source/driver-framework/obix/obix.rst similarity index 52% rename from docs/source/core_services/drivers/driver_configuration/obix.rst rename to docs/source/driver-framework/obix/obix.rst index 5717703f00..3306023d59 100644 --- a/docs/source/core_services/drivers/driver_configuration/obix.rst +++ b/docs/source/driver-framework/obix/obix.rst @@ -1,21 +1,29 @@ -.. _Obix-config: +.. _Obix-Driver: + +=========== +Obix Driver +=========== + + +.. _Obix-Config: Obix Driver Configuration -------------------------- +========================= + VOLTTRON's uses Obix's restful interface to facilitate communication. -This driver does *not* handle reading data from the history section of the interface. -If the user wants data published from the management systems historical data use -the :ref:`Obix-history` agent. +This driver does *not* handle reading data from the history section of the interface. If the user wants data published +from the management systems historical data use the :ref:`Obix History ` agent. -driver_config -************* -There are three arguments for the **driver_config** section of the device configuration file: +Driver Configuration +-------------------- - - **url** - URL of the interface. - - **username** - User name for site.. - - **password** - Password for username. +There are three arguments for the ``driver_config`` section of the device configuration file: + + - ``url`` - URL of the Obix remote API interface + - ``username`` - User's username for the Obix remote API + - ``password`` - Users' password corresponding to the username Here is an example device configuration file: @@ -31,31 +39,41 @@ Here is an example device configuration file: "timezone": "UTC" } -A sample Obix configuration file can be found in the VOLTTRON repository in ``examples/configurations/drivers/obix.config`` +A sample Obix configuration file can be found in the VOLTTRON repository in +`examples/configurations/drivers/obix.config` -.. _Obix-Driver: +.. _Obix-Registry-Config: Obix Registry Configuration File -******************************** +-------------------------------- -The registry configuration file is a `CSV `_ file. Each row configures a point on the device. +The registry configuration file is a `CSV `_ file. Each row +configures a point on the device. The following columns are required for each row: - - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this point. For instance, if the Volttron Point Name is HeatCall1 then an agent would use ``/HeatCall1`` to refer to the point when using the RPC interface of the actuator agent. - - **Obix Point Name** - Name of the point on the obix interface. Escaping of spaces and dashes for use with the interface is handled internaly. - - **Obix Type** - One of "bool", "int", or "real" without quotes. + - **Volttron Point Name** - The name by which the platform and agents running on the platform will refer to this + point. For instance, if the Volttron Point Name is HeatCall1 then an agent would use `/HeatCall1` + to refer to the point when using the RPC interface of the actuator agent. + - **Obix Point Name** - Name of the point on the Obix interface. Escaping of spaces and dashes for use with the + interface is handled internally. + - **Obix Type** - One of `bool`, `int`, or `real` - **Units** - Used for meta data when creating point information on the historian. - - **Writable** - Either "TRUE" or "FALSE". Determines if the point can be written to. Only points labeled TRUE can be written to through the ActuatorAgent. This can be used to protect points that should not be accessed by the platform. + - **Writable** - Either `TRUE` or `FALSE`. Determines if the point can be written to. Only points labeled + **TRUE** can be written to through the ActuatorAgent. This can be used to protect points that should not be + accessed by the platform. The following column is optional: - - **Default Value** - The default value for the point. When the point is reverted by an agent it will change back to this value. If this value is missing it will revert to the last known value not set by an agent. + - **Default Value** - The default value for the point. When the point is reverted by an agent it will change back to + this value. If this value is missing it will revert to the last known value not set by an agent. -Any additional columns will be ignored. It is common practice to include a **Point Name** or **Reference Point Name** to include the device documentation's name for the point and **Notes** and **Unit Details** for additional information about a point. +Any additional columns will be ignored. It is common practice to include a `Point Name` or `Reference Point Name` to +include the device documentation's name for the point and `Notes` and `Unit Details` for additional information +about a point. -The following is an example of a Obix registry confugration file: +The following is an example of a Obix registry configuration file: .. csv-table:: Obix :header: Volttron Point Name,Obix Point Name,Obix Type,Units,Writable,Notes @@ -76,22 +94,27 @@ The following is an example of a Obix registry confugration file: HomeImportSPFromCDH,HomeImportSPFromCDH,real,kilowatt,FALSE,Precision: 0 ThermalFollowingAlarm,ThermalFollowingAlarm,bool,,FALSE, -A sample Obix configuration can be found in the VOLTTRON repository in ``examples/configurations/drivers/obix.csv`` +A sample Obix configuration can be found in the VOLTTRON repository in `examples/configurations/drivers/obix.csv` + -.. _Obix-AutoConfiguration: +.. _Obix-Auto-Configuration: Automatic Obix Configuration File Creation -****************************************** -A script that will automatically create both a device and register -configuration file for a site is located in the repository at ``scripts/obix/get_obix_driver_config.py``. +------------------------------------------ + +A script that will automatically create both a device and register configuration file for a site is located in the +repository at `scripts/obix/get_obix_driver_config.py`. The utility is invoked with the command: - ``python get_obix_driver_config.py -u -p `` +.. code-block:: bash + + python get_obix_driver_config.py -u -p -If either the registry_file or driver_file is omitted the script will output those files to stdout. +If either the `registry_file` or `driver_file` is omitted the script will output those files to stdout. -If either the username or password arguments are left out the script will ask for them on the command line before proceeding. +If either the username or password arguments are left out the script will ask for them on the command line before +proceeding. -The registry file produced by this script assumes that the `Volttron Point Name` and the `Obix Point Name` have the same value. -Also, it is assumed that all points should be read only. Users are expected to fix this as appropriate. +The registry file produced by this script assumes that the `Volttron Point Name` and the `Obix Point Name` have the same +value. Also, it is assumed that all points should be read only. Users are expected to fix this as appropriate. diff --git a/docs/source/core_services/drivers/files/ted-spyders.png b/docs/source/driver-framework/ted-driver/files/ted-spyders.png similarity index 100% rename from docs/source/core_services/drivers/files/ted-spyders.png rename to docs/source/driver-framework/ted-driver/files/ted-spyders.png diff --git a/docs/source/core_services/drivers/driver_configuration/the-energy-detective-driver.rst b/docs/source/driver-framework/ted-driver/the-energy-detective-driver.rst similarity index 76% rename from docs/source/core_services/drivers/driver_configuration/the-energy-detective-driver.rst rename to docs/source/driver-framework/ted-driver/the-energy-detective-driver.rst index 82c4e116e5..f70959b5de 100644 --- a/docs/source/core_services/drivers/driver_configuration/the-energy-detective-driver.rst +++ b/docs/source/driver-framework/ted-driver/the-energy-detective-driver.rst @@ -1,25 +1,20 @@ .. _The-Energy-Detective-Driver: +================================= The Energy Detective Meter Driver ------------------------------------- +================================= +The TED-Pro is an energy monitoring system that can measure energy consumption of multiple mains and supports +sub-metering of individual circuits. This driver connects to a TED Pro Energy Control Center (ECC) and can collect +information from multiple Measuring Transmitting Units (MTUs) and Spyder sub-metering devices connected to the ECC. -Introduction ------------- -The TED-Pro is an energy monitoring system that can measure energy consumption -of multiple mains and supports submetering of individual circuits. -This driver connects to a TED Pro Energy Control Center (ECC) and can collect -information from multiple Measuring Transmiting Units (MTUs) and Spyder submetering -devices connected to the ECC. +Configuration +============= -configuration -------------- - -The TED Pro device interface is configured as follows. You'll need the ip address -or hostname of the ECC on a network segment accessible from the Volttron instance, -if configured to use a port other than 80, you can provide it as shown below, -following a colon after the host address. +The TED Pro device interface is configured as follows. You'll need the ip address or hostname of the ECC on a network +segment accessible from the VOLTTRON instance, if configured to use a port other than 80, you can provide it as shown +below, following a colon after the host address. .. code-block:: json @@ -34,23 +29,24 @@ following a colon after the host address. } } + Parameters -********** +---------- - **username** - Username if the TED Pro is configured with Basic Authentication - **password** - Password if the TED Pro is configured with Basic Authentication - **device_address** - Hostname or IP address of the TED Pro ECC, a non-standard port can be included if needed - - **scrape_spyder** - Default true, enables or disables collection of the submetering data from spyder devices - connected to the TED Pro + - **scrape_spyder** - Default true, enables or disables collection of the sub-metering data from spyder devices + connected to the TED Pro - **track_totalizers** - Default true, enables or disables tracking of lifetime totals in the VOLTTRON Driver .. note:: - The TED Pro does not expose its internal lifetime totalized metering, instead offering month to date (MTD) - and daily totals (TDY). Using the "track_totalizers" setting, the ted-meter driver will attempt to maintain - monotonically increasing lifetime totalizers. To do so, it must retain state regarding the running total and - the last read value. The driver makes use of the VOLTTRON Config subsystem to store this state. - To reset these totals, delete the state/ted_meter/ config from the master driver config store and restart the + The TED Pro does not expose its internal lifetime "totalized" metering, instead offering month to date (MTD) + and daily totals (TDY). Using the "track_totalizers" setting, the ted-meter driver will attempt to maintain + monotonically increasing lifetime totalizers. To do so, it must retain state regarding the running total and + the last read value. The driver makes use of the VOLTTRON Config subsystem to store this state. To reset these + totals, delete the 1state/ted_meter/1 config from the master driver config store and restart the master driver. .. note:: @@ -58,15 +54,15 @@ Parameters This driver does not make use of the registry config. Because it is able to determine the configuration of the TED Pro Device via the API, it simply creates registers for each data source on the TED Pro - .. note:: - This driver is internally aware of the appropriate HayStack Tags for its registers, however, the - MasterDriver Framework makes no provision for publishing those tags during a scrape. Therefore, - integration of the tagging data is left to the end user. + This driver is internally aware of the appropriate HayStack Tags for its registers, however, the Master Driver makes + no provision for publishing those tags during a scrape. Therefore, integration of the tagging data is left to the + end user. + Examples -******** +-------- |TED Pro showing spyder outputs| @@ -137,4 +133,4 @@ The above configuration in the TED will result in the following scrape from the } ] -.. |TED Pro showing spyder outputs| image:: ../files/ted-spyders.png \ No newline at end of file +.. |TED Pro showing spyder outputs| image:: files/ted-spyders.png diff --git a/docs/source/setup/images/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png b/docs/source/files/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png similarity index 100% rename from docs/source/setup/images/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png rename to docs/source/files/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png diff --git a/docs/source/files/dashboard-blank.png b/docs/source/files/dashboard-blank.png deleted file mode 100755 index 2e62a2331d..0000000000 Binary files a/docs/source/files/dashboard-blank.png and /dev/null differ diff --git a/docs/source/images/VOLTTRON_User_Guide.pdf b/docs/source/images/VOLTTRON_User_Guide.pdf deleted file mode 100755 index 4c8b77a890..0000000000 Binary files a/docs/source/images/VOLTTRON_User_Guide.pdf and /dev/null differ diff --git a/docs/source/images/clone-existing.png b/docs/source/images/clone-existing.png deleted file mode 100755 index 465f351bf0..0000000000 Binary files a/docs/source/images/clone-existing.png and /dev/null differ diff --git a/docs/source/images/dashboard-blank.png b/docs/source/images/dashboard-blank.png deleted file mode 100755 index 2e62a2331d..0000000000 Binary files a/docs/source/images/dashboard-blank.png and /dev/null differ diff --git a/docs/source/images/eclipse-marketplace.png b/docs/source/images/eclipse-marketplace.png deleted file mode 100755 index cf6a0136b2..0000000000 Binary files a/docs/source/images/eclipse-marketplace.png and /dev/null differ diff --git a/docs/source/images/eclipse-marketplace2.png b/docs/source/images/eclipse-marketplace2.png deleted file mode 100755 index 91b4b5f706..0000000000 Binary files a/docs/source/images/eclipse-marketplace2.png and /dev/null differ diff --git a/docs/source/images/finish-import.png b/docs/source/images/finish-import.png deleted file mode 100755 index 5a32e44151..0000000000 Binary files a/docs/source/images/finish-import.png and /dev/null differ diff --git a/docs/source/images/general-project.png b/docs/source/images/general-project.png deleted file mode 100755 index 106228a41d..0000000000 Binary files a/docs/source/images/general-project.png and /dev/null differ diff --git a/docs/source/images/git-view.png b/docs/source/images/git-view.png deleted file mode 100755 index 0b59e8fcc3..0000000000 Binary files a/docs/source/images/git-view.png and /dev/null differ diff --git a/docs/source/images/import-project.png b/docs/source/images/import-project.png deleted file mode 100755 index 5ac524f41d..0000000000 Binary files a/docs/source/images/import-project.png and /dev/null differ diff --git a/docs/source/images/listener-all-vars.png b/docs/source/images/listener-all-vars.png deleted file mode 100755 index f4fdde5296..0000000000 Binary files a/docs/source/images/listener-all-vars.png and /dev/null differ diff --git a/docs/source/images/logout-button.png b/docs/source/images/logout-button.png deleted file mode 100755 index 81fab0beff..0000000000 Binary files a/docs/source/images/logout-button.png and /dev/null differ diff --git a/docs/source/images/new-python-run.png b/docs/source/images/new-python-run.png deleted file mode 100755 index 580b462c8f..0000000000 Binary files a/docs/source/images/new-python-run.png and /dev/null differ diff --git a/docs/source/images/pick-python.png b/docs/source/images/pick-python.png deleted file mode 100755 index 2d3e8eafaf..0000000000 Binary files a/docs/source/images/pick-python.png and /dev/null differ diff --git a/docs/source/images/pin-to-dashboard.png b/docs/source/images/pin-to-dashboard.png deleted file mode 100755 index b4041b15df..0000000000 Binary files a/docs/source/images/pin-to-dashboard.png and /dev/null differ diff --git a/docs/source/images/platform-run-config.png b/docs/source/images/platform-run-config.png deleted file mode 100755 index 083e157d83..0000000000 Binary files a/docs/source/images/platform-run-config.png and /dev/null differ diff --git a/docs/source/images/pydev-python.png b/docs/source/images/pydev-python.png deleted file mode 100755 index e20f45bdb2..0000000000 Binary files a/docs/source/images/pydev-python.png and /dev/null differ diff --git a/docs/source/images/register-new-platform-authorization.png b/docs/source/images/register-new-platform-authorization.png deleted file mode 100755 index 5a03488d0b..0000000000 Binary files a/docs/source/images/register-new-platform-authorization.png and /dev/null differ diff --git a/docs/source/images/run-results.png b/docs/source/images/run-results.png deleted file mode 100755 index 5568a59585..0000000000 Binary files a/docs/source/images/run-results.png and /dev/null differ diff --git a/docs/source/images/select-path.png b/docs/source/images/select-path.png deleted file mode 100755 index 7f690f5696..0000000000 Binary files a/docs/source/images/select-path.png and /dev/null differ diff --git a/docs/source/images/select-repo.png b/docs/source/images/select-repo.png deleted file mode 100755 index d74fd20b45..0000000000 Binary files a/docs/source/images/select-repo.png and /dev/null differ diff --git a/docs/source/images/set-as-pydev.png b/docs/source/images/set-as-pydev.png deleted file mode 100755 index 681eeefd60..0000000000 Binary files a/docs/source/images/set-as-pydev.png and /dev/null differ diff --git a/docs/source/images/setup-python.png b/docs/source/images/setup-python.png deleted file mode 100755 index 6e90e4537c..0000000000 Binary files a/docs/source/images/setup-python.png and /dev/null differ diff --git a/docs/source/images/start-agent.png b/docs/source/images/start-agent.png deleted file mode 100755 index 57b3b0006a..0000000000 Binary files a/docs/source/images/start-agent.png and /dev/null differ diff --git a/docs/source/images/volttron-console.png b/docs/source/images/volttron-console.png deleted file mode 100755 index 1a63411b9a..0000000000 Binary files a/docs/source/images/volttron-console.png and /dev/null differ diff --git a/docs/source/images/volttron-main.png b/docs/source/images/volttron-main.png deleted file mode 100755 index 2443671f96..0000000000 Binary files a/docs/source/images/volttron-main.png and /dev/null differ diff --git a/docs/source/images/volttron-pick-main.png b/docs/source/images/volttron-pick-main.png deleted file mode 100755 index 13168ba066..0000000000 Binary files a/docs/source/images/volttron-pick-main.png and /dev/null differ diff --git a/docs/source/index.rst b/docs/source/index.rst index bad977bc7e..e2a64841d1 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,61 +1,130 @@ -.. VOLTTRON documentation master file, created by - sphinx-quickstart on Thu Feb 4 21:15:08 2016. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. +.. VOLTTRON documentation master file ========================== |VOLTTRON| documentation! ========================== - - |VOLTTRON Tagline| -|VOLTTRON| is an open-source platform for distributed sensing and control. The platform provides services for collecting and storing data from buildings and devices and provides an environment for developing applications -that interact with that data. +|VOLTTRON| is an open-source platform for distributed sensing and control. The platform is an open source tool for +performing simulations, improving building system performance, and creating a more flexible and reliable power grid. Features --------- +======== + +- a secure :ref:`message bus ` allowing connectivity between modules on individual platforms and + between platform instances in large scale deployments +- a flexible :ref:`agent framework ` allowing users to adapt the platform to their unique use-cases +- a configurable :ref:`driver framework ` for collecting data from and sending control + signals to buildings and devices +- automatic data capture and retrieval through our :ref:`historian framework ` +- an extensible :ref:`web framework ` allowing users and services to securely connect to the platform + from anywhere + +VOLTTRON™ is open source and publicly available from `GitHub `_. The project +is supported by the U.S. Department of Energy and receives ongoing updates from a team of core developers at PNNL. The +VOLTTRON team encourages and appreciates community involvement including issues and pull requests on Github, meetings +at our bi-weekly office-hours and on Slack. To be invited to office-hours or slack, please `send the team an email +`_. + + +.. toctree:: + :caption: Introduction + :hidden: + :titlesonly: + :maxdepth: 1 + + introduction/what-is-volttron + introduction/how-does-it-work + introduction/platform-install + introduction/definitions + introduction/license + -Out of the box VOLTTRON provides: +.. toctree:: + :caption: Developing in VOLTTRON + :hidden: + :titlesonly: + :maxdepth: 1 + + developing-volttron/contributing/community + developing-volttron/development-environment/index + developing-volttron/developing-agents/agent-development-walk-through + developing-volttron/developing-drivers/driver-development-walk-through + developing-volttron/jupyter/jupyter-notebooks + developing-volttron/python-for-matlab-users -- a secure :ref:`message bus ` allowing agents to subscribe to data sources and publish results and messages. -- secure connectivity between multiple VOLTTRON instances. -- BACnet, ModBus and other device/system protocol connectivity through our :ref:`driver framework ` for collecting data from and sending control actions to buildings and devices. -- automatic data capture and retrieval through our :ref:`historian framework `. -- platform based :ref:`agent lifecycle management `. -- a :ref:`web based management ` tool for managing several instances from a central instance. -- the ability to easily extend the functionality of existing agents or create new ones for your specific purposes. +.. toctree:: + :caption: Deploying VOLTTRON + :hidden: + :titlesonly: + :maxdepth: 1 -Background ----------- + deploying-volttron/deployment-walk-through + deploying-volttron/single-machine-walk-through + deploying-volttron/multi-platform-walk-through + deploying-volttron/volttron-central + deploying-volttron/platform-hardening -|VOLTTRON| is written in Python 3.6 and runs on Linux Operating Systems. For users unfamiliar with those technologies, the following resources are recommended: -- https://docs.python.org/3.6/tutorial/ -- http://ryanstutorials.net/linuxtutorial/ +.. toctree:: + :caption: Agent Framework + :hidden: + :titlesonly: + :maxdepth: 1 + + agent-framework/agents-overview + agent-framework/core-service-agents/index + agent-framework/historian-agents/historian-framework + agent-framework/operations-agents/index + agent-framework/web-framework + agent-framework/platform-service-standardization + agent-framework/third-party-agents -License -------- -The project is :ref:`licensed ` under Apache 2 license. +.. toctree:: + :caption: Driver Framework + :hidden: + :titlesonly: + :maxdepth: 1 + + driver-framework/drivers-overview + driver-framework/fake-driver/fake-driver + driver-framework/actuator/actuator-agent + driver-framework/bacnet/bacnet-driver + driver-framework/chargepoint/chargepoint-driver + driver-framework/dnp3-driver/dnp3-driver + driver-framework/ecobee/ecobee-web-driver + driver-framework/ieee-2030_5/ieee-2030-driver + driver-framework/modbus/modbus-driver + driver-framework/modbus/modbus-tk-driver + driver-framework/obix/obix + driver-framework/ted-driver/the-energy-detective-driver + + +.. toctree:: + :caption: Platform Features + :hidden: + :titlesonly: + :maxdepth: 1 + platform-features/message-bus/index + platform-features/control/index + platform-features/config-store/configuration-store + platform-features/security/volttron-security -Contents: .. toctree:: - :maxdepth: 2 + :caption: VOLTTRON Topics + :hidden: + :titlesonly: + :maxdepth: 1 - overview/index - community_resources/index - setup/index - devguides/index - core_services/index - specifications/index - volttron_applications/index - VOLTTRON Platform API + volttron-topics/troubleshooting/index + volttron-topics/volttron-applications/index + volttron-topics/change-log/index Indices and tables @@ -63,9 +132,8 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` -* :ref:`search` .. |VOLTTRON Logo| image:: images/volttron-webimage.jpg .. |VOLTTRON| unicode:: VOLTTRON U+2122 -.. |VOLTTRON Tagline| image:: images/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png +.. |VOLTTRON Tagline| image:: files/VOLLTRON_Logo_Black_Horizontal_with_Tagline.png diff --git a/docs/source/introduction/definitions.rst b/docs/source/introduction/definitions.rst new file mode 100644 index 0000000000..c298f8dc9c --- /dev/null +++ b/docs/source/introduction/definitions.rst @@ -0,0 +1,109 @@ +.. _Definitions: + +=================== +Definition of Terms +=================== + +This page lays out a common terminology for discussing the components and underlying technologies used by the platform. +The first section discusses capabilities and industry standards that VOLTTRON conforms to while the latter is specific +to the VOLTTRON domain. + + +Industry Terms +============== + +- **BACNet**: Building Automation and Control network, that leverages ASHRAE, ANSI, and IOS 16484-5 standard protocols +- **JSON-RPC**: JSON-encoded Remote Procedure Call +- **JSON**: JavaScript object notation is a text-based, human-readable, open data interchange format, similar to XML, + but less verbose +- **Modbus**: Communications protocol for talking with industrial electronic devices +- **Publish/subscribe**: A message delivery pattern where senders (publishers) and receivers (subscribers) do not + communicate directly nor necessarily have knowledge of each other, but instead exchange messages through an + intermediary based on a mutual class or topic +- **RabbitMQ**: +- **SSH**: Secure shell is a network protocol providing encryption and authentication of data using public-key + cryptography +- **SSL**: Secure sockets layer is a technology for encryption and authentication of network traffic based on a chain + of trust +- **TLS**: Transport layer security is the successor to SSL +- **ZeroMQ or ØMQ**: A library used for inter-process and inter-computer communication + + +VOLTTRON Terms +============== + + +.. _Activated-Environment: + +Activated Environment +--------------------- + + An activated environment is the environment a VOLTTRON instance is run in. The bootstrap process creates the + environment from the shell and to activate it the following command is executed. + + .. code-block:: bash + + user@computer> source env/bin/activate + + # Note once the above command has been run the prompt will have changed + (volttron)user@computer> + + +.. _AIP: + +AIP +--- + + Agent Instantiation and Packaging - this is the module responsible for creating agent wheels, the agent execution + environment and running agents. Found in the VOLTTRON repository in the `volttron/platform` directory. + + +.. _Bootstrap-Environment: + +Bootstrap Environment +--------------------- + + The process by which an operating environment (activated environment) is produced. From the + :ref:`VOLTTRON_ROOT` directory executing `python bootstrap.py` will start the bootstrap process. + + +.. _VOLTTRON_HOME: + +VOLTTRON_HOME +------------- + + The location for a specific :ref:`VOLTTRON_INSTANCE` to store its specific information. There can be many + VOLTTRON_HOMEs on a single computing resource(VM, machine, etc.), and each `VOLTTRON_HOME` will correspond to a + single instance of VOLTTRON. + + +.. _VOLTTRON_INSTANCE: + +VOLTTRON_INSTANCE +----------------- + A single volttron process executing instructions on a computing resource. For each VOLTTRON_INSTANCE there WILL + BE only one :ref:`VOLTTRON_HOME` associated with it. In order for a VOLTTRON_INSTANCE to be able to + participate outside its computing resource it must be bound to an external ip address. + + +.. _VOLTTRON_ROOT: + +VOLTTRON_ROOT +------------- + + The cloned directory from Github. When executing the command + + .. code-block:: bash + + git clone http://github.com/VOLTTRON/volttron + + the top level volttron folder is the VOLTTRON_ROOT + + +.. _VIP: + +VIP +--- + + VOLTTRON Interconnect Protocol is a secure routing protocol that facilitates communications between agents, + controllers, services and the supervisory :ref:`VOLTTRON_INSTANCE`. diff --git a/docs/source/overview/files/overview.png b/docs/source/introduction/files/overview.png similarity index 100% rename from docs/source/overview/files/overview.png rename to docs/source/introduction/files/overview.png diff --git a/docs/source/introduction/how-does-it-work.rst b/docs/source/introduction/how-does-it-work.rst new file mode 100644 index 0000000000..1326b2983a --- /dev/null +++ b/docs/source/introduction/how-does-it-work.rst @@ -0,0 +1,53 @@ +.. _How-it-Works: + +================= +How Does it Work? +================= + +The VOLTTRON platform is built around the concept of software agents. Software agents perform autonomous functions on +behalf of a user. The VOLTTRON platform was created to allow a suite of agents installed by a user to work together to +achieve the user's goals. + + +Major Components +================ + +The platform comprises several components that allow agents to operate and connect to the platform. + +* The :ref:`Message Bus ` is central to the platform. All other VOLTTRON components communicate through it + using :ref:`VOLTTRON Interconnect Protocol` (VIP). VIP implements the publish/subscribe paradigm over a + variety of topics or directed communication using :ref:`Remote Procedure Calls `. + +* :ref:`Agents ` on the platform extend the base agent which provides a VIP connection to the message + bus and an agent lifecycle. Agents subscribe to topics which allow it to read. The agent lifecycle is controlled + by the :ref:`Agent Instantiation and Packaging ` (AIP) component which launches + agents in an agent execution environment. + +* The :ref:`Master Driver Agent ` can be configured with a number of driver configurations and will spawn + corresponding driver instances. Each driver instance provides functions for collecting device data and setting values + on the device. These functions implement device protocol or remote communication endpoint interfaces. Driver data + is published to the message bus or if requested by an agent will be delivered in an RPC response. + +* Agents can control devices by interacting with the :ref:`Actuator Agent ` to schedule and send + commands. + +* The :ref:`Historian ` framework subscribes to data published on the messages bus and stores it to + a database or file, or sends it to another location. + + +Usability Components +==================== + +Usability components exist to enhance the base capabilities of the platform for deployments. + +* :ref:`VOLTTRON Control ` is the command line interface to controlling a platform instance. VOLTTRON + Control can be used to operate agents, configure drivers, get status and health details, etc. + +* Data collection, command and control can be achieved in large deployments by + :ref:`connecting multiple platform instances `. + +* :ref:`VOLTTRON Central ` is an agent which can be installed on a platform to provide a single + management interface to multiple VOLTTRON platform instances. + +* JSON, static and websocket endpoints can be registered to agents via the :ref:`Web Framework ` + and platform web server. This allows remote agent communication as well as for agents to serve web pages. diff --git a/docs/source/overview/license.rst b/docs/source/introduction/license.rst similarity index 98% rename from docs/source/overview/license.rst rename to docs/source/introduction/license.rst index cbeb07bb95..ccbc5801e1 100644 --- a/docs/source/overview/license.rst +++ b/docs/source/introduction/license.rst @@ -1,4 +1,4 @@ -.. _license: +.. _License: ======= License @@ -14,9 +14,9 @@ The patent license grant shall only be applicable to the following patent and pa Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -======= + Terms -======= +===== This material was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor the United States Department of Energy, nor Battelle, nor any of their employees, nor any jurisdiction or organization that has cooperated in the development of these materials, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness or any information, apparatus, product, software, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or Battelle Memorial Institute. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. diff --git a/docs/source/introduction/platform-install.rst b/docs/source/introduction/platform-install.rst new file mode 100644 index 0000000000..97613f4dfd --- /dev/null +++ b/docs/source/introduction/platform-install.rst @@ -0,0 +1,493 @@ +.. _Platform-Installation: + +.. role:: bash(code) + :language: bash + +======================= +Installing the Platform +======================= + +VOLTTRON is written in Python 3.6+ and runs on Linux Operating Systems. For users unfamiliar with those technologies, +the following resources are recommended: + +- `Python 3.6 Tutorial `_ +- `Linux Tutorial `_ + +This guide will specify commands to use to successfully install the platform on supported Linux distributions, but a +working knowledge of Linux will be helpful for troubleshooting and may improve your ability to get more out of your +deployment. + +.. note:: + + Volttron version 7.0rc1 is currently tested for Ubuntu versions 18.04 and 18.10 as well as Linux Mint version 19.3. + Version 6.x is tested for Ubuntu versions 16.04 and 18.04 as well as Linux Mint version 19.1. + + +.. _Platform-Prerequisites: + +Step 1 - Install prerequisites +============================== + +The following packages will need to be installed on the system: + +* git +* build-essential +* python3.6-dev +* python3.6-venv +* openssl +* libssl-dev +* libevent-dev + +On **Debian-based systems**, these can all be installed with the following command: + +.. code-block:: bash + + sudo apt-get update + sudo apt-get install build-essential python3-dev python3-venv openssl libssl-dev libevent-dev git + +On Ubuntu-based systems, available packages allow you to specify the Python3 version, 3.6 or greater is required +(Debian itself does not provide those packages). + +.. code-block:: bash + + sudo apt-get install build-essential python3.6-dev python3.6-venv openssl libssl-dev libevent-dev git + + +On arm-based systems (including, but not limited to, Raspbian), you must also install libffi-dev, you can do this with: + +.. code-block:: bash + + sudo apt-get install libffi-dev + +On **Redhat or CENTOS systems**, these can all be installed with the following +command: + +.. code-block:: bash + + sudo yum update + sudo yum install make automake gcc gcc-c++ kernel-devel python3-devel openssl openssl-devel libevent-devel git + +.. warning:: + Python 3.6 or greater is required, please ensure you have installed a supported version with :bash:`python3 --version` + +If you have an agent which requires the pyodbc package, install the following additional requirements: + +* freetds-bin +* unixodbc-dev + +On **Debian-based systems** these can be installed with the following command: + +.. code-block:: bash + + sudo apt-get install freetds-bin unixodbc-dev + +On **Redhat or CentOS systems**, these can be installed from the Extra Packages for Enterprise Linux (EPEL) repository: + +.. code-block:: bash + + sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm + sudo yum install freetds unixODBC-devel + +.. note:: + The above command to install the EPEL repository is for Centos/Redhat 8. Change the number to match your OS version. + EPEL packages are included in Fedora repositories, so installing EPEL is not required on Fedora. + +It may be possible to deploy VOLTTRON on a system not listed above but may involve some troubleshooting and dependency +management on the part of the user. + + +Step 2 - Clone VOLTTRON code +============================ + + +.. _Repository-Structure: + +Repository Structure +-------------------- + +There are several options for using the VOLTTRON code depending on whether you require the most stable version of the +code or want the latest updates as they happen. In order of decreasing stability and increasing currency: + +* `Master` - Most stable release branch, current major release is 7.0. This branch is default. +* `develop` - contains the latest `finished` features as they are developed. When all features are stable, this branch + will be merged into `Master`. + + .. note:: + + This branch can be cloned by those wanting to work from the latest version of the platform but should not be + used in deployments. + +* Features are developed on “feature” branches or developers' forks of the main repository. It is not recommended to + clone these branches except for exploring a new feature. + +.. note:: + + VOLTTRON versions 6.0 and newer support two message buses - ZMQ and RabbitMQ. + +.. code-block:: bash + + git clone https://github.com/VOLTTRON/volttron --branch + + +Step 3 - Setup virtual environment +================================== + +The :ref:`bootstrap.py ` script in the VOLTTRON root directory will create a +`virtual environment `_ and install the package's Python dependencies. +Options exist for upgrading or rebuilding existing environments, and for adding additional dependencies for optional +drivers and agents included in the repository. + +.. note:: + + The :bash:`--help` option for `bootstrap.py` can specified to display all available optional parameters. + + +.. _ZeroMQ-Install: + +Steps for ZeroMQ +---------------- + +Run the following command to install all required packages: + +.. code-block:: bash + + cd + python3 bootstrap.py + +Then activate the Python virtual environment: + +.. code-block:: bash + + source env/bin/activate + +Proceed to step 4. + +.. note:: + + You can deactivate the environment at any time by running `deactivate`. + + +.. _RabbitMQ-Install: + +Steps for RabbitMQ +------------------ + + +Step 1 - Install Erlang packages +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For RabbitMQ based VOLTTRON, some of the RabbitMQ specific software packages have to be installed. + + +On Debian based systems and CentOS 6/7 +"""""""""""""""""""""""""""""""""""""" + +If you are running a Debian or CentOS system, you can install the RabbitMQ dependencies by running the +"rabbit_dependencies.sh" script, passing in the OS name and appropriate distribution as parameters. The +following are supported: + +* `debian bionic` (for Ubuntu 18.04) + +* `debian xenial` (for Ubuntu 16.04 or Linux Mint 18.04) + +* `debian stretch` (for Debian Stretch) + +* `debian buster` (for Debian Buster) + +* `raspbian buster` (for Raspbian/Raspberry Pi OS Buster) + +Example command: + +.. code-block:: bash + + ./scripts/rabbit_dependencies.sh debian xenial + + +Alternatively +""""""""""""" + +You can download and install Erlang from [Erlang Solutions](https://www.erlang-solutions.com/resources/download.html). +Please include OTP/components - ssl, public_key, asn1, and crypto. +Also lock your version of Erlang using the [yum-plugin-versionlock](https://access.redhat.com/solutions/98873) + +.. note:: + Currently VOLTTRON only officially supports specific versions of Erlang for each operating system: + * 1:22.1.8.1-1 for Debian + * 1:21.2.6+dfsg-1 for Raspbian + * Specific Erlang 21.x versions correspond to CentOS versions 6, 7, and 8, these can be found + `here `_ + + +Step 2 - Configure hostname +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Make sure that your hostname is correctly configured in /etc/hosts. +See (). +If you are testing with VMs make please make sure to provide unique host names for each of the VMs you are using. + +The hostname should be resolvable to a valid IP when running on bridged mode. RabbitMQ checks for this during initial +boot. Without this (for example, when running on a VM in NAT mode) RabbitMQ start-up would fail with the error "unable +to connect to empd (port 4369) on ." + +.. note:: + + RabbitMQ startup error would show up in the VM's syslog (/var/log/messages) file and not in RabbitMQ logs + (/var/log/rabbitmq/rabbitmq@hostname.log) + + +Step 3 - Bootstrap the environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + cd volttron + python3 bootstrap.py --rabbitmq [optional install directory. defaults to /rabbitmq_server] + +This will build the platform and create a virtual Python environment and dependencies for RabbitMQ. It also installs +RabbitMQ server as the current user. If an install path is provided, that path should exist and the user should have +write permissions. RabbitMQ will be installed under `/rabbitmq_server-3.7.7`. The rest of the +documentation refers to the directory `/rabbitmq_server-3.7.7` as `$RABBITMQ_HOME`. + +.. note:: + + There are many additional :ref:`options for bootstrap.py ` for including dependencies, altering + the environment, etc. + +You can check if the RabbitMQ server is installed by checking its status: + +.. code-block:: bash + + service rabbitmq status + +.. note:: + + The `RABBITMQ_HOME` environment variable can be set in ~/.bashrc. If doing so, it needs to be set to the RabbitMQ + installation directory (default path is `/rabbitmq_server/rabbitmq_server-3.7.7`) + +.. code-block:: bash + + echo 'export RABBITMQ_HOME=$HOME/rabbitmq_server/rabbitmq_server-3.7.7'|sudo tee --append ~/.bashrc + source ~/.bashrc + $RABBITMQ_HOME/sbin/rabbitmqctl status + + +Step 4 - Activate the environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + source env/bin/activate + +.. note:: + + You can deactivate the environment at any time by running :bash:`deactivate`. + + +Step 5 - Configure RabbitMQ setup for VOLTTRON +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + vcfg --rabbitmq single [optional path to rabbitmq_config.yml] + +Refer to [examples/configurations/rabbitmq/rabbitmq_config.yml](examples/configurations/rabbitmq/rabbitmq_config.yml) +for a sample configuration file. At a minimum you will need to provide the host name and a unique common-name +(under certificate-data) in the configuration file. + +.. note:: + + common-name must be unique and the general convention is to use `-root-ca`. + +Running the above command without the optional configuration file parameter will cause the user user to be prompted for +all the required data in the command prompt. "vcfg" will use that data to generate a rabbitmq_config.yml file in the +`VOLTTRON_HOME` directory. + +.. note:: + + If the above configuration file is being used as a basis for creating your own configuration file, be sure to update + it with the hostname of the deployment (this should be the fully qualified domain name of the system). + +This script creates a new virtual host and creates SSL certificates needed for this VOLTTRON instance. These +certificates get created under the subdirectory "certificates" in your VOLTTRON home (typically in ~/.volttron). It +then creates the main VIP exchange named "volttron" to route message between the platform and agents and alternate +exchange to capture unrouteable messages. + +.. note:: + + We configure the RabbitMQ instance for a single volttron_home and volttron_instance. This script will confirm with + the user the volttron_home to be configured. The VOLTTRON instance name will be read from volttron_home/config + if available, if not the user will be prompted for VOLTTRON instance name. To run the scripts without any prompts, + save the the VOLTTRON instance name in volttron_home/config file and pass the VOLTTRON home directory as a command + line argument. For example: `vcfg --vhome /home/vdev/.new_vhome --rabbitmq single` + +The Following are the example inputs for `vcfg --rabbitmq single` command. Since no config file is passed the script +prompts for necessary details. + +.. code-block:: console + + Your VOLTTRON_HOME currently set to: /home/vdev/new_vhome2 + + Is this the volttron you are attempting to setup? [Y]: + Creating rmq config yml + RabbitMQ server home: [/home/vdev/rabbitmq_server/rabbitmq_server-3.7.7]: + Fully qualified domain name of the system: [cs_cbox.pnl.gov]: + + Enable SSL Authentication: [Y]: + + Please enter the following details for root CA certificates + Country: [US]: + State: Washington + Location: Richland + Organization: PNNL + Organization Unit: Volttron-Team + Common Name: [volttron1-root-ca]: + Do you want to use default values for RabbitMQ home, ports, and virtual host: [Y]: N + Name of the virtual host under which RabbitMQ VOLTTRON will be running: [volttron]: + AMQP port for RabbitMQ: [5672]: + http port for the RabbitMQ management plugin: [15672]: + AMQPS (SSL) port RabbitMQ address: [5671]: + https port for the RabbitMQ management plugin: [15671]: + INFO:rmq_setup.pyc:Starting rabbitmq server + Warning: PID file not written; -detached was passed. + INFO:rmq_setup.pyc:**Started rmq server at /home/vdev/rabbitmq_server/rabbitmq_server-3.7.7 + INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost + INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost + INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost + INFO:rmq_setup.pyc: + Checking for CA certificate + + INFO:rmq_setup.pyc: + Root CA (/home/vdev/new_vhome2/certificates/certs/volttron1-root-ca.crt) NOT Found. Creating root ca for volttron instance + Created CA cert + INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost + INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost + INFO:rmq_setup.pyc:**Stopped rmq server + Warning: PID file not written; -detached was passed. + INFO:rmq_setup.pyc:**Started rmq server at /home/vdev/rabbitmq_server/rabbitmq_server-3.7.7 + INFO:rmq_setup.pyc: + + ####################### + + Setup complete for volttron home /home/vdev/new_vhome2 with instance name=volttron1 + Notes: + + - Please set environment variable `VOLTTRON_HOME` to `/home/vdev/new_vhome2` before starting volttron + + - On production environments, restrict write access to + /home/vdev/new_vhome2/certificates/certs/volttron1-root-ca.crt to only admin user. For example: sudo chown root /home/vdev/new_vhome2/certificates/certs/volttron1-root-ca.crt + + - A new admin user was created with user name: volttron1-admin and password=default_passwd. + You could change this user's password by logging into Please update /home/vdev/new_vhome2/rabbitmq_config.yml if you change password + + ####################### + + +Test the VOLTTRON Deployment +============================ + +We are now ready to start VOLTTRON instance. If configured with RabbitMQ message bus a config file would have been +generated in `$VOLTTRON_HOME/config` with the entry ``message-bus=rmq``. If you need to revert back to ZeroMQ based +VOLTTRON, you will have to either remove the ``message-bus`` parameter or set it to the default "zmq" in +`$VOLTTRON_HOME/config`. + +The following command starts volttron process in the background: + +.. code-block:: bash + + volttron -vv -l volttron.log& + +This enters the virtual Python environment and then starts the platform in debug (vv) mode with a log file +named volttron.log. Alternatively you can use the utility script start-volttron script that does the same. + +.. code-block:: bash + + ./start-volttron + +To stop the platform, use the `vct` command: + +.. code-block:: bash + + volttron-ctl shutdown --platform + +or use the included `stop-volttron` script: + +.. code-block:: bash + + ./stop-volttron + + +.. warning:: + If you plan on running VOLTTRON in the background and detaching it from the + terminal with the ``disown`` command be sure to redirect stderr and stdout to ``/dev/null``. + Some libraries which VOLTTRON relies on output directly to stdout and stderr. + This will cause problems if those file descriptors are not redirected to ``/dev/null`` + + :: + + #To start the platform in the background and redirect stderr and stdout + #to /dev/null + volttron -vv -l volttron.log > /dev/null 2>&1& + + + +Installing and Running Agents +----------------------------- + +VOLTTRON platform comes with several built in services and example agents out of the box. To install a agent +use the script `install-agent.py` + +.. code-block:: bash + + python scripts/install-agent.py -s [-c ] + + +For example, we can use the command to install and start the Listener Agent - a simple agent that periodically publishes +heartbeat message and listens to everything on the message bus. Install and start the Listener agent using the +following command: + +.. code-block:: bash + + python scripts/install-agent.py -s examples/ListenerAgent --start + + +Check volttron.log to ensure that the listener agent is publishing heartbeat messages. + +.. code-block:: bash + + tail volttron.log + +.. code-block:: console + + 2016-10-17 18:17:52,245 (listeneragent-3.2 11367) listener.agent INFO: Peer: 'pubsub', Sender: 'listeneragent-3.2_1':, Bus: u'', Topic: 'heartbeat/listeneragent-3.2_1', Headers: {'Date': '2016-10-18T01:17:52.239724+00:00', 'max_compatible_version': u'', 'min_compatible_version': '3.0'}, Message: {'status': 'GOOD', 'last_updated': '2016-10-18T01:17:47.232972+00:00', 'context': 'hello'} + + +You can also use the `volttron-ctl` (or `vctl`) command to start, stop or check the status of an agent + +.. code-block:: console + + (volttron)volttron@volttron1:~/git/rmq_volttron$ vctl status + AGENT IDENTITY TAG STATUS HEALTH + 6 listeneragent-3.2 listeneragent-3.2_1 running [13125] GOOD + f master_driveragent-3.2 platform.driver master_driver + +.. code-block:: bash + + vctl stop + + +.. note:: + + The default working directory is ~/.volttron. The default directory for creation of agent packages is + `~/.volttron/packaged` + + +Next Steps +========== + +There are several walk-throughs and detailed explanations of platform features to explore additional aspects of the +platform: + +* :ref:`Agent Framework ` +* :ref:`Driver Framework ` +* Demonstration of the :ref:`management UI ` +* :ref:`RabbitMQ setup ` with Federation and Shovel plugins diff --git a/docs/source/introduction/what-is-volttron.rst b/docs/source/introduction/what-is-volttron.rst new file mode 100644 index 0000000000..588fa323e4 --- /dev/null +++ b/docs/source/introduction/what-is-volttron.rst @@ -0,0 +1,47 @@ +.. _What-is-Volttron: + +================= +What is VOLTTRON? +================= + +VOLTTRON™ is a software platform on which software modules called "agents" and device driver modules to connect to a +message bus to interact. Users may configure included drivers for industry standard device communication protocols such +as BACnet or Modbus, or develop and configure their own. Additionally, agents can be installed or developed to perform +a vast variety of tasks. + + +Design Philosophy +================= + +VOLTTRON was designed by Pacific Northwest National Laboratory to service building efficiency, building-grid integration +and transactive controls systems. These systems are working to improve energy efficiency and resiliency in critical +infrastructure. To this end, VOLTTRON was built with the following pillars in mind: + + * Cost-Effectiveness - Open source software (free to users) and can be hosted on inexpensive computing resources + * Scalability - Can be used in one building or a fleet of buildings + * Interoperability - Enables interaction/connection with various systems and subsystems, in and out of the energy + sector + * Security - Underpinned with a robust security foundation to combat today’s cyber vulnerabilities and attacks + + +Basic Components +================ + +* :ref:`Message bus ` - The VOLTTRON message bus uses + `message queueing software `_ to exchange messages + between agents and drivers installed on the platform. VOLTTRON messages are exchanged using a + :ref:`publish/suscriber paradigm `, or messages can be routed to specific agents through the bus using + :ref:`remote procedure calls `. + +* :ref:`Agents ` - Agents are software modules which autonomously perform a set of desired functions on + behalf of a user. VOLTTRON agents are often use to collect data, send control signals to devices, implement control + algorithms or perform simulations. + +* :ref:`Drivers ` - Drivers can be installed on the platform and configured to communicate with + industrial or Internet of Things devices. Drivers provide a set of pre-defined functions which can be mapped to + device communication methods to read or set values on the device. + +* :ref:`Historians ` - Historians are special purpose agents which are used to subscribe to sources broadcasting on + the message bus and store their messages for later use. + +* :ref:`Web Framework ` - The VOLTTRON web framework diff --git a/docs/source/overview/DefinitionOfTerms.rst b/docs/source/overview/DefinitionOfTerms.rst deleted file mode 100644 index 44298a6b50..0000000000 --- a/docs/source/overview/DefinitionOfTerms.rst +++ /dev/null @@ -1,83 +0,0 @@ -.. _definitions: - -=================== -Definition of Terms -=================== - -This page lays out a common terminology for discussing the components and -underlying technologies used by the platform. The first -section discusses capabilities and industry standards that volttron -conforms to while the latter is specific to the VOLTTRON domain. - -Industry Terms -~~~~~~~~~~~~~~ - -- **BACNet**: Building Automation and Control network, that leverages ASHRAE, ANSI, and IOS 16484-5 standard protocols. -- **JSON-RPC**: JSON-encoded remote procedure call -- **JSON**: JavaScript object notation is a text-based, human-readable, open data interchange format, similar to XML, but less verbose -- **Publish/subscribe**: A message delivery pattern where senders (publishers) and receivers (subscribers) do not communicate directly nor necessarily have knowledge of each other, but instead exchange messages through an intermediary based on a mutual class or topic -- **ZeroMQ or ØMQ**: A library used for inter-process and inter-computer communication -- **Modbus**: Communications protocol for talking with industrial electronic devices -- **SSH**: Secure shell is a network protocol providing encryption and authentication of data using public-key cryptography -- **SSL**: Secure sockets layer is a technology for encryption and authentication of network traffic based on a chain of trust -- **TLS**: Transport layer security is the successor to SSL - - -VOLTTRON Terms -~~~~~~~~~~~~~~ - - .. _activated-environment: - - Activated Environment - An activated environment is the environment a VOLTTRON instance is run in. - The bootstrap process creates the environment from the shell and to activate - it the following command is executed. - - .. code-block:: bash - - user@computer> source env/bin/activate - - # Note once the above command has been run the prompt will have changed - (volttron)user@computer> - - .. _bootstrap-environment: - - Bootstrap Environment - The process by which an operating environment (activated environment) - is produced. From the :ref:`VOLTTRON_ROOT` directory executing - ``python bootstrap.py`` will start the bootstrap process. - - .. _VOLTTRON_HOME: - - VOLTTRON_HOME - The location for a specific :ref:`VOLTTRON_INSTANCE` to store its specific - information. There can be many VOLTTRON_HOMEs on a single computing - resource(VM, machine, etc.) - - .. _VOLTTRON_INSTANCE: - - VOLTTRON_INSTANCE - A single volttron process executing instructions on a computing resource. - For each VOLTTRON_INSTANCE there WILL BE only one :ref:`VOLTTRON_HOME` - associated with it. In order for a VOLTTRON_INSTANCE to be able to - participate outside its computing resource it must be bound to an - external ip address. - - .. _VOLTTRON_ROOT: - - VOLTTRON_ROOT - The cloned directory from github. When executing the command - - .. code-block:: bash - - git clone http://github.com/VOLTTRON/volttron - - the top volttron folder is the VOLTTRON_ROOT - - .. _VIP: - - VIP - VOLTTRON Interconnect Protocol is a secure routing protocol that facilitates - communications between agents, controllers, services and the supervisory - :ref:`VOLTTRON_INSTANCE`. - diff --git a/docs/source/overview/agents-overview.rst b/docs/source/overview/agents-overview.rst deleted file mode 100644 index 7bca29594f..0000000000 --- a/docs/source/overview/agents-overview.rst +++ /dev/null @@ -1,20 +0,0 @@ -.. _agents-overview: - -====================== -Agents in the Platform -====================== - -Agents deployed on VOLTTRON can perform one or more roles which can be broadly classified into the following groups: - -- Platform Agents: Agents which are part of the platform and provide a service to other agents. Examples are agents which interface with devices to publish readings and handle control signals from other agents. -- Cloud Agents: These agents represent a remote application which needs access to the messages and data on the platform. This agent would subscribe to topics of interest to the remote application and would also allow it publish data to the platform. -- Control Agents: These agents control the devices of interest and interact with other resources to achieve some goal. - -Platform Services: - -- Message Bus: All agents and services publish and subscribe to topics on the message bus. This provides a single interface that abstracts the details of devices and agents from each other. Components in the platform basically produce and consume events. -- Weather Information: This agent periodically retrieves data from the Weather Underground site. It then reformats it and publishes it out to the platform on a weather topic. -- Modbus-based device interface: The Modbus driver publishes device data onto the message bus. It also handles the locking of devices to prevent multiple conflicting directives. -- Application Scheduling: This service allows the scheduling of agents’ access to devices in order to prevent conflicts. -- Logging service: Agents can publish arbitrary strings to a logging topic and this service will push them to a historian for later analysis. - diff --git a/docs/source/overview/components.rst b/docs/source/overview/components.rst deleted file mode 100644 index 88db8e3743..0000000000 --- a/docs/source/overview/components.rst +++ /dev/null @@ -1,25 +0,0 @@ -.. _components: - -========== -Components -========== - -An overview of the VOLTTRON platform components is illustrated in the figure below. The platform -comprises several components -and agents that provide services to other agents. Of these components, the Information Exchange Bus (IEB), -or :ref:`Message Bus ` is central to the -platform. All other VOLTTRON components communicate through it using the publish/subscribe paradigm over a variety of -topics. - -:ref:`Drivers ` communicate with devices allowing their data to be published on the IEB. -Agents can control devices by interacting with the :ref:`Actuator Agent ` to schedule and send commands. -The :ref:`Historian ` framework takes data published on the messages bus and stores it to a database, -file, or sends it to another location. - -The agent lifecycle is controlled by the Agent Instantiation and Packaging (AIP) component which launches agents in an -Agent Execution Environment. This isolates agents from the platform while allowing them to interact with the IEB. - - -|Overview of the VOLTTRON platform| - -.. |Overview of the VOLTTRON platform| image:: files/overview.png diff --git a/docs/source/overview/index.rst b/docs/source/overview/index.rst deleted file mode 100644 index 36a259fce3..0000000000 --- a/docs/source/overview/index.rst +++ /dev/null @@ -1,33 +0,0 @@ -.. _overview: - -======== -Overview -======== - -VOLTTRON™ is an open-source distributed control and sensing platform for integrating buildings -and the power grid. VOLTTRON connects devices, agents in the platform, agents in the Cloud, and -signals from the power grid. The platform also supports use cases such as demand response and -integration of distributed renewable energy sources. - -VOLTTRON provides an environment for agent execution and serves as a single point of contact for -interfacing with devices (rooftop units, building systems, meters, etc.), external resources, and platform -services such as data archival and retrieval. VOLTTRON applications are referred to as agents since -VOLTTRON provides an agent-based programming paradigm to ease application development and -minimize the lines of code that need to be written by domain experts such as buildings engineers. -VOLTTRON provides a collection of utility and helper classes that simplifies agent development. - -The VOLTTRON white paper provides an overview of the capabilities of the platform: -https://volttron.org/sites/default/files/publications/PNNL-25499_VOLTTRON_2016.pdf - - -.. toctree:: - :glob: - :maxdepth: 1 - - background - components - agents-overview - DefinitionOfTerms - version-history - license - diff --git a/docs/source/platform-features/config-store/agent-configuration-store.rst b/docs/source/platform-features/config-store/agent-configuration-store.rst new file mode 100644 index 0000000000..fca287ba98 --- /dev/null +++ b/docs/source/platform-features/config-store/agent-configuration-store.rst @@ -0,0 +1,373 @@ +.. _Agent-Configuration-Store: + +========================= +Agent Configuration Store +========================= + +This document describes the configuration store feature and explains how an agent uses it. + +The configuration store enables users to store agent configurations on the platform and allows the agent to +automatically retrieve them during runtime. Users may update the configurations and the agent will automatically be +informed of the changes. + + +Compatibility +============= + +Supporting the configuration store will *not* be required by Agents, however the usage will be strongly encouraged as it +should substantially improve user experience. + +The previous method for configuring an agent will still be available to agents (and in some cases required), however +agents can be created to only work with the configuration store and not support the old method at all. + +It will be possible to create an agent to use the traditional method for configuration to establish defaults if no +configuration exist in the platform configuration store. + + +Configuration Names and Paths +============================= + +Any valid OS file path name is a valid configuration name. Any leading or trailing "/", "\" and whitespace is removed +by the store. + +The canonical name for the main agent configuration is `config`. + +The configuration subsystem remembers the case of configuration names. Name matching is case insensitive both on the +Agent and platform side. Configuration names are reported to agent callbacks in the original case used when adding them +to the configuration. If a new configuration is store with a different case of an existing name the new name case is +used. + + +Configuration Ownership +======================= + +Each configuration belongs to one agent and one agent only. When an agent refers to a configuration file via it's path +it does not need to supply any information about its identity to the platform in the file path. The only configurations +an agent has direct access to are it's own. The platform will only inform the owning agent configuration changes. + + +Configuration File Types +======================== + +Configurations files come in three types: `json`, `csv`, and `raw`. The type of a configuration file is declared when +it is added to or changed in the store. + +The parser assumes the first row of every CSV file is a header. + +Invalid JSON or CSV files are rejected at the time they are added to the store. + +Raw files are unparsed and accepted as is. + +Other parsed types may be added in the future. + + +Configuration File Representation to Agents +=========================================== + +JSON +---- + +A JSON file is parsed and represented as appropriate data types to the requester. + +Consider a file with the following contents: + +.. code-block:: json + + { + "result": "PREEMPTED", + "info": null, + "data": { + "agentID": "my_agent", + "taskID": "my_task" + } + } + +The file will be parsed and presented as a dictionary with 3 values to the requester. + + +CSV +--- + +A CSV file is represented as a list of objects. Each object represents a row in the CSV file. + +For instance this (simplified) CSV file: + +.. csv-table:: Example CSV + :header: Volttron Point Name,Modbus Register,Writable,Point Address + + ReturnAirCO2,>f,FALSE,1001 + ReturnAirCO2Stpt,>f,TRUE,1011 + +will be represented like this: + +.. code-block:: json + + [ + { + "Volttron Point Name": "ReturnAirCO2", + "Modbus Register": ">f", + "Writable": "FALSE", + "Point Address": "1001" + }, + { + "Volttron Point Name": "ReturnAirCO2Stpt", + "Modbus Register": ">f", + "Writable": "TRUE", + "Point Address": "1011" + } + ] + + +Raw +--- + +Raw files are represented as a string containing the contents of the file. + + +File references +=============== + +The `Platform Configuration Store` supports referencing one configuration file from another. If a referenced file +exists the contents of that file will replace the file reference when the file is sent to the owning agent. Otherwise +the reference will be replaced with None. + +Only configurations that are parsed by the platform (currently "json" or "csv") will be examined for references. If the +file referenced is another parsed file type (JSON or CSV, currently) then the replacement will be the parsed contents of +the file. + +In a JSON object the name of a value will never be considered a reference. + +A file reference is any value string that starts with ``config://``. The rest of the string is the path in the config +store to that configuration. The config store path is converted to lower case for comparison purposes. + +Consider the following configuration files named `devices/vav1.config` and `registries/vav.csv`, respectively: + +.. code-block:: json + + { + "driver_config": {"device_address": "10.1.1.5", + "device_id": 500}, + + "driver_type": "bacnet", + "registry_config":"config://registries/vav.csv", + "campus": "pnnl", + "building": "isb1", + "unit": "vav1" + } + +.. csv-table:: vav.csv + :header: Volttron Point Name,Modbus Register,Writable,Point Address + + ReturnAirCO2,>f,FALSE,1001 + ReturnAirCO2Stpt,>f,TRUE,1011 + +The resulting configuration returns when an agent asks for `devices/vav1.config`. The Python object will have the +following configuration: + +.. code-block:: python + + { + "driver_config": {"device_address": "10.1.1.5", + "device_id": 500}, + + "driver_type": "bacnet", + "registry_config":[ + { + "Volttron Point Name": "ReturnAirCO2", + "Modbus Register": ">f", + "Writable": "FALSE", + "Point Address": "1001" + }, + { + "Volttron Point Name": "ReturnAirCO2Stpt", + "Modbus Register": ">f", + "Writable": "TRUE", + "Point Address": "1011" + } + ], + "campus": "pnnl", + "building": "isb1", + "unit": "vav1" + } + +Circular references are not allowed. Adding a file that creates a circular reference will cause that file to be +rejected by the platform. + +If a file is changed in anyway (`NEW`, `UPDATE`, or `DELETE`) and that file is referred to by another file then the +platform considers the referring configuration as changed. The configuration subsystem on the Agent will call every +callback listening to a file or any file referring to that file either directly or indirectly. + + +Agent Configuration Sub System +============================== + +The configuration store shall be implemented on the Agent(client) side in the form of a new subsystem called config. + +The subsystem caches configurations as the platform updates the state to the agent. Changes to the cache triggered by +an RPC call from the platform will trigger callbacks in the agent. + +No callback methods are called until the `onconfig` phase of agent startup. A new phase to agent startup called +`onconfig` will be added to the `Core `class. Originally it was planned to have this run after the `onstart` phase has +completed but that is currently not possible. Ideally if an agent is using the config store feature it will not need +any `onstart` methods. + +When the `onconfig` phase is triggered the subsystem will retrieve the current configuration state from the platform and +call all callbacks registered to a configuration in the store to the `NEW` action. No callbacks are called before this +point in agent startup. + +The first time callbacks are called at agent startup any callbacks subscribed to a configuration called `config` are +called first. + + +Configuration Subsystem Agent Methods +------------------------------------- + +These methods are part of the interface available to the Agent. + + **config.get( config_name="config")** - Get the contents of a configuration. + If no name is provided the contents of the main agent configuration "config" is returned. This may not be called + before `onstart` methods are called. If called during the `onstart` phase it will trigger the subsystem to + initialize early but will not trigger any callbacks. + + **config.subscribe(callback, action=("NEW", "UPDATE", "DELETE"), pattern="*")** - Sets up a callback for handling a + configuration change. The platform will automatically update the agent when a configuration changes ultimately + triggering all callbacks that match the pattern specified. The action argument describes the types of configuration + change action that will trigger the callback. Possible actions are `NEW`, `UPDATE`, and `DELETE` or a tuple of any + combination of actions. If no action is supplied the callback happens for all changes. A list of actions can be + supplied if desired. If no file name pattern is supplied then the callback is called for all configurations. The + pattern is an regex used match the configuration name. + + The callback will also be called if any file referenced by a configuration file is changed. + + The signature of the callback method is ``callback(config_name, action, contents)`` where `file_name` is the file + that triggered the callback, action is the action that triggered the callback, and contents are the new contents of + the configuration. Contents will be ``None`` on a `DELETE` action. All callbacks registered for `NEW` events will + be called at agent startup after all `osntart` methods have been called. Unlike pubsub subscriptions, this may be + called at any point in an agent's lifetime. + + **config.unsubscribe(callback=None, config_name_pattern=None)** - Unsubscribe from configuration changes. + Specifying a callback only will unsubscribe that callback from all config name patterns they have been bound to. + If a pattern only is specified then all callbacks bound to that pattern will be removed. Specifying both will + remove that callback from that pattern. Calling with no arguments will remove all subscriptions. + + **config.unsubscribe_all()** - Unsubscribe from all configuration changes. + + **config.set( config_name, contents, trigger_callback=False )** - Set the contents of a configuration. This may not + be called before `onstart` methods are called. This can be used by an agent to store agent state across agent + installations. This will *NOT* trigger any callbacks unless `trigger_callback` is set to `True`. To prevent + deadlock with the platform this method may not be called from a configuration callback function. Doing so will + raise a `RuntimeError` exception. + + This will not modify the local configuration cache the Agent maintains. It will send the configuration change to + the platform and rely on the subsequent `update_config` call. + + **config.delete( config_name, trigger_callback=False)** - Remove the configuration from the store. This will *NOT* + trigger any callbacks unless trigger_callback is `True`. To prevent deadlock with the platform this method may not + be called from a configuration callback function. Doing so will raise a `RuntimeError` exception. + + **config.list( )** - Returns a list of configuration names. + + **config.set_default(config_name, contents, trigger_callback=False)** - Set a default value for a configuration. + *DOES NOT* modify the platform's configuration store but creates a default configuration that is used for agent + configuration callbacks if the configuration does not exist in the store or the configuration is deleted from the + store. The callback will only be triggered if `trigger_callback` is true and the configuration store subsystem on + the agent is not aware of a configuration with that name from the platform store. + + Typically this will be called in the `__init__` method of an agent with the parsed contents of the packaged + configuration file. This may not be called from a configuration callback. Doing so will raise a `RuntimeError`. + + **config.delete_default(config_name, trigger_callback=False)** - Delete a default value for a configuration. This + method is included for for completeness and is unlikely to be used in agent code. This may not be called from a + configuration callback. Doing so will raise a `RuntimeError`. + + +Configuration Sub System RPC Methods +------------------------------------ + +These methods are made available on each agent to allow the platform to communicate changes to a configuration to the +affected agent. As these methods are not part of the exposed interface they are subject to change. + +**config.update( config_name, action, contents=None, trigger_callback=True)** - called by the platform when a +configuration was changed by some method other than the Agent changing the configuration itself. Trigger callback tells +the agent whether or not to call any callbacks associate with the configuration. + + +Notes on trigger_callback +------------------------- + +As the configuration subsystem calls all callbacks in the `onconfig` phase and none are called beforehand the +`trigger_callback` setting is effectively ignored if an agent sets a configuration or default configuration before the +end of the `onstart` phase. + + +Platform Configuration Store +============================ + +The platform configuration store handles the storage and maintenance of configuration states on the platform. + +As these methods are not part of the exposed interface they are subject to change. + + +Platform RPC Methods +-------------------- + + +Methods for Agents +^^^^^^^^^^^^^^^^^^ + +Agent methods that change configurations do not trigger any callbacks unless trigger_callback is True. + +**set_config(config_name, contents, trigger_callback=False)** - Change/create a configuration file on the platform. + +**get_configs()** - Get all of the configurations for an Agent. + +**delete_config(config_name, trigger_callback=False)** - Delete a configuration. + + +Methods for Management +^^^^^^^^^^^^^^^^^^^^^^ + +**manage_store_config(identity, config_name, contents, config_type="raw")** - Change/create a configuration on the +platform for an agent with the specified identity + +**manage_delete_config(identity, config_name)** - Delete a configuration for an agent with the specified identity. +Calls the agent's update_config with the action `DELETE_ALL` and no configuration name. + +**manage_delete_store(identity)** - Delete all configurations for a VIP IDENTITY. + +**manage_list_config(identity)** - Get a list of configurations for an agent with the specified identity. + +**manage_get_config(identity, config_name, raw=True)** - Get the contents of a configuration file. If raw is set to +`True` this function will return the original file, otherwise it will return the parsed representation of the file. + +**manage_list_stores()** - Get a list of all the agents with configurations. + + +Direct Call Methods +^^^^^^^^^^^^^^^^^^^ + +Services local to the platform who wish to use the configuration store may use two helper methods on the agent class +created for this purpose. This allows the auth service to use the config store before the router is started. + +**delete(self, identity, config_name, trigger_callback=False)** - Same as functionality as `delete_config`, but the +caller must specify the identity of the config store. + +**store(self, identity, config_name, contents, trigger_callback=False)** - Same functionality as set_config, but the +caller must specify the identity of the config store. + + +Command Line Interface +^^^^^^^^^^^^^^^^^^^^^^ + +The command line interface will consist of a new commands for the `volttron-ctl` program called `config` with four +sub-commands called `store`, `delete`, `list`, `get`. These commands will map directly to the management RPC functions +in the previous section. + + +Disabling the Configuration Store +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Agents may optionally disable support for the configuration store by passing ``enable_store=False`` to the `__init__` +method of the Agent class. This allows temporary agents to not spin up the subsystem when it is not needed. Platform +service agents that do not yet support the configuration store and the temporary agents used by `volttron-ctl` will set +this value. diff --git a/docs/source/core_services/config_store/Commandline-Interface.rst b/docs/source/platform-features/config-store/commandline-interface.rst similarity index 95% rename from docs/source/core_services/config_store/Commandline-Interface.rst rename to docs/source/platform-features/config-store/commandline-interface.rst index ea9c729027..a8b3d10970 100644 --- a/docs/source/core_services/config_store/Commandline-Interface.rst +++ b/docs/source/platform-features/config-store/commandline-interface.rst @@ -1,8 +1,12 @@ -Configuration Store Command Line Tools -====================================== +.. _Commandline-Interface: + +=============================== +Config Store Command Line Tools +=============================== Command line management of the Configuration Store is done with the `vctl config` sub-commands. + Store Configuration ------------------- @@ -22,6 +26,7 @@ Optionally you may specify the file type of the file. Defaults to ``--json``. - ``--csv`` - Interpret the file as CSV. - ``--raw`` - Interpret the file as raw data. + Delete Configuration -------------------- @@ -58,6 +63,7 @@ By default this command will return the json representation of what is stored. - ``--raw`` - Return the raw version of the file. + List Configurations ------------------- @@ -94,6 +100,3 @@ The configuration must exist in the store to be edited. By default `edit` will try to open the file with the `nano` editor. The `edit` command will respect the `EDITOR` environment variable. You may override this with the `--editor` option. - - - diff --git a/docs/source/core_services/config_store/Configuration-Store.rst b/docs/source/platform-features/config-store/configuration-store.rst similarity index 63% rename from docs/source/core_services/config_store/Configuration-Store.rst rename to docs/source/platform-features/config-store/configuration-store.rst index a3bd520196..1f4d343d7a 100644 --- a/docs/source/core_services/config_store/Configuration-Store.rst +++ b/docs/source/platform-features/config-store/configuration-store.rst @@ -1,57 +1,60 @@ -============================ -Platform Configuration Store -============================ +.. _Configuration-Store: + +=================== +Configuration Store +=================== The Platform Configuration Store is a mechanism provided by the platform to facilitate the dynamic configuration -of agents. The Platform Configuration Store works by informing agents of changes to their configuration store and +of agents. The Platform Configuration Store works by informing agents of changes to their configuration store and the agent responding to those changes by updating any settings, subscriptions, or processes that are affected by the configuration of the Agent. -**Support for the Configuration Store is not automatically available in existing agents.** An agent must be updated -in order to support this feature. Currently only the Master Driver Agent, the Aggregation Agent, and the Actuator -Agent support the Configuration Store. Configurations and Agents -************************* +========================= Each agent has it's own configuration store (or just store). Agents are not given access to any other agent's store. The existence of a store is not dependent on the existence of an agent installed on the platform. -Each store has a unique identity. Stores are matched to agents at agent runtime via the agent's VIP IDENTITY. +Each store has a unique identity. Stores are matched to agents at agent runtime via the agent's VIP IDENTITY. Therefore the store for an agent is the store with the same identity as the agent's VIP IDENTITY. -When a user updates a configuration in the store the platform immediately informs the agent of the change. -The platform will not send another update until the Agent finishes processing the first. The platform -will send updates to the agent, one file at a time, in the order the changes were received. +When a user updates a configuration in the store the platform immediately informs the agent of the change. The platform +will not send another update until the Agent finishes processing the first. The platform will send updates to the +agent, one file at a time, in the order the changes were received. + Configuration Names -******************* +=================== -Every configuration in an agent's store has a unique name. When a configuration is added to an agent's store -with the same name as an existing configuration it will replace the existing configuration. The store will +Every configuration in an agent's store has a unique name. When a configuration is added to an agent's store +with the same name as an existing configuration it will replace the existing configuration. The store will remove any leading or trailing whitespace, "/", and "\\" from the name. + Configuration File Types -************************ +======================== -The configuration store will automatically parse configuration files before presenting them to an agent. Additionally, the -configuration store does support storing raw data and giving to the agent unparsed. Most Agents will require the -configuration to be parsed. Any Agent that requires raw data will specifically mention the requirement in its documentation. +The configuration store will automatically parse configuration files before presenting them to an agent. Additionally, +the configuration store does support storing raw data and giving to the agent unparsed. Most Agents will require the +configuration to be parsed. Any Agent that requires raw data will specifically mention the requirement in its +documentation. -This system removes the requirement that configuration files for an agent be in a specific format. For instance -a registry configuration for a driver may be JSON instead of CSV if that is more convenient for the user. This +This system removes the requirement that configuration files for an agent be in a specific format. For instance +a registry configuration for a driver may be JSON instead of CSV if that is more convenient for the user. This will work as long as the JSON parses into an equivalent set of objects as an appropriate CSV file. Currently the store supports parsing JSON and CSV files with support for more files types to come. + JSON ---- The store uses the same JSON parser that agents use to parse their configuration files. Therefore it supports Python style comments and must create an object or list when parsed. -.. code-block:: json +:: { "result": "PREEMPTED", #This is a comment. @@ -94,23 +97,24 @@ Is the equivalent to this JSON file: } ] + File references -*************** +=============== -The Platform Configuration Store supports referencing one configuration file from another. If a referenced -file exists the contents of that file will replace the file reference when the file is processed by the -agent. Otherwise the reference will be replaced with null (or in Python, None). +The Platform Configuration Store supports referencing one configuration file from another. If a referenced file exists +the contents of that file will replace the file reference when the file is processed by the agent. Otherwise the +reference will be replaced with null (or in Python, ``None``). -Only configurations that are parsed by the platform (currently JSON or CSV) will be examined for -references. If the file referenced is another parsed file type (JSON or CSV, currently) then the replacement -will be the parsed contents of the file, otherwise it will be the raw contents of the file. +Only configurations that are parsed by the platform (currently JSON or CSV) will be examined for references. If the +file referenced is another parsed file type (JSON or CSV, currently) then the replacement will be the parsed contents of +the file, otherwise it will be the raw contents of the file. In a JSON object the name of a value will never be considered a reference. -A file reference is any value string that starts with "config://". The rest of the string is the name -of another configuration. The configuration name is converted to lower case for comparison purposes. +A file reference is any value string that starts with ``config://``. The rest of the string is the name of another +configuration. The configuration name is converted to lower case for comparison purposes. -Consider the following configuration files named "devices/vav1.config" and "registries/vav.csv", respectively: +Consider the following configuration files named `devices/vav1.config` and `registries/vav.csv`, respectively: .. code-block:: json @@ -131,7 +135,7 @@ Consider the following configuration files named "devices/vav1.config" and "regi ReturnAirCO2,>f,FALSE,1001 ReturnAirCO2Stpt,>f,TRUE,1011 -The resulting configuration returns when an agent asks for "devices/vav1.config". +The resulting configuration returns when an agent asks for `devices/vav1.config`. .. code-block:: python @@ -159,17 +163,24 @@ The resulting configuration returns when an agent asks for "devices/vav1.config" "unit": "vav1" } -Circular references are not allowed. Adding a file that creates a circular reference will cause -that file to be rejected by the platform. +Circular references are not allowed. Adding a file that creates a circular reference will cause that file to be rejected +by the platform. If a configuration is changed in any way and that configuration is referred to by another configuration then -the agent considers the referring configuration as changed. Thus a set of configurations with references +the agent considers the referring configuration as changed. Thus a set of configurations with references can be considered one large configuration broken into pieces for the users convenience. -Multiple configurations may all reference a single configuration. For instance, when configuring drivers +Multiple configurations may all reference a single configuration. For instance, when configuring drivers in the Master Driver you may have multiple drivers reference the same registry if appropriate. + Modifying the Configuration Store -********************************* +================================= + +Currently the configuration store must be modified through the command line. See +:ref:`Commandline Interface `. + +.. toctree:: -Currently the configuration store must be modified through the command line. See :doc:`Commandline-Interface`. + commandline-interface + agent-configuration-store diff --git a/docs/source/platform-features/control/agent-management-control.rst b/docs/source/platform-features/control/agent-management-control.rst new file mode 100644 index 0000000000..0240010209 --- /dev/null +++ b/docs/source/platform-features/control/agent-management-control.rst @@ -0,0 +1,220 @@ +.. _Agent-Control-Commands: + +====================== +Agent Control Commands +====================== + +The VOLTTRON platform has several commands for controlling the lifecycle of agents. This page discusses how to use +them, for details of operation please see :ref:`Platform Configuration ` + +.. note:: + + These examples assume the VOLTTRON environment has been activated + + .. code-block:: bash + + . env/bin/activate + + If not activating the VOLTTRON virtual environment, add "bin/" to all commands + + +Agent Packaging +=============== + +The `vpkg` command is used for packaging and configuring agents. It is not necessary to have the platform running to +use this command. The platform uses `Python Wheel `__ for its packaging and follows +the Wheel naming `convention `__. + +To create an agent package, call: + +.. code-block:: bash + + vpkg + +For instance: ``vpkg package examples/ListenerAgent`` + +The ``package`` command uses the `setup.py` in the agent directory to create the package. The name and version number +portion of the Wheel filename come from this. The resulting wheels are created at `~/.volttron/packaged`. For example: +``~/.volttron/packaged/listeneragent-3.0-py2-none-any.whl``. + + +Agent Configuration +=================== + +Agent packages are configured with: + +.. code-block:: bash + + vpkg configure + +It is suggested that this file use JSON formatting but the agent can be written to interpret any format it requires. +The configuration of a particular agent is opaque to the VOLTTRON platform. The location of the agent config file is +passed as an environmental variable `AGENT_CONFIG` which the provided utilities read in and pass to the agent. + +An example config file passing in some parameters: + +.. code-block:: json + + { + + "agentid": "listener1", + "message": "hello" + } + + +Agent Installation and Removal +============================== + +Agents are installed into the platform using: + +.. code-block:: bash + + vctl install + +When agents are installed onto a platform, it creates a uuid for that instance of an agent. This allows multiple +instances of the same agent package to be installed on the platform. + +This allows the user to refer to the agent with ``--tag `` instead of the uuid when issuing commands. This tag can +also distinguish instances of an agent from each other. + +A stopped agent can be removed with: + +- ``vctl remove `` +- ``vctl remove --tag `` +- ``vctl remove --name `` + + +.. _Agent-Tag: + +Tagging Agents +-------------- + +Agents can be tagged as they are installed with: + +``vctl install =`` + +Agents can be tagged after installation with: + +``vctl tag `` + +Agents can be "tagged" to provide a meaningful user defined way to reference the agent instead of the uuid or the name. +This allows users to differentiate between instances of agents which use the same codebase but are configured +differently. + + +Example +^^^^^^^ + +A user installs two instances of the Listener Agent, tagged with `listen1` and `listen2` respectively: + +.. code-block:: bash + + python scripts/install-agent.py -s examples/ListenerAgent --tag listener1 + python scripts/install-agent.py -s examples/ListenerAgent --tag listener2 + +``vctl status`` displays: + +.. code-block:: console + + AGENT IDENTITY TAG STATUS HEALTH + a listeneragent-3.3 listeneragent-3.3_2 listener2 + 6 listeneragent-3.3 listeneragent-3.3_1 listener1 + +Commands which operate off an agent's UUID can optionally operate off the tag by using "--tag ". This can use wildcards +to catch multiple agents at once. For example, ``vctl start --tag listener*`` will start both `listener1` and +`listener2`. + +.. warning:: + + Removal by tag and name potentially allows multiple agents to be removed at once and should be used with caution. A + "-f" option is required to delete more than one agent at a time. + + +Agent Control +============= + +Starting and Stopping an Agent +------------------------------ + +Agent that are installed in the platform can be launched with the `start` command. By default this operates off the +agent's UUID but can be used with ``--tag`` or ``--name`` to launch agents by those attributes. + +This can allow multiple agents to be started at once. For instance: ``vctl start --name myagent-0.1`` would start all +instances of that agent regardless of their uuid, tag, or configuration information. + +After an agent is started, it will show up in :ref:`Agent Status ` as "running" with a process id. + +Similarly, ``volttron-ctl stop `` can also operate off the tag and name of agent(s). After an agent is stopped, +it will show an exit code of 0 in :ref:`Agent Status ` + +Running an agent +---------------- + +For testing purposes, an agent package not installed in the platform can +be run by using: + +.. code-block:: bash + + vctl run + + +.. _Agent-Status: + +Agent Status +============ + +``vctl list`` shows the agents which have been installed on the platform along with their uuid, associated +:ref:`tag ` and :ref:`priority `. + +- `uuid` is the first column of the display and is displayed as the shorted unique portion. Using this portion, agents + can be started, stopped, removed, etc. +- `AGENT` is the "name" of this agent based on the name of the wheel file which was installed. Agents can be + controlled with this using ``--name``. + + .. note:: + + If multiple instances of a wheel are installed they will all have the same name and can be controlled as a group. + +- IDENTITY is the VIP platform identity assigned to the agent which can be used to make RPC calls, etc. with the + platform +- :ref:`TAG ` is a user provided tag which makes it simpler to track and refer to agents. ``--tag `` + can used in most agent control commands instead of the UUID to control that agent or multiple agents with a pattern. +- PRI is the priority for agents which have been "enabled" using the ``vctl enable`` command. When enabled, agents + will be automatically started in priority order along with the platform. + + +.. code-block:: console + + AGENT IDENTITY TAG PRI + a listeneragent-3.3 listeneragent-3.3_2 listener2 + 6 listeneragent-3.3 listeneragent-3.3_1 listener1 + + +The ``vctl status`` command shows the list of installed agents and whether they are running or have exited. + +.. code-block:: console + + AGENT IDENTITY TAG STATUS HEALTH + a listeneragent-3.3 listeneragent-3.3_2 listener2 running [12872] GOOD + 6 listeneragent-3.3 listeneragent-3.3_1 listener1 running [12873] GOOD + +- `AGENT`, `IDENTITY` and `TAG` are the same as in the ``vctl list`` command +- `STATUS` is the current condition of the agent. If the agent is currently executing, it has "running" and the process + id of the agent. If the agent is not running, the exit code is shown. +- `HEALTH` represents the current state of the agent. `GOOD` health is displayed while the agent is operating as + expected. If an agent enters an error state the health will display as `BAD` + + +.. _Agent-Autostart: + +Agent Autostart +=============== + +An agent can be setup to start when the platform is started with the `enable` command. This command also allows a +priority to be set (0-100, default 50) so that agents can be started after any dependencies. This command can also be +used with the ``--tag`` or ``--name`` options. + +.. code-block:: bash + + vctl enable + diff --git a/docs/source/devguides/walkthroughs/Agent-Authentication-Walkthrough.rst b/docs/source/platform-features/control/authentication-commands.rst similarity index 76% rename from docs/source/devguides/walkthroughs/Agent-Authentication-Walkthrough.rst rename to docs/source/platform-features/control/authentication-commands.rst index 76eff66200..74d0ca3217 100644 --- a/docs/source/devguides/walkthroughs/Agent-Authentication-Walkthrough.rst +++ b/docs/source/platform-features/control/authentication-commands.rst @@ -1,10 +1,81 @@ -.. _AgentAuthentication: +.. _Agent-Authentication-Commands: -How to authenticate an agent to communicate with VOLTTRON platform: -=================================================================== +======================= +Authentication Commands +======================= -An administrator can allow an agent to communicate with VOLTTRON platform by creating an authentication record for that agent. -An authentication record is created by using :code:`vctl auth add` command and entering values to asked arguments. +All authentication sub-commands can be viewed by entering following command. + +.. code-block:: console + + vctl auth --help + +.. code-block:: console + + optional arguments: + -h, --help show this help message and exit + -c FILE, --config FILE + read configuration from FILE + --debug show tracbacks for errors rather than a brief message + -t SECS, --timeout SECS + timeout in seconds for remote calls (default: 30) + --vip-address ZMQADDR + ZeroMQ URL to bind for VIP connections + --keystore-file FILE use keystore from FILE + --known-hosts-file FILE + get known-host server keys from FILE + + subcommands: + add add new authentication record + add-group associate a group name with a set of roles + add-known-host add server public key to known-hosts file + add-role associate a role name with a set of capabilities + keypair generate CurveMQ keys for encrypting VIP connections + list list authentication records + list-groups show list of group names and their sets of roles + list-known-hosts list entries from known-hosts file + list-roles show list of role names and their sets of capabilities + publickey show public key for each agent + remove removes one or more authentication records by indices + remove-group disassociate a group name from a set of roles + remove-known-host remove entry from known-hosts file + remove-role disassociate a role name from a set of capabilities + serverkey show the serverkey for the instance + update updates one authentication record by index + update-group update group to include (or remove) given roles + update-role update role to include (or remove) given capabilities + + +Authentication record +--------------------- + +An authentication record consist of following parameters + +.. code-block:: console + + domain []: + address []: Either a single agent identity or an array of agents identities + user_id []: Arbitrary string to identify the agent + capabilities (delimit multiple entries with comma) []: Array of strings referring to authorized capabilities defined by exported RPC methods + roles (delimit multiple entries with comma) []: + groups (delimit multiple entries with comma) []: + mechanism [CURVE]: + credentials []: Public key string for the agent + comments []: + enabled [True]: + +For more details on how to create authentication record, please see section +:ref:`Agent Authentication ` + + +.. _Agent-Authentication: + +How to authenticate an agent to communicate with VOLTTRON platform +================================================================== + +An administrator can allow an agent to communicate with VOLTTRON platform by creating an authentication record for that +agent. An authentication record is created by using :code:`vctl auth add` command and entering values to asked +arguments. .. code-block:: console @@ -206,7 +277,6 @@ auth.json file entry for the above command would be: } - Roles: ------- A role is a name for a set of capabilities. Roles can be used to grant an agent @@ -259,6 +329,7 @@ To remove a capability from a role: vctl auth update-role BUILDING_A_ADMIN TRIGGER_ALARM --remove + Groups: ------- Groups provide one more layer of *grouping*. A group is a named set of roles. @@ -285,10 +356,12 @@ such agents would implicity be granted the following capabilities: ``READ_BUILDING_A_TEMP``, ``SET_BUILDING_A_TEMP``, ``READ_BUILDLING_B_TEMP``, and ``SET_BUILDING_B_TEMP``. + Mechanism: ----------- Mechanism is the authentication method by which the agent will communicate with VOLTTRON platform. Currently VOLTTRON uses only CURVE mechanism to authenticate agents. + Credentials: ------------- @@ -308,7 +381,3 @@ Enabled: --------- TRUE of FALSE value to enable or disable the authentication record. Record will only be used if this value is True - - - - diff --git a/docs/source/platform-features/control/index.rst b/docs/source/platform-features/control/index.rst new file mode 100644 index 0000000000..77e8ed823d --- /dev/null +++ b/docs/source/platform-features/control/index.rst @@ -0,0 +1,16 @@ +.. _Control: + +================ +VOLTTRON Control +================ + +The base platform functionality focuses on the agent lifecycle, management of the platform itself, and security. This +section describes how to use the commands included with VOLTTRON to configure and control the platform, agents and +drivers. + +.. toctree:: + + platform-configuration + platform-commands + agent-management-control + authentication-commands diff --git a/docs/source/core_services/control/PlatformCommands.rst b/docs/source/platform-features/control/platform-commands.rst similarity index 75% rename from docs/source/core_services/control/PlatformCommands.rst rename to docs/source/platform-features/control/platform-commands.rst index 6adefd115f..3dbad61070 100644 --- a/docs/source/core_services/control/PlatformCommands.rst +++ b/docs/source/platform-features/control/platform-commands.rst @@ -1,34 +1,27 @@ -.. _PlatformCommands: +.. _Platform-Commands: +================= Platform Commands ================= -VOLTTRON files for -a platform instance are stored under a single directory known as the -VOLTTRON home. This home directory is set via the VOLTTRON\_HOME -environment variable and defaults to ~/.volttron. Multiple instances of -the platform may exist under the same account on a system by setting the -VOLTTRON\_HOME environment variable appropriately before executing -VOLTTRON commands. - -Configuration files use a modified INI format where section names are -command names for which the settings in the section apply. Settings -before the first section are considered global and will be used by all -commands for which the settings are valid. Settings keys are long -options (with or without the opening --) and are followed by a colon (:) -or equal (=) and then the value. Boolean options need not include the -separator or value, but may specify a value of 1, yes, or true for true -or 0, no, or false for false. - -A default configuration file, $VOLTTRON\_HOME/config, may be created to -override default options. If it exists, it will be automatically parsed -before all other command-line options. To skip parsing the default -configuration file, either move the file out of the way or set the -SKIP\_VOLTTRON\_CONFIG environment variable. - -All commands and sub-commands have help available with "-h" or "--help". -Additional configuration files may be specified with "-c" or "--config". -To specify a log file, use "-l" or "--log". +VOLTTRON files for a platform instance are stored under a single directory known as the VOLTTRON home. This home +directory is set via the `VOLTTRON_HOME` environment variable and defaults to ``~/.volttron``. Multiple instances of +the platform may exist under the same account on a system by setting the `VOLTTRON_HOME` environment variable +appropriately before executing VOLTTRON commands. + +Configuration files use a modified INI format where section names are command names for which the settings in the +section apply. Settings before the first section are considered global and will be used by all commands for which the +settings are valid. Settings keys are long options (with or without the opening "--") and are followed by a colon (:) +or equal (=) and then the value. Boolean options need not include the separator or value, but may specify a value of 1, +yes, or `true` for true or 0, no, or `false` for false. + +A default configuration file, `$VOLTTRON_HOME/config`, may be created to override default options. If it exists, it +will be automatically parsed before all other command-line options. To skip parsing the default +configuration file, either move the file out of the way or set the `SKIP_VOLTTRON_CONFIG` environment variable. + +All commands and sub-commands have help available with ``-h`` or ``--help``. +Additional configuration files may be specified with ``-c`` or ``-config``. +To specify a log file, use ``-l`` or ``--log``. .. code-block:: bash @@ -85,19 +78,19 @@ Full options: volttron-ctl Commands ---------------------- +===================== volttron-ctl is used to issue commands to the platform from the command line. Through volttron-ctl it is possible to install and removed agents, start and stop agents, manage the configuration store, get the platform status, and shutdown the platform. -In more recent versions of VOLTTRON, the commands 'vctl', 'vpkg', and 'vcfg' -have been added to be used as a stand-in for 'volttron-ctl', 'volttron-pkg', and -'volttron-cfg' in the CLI. The VOLTTRON documentation will often use this convention. +In more recent versions of VOLTTRON, the commands 'vctl', 'vpkg', and 'vcfg' have been added to be used as a stand-in +for 'volttron-ctl', 'volttron-pkg', and 'volttron-cfg' in the CLI. The VOLTTRON documentation will often use this +convention. .. warning:: - volttron-ctl creates a special temporary agent ito communicate with the - platform with a specific VIP IDENTITY, thus multiple instances of volttron-ctl - cannot run at the same time. Attempting to do so will result in a conflicting + + volttron-ctl creates a special temporary agent ito communicate with the platform with a specific VIP IDENTITY, thus + multiple instances of volttron-ctl cannot run at the same time. Attempting to do so will result in a conflicting identity error. .. code-block:: console @@ -147,8 +140,9 @@ have been added to be used as a stand-in for 'volttron-ctl', 'volttron-pkg', and send send agent and start on a remote platform stats manage router message statistics tracking + vctl auth subcommands -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------- .. code-block:: console @@ -163,8 +157,9 @@ vctl auth subcommands serverkey show the serverkey for the instance update updates one authentication record by index + vctl config subcommands -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- .. code-block:: console @@ -175,10 +170,24 @@ vctl config subcommands list list stores or configurations in a store get get the contents of a configuration +rpc subcommands +~~~~~~~~~~~~ + +.. code-block:: console + + subcommands: + code shows how to use rpc call in other agents + -v, --verbose list all subsystem rpc methods in addition to the + agent's rpc methods + list lists all agents and their rpc methods + -i, --vip filter by vip identity + -v, --verbose list all subsystem rpc methods in addition to the + agent's rpc methods. If a method is specified, display + the doc-string associated with the method. vpkg Commands ---------------------- +------------- .. code-block:: console @@ -199,8 +208,7 @@ vpkg Commands agent. configure add a configuration file to an agent package -vpkg commands (with Volttron Restricted package installed and -enabled): +vpkg commands (with Volttron Restricted package installed and enabled): .. code-block:: console @@ -234,13 +242,13 @@ enabled): sign sign a package verify verify an agent package + volttron-cfg Commands --------------------- -volttron-cfg (vcfg) is a tool aimed at making it easier to get up and running with -Volttron and a handful of agents. Running the tool without any arguments -will start a *wizard* with a walk through for setting up instance configuration -options and available agents.If only individual agents need to be configured -they can be listed at the command line. + +volttron-cfg (vcfg) is a tool aimed at making it easier to get up and running with VOLTTRON and a handful of agents. +Running the tool without any arguments will start a *wizard* with a walk through for setting up instance configuration +options and available agents. If only individual agents need to be configured they can be listed at the command line. .. code-block:: console diff --git a/docs/source/core_services/control/VOLTTRON-Config.rst b/docs/source/platform-features/control/platform-configuration.rst similarity index 70% rename from docs/source/core_services/control/VOLTTRON-Config.rst rename to docs/source/platform-features/control/platform-configuration.rst index 310169d98d..24f7888da9 100644 --- a/docs/source/core_services/control/VOLTTRON-Config.rst +++ b/docs/source/platform-features/control/platform-configuration.rst @@ -1,3 +1,72 @@ +.. _Platform-Configuration: + +==================== +VOLTTRON Environment +==================== + +By default, the VOLTTRON projects bases its files out of `VOLTTRON_HOME` +which defaults to `~/.volttron`. + +- ``$VOLTTRON_HOME/agents`` contains the agents installed on the + platform +- ``$VOLTTRON_HOME/certificates`` contains the certificates for use + with the Licensed VOLTTRON code. +- ``$VOLTTRON_HOME/run`` contains files create by the platform during + execution. The main ones are the 0MQ files created for publish and + subcribe. +- ``$VOLTTRON_HOME/ssh`` keys used by agent mobility in the Licensed + VOLTTRON code +- ``$VOLTTRON_HOME/config`` Default location to place a config file to + override any platform settings. +- ``$VOLTTRON_HOME/packaged`` is where agent packages created with `volttron-pkg` are created + + +.. _Platform-Config-File: + +VOLTTRON Config File +==================== + +The VOLTTRON platform config file can contain any of the command line arguments for starting the platform... + +.. code-block:: console + + -c FILE, --config FILE + read configuration from FILE + -l FILE, --log FILE send log output to FILE instead of stderr + -L FILE, --log-config FILE + read logging configuration from FILE + -q, --quiet decrease logger verboseness; may be used multiple + times + -v, --verbose increase logger verboseness; may be used multiple + times + --verboseness LEVEL set logger verboseness + --help show this help message and exit + --version show program's version number and exit + +agent options: + +.. code-block:: console + + --autostart automatically start enabled agents and services + --publish-address ZMQADDR + ZeroMQ URL for used for agent publishing + --subscribe-address ZMQADDR + ZeroMQ URL for used for agent subscriptions + +control options: + +.. code-block:: console + + --control-socket FILE + path to socket used for control messages + --allow-root allow root to connect to control socket + --allow-users LIST users allowed to connect to control socket + --allow-groups LIST user groups allowed to connect to control socket + +Boolean options, which take no argument, may be inverted by prefixing the option with no '-' (e.g. ``--autostart`` may +be inverted using ``--no-autostart``). + + .. _VOLTTRON-Config: VOLTTRON Config @@ -9,13 +78,13 @@ historian, VOLTTRON Central UI, and platform agent. example vcfg output: -.. note:: +.. note:: - In this example, represents the user's home directory, and represents the machine's localhost. - If an option was not specified during bootstrapping i.e. "--web", "--rabbitmq", or "--driver", and an option is selected during the vcfg wizard that requires that option, the necessary dependencies will be installed automatically. -.. code-block:: console +.. code-block:: console Your VOLTTRON_HOME currently set to: /home//.volttron @@ -103,5 +172,3 @@ example vcfg output: If you need to change the instance configuration you can edit the config file is at /home//.volttron/config - - diff --git a/docs/source/platform-features/message-bus/index.rst b/docs/source/platform-features/message-bus/index.rst new file mode 100644 index 0000000000..54f3432686 --- /dev/null +++ b/docs/source/platform-features/message-bus/index.rst @@ -0,0 +1,24 @@ +.. _Message-Bus: + +=========== +Message Bus +=========== + +The VOLTTRON message bus is the mechanism responsible for enabling communication between agents, drivers, and platform +instances. The message bus supports communication using the :ref:`Publish/Subscribe Paradigm ` and +:ref:`JSON RPC `. +Currently VOLTTRON may be configured to use either Zero MQ or RabbitMQ messaging software to perform messaging. + +To standardize message bus communication, VOLTTRON implements VIP - VOLTTRON Interconnect Protocol. VIP defines +patterns for pub/sub communication as well as JSON-RPC, and allows for the creation of agent communication subsystems. + +For more information on messaging, VIP, multi-platform communication and more, please explore the message bus +documentation linked below: + + +.. toctree:: + :caption: Message Bus Topics + + topics + vip/vip-overview + multi-platform/multi-platform-communication diff --git a/docs/source/setup/RabbitMQ/images/central_no_pending.png b/docs/source/platform-features/message-bus/multi-platform/images/central_no_pending.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/central_no_pending.png rename to docs/source/platform-features/message-bus/multi-platform/images/central_no_pending.png diff --git a/docs/source/setup/RabbitMQ/images/central_pending.png b/docs/source/platform-features/message-bus/multi-platform/images/central_pending.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/central_pending.png rename to docs/source/platform-features/message-bus/multi-platform/images/central_pending.png diff --git a/docs/source/setup/RabbitMQ/images/csr-approve.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-approve.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-approve.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-approve.png diff --git a/docs/source/setup/RabbitMQ/images/csr-collector-forwarder-approved.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-collector-forwarder-approved.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-collector-forwarder-approved.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-collector-forwarder-approved.png diff --git a/docs/source/setup/RabbitMQ/images/csr-collector-forwarder-request.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-collector-forwarder-request.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-collector-forwarder-request.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-collector-forwarder-request.png diff --git a/docs/source/setup/RabbitMQ/images/csr-collector-vcp-approve.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-collector-vcp-approve.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-collector-vcp-approve.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-collector-vcp-approve.png diff --git a/docs/source/setup/RabbitMQ/images/csr-collector-vcp-request.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-collector-vcp-request.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-collector-vcp-request.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-collector-vcp-request.png diff --git a/docs/source/setup/RabbitMQ/images/csr-initial-state.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-initial-state.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-initial-state.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-initial-state.png diff --git a/docs/source/setup/RabbitMQ/images/csr-login-page.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-login-page.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-login-page.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-login-page.png diff --git a/docs/source/setup/RabbitMQ/images/csr-no-requests-page.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-no-requests-page.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-no-requests-page.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-no-requests-page.png diff --git a/docs/source/setup/RabbitMQ/images/csr-request.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-request.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-request.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-request.png diff --git a/docs/source/setup/RabbitMQ/images/csr-sequence-approval.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-sequence-approval.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-sequence-approval.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-sequence-approval.png diff --git a/docs/source/setup/RabbitMQ/images/csr-sequence-deny.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-sequence-deny.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-sequence-deny.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-sequence-deny.png diff --git a/docs/source/setup/RabbitMQ/images/csr-set-admin.png b/docs/source/platform-features/message-bus/multi-platform/images/csr-set-admin.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/csr-set-admin.png rename to docs/source/platform-features/message-bus/multi-platform/images/csr-set-admin.png diff --git a/docs/source/setup/RabbitMQ/images/node-rmq-central-vcfg.png b/docs/source/platform-features/message-bus/multi-platform/images/node-rmq-central-vcfg.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/node-rmq-central-vcfg.png rename to docs/source/platform-features/message-bus/multi-platform/images/node-rmq-central-vcfg.png diff --git a/docs/source/setup/RabbitMQ/images/node-rmq-collector2-vcfg.png b/docs/source/platform-features/message-bus/multi-platform/images/node-rmq-collector2-vcfg.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/node-rmq-collector2-vcfg.png rename to docs/source/platform-features/message-bus/multi-platform/images/node-rmq-collector2-vcfg.png diff --git a/docs/source/setup/RabbitMQ/images/node-zmq-collector1-vcfg.png b/docs/source/platform-features/message-bus/multi-platform/images/node-zmq-collector1-vcfg.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/node-zmq-collector1-vcfg.png rename to docs/source/platform-features/message-bus/multi-platform/images/node-zmq-collector1-vcfg.png diff --git a/docs/source/setup/RabbitMQ/images/remote_rmq_pending.png b/docs/source/platform-features/message-bus/multi-platform/images/remote_rmq_pending.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/remote_rmq_pending.png rename to docs/source/platform-features/message-bus/multi-platform/images/remote_rmq_pending.png diff --git a/docs/source/setup/RabbitMQ/images/rmq_remote_forwarder_accepted.png b/docs/source/platform-features/message-bus/multi-platform/images/rmq_remote_forwarder_accepted.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/rmq_remote_forwarder_accepted.png rename to docs/source/platform-features/message-bus/multi-platform/images/rmq_remote_forwarder_accepted.png diff --git a/docs/source/setup/RabbitMQ/images/rmq_remote_forwarder_pending.png b/docs/source/platform-features/message-bus/multi-platform/images/rmq_remote_forwarder_pending.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/rmq_remote_forwarder_pending.png rename to docs/source/platform-features/message-bus/multi-platform/images/rmq_remote_forwarder_pending.png diff --git a/docs/source/setup/RabbitMQ/images/vc-auth-failure.png b/docs/source/platform-features/message-bus/multi-platform/images/vc-auth-failure.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/vc-auth-failure.png rename to docs/source/platform-features/message-bus/multi-platform/images/vc-auth-failure.png diff --git a/docs/source/setup/RabbitMQ/images/vc-collector1-forwarder.png b/docs/source/platform-features/message-bus/multi-platform/images/vc-collector1-forwarder.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/vc-collector1-forwarder.png rename to docs/source/platform-features/message-bus/multi-platform/images/vc-collector1-forwarder.png diff --git a/docs/source/setup/RabbitMQ/images/vc-collector2-forwarder.png b/docs/source/platform-features/message-bus/multi-platform/images/vc-collector2-forwarder.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/vc-collector2-forwarder.png rename to docs/source/platform-features/message-bus/multi-platform/images/vc-collector2-forwarder.png diff --git a/docs/source/setup/RabbitMQ/images/vc_platforms.png b/docs/source/platform-features/message-bus/multi-platform/images/vc_platforms.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/vc_platforms.png rename to docs/source/platform-features/message-bus/multi-platform/images/vc_platforms.png diff --git a/docs/source/setup/RabbitMQ/images/zmq_pending_credential_1.png b/docs/source/platform-features/message-bus/multi-platform/images/zmq_pending_credential_1.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/zmq_pending_credential_1.png rename to docs/source/platform-features/message-bus/multi-platform/images/zmq_pending_credential_1.png diff --git a/docs/source/setup/RabbitMQ/images/zmq_pending_credential_1_approved.png b/docs/source/platform-features/message-bus/multi-platform/images/zmq_pending_credential_1_approved.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/zmq_pending_credential_1_approved.png rename to docs/source/platform-features/message-bus/multi-platform/images/zmq_pending_credential_1_approved.png diff --git a/docs/source/setup/RabbitMQ/images/zmq_pending_credential_2.png b/docs/source/platform-features/message-bus/multi-platform/images/zmq_pending_credential_2.png similarity index 100% rename from docs/source/setup/RabbitMQ/images/zmq_pending_credential_2.png rename to docs/source/platform-features/message-bus/multi-platform/images/zmq_pending_credential_2.png diff --git a/docs/source/core_services/multiplatform/Multiplatform-Communication.rst b/docs/source/platform-features/message-bus/multi-platform/multi-platform-communication.rst similarity index 63% rename from docs/source/core_services/multiplatform/Multiplatform-Communication.rst rename to docs/source/platform-features/message-bus/multi-platform/multi-platform-communication.rst index 51eda65b2f..600d5dcfb4 100644 --- a/docs/source/core_services/multiplatform/Multiplatform-Communication.rst +++ b/docs/source/platform-features/message-bus/multi-platform/multi-platform-communication.rst @@ -5,36 +5,40 @@ Multi-Platform Communication ============================ To connect to remote VOLTTRON platforms, we would need platform discovery information of the remote platforms. This -information contains the platform name, VIP address and serverkey of the remote platforms and we need to provide this -as part of Multiplatform configuration. +information contains the platform name, VIP address and `serverkey` of the remote platforms and we need to provide this +as part of multi-platform configuration. + Configuration ************* The configuration and authentication for multi-platform connection can be setup either manually or by running the -platforms in set up mode. Both the setups are described below. +platforms in set up mode. Both the setups are described below. + Setup Mode For Automatic Authentication *************************************** -.. note:: It is necessary for each platform to have a web server if running in setup mode. +.. note:: + + It is necessary for each platform to have a web server if running in setup mode. For ease of use and to support multi-scale deployment, the process of obtaining the platform discovery information and -authenticating the new platform connection is automated. We can now bypass the manual process of adding auth keys (i.e., -either by using the volttron-ctl utility or directly updating the auth.json config file). +authenticating the new platform connection is automated. We can now bypass the manual process of adding auth keys +(i.e., either by using the `volttron-ctl` utility or directly updating the `auth.json` config file). -A config file containing list of web addresses (one for each platform) need to be made available in VOLTTRON_HOME +A config file containing list of web addresses (one for each platform) need to be made available in `VOLTTRON_HOME` directory. -Name of the file: external_address.json +Name of the file: `external_address.json` Directory path: Each platform’s VOLTTRON_HOME directory. -For example: /home/volttron/.volttron1 +For example: `/home/volttron/.volttron1` Contents of the file: -.. code-block:: json +:: [ "http://:", @@ -46,24 +50,23 @@ Contents of the file: We then start each VOLTTRON platform with setup mode option in this way. - :: - - volttron -vv -l volttron.log --setup-mode& +.. code-block:: bash + volttron -vv -l volttron.log --setup-mode& Each platform will obtain the platform discovery information of the remote platform that it is trying to connect through a HTTP discovery request and store the information in a configuration file -($VOLTTRON_HOME/external_platform_discovery.json). It will then use the VIP address and serverkey to connect to the -remote platform. The remote platform shall authenticate the new connection and store the auth keys (public key) of the +(`$VOLTTRON_HOME/external_platform_discovery.json`). It will then use the VIP address and `serverkey` to connect to the +remote platform. The remote platform shall authenticate the new connection and store the auth keys (public key) of the connecting platform for future use. -The platform discovery information will be stored in VOLTTRON_HOME directory and looks like below: +The platform discovery information will be stored in `VOLTTRON_HOME` directory and looks like below: -Name of config file: external_platform_discovery.json +Name of config file: `external_platform_discovery.json` Contents of the file: -.. code-block:: json +:: {"": {"vip-address":"tcp://:", "instance-name":"", @@ -85,24 +88,23 @@ Each platform will use this information for future connections. Once the keys have been exchanged and stored in the auth module, we can restart all the VOLTTRON platforms in normal mode. - :: - - ./stop-volttron +.. code-block:: bash - ./start-volttron + ./stop-volttron + ./start-volttron Manual Configuration of External Platform Information ***************************************************** -Platform discovery configuration file can also be built manually and it needs to be added inside VOLTTRON_HOME directory -of each platform. +Platform discovery configuration file can also be built manually and it needs to be added inside `VOLTTRON_HOME` +directory of each platform. -Name of config file: external_platform_discovery.json +Name of config file: `external_platform_discovery.json` Contents of the file: -.. code-block:: json +:: {"": {"vip-address":"tcp://:", "instance-name":"", @@ -120,12 +122,23 @@ Contents of the file: } With this configuration, platforms can be started in normal mode. - :: - ./start-volttron +.. code-block:: bash + + ./start-volttron For external platform connections to be authenticated, we would need to add the credentials of the connecting platforms -in each platform using the volttron-ctl auth utility. For more details -:ref:`Agent authentication walkthrough `. +in each platform using the `volttron-ctl auth` utility. For more details +:ref:`Agent authentication walk-through `. + +.. seealso:: + + :ref:`Multi-Platform Walk-through ` + + +.. toctree:: + :caption: Multi-platform Message Bus Topics -.. seealso:: :ref:`Multi-Platform Walkthrough ` + multi-platform-pubsub + multi-platform-rpc + multi-platform-rmq-csr-walk-through diff --git a/docs/source/core_services/multiplatform/Multiplatform-PubSub.rst b/docs/source/platform-features/message-bus/multi-platform/multi-platform-pubsub.rst similarity index 100% rename from docs/source/core_services/multiplatform/Multiplatform-PubSub.rst rename to docs/source/platform-features/message-bus/multi-platform/multi-platform-pubsub.rst diff --git a/docs/source/setup/RabbitMQ/Multi-Platform-RMQ-CSR-Walkthrough.rst b/docs/source/platform-features/message-bus/multi-platform/multi-platform-rmq-csr-walk-through.rst similarity index 94% rename from docs/source/setup/RabbitMQ/Multi-Platform-RMQ-CSR-Walkthrough.rst rename to docs/source/platform-features/message-bus/multi-platform/multi-platform-rmq-csr-walk-through.rst index e871e81afa..e5029873c7 100644 --- a/docs/source/setup/RabbitMQ/Multi-Platform-RMQ-CSR-Walkthrough.rst +++ b/docs/source/platform-features/message-bus/multi-platform/multi-platform-rmq-csr-walk-through.rst @@ -1,5 +1,6 @@ -.. _Multi_Platform_Walkthrough: +.. _Multi-Platform-Multi-Bus-Walk-through: +===================================== Multi-Platform Multi-Bus Walk-through ===================================== @@ -13,9 +14,9 @@ with VOLTTRON instance. Node Setup ---------- -For this example we will have two types of nodes; a data collector and a central node. Each of the data -collectors will have different message buses (VOLTTRON supports both RabbitMQ and ZeroMQ). The nodes will -be configured as in the following table. +For this example we will have two types of nodes; a data collector and a central node. Each of the data collectors will +have different message buses (VOLTTRON supports both RabbitMQ and ZeroMQ). The nodes will be configured as in the +following table. .. csv-table:: Node Configuration :header: "", "Central", "Node-ZMQ", "Node-RMQ" @@ -34,14 +35,16 @@ be configured as in the following table. The goal of this is to be able to see the data from Node-ZMQ and Node-RMQ in the Central SQL Historian and on the trending charts of Volttron Central. + Virtual Machine Setup --------------------- The first step in creating a VOLTTRON instance is to make sure the machine is ready for VOLTTRON. Each machine -should have its hostname setup. For this walkthrough, the hostnames "central", "node-zmq" and "node-rmq" will be used. +should have its hostname setup. For this walk-through, the hostnames "central", "node-zmq" and "node-rmq" will be used. + +For Central and Node-RMQ follow the instructions :ref:`platform installation steps for RMQ `. For +Node-ZMQ use :ref:`Platform Installation steps for ZeroMQ `. -For Central and Node-RMQ follow the instructions :ref:`Building-VOLTTRON#steps-for-rabbitmq`. For Node-ZMQ use -:ref:`Building-VOLTTRON#steps-for-zmq`. Instance Setup -------------- @@ -49,18 +52,22 @@ Instance Setup The following conventions/assumptions are made for the rest of this document: - Commands should be run from the VOLTTRON root - - Default values are used for VOLTTRON_HOME($HOME/.volttron), vip port (22916), HTTPS port (8443), rabbitmq ports( 5671 for AMQPs and 15671 for RabbitMQ management interface). If using different VOLTTRON_HOME or ports, please replace accordingly. + - Default values are used for VOLTTRON_HOME($HOME/.volttron), VIP port (22916), HTTPS port (8443), rabbitmq ports + (5671 for AMQPs and 15671 for RabbitMQ management interface). If using different `VOLTTRON_HOME` or ports, please + replace accordingly. - Replace central, node-zmq and node-rmq with your own hostnames. - user will represent your current user. -The following will use vcfg (volttron-cfg) to configure the individual platforms. +The following will use `vcfg` (volttron-cfg) to configure the individual platforms. + Central Instance Setup ---------------------- .. note:: - This instance must have been bootstrapped using --rabbitmq see :ref:`Building-VOLTTRON#steps-for-rabbitmq`. + This instance must have been bootstrapped using ``--rabbitmq`` see + :ref:`RabbitMq installation instructions `. Next step would be to configure the instance to have a web interface to accept/deny incoming certificate signing requests from other instances. Additionally, we will need to install a Volttron Central agent, Volttron Central @@ -197,6 +204,7 @@ Approve the CSR request to allow authenticated SSL based connection to the "cent Go back to the terminal and check the status of Volttron Central Platform agent. It should be set to "GOOD". + Node-ZMQ Instance Setup ----------------------- On the "node-zmq" VM, setup a ZeroMQ based VOLTTRON instance. Using "vcfg" command, install Volttron Central Platform agent, @@ -344,7 +352,8 @@ Node-RMQ Instance Setup .. note:: - This instance must have been bootstrapped using --rabbitmq see :ref:`Building-VOLTTRON#steps-for-rabbitmq`. + This instance must have been bootstrapped using --rabbitmq see + :ref:`RabbitMq installation instructions `. Using "vcfg" command, install Volttron Central Platform agent, a master driver agent with fake driver. The instance @@ -361,7 +370,7 @@ name is set to "collector2". Name of this volttron instance: [volttron1]: collector2 RabbitMQ server home: [/home/user/rabbitmq_server/rabbitmq_server-3.7.7]: Fully qualified domain name of the system: [node-rmq]: - Would you like to create a new self signed root CAcertificate for this instance: [Y]: + Would you like to create a new self signed root CA certificate for this instance: [Y]: Please enter the following details for root CA certificate Country: [US]: diff --git a/docs/source/core_services/multiplatform/Multiplatform-RPC.rst b/docs/source/platform-features/message-bus/multi-platform/multi-platform-rpc.rst similarity index 100% rename from docs/source/core_services/multiplatform/Multiplatform-RPC.rst rename to docs/source/platform-features/message-bus/multi-platform/multi-platform-rpc.rst diff --git a/docs/source/platform-features/message-bus/topics.rst b/docs/source/platform-features/message-bus/topics.rst new file mode 100644 index 0000000000..ff83282832 --- /dev/null +++ b/docs/source/platform-features/message-bus/topics.rst @@ -0,0 +1,52 @@ +.. _Messaging-Topics: + +==================== +Messaging and Topics +==================== + + +Introduction +============ + +Agents in |VOLTTRON| communicate with each other using a publish/subscribe mechanism built on the Zero MQ or RabbitMQ +Python libraries. This allows for great flexibility as topics can be created dynamically and the messages sent can be +any format as long as the sender and receiver understand it. An agent with data to share publishes to a topic, then +any agents interested in that data subscribe to that topic. + +While this flexibility is powerful, it also could also lead to confusion if some standard is not followed. The current +conventions for communicating in the VOLTTRON are: + +- Topics and subtopics follow the format: ``topic/subtopic/subtopic`` +- Subscribers can subscribe to any and all levels. Subscriptions to `topic` will include messages for the base topic + and all subtopics. Subscriptions to ``topic/subtopic1`` will only receive messages for that subtopic and any + children subtopics. Subscriptions to empty string ("") will receive ALL messages. This is not recommended. + + - All agents should subscribe to the ``platform`` topic. This is the topic the VOLTTRON will use to send messages + to agents, such as `shutdown`. + +Agents should set the `From` header. This will allow agents to filter on the `To` message sent back. + + +Topics +====== + + +In VOLTTRON +----------- + +- **platform** - Base topic used by the platform to inform agents of platform events +- **platform/shutdown** - General shutdown command. All agents should exit upon receiving this. Message content will + be a reason for the shutdown +- **platform/shutdown_agent** - This topic will provide a specific agent id. Agents should subscribe to this topic and + exit if the id in the message matches their id. +- **devices** - Base topic for data being published by drivers +- **datalogger** - Base topic for agents wishing to record time series data +- record - Base topic for agents to record data in an arbitrary format. + + +Controller Agent Topics +----------------------- + +See the documentation for the :ref:`Actuator Agent `. + +.. |VOLTTRON| unicode:: VOLTTRON U+2122 diff --git a/docs/source/core_services/messagebus/VIP/VIP-Authentication.rst b/docs/source/platform-features/message-bus/vip/vip-authentication.rst similarity index 97% rename from docs/source/core_services/messagebus/VIP/VIP-Authentication.rst rename to docs/source/platform-features/message-bus/vip/vip-authentication.rst index 8166eea3bc..f462d2d21c 100644 --- a/docs/source/core_services/messagebus/VIP/VIP-Authentication.rst +++ b/docs/source/platform-features/message-bus/vip/vip-authentication.rst @@ -55,7 +55,7 @@ The auth file should not be modified directly. To change the auth file, use ``vctl auth`` subcommands: ``add``, ``list``, ``remove``, and ``update``. (Run ``vctl auth --help`` for more details and see the -:ref:`authentication commands documentation<_AuthenticationCommands>`.) +:ref:`authentication commands documentation `.) Here are some example entries:: @@ -210,4 +210,4 @@ Now if agent ``A`` can successfully connect to platform ``B``, and platform 2016-10-19 14:26:16,446 () volttron.platform.auth INFO: authentication success: domain='vip', address='127.0.0.1', mechanism='CURVE', credentials=['HOVXfTspZWcpHQcYT_xGcqypBHzQHTgqEzVb4iXrcDg'], user_id='Agent-A' -For a more details see the :ref:`authentication walkthrough`. +For a more details see the :ref:`authentication walk-through `. diff --git a/docs/source/core_services/messagebus/VIP/VIP-Authorization.rst b/docs/source/platform-features/message-bus/vip/vip-authorization.rst similarity index 64% rename from docs/source/core_services/messagebus/VIP/VIP-Authorization.rst rename to docs/source/platform-features/message-bus/vip/vip-authorization.rst index d3829123cf..56a41dabb5 100644 --- a/docs/source/core_services/messagebus/VIP/VIP-Authorization.rst +++ b/docs/source/platform-features/message-bus/vip/vip-authorization.rst @@ -4,29 +4,26 @@ VIP Authorization ================= -VIP :ref:`authentication ` and authorization go hand in -hand. When an agent authenticates to a VOLTTRON platform that agent proves its -identity to the platform. Once authenticated, an agent is allowed to -connect to the :ref:`message bus<_VOLTTRON-Message-Bus>`. VIP -authorization is about giving a platform owner the ability to limit the -capabilities of authenticated agents. +VIP :ref:`authentication ` and authorization go hand in hand. When an agent authenticates to a +VOLTTRON platform that agent proves its identity to the platform. Once authenticated, an agent is allowed to connect to +the :ref:`message bus `. VIP authorization is about giving a platform owner the ability to limit +the capabilities of authenticated agents. There are two parts to authorization: #. Required capabilities (specified in agent's code) #. Authorization entries (specified via ``volttron-ctl auth`` commands) -The following example will walk through how to specify required capabilities -and grant those capabilities in authorization entries. +The following example will walk through how to specify required capabilities and grant those capabilities in +authorization entries. + Single Capability ----------------- -For this example suppose there is a temperature agent that can read and set the -temperature of a particular room. The agent author anticipates that building -managers will want to limit which agents can set the temperature. +For this example suppose there is a temperature agent that can read and set the temperature of a particular room. The +agent author anticipates that building managers will want to limit which agents can set the temperature. -In the temperature agent, a required capability is specified by -using the ``RPC.allow`` decorator: +In the temperature agent, a required capability is specified by using the ``RPC.allow`` decorator: .. code:: Python @@ -39,21 +36,22 @@ using the ``RPC.allow`` decorator: def set_temperature(temp): ... -In the code above, any agent can call the ``get_temperature`` method, but only -agents with the ``CAP_SET_TEMP`` capability can call ``set_temperature``. -(Note: capabilities are arbitrary strings. This example follows the general -style used for Linux capabilities, but it is up to the agent author.) +In the code above, any agent can call the ``get_temperature`` method, but only agents with the ``CAP_SET_TEMP`` +capability can call ``set_temperature``. + +.. Note:: -Now that a required capability has been specified, suppose a VOLLTRON platform -owner wants to allow a specific agent, say AliceAgent, to set the temperature. + Capabilities are arbitrary strings. This example follows the general style used for Linux capabilities, but it is + up to the agent author. -The platform owner runs ``vctl auth add`` to add new authorization -entries or ``vctl auth update`` to update an existing entry. -If AliceAgent is installed on the platform, then it already has an -authorization entry. Running ``vctl auth list`` shows the existing -entries: +Now that a required capability has been specified, suppose a VOLTTRON platform owner wants to allow a specific agent, +say `Alice Agent`, to set the temperature. -.. code:: JSON +The platform owner runs ``vctl auth add`` to add new authorization entries or ``vctl auth update`` to update an existing +entry. If `Alice Agent` is installed on the platform, then it already has an authorization entry. Running +``vctl auth list`` shows the existing entries: + +:: ... INDEX: 3 @@ -71,9 +69,8 @@ entries: } ... -Currently AliceAgent cannot set the temperature because it does -not have the ``CAP_SET_TEMP`` capability. To grant this capability -the platform owner runs ``vctl auth update 3``: +Currently AliceAgent cannot set the temperature because it does not have the ``CAP_SET_TEMP`` capability. To grant this +capability the platform owner runs ``vctl auth update 3``: .. code:: Bash @@ -91,18 +88,20 @@ the platform owner runs ``vctl auth update 3``: updated entry at index 3 -Now AliceAgent can call ``set_temperature`` via RPC. -If other agents try to call that method they will get the following -exception:: +Now `Alice Agent` can call ``set_temperature`` via RPC. If other agents try to call that method they will get the +following exception: + +.. code-block:: console error: method "set_temperature" requires capabilities set(['CAP_SET_TEMP']), but capability list [] was provided + Multiple Capabilities --------------------- -Expanding on the temperature-agent example, the ``set_temperature`` method can -require agents to have multiple capabilities: +Expanding on the temperature-agent example, the ``set_temperature`` method can require agents to have multiple +capabilities: .. code:: Python @@ -111,9 +110,8 @@ require agents to have multiple capabilities: def set_temperature(): ... -This requires an agent to have both the ``CAP_SET_TEMP`` and the -``CAP_FOO_BAR`` capabilities. Multiple capabilities can also -be specified by using multiple ``RPC.allow`` decorators: +This requires an agent to have both the ``CAP_SET_TEMP`` and the ``CAP_FOO_BAR`` capabilities. Multiple capabilities can +also be specified by using multiple ``RPC.allow`` decorators: .. code:: Python @@ -123,32 +121,33 @@ be specified by using multiple ``RPC.allow`` decorators: def temperature(): ... + Capability with parameter restriction ------------------------------------- -Capabilities can also be used to restrict access to a rpc method only with certain parameter values. For example, if AgentA -exposes a method bar which accepts parameter x - +Capabilities can also be used to restrict access to a rpc method only with certain parameter values. For example, if +`Agent A` exposes a method bar which accepts parameter `x`. AgentA's capability enabled exported RPC method: -:: +.. code-block:: python @RPC.export @RPC.allow('can_call_bar') def bar(self, x): return 'If you can see this, then you have the required capabilities' -You can restrict access to AgentA's bar method to AgentB with x=1. To add this auth entry use the vctl auth add command -as show below +You can restrict access to `Agent A`'s `bar` method to `Agent B` with ``x=1``. To add this auth entry use the +``vctl auth add`` command as show below: -:: +.. code-block:: bash vctl auth add --capabilities '{"test1_cap2":{"x":1}}' --user_id AgentB --credential vELQORgWOUcXo69DsSmHiCCLesJPa4-CtVfvoNHwIR0 -auth.json file entry for the above command would be -:: +The auth.json file entry for the above command would be: + +.. code-block:: json { "domain": null, @@ -168,10 +167,9 @@ auth.json file entry for the above command would be } +Parameter values can also be regular expressions: -Parameter values can also be regular expressions - -:: +.. code-block:: console (volttron)volttron@volttron1:~/git/myvolttron$ vctl auth add domain []: @@ -187,9 +185,9 @@ Parameter values can also be regular expressions added entry domain=None, address=None, mechanism='CURVE', credentials=u'vELQORgWOUcXo69DsSmHiCCLesJPa4-CtVfvoNHwIR0', user_id='b22e041d-ec21-4f78-b32e-ab7138c22373' -auth.json file entry for the above command would be: +The auth.json file entry for the above command would be: -:: +.. code-block:: json { "domain": null, diff --git a/docs/source/platform-features/message-bus/vip/vip-enhancements.rst b/docs/source/platform-features/message-bus/vip/vip-enhancements.rst new file mode 100644 index 0000000000..7e2967dfc7 --- /dev/null +++ b/docs/source/platform-features/message-bus/vip/vip-enhancements.rst @@ -0,0 +1,85 @@ +.. _VIP-Enhancements: + +================ +VIP Enhancements +================ + +When creating VIP for VOLTTRON 3.0 we wanted to address two security concerns and one user request: + +- Security Concern 1: Agents can spoof each other on the VOLTTRON message bus and fake messages. +- Security Concern 2: Agents can subscribe to topics that they are not authorized to subscribe to. +- User Request 1: Several users requested means to transfer large + amounts of data between agents without using the message bus. + +VOLTTRON Interconnect Protocol (VIP) was created to address these issues but unfortunately, it broke the easy to use +pub-sub messaging model of VOLTTRON. Additionally to use the security features of VOLTTRON in 3.0 code has become an +ordeal especially when multiple platforms are concerned. Finally, VIP has introduced the requirement for knowledge of +specific other platforms to agents written by users in order to be able to communicate. The rest of this memo focuses +on defining the way VOLTTRON message bus will work going forward indefinitely and should be used as the guiding +principles for any future work on VIP and VOLTTRON. + + +VOLTTRON Message Bus Guiding Principles: +---------------------------------------- + +#. | All communications between two or more different VOLTTRON platforms MUST go through the VIP Router. Said another + way, a user agent (application) should have *NO* capability to reach out to an agent on a different VOLTTRON + platform directly. + + | All communications between two or more VOLTTRON platforms must be in the form of topics on the message bus. Agents + *MUST* not use a distinct platform address or name to communicate via a direct connection between two platforms. + +#. VOLTTRON will use two TCP ports. One port is used to extend VIP across platforms. A second port is used for the + VOLTTRON discovery protocol (more on this to come on a different document). VIP will establish bi-directional + communication via a single TCP port. + +#. In order to solve the bootstrapping problem that CurveMQ has punted on, we will modify VIP to operate similar + (behaviorally) to SSH. + +A. On a single VOLTTRON platform, the platform’s public key will be made available via an API so that all agents will be + able to communicate with the platform. Additionally, the behavior of the platform will be changed so that agents on + the same platform will automatically be added to the `auth.json` file. No more need for user to add the agents + manually to the file. The desired behavior is similar to how SSH handles `known_hosts`. + + .. Note:: + + This behavior still addresses the security request 1 & 2. + +B. When connecting VOLTTRON platforms, VOLTTRON Discovery Protocol (VDP) will be used to discover the other platforms + public key to establish the router to router connection. Note that since we *BANNED* agent to agent communication + between two platforms, we have prevented an "O(N^2)" communication pattern and key bootstrapping problem. + +#. Authorization determines what agents are allowed to access what topics. Authorization MUST be managed by the + VOLTTRON Central platform on a per organization basis. It is not recommended to have different authorization + profiles on different VOLTTRON instances belonging to the same organization. + +#. VOLTTRON message bus uses topics such as and will adopt an information model agreed upon by the VOLTTRON community + going forward. Our initial information model is based on the OpenEIS schema going forward. A different document + will describe the information model we have adopted going forward. All agents are free to create their own topics + but the VOLTTRON team (going forward) will support the common VOLTTRON information model and all agents developed by + PNNL will be converted to use the new information model. + +#. Two connected VOLTTRON systems will exchange a list of available topics via the message router. This will allow each + VIP router to know what topics are available at what VOLTTRON platform. + +#. Even though each VOLTTRON platform will have knowledge of what topics are available around itself, no actual messages + will be forwarded between VOLTTRON platforms until an agent on a specific platform subscribes to a topic. When an + agent subscribes to a topic that has a publisher on a different VOLTTRON platform, the VIP router will send a request + to its peer routers so that the messages sent to that topic will be forwarded. There will be cases (such as clean + energy transactive project) where the publisher to a topic may be multiple hops away. In this case, the subscribe + request will be sent towards the publisher through other VIP routers. In order to find the most efficient path, we + may need to keep track of the total number of hops (in terms of number of VIP routers). + +#. The model described in steps 5/6/7 applies to data collection. For control applications, VOLTTRON team only allows + control actions to be originated from the VOLTTRON instance that is directly connected to that controlled device. + This decision is made to increase the robustness of the control agent and to encourage truly distributed applications + to be developed. + +#. Direct agent to agent communication will be supported by creation of an ephemeral topic under the topic hierarchy. + Our measurements have shown repeatedly that the overhead of using the ZeroMQ message pub/sub is minimal and has zero + impact on communications throughput. + +In summary, by making small changes to the way VIP operates, I believe that we can significantly increase the usability +of the platform and also correct the mixing of two communication platforms into VIP. VOLTTRON message bus will return +to being a pub/sub messaging system going forward. Direct agent to agent communication will be supported through the +message bus. diff --git a/docs/source/platform-features/message-bus/vip/vip-json-rpc.rst b/docs/source/platform-features/message-bus/vip/vip-json-rpc.rst new file mode 100644 index 0000000000..27d3ca22ee --- /dev/null +++ b/docs/source/platform-features/message-bus/vip/vip-json-rpc.rst @@ -0,0 +1,305 @@ +.. _Remote-Procedure-Calls: + +====================== +Remote Procedure Calls +====================== + +Remote procedure calls (RPC) is a feature of VOLTTRON Interconnect Protocol :ref:`VIP `. VIP includes the +ability to create new point-to-point protocols, called subsystems, enabling the implementation of +`JSON-RPC 2.0 `_. This provides a simple method for agent authors to write +methods and expose or export them to other agents, making request-reply or notify communications patterns as +simple as writing and calling methods. + + +Exporting Methods +================= + +The ``export()`` method, defined on the RPC subsystem class, is used to mark a method as remotely accessible. This +``export()`` method has a dual use: + +* The class method can be used as a decorator to statically mark methods when the agent class is defined. +* The instance method dynamically exports methods, and can be used with methods not defined on the agent + class. + +Each take an optional export name argument, which defaults to the method name. Here are the two export method +signatures: + +Instance method: + +.. code-block:: python + + RPC.export(method, name=None) + +Class method: + +.. code-block:: python + + RPC.export(name=None) + +And here is an example agent definition using both methods: + +.. code-block:: python + + from volttron.platform.vip import Agent, Core, RPC + + def add(a, b): + '''Add two numbers and return the result''' + return a + b + + + class ExampleAgent(Agent): + @RPC.export + def say_hello(self, name): + '''Build and return a hello string''' + return 'Hello, %s!' % (name,) + + @RPC.export('say_bye') + def bye(self, name): + '''Build and return a goodbye string''' + return 'Goodbye, %s.' % (name,) + + @Core.receiver('setup') + def onsetup(self, sender, **kwargs): + self.vip.rpc.export('add') + + +Calling exported methods +======================== + +The RPC subsystem provides three methods for calling exported RPC methods: + +.. code-block:: python + + RPC.call(peer, method, *args, **kwargs) + +Call the remote ``method`` exported by ``peer`` with the given arguments. Returns a `gevent` `AsyncResult` object. + +.. code-block:: python + + RPC.batch(peer, requests) + +Batch call remote methods exported by `peer`. `requests` must be an iterable of 4-tuples +``(notify, method, args, kwargs)``, where ``notify`` is a boolean indicating whether this is a notification or standard +call, ``method`` is the method name, ``args`` is a list and ``kwargs`` is a dictionary. Returns a list of `AsyncResult` +objects for any standard calls. Returns ``None`` if all requests were notifications. + +.. code-block:: python + + RPC.notify(peer, method, *args, **kwargs) + +Send a one-way notification message to `peer` by calling `method` without returning a result. + +Here are some examples: + +.. code-block:: python + + self.vip.rpc.call(peer, 'say_hello', 'Bob').get() + results = self.vip.rpc.batch(peer, [(False, 'say_bye', 'Alice', {}), (True, 'later', [], {})]) + self.vip.rpc.notify(peer, 'ready') + + +Inspection +---------- + +A list of methods is available by calling the `inspect` method. Additional information can be returned for any method +by appending ``.inspect`` to the method name. Here are a couple examples: + +.. code-block:: python + + self.vip.rpc.call(peer, 'inspect') # Returns a list of exported methods + self.vip.rpc.call(peer, 'say_hello.inspect') # Return metadata on say_hello method + + +VCTL RPC Commands +~~~~~~~~~~~~~~~~~ + +There are two rpc subcommands available through vctl, *list* and *code*. + +The list subcommand displays all of the agents that have a peer +connection to the instance and which methods are available from +each of these agents. + +.. code-block:: console + + vctl rpc list + config.store + delete_config + get_configs + manage_delete_config + manage_delete_store + manage_get + manage_get_metadata + manage_list_configs + manage_list_stores + manage_store + set_config + . + . + . + + platform.historian + get_aggregate_topics + get_topic_list + get_topics_by_pattern + get_topics_metadata + get_version + insert + query + volttron.central + get_publickey + is_registered + +If a single agent is specified, it will list all methods available for that agent. + +.. code-block:: console + + vctl rpc list platform.historian + platform.historian + get_aggregate_topics + get_topic_list + get_topics_by_pattern + get_topics_metadata + get_version + insert + query + +If the -v option is selected, all agent subsystem rpc methods will be displayed +for each selected agent as well. + +.. code-block:: console + + vctl rpc list -v platform.historian + platform.historian + get_aggregate_topics + get_topic_list + get_topics_by_pattern + get_topics_metadata + get_version + insert + query + agent.version + health.set_status + health.get_status + health.get_status_json + health.send_alert + heartbeat.start + heartbeat.start_with_period + heartbeat.stop + heartbeat.restart + heartbeat.set_period + config.update + config.initial_update + auth.update + +If an agent is specified, and then a method (or methods) are specified, +all parameters associated with the method(s) will be output. + +.. code-block:: console + + vctl rpc list platform.historian get_version query + platform.historian + get_version + Parameters: + query + Parameters: + topic: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': None} + start: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': None} + end: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': None} + agg_type: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': None} + agg_period: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': None} + skip: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': 0} + count: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': None} + order: + {'kind': 'POSITIONAL_OR_KEYWORD', 'default': 'FIRST_TO_LAST'} + + +By adding the '-v' option to this stage, the doc-string description +of the method will be displayed along with the method and parameters if available. + +.. code-block:: console + + vctl rpc list -v platform.historian get_version + platform.historian + get_version + Documentation: + RPC call to get the version of the historian + + :return: version number of the historian used + :rtype: string + + Parameters: + + vctl rpc code + vctl rpc list + vctl rpc list + vctl rpc list -v + vctl rpc list -v + vctl rpc code -v + vctl rpc code + vctl rpc code + +The code subcommand functions similarly to list, except that it will output the code +to be used in an agent when writing an rpc call. Any available parameters are included +as a list in the line of code where the parameters will need to be provided. These will +need to be modified based on the use case. + +.. code-block:: console + + vctl rpc code + self.vip.rpc.call(config.store, delete_config, ['config_name', 'trigger_callback', 'send_update']).get() + self.vip.rpc.call(config.store, get_configs).get() + self.vip.rpc.call(config.store, manage_delete_config, ['args', 'kwargs']).get() + self.vip.rpc.call(config.store, manage_delete_store, ['args', 'kwargs']).get() + self.vip.rpc.call(config.store, manage_get, ['identity', 'config_name', 'raw']).get() + self.vip.rpc.call(config.store, manage_get_metadata, ['identity', 'config_name']).get() + self.vip.rpc.call(config.store, manage_list_configs, ['identity']).get() + self.vip.rpc.call(config.store, manage_list_stores).get() + self.vip.rpc.call(config.store, manage_store, ['args', 'kwargs']).get() + self.vip.rpc.call(config.store, set_config, ['config_name', 'contents', 'trigger_callback', 'send_update']).get() + . + . + . + self.vip.rpc.call(platform.historian, get_aggregate_topics).get() + self.vip.rpc.call(platform.historian, get_topic_list).get() + self.vip.rpc.call(platform.historian, get_topics_by_pattern, ['topic_pattern']).get() + self.vip.rpc.call(platform.historian, get_topics_metadata, ['topics']).get() + self.vip.rpc.call(platform.historian, get_version).get() + self.vip.rpc.call(platform.historian, insert, ['records']).get() + self.vip.rpc.call(platform.historian, query, ['topic', 'start', 'end', 'agg_type', 'agg_period', 'skip', 'count', 'order']).get() + self.vip.rpc.call(volttron.central, get_publickey).get() + self.vip.rpc.call(volttron.central, is_registered, ['address_hash', 'address']).get() + +As with rpc list, the code subcommand can be filtered based on the vip identity and/or the method(s). + +.. code-block:: console + + vctl rpc code platform.historian + self.vip.rpc.call(platform.historian, get_aggregate_topics).get() + self.vip.rpc.call(platform.historian, get_topic_list).get() + self.vip.rpc.call(platform.historian, get_topics_by_pattern, ['topic_pattern']).get() + self.vip.rpc.call(platform.historian, get_topics_metadata, ['topics']).get() + self.vip.rpc.call(platform.historian, get_version).get() + self.vip.rpc.call(platform.historian, insert, ['records']).get() + self.vip.rpc.call(platform.historian, query, ['topic', 'start', 'end', 'agg_type', 'agg_period', 'skip', 'count', 'order']).get() + +.. code-block:: console + + vctl rpc code platform.historian query + self.vip.rpc.call(platform.historian, query, ['topic', 'start', 'end', 'agg_type', 'agg_period', 'skip', 'count', 'order']).get() + + +Implementation +-------------- + +See the `RPC module `_ +for implementation details. + +Also see :ref:`Multi-Platform RPC Communication ` and :ref:`RPC in RabbitMQ ` for +additional resources. diff --git a/docs/source/core_services/messagebus/VIP/VIP-Known-Identities.rst b/docs/source/platform-features/message-bus/vip/vip-known-identities.rst similarity index 97% rename from docs/source/core_services/messagebus/VIP/VIP-Known-Identities.rst rename to docs/source/platform-features/message-bus/vip/vip-known-identities.rst index 77c237e561..17b43a7e32 100644 --- a/docs/source/core_services/messagebus/VIP/VIP-Known-Identities.rst +++ b/docs/source/platform-features/message-bus/vip/vip-known-identities.rst @@ -1,12 +1,11 @@ -.. _VIP-Kwown-Identities: +.. _VIP-Known-Identities: +==================== VIP Known Identities ==================== -It is critical for systems to have known locations for receiving -resources and services from in a networked environment. The following -table details the vip identities that are reserved for VOLTTRON specific -usage. +It is critical for systems to have known locations for receiving resources and services from in a networked environment. +The following table details the vip identities that are reserved for VOLTTRON specific usage. +----------------------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | VIP Identity | Sphere of Influence | Notes | diff --git a/docs/source/core_services/messagebus/VIP/VIP-Overview.rst b/docs/source/platform-features/message-bus/vip/vip-overview.rst similarity index 65% rename from docs/source/core_services/messagebus/VIP/VIP-Overview.rst rename to docs/source/platform-features/message-bus/vip/vip-overview.rst index 32fc98f9bb..564886d3f4 100644 --- a/docs/source/core_services/messagebus/VIP/VIP-Overview.rst +++ b/docs/source/platform-features/message-bus/vip/vip-overview.rst @@ -1,9 +1,13 @@ .. _VIP-Overview: -VIP - VOLTTRON™ Interconnect Protocol -+++++++++++++++++++++++++++++++++++++ +=============================== +VOLTTRON™ Interconnect Protocol +=============================== -This document specifies VIP, the VOLTTRON™ Interconnect Protocol. The use case for VIP is to provide communications between *agents*, *controllers*, *services*, and the supervisory *platform* in an abstract fashion so that additional protocols can be built and used above VIP. VIP defines how *peers* connect to the *router* and the messages they exchange. +This document specifies VIP, the VOLTTRON™ Interconnect Protocol. The use case for VIP is to provide communications +between *agents*, *controllers*, *services*, and the supervisory *platform* in an abstract fashion so that additional +protocols can be built and used above VIP. VIP defines how *peers* connect to the *router* and the messages they +exchange. * Name: github.com/VOLTTRON/volttron/wiki/VOLTTRON-Interconnect-Protocol * Editor: Brandon Carpenter @@ -16,57 +20,65 @@ This document specifies VIP, the VOLTTRON™ Interconnect Protocol. The use case .. _ZAP: http://rfc.zeromq.org/spec:27/ZAP. -Preamble -======== +.. toctree:: + :caption: VIP Topics -Copyright 2019, Battelle Memorial Institute. + vip-json-rpc + vip-known-identities + vip-authentication + vip-authorization + vip-enhancements -Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at -http://www.apache.org/licenses/LICENSE-2.0 - -The patent license grant shall only be applicable to the following patent and patent application (Battelle IPID 17008-E), as assigned to the Battelle Memorial Institute, as used in conjunction with this Work: • US Patent No. 9,094,385, issued 7/28/15 • USPTO Patent App. No. 14/746,577, filed 6/22/15, published as US 2016-0006569. - -Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in `RFC 2119`_. - -.. _RFC 2119: http://tools.ietf.org/html/rfc2119 - - -Overall Design -============== +Design Overview +=============== What Problems does VIP Address? ------------------------------- -When VOLTTRON agents, controllers, or other entities needed to exchange data, they previously used the first generation pub/sub messaging mechanism and ad-hoc methods to set up direct connections. While the pub/sub messaging is easy to implement and use, it suffers from several limitations: +When VOLTTRON agents, controllers, or other entities needed to exchange data, they previously used the first generation +pub/sub messaging mechanism and ad-hoc methods to set up direct connections. While the pub/sub messaging is easy to +implement and use, it suffers from several limitations: * It requires opening two listening sockets: one each for publishing and subscribing. * There is no trivial way to prevent message spoofing. * There is no trivial way to enable private messaging * It is not ideal for peer-to-peer communications. -These limitations have severe security implications. For improved security in VOLTTRON, the communications protocol must provide a method for secure data exchange that is intuitive and simple to implement and use. +These limitations have severe security implications. For improved security in VOLTTRON, the communications protocol +must provide a method for secure data exchange that is intuitive and simple to implement and use. + +Many messaging platforms already provides many of the building blocks to implement encrypted and authenticated +communications over a shared socket. They include a socket type implementing the router pattern. What remains is a +protocol built on the ZeroMQ and/or RabbitMQ to provide a single connection point, secure message passing, and retain +the ability for entities to come and go as they please. -ZeroMQ already provides many of the building blocks to implement encrypted and authenticated communications over a shared socket. It already includes a socket type implementing the router pattern. What remains is a protocol built on ZeroMQ to provide a single connection point, secure message passing, and retain the ability for entities to come and go as they please. +VIP is VOLTTRON protocol implementation targeting the limitations above. -VIP is just that protocol, specifically targeting the limitations above. +ZeroMQ +====== Why ZeroMQ? ----------- -Rather than reinvent the wheel, VIP makes use of many features already implemented in ZeroMQ, including ZAP and CurveMQ. While VIP doesn't require the use of ZAP or CurveMQ, their use substantially improves security by encrypting traffic over public networks and limiting connections to authenticated peers. +Rather than reinvent the wheel, VIP makes use of many features already implemented in ZeroMQ, including ZAP and CurveMQ. +While VIP doesn't require the use of ZAP or CurveMQ, their use substantially improves security by encrypting traffic +over public networks and limiting connections to authenticated peers. -ZeroMQ also provides reliable transports with built-in framing, automatic reconnection, in-process zero-copy message passing, abstractions for underlying protocols, and so much more. While some of these features create other pain points, they are minimal compared with the effort of either reimplementing or cobbling together libraries. +ZeroMQ also provides reliable transports with built-in framing, automatic reconnection, in-process zero-copy message +passing, abstractions for underlying protocols, and so much more. While some of these features create other pain +points, they are minimal compared with the effort of either reimplementing or cobbling together libraries. VIP is a routing protocol ------------------------- -VIP uses the ZeroMQ router pattern. Specifically, the router binds a ROUTER socket and peers connect using a DEALER or ROUTER socket. Unless the peer is connecting a single socket to multiple routers, using the DEALER socket is easiest, but there are instances where using a ROUTER is more appropriate. One must just exercise care to include the proper address envelope to ensure proper routing. +VIP uses the ZeroMQ router pattern. Specifically, the router binds a ROUTER socket and peers connect using a DEALER or +ROUTER socket. Unless the peer is connecting a single socket to multiple routers, using the DEALER socket is easiest, +but there are instances where using a ROUTER is more appropriate. One must just exercise care to include the proper +address envelope to ensure proper routing. Extensible Security @@ -78,7 +90,8 @@ VIP makes no assumptions about the security mechanisms used. It works equally we ZeroMQ Compatibility -------------------- -For enhanced security, VOLTTRON recommends libzmq version 4.1 or greater, however, most features of VIP are available with older versions. The following is an incomplete list of core features available with recent versions of libzmq. +For enhanced security, VOLTTRON recommends libzmq version 4.1 or greater, however, most features of VIP are available +with older versions. The following is an incomplete list of core features available with recent versions of libzmq. * Version 3.2: @@ -97,7 +110,11 @@ For enhanced security, VOLTTRON recommends libzmq version 4.1 or greater, howeve Message Format and Version Detection ------------------------------------ -VIP uses a simple, multi-frame format for its messages. The first one (for peers) or two (for router) frames contain the delivery address(es) and are follow immediately by the VIP signature ``VIP1``. The first characters of the signature are used to match the protocol and the last character digit indicates the protocol version, which will be incremented as the protocol is revised. This allows for fail-fast behavior and backward compatibility while being simple to implement in any language supported by ZeroMQ. +VIP uses a simple, multi-frame format for its messages. The first one (for peers) or two (for router) frames contain +the delivery address(es) and are follow immediately by the VIP signature ``VIP1``. The first characters of the +signature are used to match the protocol and the last character digit indicates the protocol version, which will be +incremented as the protocol is revised. This allows for fail-fast behavior and backward compatibility while being +simple to implement in any language supported by ZeroMQ. Formal Specification @@ -107,7 +124,8 @@ Formal Specification Architecture ------------ -VIP defines a message-based dialog between a *router* that transfers data between *peers*. The *router* and *peers* SHALL communicate using the following socket types and transports: +VIP defines a message-based dialog between a *router* that transfers data between *peers*. The *router* and *peers* +SHALL communicate using the following socket types and transports: * The router SHALL use a ROUTER socket. * Peers SHALL use a DEALER or ROUTER socket. @@ -119,7 +137,8 @@ VIP defines a message-based dialog between a *router* that transfers data betwee Message Format -------------- -A routing exchange SHALL consist of a peer sending a message to the router followed by the router receiving the message and sending it to the destination peer. +A routing exchange SHALL consist of a peer sending a message to the router followed by the router receiving the message +and sending it to the destination peer. Messages sent to the router by peers SHALL consist of the following message frames: @@ -130,7 +149,11 @@ Messages sent to the router by peers SHALL consist of the following message fram * The *subsystem*, which SHALL contain a string. * The *data*, which SHALL be zero or more subsystem-specific opaque frames. -Messages received from a peer by the router will automatically have a *sender* frame prepended to the message by the ROUTER socket. When the router forwards the message, the sender and recipient fields are swapped so that the *recipient* is in the first frame and the *sender* is in the second frame. The *recipient* frame is automatically stripped by the ROUTER socket during delivery. Peers using ROUTER sockets must prepend the message with an *intermediary* frame, which SHALL contain the identity of a router socket. +Messages received from a peer by the router will automatically have a *sender* frame prepended to the message by the +ROUTER socket. When the router forwards the message, the sender and recipient fields are swapped so that the *recipient* +is in the first frame and the *sender* is in the second frame. The *recipient* frame is automatically stripped by the +ROUTER socket during delivery. Peers using ROUTER sockets must prepend the message with an *intermediary* frame, which +SHALL contain the identity of a router socket. Messages received from the router by peers SHALL consist of the following message frames: @@ -146,22 +169,33 @@ The various fields have these meanings: * sender: the ZeroMQ DEALER or ROUTER identity of the sending (source) peer. * recipient: the ZeroMQ DEALER or ROUTER identity of the recipient (destination) peer. * intermediary: the ZeroMQ ROUTER identity of the intermediary router. -* user id: VIP authentication metadata set in the authenticator. See the discussion below for more information on this value. -* request id: the meaning of this field is defined by the sending peer. Replies SHALL echo the request id without modifying it. -* subsystem: this specifies the peer subsystem the data is intended for. The length of a subsystem name SHALL NOT exceed 255 characters and MUST only contain ASCII characters. -* data: provides the data for the given subsystem. The number of frames required is defined by each subsystem. +* user id: VIP authentication metadata set in the authenticator. See the discussion below for more information on this + value. +* request id: the meaning of this field is defined by the sending peer. Replies SHALL echo the request id without + modifying it. +* subsystem: this specifies the peer subsystem the data is intended for. The length of a subsystem name SHALL NOT + exceed 255 characters and MUST only contain ASCII characters. +* data: provides the data for the given subsystem. The number of frames required is defined by each subsystem. User ID ------- -The value in the *user id* frame depends on the implementation and the version of ZeroMQ. If ZAP is used with libzmq 4.1.0 or newer, peers should send an empty string for the user id and the ZAP authenticator will replace it with an authentication token which receiving peers may use to authorize access. If ZAP is not used or a version of libzmq is used which lacks support for retrieving the user id metadata, an authentication subsystem may be used to authenticate peers. The authentication subsystem SHALL provide peers with private tokens that must be sent with each message in the user id frame and which the router will substitute with a public token before forwarding. If the message cannot be authenticated, the user id received by peers SHALL be a zero-length string. +The value in the *user id* frame depends on the implementation and the version of ZeroMQ. If `ZAP` is used with libzmq +4.1.0 or newer, peers should send an empty string for the user id and the ZAP authenticator will replace it with an +authentication token which receiving peers may use to authorize access. If ZAP is not used or a version of libzmq is +used which lacks support for retrieving the user id metadata, an authentication subsystem may be used to authenticate +peers. The authentication subsystem SHALL provide peers with private tokens that must be sent with each message in the +user id frame and which the router will substitute with a public token before forwarding. If the message cannot be +authenticated, the user id received by peers SHALL be a zero-length string. Socket Types ------------ -Peers communicating via the router will typically use DEALER sockets and should not require additional handling. However, a DEALER peer may only connect to a single router. Peers may use ROUTER sockets to connect to multiple endpoints, but must prepend the routing ID of the destination. +Peers communicating via the router will typically use DEALER sockets and should not require additional handling. +However, a DEALER peer may only connect to a single router. Peers may use ROUTER sockets to connect to multiple +endpoints, but must prepend the routing ID of the destination. When using a DEALER socket: @@ -177,12 +211,14 @@ When using a ROUTER socket: Routing Identities ------------------ -Routing identities are set on a socket using the ZMQ_IDENTITY socket option and MUST be set on both ROUTER and DEALER sockets. The following additional requirements are placed on the use of peer identities: +Routing identities are set on a socket using the ZMQ_IDENTITY socket option and MUST be set on both ROUTER and DEALER +sockets. The following additional requirements are placed on the use of peer identities: * Peers SHALL set a valid identity rather than rely on automatic identity generation. * The router MAY drop messages with automatically generated identities, which begin with the zero byte ('\0'). -A zero length identity is invalid for peers and is, therefore, unroutable. It is used instead to address the router itself. +A zero length identity is invalid for peers and is, therefore, unroutable. It is used instead to address the router +itself. * Peers SHALL use a zero length recipient to address the router. * Messages sent from the router SHALL have a zero length sender address. @@ -191,7 +227,10 @@ A zero length identity is invalid for peers and is, therefore, unroutable. It is Error Handling ============== -The documented default behavior of ZeroMQ ROUTER sockets when entering the mute state (when the send buffer is full) is to silently discard messages without blocking. This behavior, however, is not consistently observed. Quietly discarding messages is not the desired behavior anyway because it prevents peers from taking appropriate action to the error condition. +The documented default behavior of ZeroMQ ROUTER sockets when entering the mute state (when the send buffer is full) is +to silently discard messages without blocking. This behavior, however, is not consistently observed. Quietly discarding +messages is not the desired behavior anyway because it prevents peers from taking appropriate action to the error +condition. * Routers SHALL set the ZMQ_SNDTIMEO socket option to 0. * Routers SHALL forward EAGAIN errors to sending peers. @@ -222,7 +261,9 @@ An error message must contain the following: Subsystems ========== -Peers may support any number of communications protocols or subsystems. For instance, there may be a remote procedure call (RPC) subsystem which defines its own protocol. These subsystems are outside the scope of VIP and this document with the exception of the *hello* and *ping* subsystems. +Peers may support any number of communications protocols or subsystems. For instance, there may be a remote procedure +call (RPC) subsystem which defines its own protocol. These subsystems are outside the scope of VIP and this document +with the exception of the *hello* and *ping* subsystems. * A router SHALL implement the hello subsystem. * All peers and routers SHALL implement the ping subsystem. @@ -231,7 +272,8 @@ Peers may support any number of communications protocols or subsystems. For inst The hello Subsystem ------------------- -The hello subsystem provides one simple RPC-style routine for peers to probe the router for version and identity information. +The hello subsystem provides one simple RPC-style routine for peers to probe the router for version and identity +information. A peer hello request message must contain the following: @@ -258,13 +300,16 @@ The hello subsystem can help a peer with the following tasks: * Discover the identity of the peer. * Discover authentication metadata. -For instance, if a peer will use a ROUTER socket for its connections, it must first know the identity of the router. The peer might first connect with a DEALER socket, issue a hello, and use the returned identity to then connect the ROUTER socket. +For instance, if a peer will use a ROUTER socket for its connections, it must first know the identity of the router. +The peer might first connect with a DEALER socket, issue a hello, and use the returned identity to then connect the +ROUTER socket. The ping Subsystem ------------------ -The *ping* subsystem is useful for testing the presence of a peer and the integrity and latency of the connection. All endpoints, including the router, must support the ping subsystem. +The *ping* subsystem is useful for testing the presence of a peer and the integrity and latency of the connection. +All endpoints, including the router, must support the ping subsystem. A peer ping request message must contain the following: @@ -282,19 +327,24 @@ A ping response message must contain the following: * The first data frame SHALL be the 4 octets 'pong'. * The remaining data frames SHALL be copied from the ping request unchanged, starting with the second data frame. -Any data can be included in the ping and should be returned unchanged in the pong, but limited trust should be placed in that data as it is possible a peer might modify it against the direction of this specification. +Any data can be included in the ping and should be returned unchanged in the pong, but limited trust should be placed in +that data as it is possible a peer might modify it against the direction of this specification. Discovery --------- -VIP does not define how to discover peers or routers. Typical options might be to hard code the router address in peers or to pass it in via the peer configuration. A well known (i.e. statically named) directory service might be used to register connected peers and allow for discovery by other peers. +VIP does not define how to discover peers or routers. Typical options might be to hard code the router address in peers +or to pass it in via the peer configuration. A well known (i.e. statically named) directory service might be used to +register connected peers and allow for discovery by other peers. Example Exchanges ================= -These examples show the messages *as sent on the wire* as sent or received by peers using DEALER sockets. The messages received or sent by peers or routers using ROUTER sockets will have an additional address at the start. We do not show the frame sizes or flags, only frame contents. +These examples show the messages *as sent on the wire* as sent or received by peers using DEALER sockets. The messages +received or sent by peers or routers using ROUTER sockets will have an additional address at the start. We do not show +the frame sizes or flags, only frame contents. Example of hello Request @@ -318,7 +368,8 @@ This shows a hello request sent by a peer, with identity "alice", to a connected | hello | Operation, "hello" in this case +-------+ -This example assumes a DEALER socket. If a peer uses a ROUTER socket, it SHALL prepend an additional frame containing the router identity, similar to the following example. +This example assumes a DEALER socket. If a peer uses a ROUTER socket, it SHALL prepend an additional frame containing +the router identity, similar to the following example. This shows the example request received by the router: @@ -478,7 +529,8 @@ This shows the example request received by "bob": | 1422573492 | Data, a single frame in this case (Unix timestamp) +------------+ -If "bob" were using a ROUTER socket, there would be an additional frame prepended to the message containing the router identity, "router" in this case. +If "bob" were using a ROUTER socket, there would be an additional frame prepended to the message containing the router +identity, "router" in this case. This shows an example reply from "bob" to "alice" diff --git a/docs/source/platform-features/security/key-stores.rst b/docs/source/platform-features/security/key-stores.rst new file mode 100644 index 0000000000..a556fdb5d2 --- /dev/null +++ b/docs/source/platform-features/security/key-stores.rst @@ -0,0 +1,45 @@ +.. _Key-Stores: + +========== +Key Stores +========== + +.. warning:: + + Most VOLTTRON users should not need to directly interact with agent key stores. These are notes for VOLTTRON + platform developers. This is not a stable interface and the implementation details are subject to change. + +Each agent has its own encryption key-pair that is used to :ref:`authenticate` itself with the +VOLTTRON platform. A key-pair comprises a public key and a private (secret) key. These keys are saved in a +"key store", which is implemented by the :py:class:`KeyStore class`. Each agent +has its own key store. + +Key Store Locations +------------------- + +There are two main locations key stores will be saved. Installed agents' key stores are in the the agent's data +directory: + +.. code-block:: bash + + $VOLTTRON_HOME/agents///keystore.json + +Agents that are not installed, such as platform services and stand-alone agents, store their key stores here: + +.. code-block:: bash + + $VOLTTRON_HOME/keystores//keystore.json + + +Generating a Key Store +---------------------- + +Agents automatically retrieve keys from their key store unless both the ``publickey`` and ``secretkey`` parameters are +specified when the agent is initialized. If an agent's key store does not exist it will automatically be generated upon +access. + +Users can generate a key pair by running the following command: + +.. code-block:: bash + + vctl auth keypair diff --git a/docs/source/platform-features/security/known-hosts-file.rst b/docs/source/platform-features/security/known-hosts-file.rst new file mode 100644 index 0000000000..943b93d2c2 --- /dev/null +++ b/docs/source/platform-features/security/known-hosts-file.rst @@ -0,0 +1,71 @@ +.. _Known-Hosts-File: + +================ +Known Hosts File +================ + +Before an agent can connect to a VOLTTRON platform that agent must know the platform's VIP address and public key (known +as the `server key`). It can be tedious to manually keep track of server keys and match them with their corresponding +addresses. + +The purpose of the known-hosts file is to save a mapping of platform addresses to server keys. This way the user only +has to specify a server key one time. + + +Saving a Server Key +------------------- + +Suppose a user wants to connect to a platform at ``192.168.0.42:22916``, and the platform's public key is +``uhjbCUm3kT5QWj5Py9w0XZ7c1p6EP8pdo4Hq4dNEIiQ``. To save this address-to-server-key association, the user can run: + +.. code-block:: bash + + volttron-ctl auth add-known-host --host 192.168.0.42:22916 --serverkey uhjbCUm3kT5QWj5Py9w0XZ7c1p6EP8pdo4Hq4dNEIiQ + +Now agents on this system will automatically use the correct server key when connecting to the platform at +``192.168.0.42:22916``. + + +Server Key for Local Platforms +------------------------------ + +When a platform starts it automatically adds its public key to the known-hosts file. Thus agents connecting to the +local VOLTTRON platform (on the same system and using the same ``$VOLTTRON_HOME``) will automatically be able to +retrieve the platform's public key. + + +Know-Host-File Details +---------------------- + +.. note:: + + The following details regarding the known-hosts file are subject to change. These notes are primarily for + developers, but the may be helpful if troubleshooting an issue. **The known-hosts file should not be edited + directly.** + +File Location +^^^^^^^^^^^^^ + +The known-hosts-file is stored at ``$VOLTTRON_HOME/known_hosts``. + + +File Contents +^^^^^^^^^^^^^ + +Here are the contents of an example known-hosts file: + +.. code:: JSON + + { + "@": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", + "127.0.0.1:22916": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", + "127.0.0.2:22916": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", + "127.0.0.1:12345": "FSG7LHhy3v8tdNz3gK35G6-oxUcyln54pYRKu5fBJzU", + "192.168.0.42:22916": "uhjbCUm3kT5QWj5Py9w0XZ7c1p6EP8pdo4Hq4dNEIiQ" + } + +The first four entries are for the local platform. (They were automatically added when the platform started.) The first +entry with the ``@`` key is for IPC connections, and the entries with the ``127.0.0.*`` keys are for local TCP +connections. Note that a single VOLTTRON platform can bind to multiple TCP addresses, and each address will be +automatically added to the known-hosts file. The last entry is for a remote VOLTTRON platform. (It was added in the +`Saving a Server Key`_ section.) diff --git a/docs/source/platform-features/security/protecting-pubsub-topics.rst b/docs/source/platform-features/security/protecting-pubsub-topics.rst new file mode 100644 index 0000000000..567bf0a099 --- /dev/null +++ b/docs/source/platform-features/security/protecting-pubsub-topics.rst @@ -0,0 +1,66 @@ +.. _Protected-Topics: + +========================= +Protecting Pub/Sub Topics +========================= + +VIP :ref:`authorization ` enables VOLTTRON platform owners to protect pub/sub topics. More +specifically, a platform owner can limit who can publish to a given topic. This protects subscribers on that platform +from receiving messages (on the protected topic) from unauthorized agents. + + +Example +------- + +To protect a topic, add the topic name to ``$VOLTTRON_HOME/protected_topics.json``. For example, the following +protected-topics file declares that the topic ``foo`` is protected: + +.. code:: JSON + + { + "write-protect": [ + {"topic": "foo", "capabilities": ["can_publish_to_foo"]} + ] + } + +.. note:: + + The capability name ``can_publish_to_foo`` is not special; It can be any string, but it is easier to manage + capabilities with meaningful names. + +Now only agents with the capability ``can_publish_to_foo`` can publish to the topic ``foo``. To add this capability to +authenticated agents, run ``vctl auth update`` (or ``volttron-ctl auth add`` for new authentication entries), and enter +``can_publish_to_foo`` in the capabilities field: + +.. code:: Bash + + capabilities (delimit multiple entries with comma) []: can_publish_to_foo + +Agents that have the ``can_publish_to_foo`` capabilities can publish to topic ``foo``. That is, such agents can call: + +.. code:: Python + + self.vip.pubsub.publish('pubsub', 'foo', message='Here is a message') + +If unauthorized agents try to publish to topic ``foo`` they will get an exception: + +.. code-block:: console + + to publish to topic "foo" requires capabilities ['can_publish_to_foo'], but capability list [] was provided + + +Regular Expressions +------------------- + +Topic names in ``$VOLTTRON_HOME/protected_topics.json`` can be specified as regular expressions. In order to use a +regular expression, the topic name must begin and end with a "/". For example: + +.. code:: JSON + + { + "write-protect": [ + {"topic": "/foo/*.*/", "capabilities": ["can_publish_to_foo"]} + ] + } + +This protects topics such as ``foo/bar`` and ``foo/anything``. diff --git a/docs/source/platform-features/security/running-agent-as-user.rst b/docs/source/platform-features/security/running-agent-as-user.rst new file mode 100644 index 0000000000..e82efb339f --- /dev/null +++ b/docs/source/platform-features/security/running-agent-as-user.rst @@ -0,0 +1,118 @@ +.. _Running-Agents-as-Unix-User: + +============================ +Running Agents as Unix Users +============================ + +This VOLTTRON feature will cause the platform to create a new, unique Unix user(agent users) on the host machine for +each agent installed on the platform. This user will have restricted permissions for the file system, and will be used +to run the agent process. + +.. warning:: + + The Unix user starting the VOLTTRON platform will be given limited sudo access to create and delete agent users. + +Since this feature requires system level changes (e.g. sudo access, user creation, file permission changes), the initial +step needs to be run as root or user with `sudo` access. This can be a user other than Unix user used to run the +VOLTTRON platform. + +All files and folder created by the VOLTTRON process in this mode would not have any access to others by default. +Permission for Unix group others would be provided to specific files and folder based on VOLTTRON process requirement. + +It is recommended that you use a new `VOLTTRON_HOME` to run VOLTTRON in secure mode. Converting a existing VOLTTRON +instance to secure mode is also possible but would involve some manual changes. Please see the section +`Porting existing volttron home to secure mode`_. + +.. note:: + + VOLTTRON has to be bootstrapped as prerequisite to running agents as unique users. + + +Setup agents to run using unique users +====================================== + +1. **This feature requires acl to be installed.** + + Make sure the `acl` library is installed. If you are running on a Docker image `acl` might not be installed by + default. + + .. code-block:: bash + + apt-get install acl + +2. Agents now run as a user different from VOLTTRON platform user. Agent users should have `read` and `execute` + permissions to all directories in the path to the Python executable used by VOLTTRON. For example, if VOLTTRON is + using a virtual environment, then agent users should have *read* permissions to `/bin/python` and *read + and execute* permission to all the directories in the path `/bin`. This can be achieved by running: + + .. code-block:: bash + + chmod -R o+rx /bin + +3. **Run scripts/secure_user_permissions.sh as root or using sudo** + + This script *MUST* be run as root or using `sudo`. This script gives the VOLTTRON platform user limited `sudo` + access to create a new Unix user for each agent. All users created will be of the format `volttron_`. + + This script prompts for: + + a. **volttron platform user** - Unix user who would be running the VOLTTRON platform. This should be an existing + Unix user. On a development machine this could be the Unix user you logged in as to check out VOLTTRON source + + b. **VOLTTRON_HOME directory** - The absolute path of the volttron home directory. + + c. **Volttron instance name if VOLTTRON_HOME/config does not exist** - + + If the `VOLTTRON_HOME/config` file exists then instance name is obtained from that config file. If not, the user + will be prompted for an instance name. `volttron_` *MUST* be a 23 characters or shorter containing + only characters valid as Unix user names. + + This script will create necessary entries in `/etc/sudoers.d/volttron` to allow the VOLTTRON platform user to create + and delete agent users, the VOLTTRON agent group, and run any non-sudo command as the agent users. + + This script will also create `VOLTTRON_HOME` and the config file if given a new VOLTTRON home directory when + prompted. + +4. **Continue with VOLTTRON bootstrap and setup as normal** - point to the `VOLTTRON_HOME` that you provided in step 2. + +5. **On agent install (or agent start for existing agents)** - a unique agent user(Unix user) is created and the agent + is started as this user. The agent user name is recorded in `USER_ID` file under the agent install directory + (`VOLTTRON_HOME/agents//USER_ID`). Subsequent agent restarts will read the content of the `USER_ID` file + and start the agent process as that user. + +6. **On agent uninstall** - The agent user is deleted and the agent install directory is deleted. + + +Creating new Agents +=================== + +In this secure mode, agents will only have read write access to the agent-data directory under the agent install +directory - `VOLTTRON_HOME/agents///.agent-data`. Attempting to write in any other +folder under `VOLTTRON_HOME` **will result in permission errors**. + + +Changes to existing agents in secure mode +========================================= + +Due to the above change, **SQL historian has been modified to create its database by default under its agent-data +directory** if no path is given in the config file. If providing a path to the database in the config file, please +provide a directory where agent will have write access. This can be an external directory for which agent user +(`recorded in VOLTTRON_HOME/agents//USER_ID`) has *read, write, and execute* access. + + +Porting existing VOLTTRON home to secure mode +============================================= + +When running `scripts/secure_users_permissions.sh` you will be prompted for a `VOLTTRON_HOME` directory. If this +directory exists and contains a volttron config file, the script will update the file locations and permissions of +existing VOLTTRON files including installed directories. However this step has the following limitations: + +#. **You will NOT be able to revert to insecure mode once the changes are done.** - Once setup is complete, changing the + config file manually to make parameter `secure-agent-users` to `False`, may result inconsistent VOLTTRON behavior +#. The VOLTTRON process and all agents have to be restarted to take effect +#. **Agents can only to write to its own agent-data dir.** - If your agents writes to any directory outside + `$VOLTTRON_HOME/agents///agent-name.agent-data` move existing files and update the agent + configuration such that the agent writes to the `agent-name.agent-data` dir. For example, if you have a + `SQLHistorian` which writes a `.sqlite` file to a subdirectory under `VOLTTRON_HOME` that is not + `$VOLTTRON_HOME/agents///agent-name.agent-data` this needs to be manually updated. + diff --git a/docs/source/platform-features/security/volttron-security.rst b/docs/source/platform-features/security/volttron-security.rst new file mode 100644 index 0000000000..4b60934c94 --- /dev/null +++ b/docs/source/platform-features/security/volttron-security.rst @@ -0,0 +1,33 @@ +.. _VOLTTRON-Security: + +================= +Platform Security +================= + +There are various security-related topics throughout VOLTTRON's documentation. This is a quick roadmap for finding +security documentation. + +A core component of VOLTTRON is its :ref:`message bus`. The security of this message bus is +crucial to the entire system. The :ref:`VOLTTRON Interconnect Protocol` provides communication over the +message bus. + +VIP was built with security in mind from the ground up. VIP uses encrypted channels and enforces agent +:ref:`authentication` by default for all network communication. VIP's +:ref:`authorization` mechanism allows system administrators to limit agent capabilities with fine +granularity. + +Even with these security mechanisms built into VOLTTRON, it is important for system administrators to +:ref:`harden VOLTTRON's underlying OS `. + +The VOLTTRON team has engaged with PNNL's Secure Software Central team to create a threat profile document. You can +read about the threat assessment findings and how the VOLTTRON team is addressing them here: `SSC Threat Profile +`_ + +Additional documentation related to VIP authentication and authorization is available here: + +.. toctree:: + + protecting-pubsub-topics + running-agent-as-user + key-stores + known-hosts-file diff --git a/docs/source/services.rst b/docs/source/services.rst deleted file mode 100644 index ce633fc475..0000000000 --- a/docs/source/services.rst +++ /dev/null @@ -1,9 +0,0 @@ -Services -======== - -.. toctree:: - :maxdepth: 4 - :glob: - - apidocs/services/*/modules - diff --git a/docs/source/setup/RabbitMQ/Backward_Compatibility.rst b/docs/source/setup/RabbitMQ/Backward_Compatibility.rst deleted file mode 100644 index 86e493d6b1..0000000000 --- a/docs/source/setup/RabbitMQ/Backward_Compatibility.rst +++ /dev/null @@ -1,257 +0,0 @@ -.. RMQ-Backward-Compatability: - -Backward Compatibility With ZeroMQ Message Based VOLTTRON -========================================================= - -RabbitMQ VOLTTRON supports backward compatibility with ZeroMQ based VOLTTRON. RabbitMQ VOLTTRON has a ZeroMQ router running internally to accept incoming ZeroMQ connections and to route ZeroMQ messages coming in/going out of it's instance. There are multiple ways for an instance with a RabbitMQ message bus, and an instance with ZeroMQ message bus to connect with each other. For example, an agent from one instance can directly connect to the remote instance to publish or pull data from it. Another way is through multi-platform communication, where the VOLTTRON platform is responsible for connecting to the remote instance. For more information on multi-platform communication, see https://volttron.readthedocs.io/en/develop/core_services/multiplatform/Multiplatform-Communication.html. - - -Agent Connecting Directly to Remote Instance --------------------------------------------- - -The following steps are to demonstrate how RabbitMQ VOLTTRON is backward compatible with ZeroMQ VOLTTRON, using the Forward Historian as an example. This example shows how to forward messages from local ZeroMQ based VOLTTRON to remote RabbitMQ based VOLTTRON instance. Similar steps can be followed if you needed to move messages from local RabbiMQ based VOLTTRON to ZeroMQ based VOLTTRON. - -1. In order for RabbitMQ and ZeroMQ VOLTTRONs to communicate with each other, one needs two instances of VOLTTRON_HOME on the same VM. To create a new instance of VOLTTRON_HOME use the command. - - ``export VOLTTRON_HOME=~/.new_volttron_home`` - - It is recommended that one uses multiple terminals to keep track of both instances. - -2. Start VOLTTRON on both instances. Note: since the start-volttron script uses the volttron.log by default, the second instance will need be started manually in the background, using a separate log. For example: - - ``volttron -vv -l volttron-two.log&`` - -3. Modify the configuration file for both instances. The config file is located at ``$VOLTTRON_HOME/config`` - - For RabbitMQ VOLTTRON, the config file should look similar to: - - .. code-block:: bash - - [volttron] - message-bus = rmq - vip-address = tcp://127.0.0.1:22916 - instance-name = volttron_rmq - - The ZeroMQ config file should look similar, with all references to RMQ being replaced with ZMQ, and a different vip-address - (e.g. tcp://127.0.0.2:22916). - -4. On the instance running ZeroMQ: - - a. Install the Forward Historian agent using an upgrade script similar to: - - .. code-block:: python - - #!/bin/bash - export CONFIG=$(mktemp /tmp/abc-script.XXXXXX) - cat > $CONFIG < $CONFIG <": {"vip-address":"tcp://:", - "instance-name":"", - "serverkey":"" - }, - "": {"vip-address":"tcp://:", - "instance-name":"", - "serverkey":"" - }, - "": {"vip-address":"tcp://:", - "instance-name":"", - "serverkey":"" - }, - ...... - } - - -Additionally for different combinations of multi-bus, multi-platform setup, please refer to :ref:`Multi-Platform Multi-Bus Walk-through <_Multi_Platform_Walkthrough>` diff --git a/docs/source/setup/RabbitMQ/Monitoring_RMQ.rst b/docs/source/setup/RabbitMQ/Monitoring_RMQ.rst deleted file mode 100644 index 81957ff694..0000000000 --- a/docs/source/setup/RabbitMQ/Monitoring_RMQ.rst +++ /dev/null @@ -1,50 +0,0 @@ -.. _Monitoring-RMQ: - -Monitoring and Controlling RabbitMQ -=================================== - -Some of the important native RabbitMQ control and management commands are now -integrated with "volttron-ctl" utility. Using volttron-ctl rabbitmq management -utility, we can control and monitor the status of RabbitMQ message bus. - -.. code-block:: bash - - vctl rabbitmq --help - usage: vctl command [OPTIONS] ... rabbitmq [-h] [-c FILE] [--debug] - [-t SECS] - [--msgdebug MSGDEBUG] - [--vip-address ZMQADDR] - ... - subcommands: - - add-vhost add a new virtual host - add-user Add a new user. User will have admin privileges - i.e,configure, read and write - add-exchange add a new exchange - add-queue add a new queue - list-vhosts List virtual hosts - list-users List users - list-user-properties - List users - list-exchanges add a new user - list-exchange-properties - list exchanges with properties - list-queues list all queues - list-queue-properties - list queues with properties - list-bindings list all bindings with exchange - list-federation-parameters - list all federation parameters - list-shovel-parameters - list all shovel parameters - list-policies list all policies - remove-vhosts Remove virtual host/s - remove-users Remove virtual user/s - remove-exchanges Remove exchange/s - remove-queues Remove queue/s - remove-federation-parameters - Remove federation parameter - remove-shovel-parameters - Remove shovel parameter - remove-policies Remove policy - diff --git a/docs/source/setup/RabbitMQ/RMQ_Multiplatform_Setup.rst b/docs/source/setup/RabbitMQ/RMQ_Multiplatform_Setup.rst deleted file mode 100644 index 9a8f0a469b..0000000000 --- a/docs/source/setup/RabbitMQ/RMQ_Multiplatform_Setup.rst +++ /dev/null @@ -1,25 +0,0 @@ -.. _RMQ-Multi-Platform-Setup: - -Multi-Platform Deployment With RabbitMQ Message bus -=================================================== - -In ZeroMQ based VOLTTRON, if multiple instances needed to be connected together -and be able to send or receive messages to/from remote instances we would do it -in few different ways. - -1. Write an agent that would connect to remote instance directly and publish/subscribe to messages or - perform RPC communication directly. This is described in - :ref:`Agent connection to remote volttron instance ` - - -2. Use special agents such as forwarder/data puller agents to forward/receive - messages to/from remote instances. This can be achieved using RabbitMQ's shovel plugin and is described at - :ref:`Using Shovel Plug-in` - - -3. Configure vip address of all remote instances that an instance has to connect to - in it's $VOLTTRON_HOME/external_discovery.json and let the router module in each instance - manage the connection and take care of the message routing for us. - This is the most seamless way to do multi-platform communication. This can be achieved using RabbitMQ's federation - plugin. Setup for this is described at :ref:`Using Federation Plug-in` - diff --git a/docs/source/setup/RabbitMQ/Troubleshooting_RMQ.rst b/docs/source/setup/RabbitMQ/Troubleshooting_RMQ.rst deleted file mode 100644 index dc21a8afea..0000000000 --- a/docs/source/setup/RabbitMQ/Troubleshooting_RMQ.rst +++ /dev/null @@ -1,93 +0,0 @@ -.. _Troubleshooting-RMQ: - - -RabbitMQ Trouble Shooting -========================= - -Check the status of the federation connection ----------------------------------------------- - - .. code-block:: bash - - $RABBITMQ_HOME/sbin/rabbitmqctl eval 'rabbit_federation_status:status().' - -If everything is properly configured, then the status is set to "running". If not look for the error status. Some of the typical errors are, - -a. "failed_to_connect_using_provided_uris" - Check if RabbitMQ user is created in downstream server node. Refer to step 3b of federation setup - - -b. "unknown ca" - Check if the root CAs are copied to all the nodes correctly. Refer to step 2 of federation setup - - -c. "no_suitable_auth_mechanism" - Check if the AMPQ/S ports are correctly configured. - - -Check the status of the shovel connection ------------------------------------------- - - .. code-block:: bash - - RABBITMQ_HOME/sbin/rabbitmqctl eval 'rabbit_shovel_status:status().' - -If everything is properly configured, then the status is set to "running". -If not look for the error status. Some of the typical errors are, - -a. "failed_to_connect_using_provided_uris" - Check if RabbitMQ user is created in subscriber node. Refer to step 3 b of shovel setup - -b. "unknown ca" - Check if the root CAs are copied to remote servers correctly. Refer to step 2 of shovel setup - -c. "no_suitable_auth_mechanism" - Check if the AMPQ/S ports are correctly configured. - - -Check the RabbitMQ logs for any errors ---------------------------------------- - - .. code-block:: bash - - tail -f /rabbitmq.log - - - -Rabbitmq startup hangs ----------------------- - -a. Check for errors in rabbitmq log. There is a rabbitmq.log file in your - volttron source directory that is a symbolic link to the rabbitmq server - logs. - -b. Check for errors in syslog (/var/log/syslog or /var/log/messages) - -c. If there are no errors in either of the logs, stop rabbitmq and - starting rabbitmq server in foreground and see if there are any errors - written on the console. Once you find the error you can kill the - process by entering Ctl+C, fix the error and start rabbitmq again using - ./start-rabbitmq from volttron source directory. - - .. code-block:: bash - - ./stop-volttron - ./stop-rabbitmq - @RABBITMQ_HOME/sbin/rabbitmq-server - - -SSL trouble shooting --------------------- -There are few things that are essential for SSL certificates to work right. - -a. Please use a unique common-name for CA certificate for each volttroninstance. This is configured under certificate-data - in the rabbitmq_config.yml or if no yml file is used while configuring a volttron single instance - (using vcfg --rabbitmq single). Certificate generated for agent will automatically get agent's vip identity as the - certificate's common-name - -b. host name in ssl certificate should match hostname used to access the server. For example, if the fully qualified domain - name was configured in the certificate-data, you should use the fully qualified domain name to access - rabbitmq's management url. - -c. Check if your system time is correct especially if you are running virtual machines. If the system clock is not right, it could lead to ssl certificate errors - -DataMover troubleshooting -------------------------- - -If output from volttron.log is not as expected check for ``{'alert_key': 'historian_not_publishing'}`` in the callee node's volttron.log. Most likely cause is the historian is not running properly or credentials between caller and callee nodes was not set properly. - - diff --git a/docs/source/setup/RabbitMQ/federation_plugin.rst b/docs/source/setup/RabbitMQ/federation_plugin.rst deleted file mode 100644 index 3d8480cede..0000000000 --- a/docs/source/setup/RabbitMQ/federation_plugin.rst +++ /dev/null @@ -1,144 +0,0 @@ -.. _federation-plugin: - -RabbitMQ Multi-Platform Deployment Using Federation Plugin -========================================================== - -Federation pluggin allows us to send and receive messages to/from remote instances with -few simple connection settings. Once a federation link is established to remote instance, -the messages published on the remote instance become available to local instance as if it -were published on the local instance. Before, we illustrate the steps to setup a federation -link, let us start by defining the concept of upstream and downstream server. - -**Upstream Server** - The node that is publishing some message of interest - -**DownStream Server** - The node that wants to receive messages from the upstream server - -A federation link needs to be established from downstream server to the upstream server. The -data flows in single direction from upstream server to downstream server. For bi-directional -data flow we would need to create federation links on both the nodes. - -1. Setup two VOLTTRON instances using the instructions at :ref:`RMQ Setup`. **Please note that each instance should have a unique instance name and should be running on machine/VM that has a unique host name.** - -2. In a multi platform setup that need to communicate with each other with RabbitMQ over SSL, each VOLTTRON instance should should trust the ROOT CA of the other instance(RabbitMQ root ca) - - a. Transfer (scp/sftp/similar) voltttron_home/certificates/certs/-root-ca.crt to a temporary - location on the other volttron instance machine. For example, if you have two instance v1 and v2, - scp v1's v1-root-ca.crt to v2 and v2-root-ca.crt to v1. - - Note: If using VMs, in order to scp files between VM openssh should be installed and running. - - b. Append the contents of the transferred root ca to the instance's trusted-cas.crt file. Do this on both the instances. Now both - the instances -trusted-cas.crt will have two certificates. - - For example: - - On v1: - cat /tmp/v2-root-ca.crt >> VOLTTRON_HOME/certificates/certs/v1-trusted-cas.crt - - On v2: - cat /tmp/v1-root-ca.crt >> VOLTTRON_HOME/certificates/certs/v2-trusted-cas.crt - -3. Stop volttron, stop rabbitmq server and start volttron on both the -instances. This is required only when you update the root certificate and not -required when you add a new shovel/federation between the same hosts - -.. code-block:: bash - - ./stop-volttron - ./stop-rabbitmq - ./start-volttron - -4. Identify upstream servers (publisher nodes) and downstream servers -(collector nodes). To create a RabbitMQ federation, we have to configure -upstream servers on the downstream server and make the VOLTTRON exchange -"federated". - - a. On the downstream server (collector node) - - .. code-block:: bash - - vcfg --rabbitmq federation [optional path to rabbitmq_federation_config.yml - containing the details of the upstream hostname, port and vhost. - - - Example configuration for federation is available - in examples/configurations/rabbitmq/rabbitmq_federation_config.yml] - - - If no config file is provided, the script will prompt for - hostname (or IP address), port, and vhost of each upstream node you - would like to add. Hostname provided should match the hostname in the - SSL certificate of the upstream server. For bi-directional data flow, - we will have to run the same script on both the nodes. - - b. Create a user in the upstream server(publisher) with - username= (i.e. (instance-name)-admin) and - provide it access to the virtual host of the upstream RabbitMQ server. Run - the below command in the upstream server - - .. code-block:: bash - - vctl rabbitmq add-user - Do you want to set READ permission [Y/n] - Do you want to set WRITE permission [Y/n] - Do you want to set CONFIGURE permission [Y/n] - -5. Test the federation setup. - - a. On the downstream server run a listener agent which subscribes to messages from all platforms - - - Open the file examples/ListenerAgent/listener/agent.py. Search for @PubSub.subscribe('pubsub', '') and replace that line with @PubSub.subscribe('pubsub', 'devices', all_platforms=True) - - updgrade the listener - - .. code-block:: bash - - scripts/core/upgrade-listener - - - b. Install master driver, configure fake device on upstream server and start volttron and master driver. vcfg --agent master_driver command can install master driver and setup a fake device. - - .. code-block:: bash - - ./stop-volttron - vcfg --agent master_driver - ./start-volttron - vctl start --tag master_driver - - - c. Verify listener agent in downstream VOLTTRON instance is able to receive the messages. downstream volttron instance's volttron.log should display device data scrapped by master driver agent in upstream volttron instance - -6. Open ports and https service if needed - On Redhat based systems ports used by RabbitMQ (defaults to 5671, 15671 for - SSL, 5672 and 15672 otherwise) might not be open by default. Please - contact system administrator to get ports opened on the downstream server. - - Following are commands used on centos 7. - - .. code-block:: bash - - sudo firewall-cmd --zone=public --add-port=15671/tcp --permanent - sudo firewall-cmd --zone=public --add-port=5671/tcp --permanent - sudo firewall-cmd --reload - -7. How to remove federation link - - a. Using the management web interface - - Log into management web interface using downstream server's admin username. - Navigate to admin tab and then to federation management page. The status of the - upstream link will be displayed on the page. Click on the upstream link name and - delete it. - - b. Using "vctl" command on the upstream server. - - .. code-block:: bash - - vctl rabbitmq list-federation-parameters - NAME URI - upstream-volttron2-rabbit-2 amqps://rabbit-2:5671/volttron2?cacertfile=/home/nidd494/.volttron1/certificates/certs/volttron1-root-ca.crt&certfile=/home/nidd494/.volttron1/certificates/certs/volttron1-admin.crt&keyfile=/home/nidd494/.volttron1/certificates/private/volttron1-admin.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external&server_name_indication=rabbit-2 - - Grab the upstream link name and run the below command to remove it. - - .. code-block:: bash - - vctl rabbitmq remove-federation-parameters upstream-volttron2-rabbit-2 diff --git a/docs/source/setup/RabbitMQ/index.rst b/docs/source/setup/RabbitMQ/index.rst deleted file mode 100644 index fcee383e09..0000000000 --- a/docs/source/setup/RabbitMQ/index.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. _RabbitMQ-Resources: - -========================== -Using RabbitMQ message bus -========================== -.. toctree:: - :glob: - :maxdepth: 1 - - * diff --git a/docs/source/setup/RabbitMQ/shovel_plugin.rst b/docs/source/setup/RabbitMQ/shovel_plugin.rst deleted file mode 100644 index 4a87022212..0000000000 --- a/docs/source/setup/RabbitMQ/shovel_plugin.rst +++ /dev/null @@ -1,386 +0,0 @@ -.. _shovel-plugin: - -RabbitMQ Multi-Platform Deployment Using Shovel Plugin -====================================================== - -In RabbitMQ based VOLTTRON, forwarder and data mover agents will be replaced by shovels -to send or receive remote pubsub messages. -Shovel behaves like a well written client application that connects to its source -( can be local or remote ) and destination ( can be local or remote instance ), -reads and writes messages, and copes with connection failures. In case of shovel, apart -from configuring the hostname, port and virtual host of the remote instance, we will -also have to provide list of topics that we want to forward to remote instance. Shovels -can also be used for remote RPC communication in which case we would have to create shovel -in both the instances, one to send the RPC request and other to send the response back. - -Pubsub Communication -~~~~~~~~~~~~~~~~~~~~ - -1. Setup two VOLTTRON instances using the steps described in installation section. -Please note that each instance should have a unique instance name. - -2. In a multi platform setup that need to communicate with each other with - RabbitMQ over SSL, each VOLTTRON instance should should trust the ROOT CA of - the other instance(RabbitMQ root ca) - - a. Transfer (scp/sftp/similar) - voltttron_home/certificates/certs/-root-ca.crt to a temporary - location on the other volttron instance machine. For example, if you have two - instance v1 and v2, scp v1's v1-root-ca.crt to v2 and - v2-root-ca.crt to v1. - - b. Append the contents of the transferred root ca to the instance's root ca. - - For example: - - On v1 - - cat /tmp/v2-root-ca.crt >> VOLTTRON_HOME/certificates/v1-root-ca.crt - - On v2 - - cat /tmp/v1-root-ca.crt >> VOLTTRON_HOME/certificates/v2-root-ca.crt - -3. Identify the instance that is going to act as the "publisher" instance. Suppose - "v1" instance is the "publisher" instance and "v2" instance is the "subscriber" - instance. Then we need to create a shovel on "v1" to forward messages matching - certain topics to remote instance "v2". - - a. On the publisher node, - - .. code-block:: bash - - vcfg --rabbitmq shovel [optional path to rabbitmq_shovel_config.yml] - - rabbitmq_shovel_config.yml should contain the details of the remote hostname, port, vhost - and list of topics to forward. Example configuration for shovel is available - in examples/configurations/rabbitmq/rabbitmq_shovel_config.yml - - - For this example, let's set the topic to "devices" - - If no config file is provided, the script will prompt for - hostname (or IP address), port, vhost and list of topics for each - remote instance you would like to add. For - bi-directional data flow, we will have to run the same script on both the nodes. - - b. Create a user in the subscriber node with username set to publisher instance's - agent name ( (instance-name)-PublisherAgent ) and allow the shovel access to - the virtual host of the subscriber node. - - .. code-block:: bash - - cd $RABBITMQ_HOME - vctl add-user - -4. Test the shovel setup. - - a. Start VOLTTRON on publisher and subscriber nodes. - - b. On the publisher node, start a master driver agent that publishes messages related to - a fake device. ( Easiest way is to run volttron-cfg command and follow the steps ) - - c. On the subscriber node, run a listener agent which subscribes to messages - from all platforms (set @PubSub.subscribe('pubsub', 'devices', all_platforms=True) - instead of @PubSub.subscribe('pubsub', '') ) - - d. Verify listener agent in subscriber node is able to receive the messages - matching "devices" topic. - -5. How to remove the shovel setup. - - a. Using the management web interface - - Log into management web interface using publisher instance's admin username. - Navigate to admin tab and then to shovel management page. The status of the - shovel will be displayed on the page. Click on the shovel name and delete the shovel. - - b. Using "volttron-ctl" command on the publisher node. - - .. code-block:: bash - - vctl rabbitmq list-shovel-parameters - NAME SOURCE ADDRESS DESTINATION ADDRESS BINDING KEY - shovel-rabbit-3-devices amqps://rabbit-1:5671/volttron1?cacertfile=/home/nidd494/.volttron1/certificates/certs/volttron1-root-ca.crt&certfile=/home/nidd494/.volttron1/certificates/certs/volttron1-admin.crt&keyfile=/home/nidd494/.volttron1/certificates/private/volttron1-admin.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external&server_name_indication=rabbit-1 amqps://rabbit-3:5671/volttron3?cacertfile=/home/nidd494/.volttron1/certificates/certs/volttron1-root-ca.crt&certfile=/home/nidd494/.volttron1/certificates/certs/volttron1-admin.crt&keyfile=/home/nidd494/.volttron1/certificates/private/volttron1-admin.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external&server_name_indication=rabbit-3 __pubsub__.volttron1.devices.# - - - Grab the shovel name and run the below command to remove it. - - .. code-block:: bash - - vctl rabbitmq remove-shovel-parameters shovel-rabbit-3-devices - -RPC Communication -~~~~~~~~~~~~~~~~~ -Following are the steps to create Shovel for multi-platform RPC communication. - -1. Setup two VOLTTRON instances using the steps described in installation section. - Please note that each instance should have a unique instance name. - -2. In a multi platform setup that need to communicate with each other with - RabbitMQ over SSL, each VOLTTRON instance should should trust the ROOT CA of - the other instance(RabbitMQ root ca) - - a. Transfer (scp/sftp/similar) - voltttron_home/certificates/certs/-root-ca.crt to a temporary - location on the other volttron instance machine. For example, if you have two - instance v1 and v2, scp v1's v1-root-ca.crt to v2 and - v2-root-ca.crt to v1. - - b. Append the contents of the transferred root ca to the instance's root ca. - For example: - - On v1 - - cat /tmp/v2-root-ca.crt >> VOLTTRON_HOME/certificates/v1-root-ca.crt - - On v2 - - cat /tmp/v1-root-ca.crt >> VOLTTRON_HOME/certificates/v2-root-ca.crt - -3. Typically RPC communication is 2 way communication so we will to setup shovel in both the VOLTTRON instances. In RPC calls - there are two instances of shovel. One serving as the caller (makes RPC request) and the other acting as a callee (replies - to RPC request). Identify the instance is the "caller" and which is the "callee." Suppose "v1" instance is the "caller" - instance and "v2" instance is the "callee" instance. - - a. On both the caller and callee nodes, shovel instances need to be created. In this example, v1’s shovel would forward the - RPC call request from an agent on v1 to v2 and similarly v2’s shovel will forward the RPC reply from agent on v2 - back to v1. - - .. code-block:: bash - - vcfg --rabbitmq shovel [optional path to rabbitmq_shovel_config.yml] - - rabbitmq_shovel_config.yml should contain the details of the - **remote** hostname, port, vhost, volttron instance name (so in v1's yml file parameters would point to v2 - and vice versa), and list of agent pair identities (local caller, remote callee). Example configuration for shovel - is available in examples/configurations/rabbitmq/rabbitmq_shovel_config.yml. - - For this example, let's say that we are using the schedule-example and acutator agents. - - For v1, the agent pair identities would be: - - - [Scheduler, platform.actuator] - - For v2, they would be: - - - [platform.actuator, Scheduler] - - Indicating the flow from local agent to remote agent. - - b. On the caller node create a user with username set to callee instance's agent name ( (instance-name)-RPCCallee ) and - allow the shovel access to the virtual host of the callee node. Similarly, on the callee node, create a user with - username set to caller instance's agent name ( (instance-name)-RPCCaller ) and allow the shovel access to the virtual - host of the caller node. - - .. code-block:: bash - - cd $RABBITMQ_HOME - vctl add-user - - -4. Test the shovel setup - - a. **On caller node**: - - Make necessary changes to RPC methods of caller agent. - - For this example, in volttron/examples/SchedulerExample/schedule_example/agent.py: - - * Search for 'campus/building/unit' in publish_schedule method. Replace with - 'devices/fake-campus/fake-building/fake-device' - * Search for ['campus/building/unit3',start,end] in the use_rpc method, replace with: - - msg = ['fake-campus/fake-building/fake-device',start,end]. - * Add: kwargs = {"external_platform": 'v2'} on the line below - * On the result = self.vip.rpc.call method below, replace "msg).get(timeout=10)" with: - - .. code-block:: bash - - msg, **kwargs).get(timeout=10), - - * In the second try clause of the use_rpc method: - * Replace result['result'] with result[0]['result'] - * Add kwargs = {"external_platform": 'v2'} as the first line of the if statement - * Replace 'campus/building/unit3/some_point' with 'fake-campus/fake-building/fake-device/PowerState' - * Below 'fake-campus/fake-building/fake-device/PowerState' add: 0, - * Replace - - .. code-block:: bash - - '0.0').get(timeout=10) with **kwargs).get(timeout=10) - - - Next, install an example scheduler agent and start it: - - .. code-block:: bash - - #!/bin/bash - python /home/username/volttron/scripts/install-agent.py -c /home/username/volttron/examples/SchedulerExample/schedule-example.agent -s examples/SchedulerExample --start --force -i Scheduler - - - b. **On the callee node:** - - - Run upgrade script to install actuator agent. - - .. code-block:: bash - - #!/bin/bash - python /home/username/volttron/scripts/install-agent.py -s services/core/ActuatorAgent --start --force -i platform.actuator - - - - Run the upgrade script to install the listener agent. - - .. code-block:: bash - - scripts/core/upgrade-listener - - - - - Install master driver, configure fake device on upstream callee and start volttron and master driver. - vcfg --agent master_driver command can install master driver and setup a fake device. - - .. code-block:: bash - - ./stop-volttron - vcfg --agent master_driver - ./start-volttron - vctl start --tag master_driver - - - - Start actuator agent and listener agents. - - The output for the callee node with a successful shovel run should look similar to: - - .. code-block:: bash - - 2018-12-19 15:38:00,009 (listeneragent-3.2 13039) listener.agent INFO: Peer: pubsub, Sender: platform.driver:, Bus: , Topic: devices/fake-campus/fake-building/fake-device/all, Headers: {'Date': '2018-12-19T20:38:00.001684+00:00', 'TimeStamp': '2018-12-19T20:38:00.001684+00:00', 'min_compatible_version': '5.0', 'max_compatible_version': u'', 'SynchronizedTimeStamp': '2018-12-19T20:38:00.000000+00:00'}, Message: - [{'Heartbeat': True, 'PowerState': 0, 'ValveState': 0, 'temperature': 50.0}, - {'Heartbeat': {'type': 'integer', 'tz': 'US/Pacific', 'units': 'On/Off'}, - 'PowerState': {'type': 'integer', 'tz': 'US/Pacific', 'units': '1/0'}, - 'ValveState': {'type': 'integer', 'tz': 'US/Pacific', 'units': '1/0'}, - 'temperature': {'type': 'integer', - 'tz': 'US/Pacific', - 'units': 'Fahrenheit'}}] - - - -DataMover Communication -~~~~~~~~~~~~~~~~~~~~~~~ - -The DataMover historian running on one instance makes RPC call to platform historian running on remote -instance to store data on remote instance. Platform historian agent returns response back to DataMover -agent. For such a request-response behavior, shovels need to be created on both instances. - -1. Please ensure that preliminary steps for multi-platform communication are completed (namely, - steps 1-3 described above) . - -2. To setup a data mover to send messages from local instance (say v1) to remote instance (say v2) - and back, we would need to setup shovels on both instances. - - Example of RabbitMQ shovel configuration on v1 - - .. code-block:: json - - shovel: - # hostname of remote machine - rabbit-2: - port: 5671 - rpc: - # Remote instance name - v2: - # List of pair of agent identities (local caller, remote callee) - - [data.mover, platform.historian] - virtual-host: v1 - - This says that DataMover agent on v1 wants to make RPC call to platform historian on v2. - - .. code-block:: bash - - vcfg --rabbitmq shovel [optional path to rabbitmq_shovel_config.yml - - - Example of RabbitMQ shovel configuration on v2 - - .. code-block:: json - - shovel: - # hostname of remote machine - rabbit-1: - port: 5671 - rpc: - # Remote instance name - v1: - # List of pair of agent identities (local caller, remote callee) - - [platform.historian, data.mover] - virtual-host: v2 - - This says that Hplatform historian on v2 wants to make RPC call to DataMover agent on v1. - - a. On v1, run below command to setup a shovel from v1 to v2. - - .. code-block:: bash - - vcfg --rabbitmq shovel [optional path to rabbitmq_shovel_config.yml - - b. Create a user on v2 with username set to remote agent's username - ( for example, v1.data.mover i.e., .) and allow - the shovel access to the virtual host of v2. - - .. code-block:: bash - - cd $RABBITMQ_HOME - vctl add-user - - c. On v2, run below command to setup a shovel from v2 to v1 - - .. code-block:: bash - - vcfg --rabbitmq shovel [optional path to rabbitmq_shovel_config.yml - - d. Create a user on v1 with username set to remote agent's username - ( for example, v2.patform.historian i.e., .) and allow - the shovel access to the virtual host of the v1. - - .. code-block:: bash - - cd $RABBITMQ_HOME - vctl add-user - -3. Start Master driver agent on v1 - - .. code-block:: bash - - ./stop-volttron - vcfg --agent master_driver - ./start-volttron - vctl start --tag master_driver - -4. Install DataMover agent on v1. Contents of the install script can look like below. - - .. code-block:: bash - - #!/bin/bash - export CONFIG=$(mktemp /tmp/abc-script.XXXXXX) - cat > $CONFIG <=0.10,<0.11 - ... - + pip install --global-option --quiet --install-option --zmq=bundled --no-deps pyzmq>=14.3,<15 - ... - + pip install --global-option --quiet --editable ./lib/jsonrpc --editable . --requirement ./requirements.txt - ... - Successfully installed Smap-2.0.24c780d avro-1.7.7 configobj-5.0.6 ecdsa-0.13 flexible-jsonrpc - gevent-1.0.1 greenlet-0.4.5 monotonic-0.1 numpy-1.9.1 pandas-0.15.2 paramiko-1.15.2 - pycrypto-2.6.1 pymodbus-1.2.0 pyserial-2.7 python-dateutil-2.4.0 pytz-2014.10 requests-2.5.3 - simplejson-3.6.5 six-1.9.0 twisted-15.0.0 volttron-2.0 wheel-0.24.0 zope.interface-4.1.2 - - real 9m2.299s - user 7m51.790s - sys 0m14.450s - -Whew! The build took just over nine minutes on my nearly-4-year-old MacBook Pro running Arch Linux. In case you are wondering about my system's name, as seen in the bash prompt, *inamatus* is Latin for *unloved*. I'll leave it as an exercise for the user to determine why my system is unloved (hint: it has to do with a wonderful fruit with a bite missing from the side). - -Anyway, let's have another look at the pip download cache. - -.. code:: - - [volttron@inamatus volttron]$ find ~/.cache/pip -type f - /home/volttron/.cache/pip/http/9/a/b/2/1/9ab21efc4225c8eb9aa41d1c76abef2a53babcefa438a79fa4e981ce - /home/volttron/.cache/pip/http/9/2/6/7/2/92672ab99ac77960252018fbcb4f40984eef60ba5588229a729f18f5 - /home/volttron/.cache/pip/http/9/e/6/1/9/9e61964f51d8a05a20ecf21eef694877f28cb654a123ce1316ff77e5 - /home/volttron/.cache/pip/http/9/7/7/1/a/9771a6b64f3294ac335fdb8574cd3564e21c130924697381d72fd04d - /home/volttron/.cache/pip/http/a/a/7/e/8/aa7e8bc2af1068a43747b0f771b426b7dcf7708283ca3ce3d92a2afc - ... - /home/volttron/.cache/pip/http/8/f/9/0/d/8f90d7cf09a2b5380a319b0df8eed268be28d590b6b5f71598a3b56f - /home/volttron/.cache/pip/http/8/d/e/d/a/8deda849bcfd627b8587addf049f79bb333dd8fe1eae1d5053881039 - /home/volttron/.cache/pip/http/8/8/7/a/6/887a67fb460d57a10a50deef3658834b9ac01722244315227d334628 - /home/volttron/.cache/pip/http/5/5/4/e/2/554e2be8d96625aa74a4e0c4ee4a4b1ca10a442c2877bd3fff96e2a6 - /home/volttron/.cache/pip/http/1/d/c/8/3/1dc83c11a861a2bc20d9c0407b41089eba236796ba80c213511f1f74 - /home/volttron/.cache/pip/log/debug.log - -The output is truncated because it was long and boring. The important thing is that it now exists. Next let's remove the virtual environment and rebuild to see what effect the download cache has on our build time. - -.. code:: - - [volttron@inamatus volttron]$ rm -rf env - [volttron@inamatus volttron]$ time python3 bootstrap.py - ... - - real 8m35.387s - user 7m50.770s - sys 0m14.170s - -Notice that our CPU time was nearly the same, about 8 minutes (user + sys). So the remaining time was likely spent on I/O, which was reduced by about 30 seconds. We need something else to reduce CPU time. Enter ccache. - - -Better ------- - -What is ccache? According to the official ccache_ site, - - ccache is a compiler cache. It speeds up recompilation by caching the result of previous compilations and detecting when the same compilation is being done again. - -.. _ccache: https://ccache.samba.org/ - -Sounds like just the thing we need. ccache is already properly configured on my system, it just needs to be placed early in the ``PATH`` to be found before the official gcc compilers. - -.. code:: - - [volttron@inamatus volttron]$ which gcc - /usr/bin/gcc - [volttron@inamatus volttron]$ export PATH=/usr/lib/ccache/bin:$PATH - [volttron@inamatus volttron]$ which gcc - /usr/lib/ccache/bin/gcc - -Now to prove to ourselves that the cache will be filled during the next run, let's have a look at the cache status. - -.. code:: - - [volttron@inamatus volttron]$ ccache -s - cache directory /home/volttron/.ccache - primary config /home/volttron/.ccache/ccache.conf - secondary config (readonly) /etc/ccache.conf - cache hit (direct) 0 - cache hit (preprocessed) 0 - cache miss 0 - files in cache 0 - cache size 0.0 kB - max cache size 5.0 GB - -The cache is indeed empty. - -Nothing up my sleeve... Presto! - -.. code:: - - [volttron@inamatus volttron]$ rm -rf env - [volttron@inamatus volttron]$ time python3 bootstrap.py - ... - - real 6m3.496s - user 4m57.960s - sys 0m10.880s - -One might expect a ccache build to take slightly longer than the baseline on the first build within a single project. This build completed about two minutes faster. Let's look at the ccache status to discover why. - -.. code:: - - [volttron@inamatus volttron]$ ccache -s - cache directory /home/volttron/.ccache - primary config /home/volttron/.ccache/ccache.conf - secondary config (readonly) /etc/ccache.conf - cache hit (direct) 204 - cache hit (preprocessed) 23 - cache miss 633 - called for link 140 - called for preprocessing 95 - compile failed 1139 - preprocessor error 4 - bad compiler arguments 5 - autoconf compile/link 103 - no input file 19 - files in cache 1316 - cache size 26.1 MB - max cache size 5.0 GB - -Ah ha. There were a total of 227 cache hits, meaning that some of the files were identical across all the built packages and the cached version could be used rather than recompiling. Let's see how subsequent builds improve with few cache misses. - -.. code:: - - [volttron@inamatus volttron]$ rm -rf env - [volttron@inamatus volttron]$ time python3 bootstrap.py - ... - - real 3m15.811s - user 2m24.890s - sys 0m7.090s - -Wow! Now we're cooking with gas. Build times have been cut to nearly 1/3 of our baseline. This ccache status shows only 14 cache misses over our previous run: - -.. code:: - - [volttron@inamatus volttron]$ ccache -s - cache directory /home/volttron/.ccache - primary config /home/volttron/.ccache/ccache.conf - secondary config (readonly) /etc/ccache.conf - cache hit (direct) 1038 - cache hit (preprocessed) 35 - cache miss 647 - called for link 280 - called for preprocessing 190 - compile failed 2278 - preprocessor error 8 - bad compiler arguments 10 - autoconf compile/link 206 - no input file 38 - files in cache 1365 - cache size 35.0 MB - max cache size 5.0 GB - -So using ccache is a big win. Anyone compiling C or C++ on a Linux system should have ccache enabled. Wait, make that *must*. Go, now, and enable it on your Linux boxen. Or maybe finish reading this and then go do it. But do it! - -Best ----- - -Now you're thinking "how could it get any better," right? Well, it can. What if those compiled packages only needed to be rebuilt when a new version was required instead of every time they are installed. - -When pip installs a package, it downloads the source and executes the packages ``setup.py`` like so: ``python setup.py install``. The install command builds the package and installs it directly into the file system. What if we could package up the build results into an archive and just extract them to the file system when the package is installed. Enter **wheel**. - -pip supports the latest Python packaging format known as wheel. Typically this just means that it can install packages in the `wheel format`_. However, if the wheel_ package is installed, pip can also `build wheels`_ from source, executing ``python setup.py bdist_wheel``. By default, wheels are placed in the *wheelhouse* directory in the current working directory. But we can alter that location by setting an environment variable (read more on configuring pip here_). - -.. _wheel format: http://wheel.readthedocs.org/en/latest -.. _wheel: https://pypi.python.org/pypi/wheel -.. _build wheels: https://pip.pypa.io/en/latest/reference/pip_wheel.html - -.. code:: - - [volttron@inamatus volttron]$ export PIP_WHEEL_DIR=$HOME/.cache/pip/wheelhouse - -We also need to tell pip to look for the wheels, again using an environment variable. The directory needs to exist because while the wheel command will create the directory when creating the packages, pip may try to search the directory first. - -.. code:: - - [volttron@inamatus volttron]$ export PIP_FIND_LINKS=file://$PIP_WHEEL_DIR - [volttron@inamatus volttron]$ mkdir $PIP_WHEEL_DIR - -So to get this all working, bootstrapping now has to occur in three steps: install the virtual environment, build the wheels, and install the requirements. ``bootstrap.py`` takes options that control its behavior. The first pass requires the ``-o`` or ``--only-virtenv`` option to stop bootstrap after installing the virtual environment and prevent the update stage. - -.. code:: - - [volttron@inamatus volttron]$ rm -rf env - [volttron@inamatus volttron]$ time python3 bootstrap.py --only-virtenv - Creating virtual Python environment - Downloading virtualenv DOAP record - Downloading virtualenv 12.0.7 - New python executable in /home/volttron/volttron/env/bin/python2.7 - Also creating executable in /home/volttron/volttron/env/bin/python - Installing setuptools, pip...done. - - real 0m3.866s - user 0m1.480s - sys 0m0.230s - -The second step requires the ``-w`` or ``--wheel`` option to build the wheels. Because the virtual environment already exists, ``bootstrap.py`` must be called with the virtual environment Python, not the system Python. - -.. code:: - - [volttron@inamatus volttron]$ time env/bin/python bootstrap.py --wheel - Building required packages - + pip install --global-option --quiet wheel - ... - + pip wheel --global-option --quiet --build-option --zmq=bundled --no-deps pyzmq>=14.3,<15 - ... - + pip wheel --global-option --quiet --editable ./lib/jsonrpc --editable . --requirement ./requirements.txt - ... - Destination directory: /home/volttron/.cache/pip/wheelhouse - Successfully built numpy pandas gevent monotonic pymodbus simplejson Smap greenlet pycrypto - twisted pyserial configobj avro zope.interface - - real 3m15.431s - user 2m17.980s - sys 0m5.630s - -It took 3.25 minutes to build the wheels (with ccache still enabled). Repeating this command results in nothing new being compiled and takes only 4 seconds. Only new versions of packages meeting the requirements will be built. - -.. code:: - - [volttron@inamatus volttron]$ time env/bin/python bootstrap.py --wheel - Building required packages - ... - Skipping numpy, due to already being wheel. - Skipping pandas, due to already being wheel. - Skipping python-dateutil, due to already being wheel. - Skipping requests, due to already being wheel. - Skipping flexible-jsonrpc, due to being editable - Skipping pyzmq, due to already being wheel. - Skipping gevent, due to already being wheel. - Skipping monotonic, due to already being wheel. - Skipping paramiko, due to already being wheel. - Skipping pymodbus, due to already being wheel. - Skipping setuptools, due to already being wheel. - Skipping simplejson, due to already being wheel. - Skipping Smap, due to already being wheel. - Skipping wheel, due to already being wheel. - Skipping volttron, due to being editable - Skipping pytz, due to already being wheel. - Skipping six, due to already being wheel. - Skipping greenlet, due to already being wheel. - Skipping ecdsa, due to already being wheel. - Skipping pycrypto, due to already being wheel. - Skipping pyserial, due to already being wheel. - Skipping twisted, due to already being wheel. - Skipping configobj, due to already being wheel. - Skipping avro, due to already being wheel. - Skipping zope.interface, due to already being wheel. - - real 0m3.998s - user 0m3.580s - sys 0m0.360s - -And let's see what is in the wheelhouse. - -.. code:: - - [volttron@inamatus volttron]$ ls ~/.cache/pip/wheelhouse - Smap-2.0.24c780d-py2-none-any.whl - Twisted-15.0.0-cp27-none-linux_x86_64.whl - avro-1.7.7-py2-none-any.whl - configobj-5.0.6-py2-none-any.whl - ecdsa-0.13-py2.py3-none-any.whl - gevent-1.0.1-cp27-none-linux_x86_64.whl - greenlet-0.4.5-cp27-none-linux_x86_64.whl - monotonic-0.1-py2-none-any.whl - numpy-1.9.1-cp27-none-linux_x86_64.whl - pandas-0.15.2-cp27-none-linux_x86_64.whl - paramiko-1.15.2-py2.py3-none-any.whl - pycrypto-2.6.1-cp27-none-linux_x86_64.whl - pymodbus-1.2.0-py2-none-any.whl - pyserial-2.7-py2-none-any.whl - python_dateutil-2.4.0-py2.py3-none-any.whl - pytz-2014.10-py2.py3-none-any.whl - pyzmq-14.5.0-cp27-none-linux_x86_64.whl - requests-2.5.3-py2.py3-none-any.whl - setuptools-12.2-py2.py3-none-any.whl - simplejson-3.6.5-cp27-none-linux_x86_64.whl - six-1.9.0-py2.py3-none-any.whl - wheel-0.24.0-py2.py3-none-any.whl - zope.interface-4.1.2-cp27-none-linux_x86_64.whl - -Now ``bootstrap.py`` can be run without options to complete the bootstrap process, again using the virtual environment Python. - -.. code:: - - [volttron@inamatus volttron]$ time env/bin/python bootstrap.py - Installing required packages - + easy_install BACpypes>=0.10,<0.11 - ... - + pip install --global-option --quiet --install-option --zmq=bundled --no-deps pyzmq>=14.3,<15 - ... - + pip install --global-option --quiet --editable ./lib/jsonrpc --editable . --requirement ./requirements.txt - ... - Successfully installed Smap-2.0.24c780d avro-1.7.7 configobj-5.0.6 ecdsa-0.13 flexible-jsonrpc - gevent-1.0.1 greenlet-0.4.5 monotonic-0.1 numpy-1.9.1 pandas-0.15.2 paramiko-1.15.2 - pycrypto-2.6.1 pymodbus-1.2.0 pyserial-2.7 python-dateutil-2.4.0 pytz-2014.10 requests-2.5.3 - simplejson-3.6.5 six-1.9.0 twisted-15.0.0 volttron-2.0 zope.interface-4.1.2 - - real 0m11.137s - user 0m8.930s - sys 0m0.950s - -Installing from wheels completes in only 11 seconds. And if we blow away the environment and bootstrap again, it takes under 15 seconds. - -.. code:: - - [volttron@inamatus volttron]$ rm -rf env - [volttron@inamatus volttron]$ time python3 bootstrap.py - ... - - real 0m14.644s - user 0m10.380s - sys 0m1.240s - -Building a clean environment now occurs in less than 15 seconds instead of the 9 minute baseline. That, my friends, is fast. - - -Why care? ---------- - -The average VOLTTRON developer probably won't care or see much benefit from the wheel optimization. The typical developer workflow does not include regularly removing the virtual environment and rebuilding. This is, however, very important for continuous integration (CI). With CI, a build server should check out a fresh copy of the source code, build it in a clean environment, and perform unit tests, notifying offending users when their changes break things. Ideally, notification of breakage should happen as soon as possible. We just shaved nearly nine minutes off the turnaround time. It also reduces the load on a shared CI build server, which is nice for everyone. - - -Taking it further ------------------ - -Two additional use cases present themselves: offline installs and shared builds. - - -Offline Installs -++++++++++++++++ - -Let's say we have a system that is not connected to the Internet and, therefore, cannot download packages from PyPi_ or any other package index. Or perhaps it doesn't have a suitable compiler. Wheels can be built on another *similar*, connected system and transferred by USB drive to the offline system, where they can then be installed. Note that the architecture must be identical and the OS must be very similar between the two systems for this to work. - -If the two systems differ too much for a compatible binary build and the offline system has a suitable compiler, then source files can be copied from the pip download cache and transferred from the online system to the offline system for building. - - -Shared Builds -+++++++++++++ - -If many developers are working on the same project, why not share the results of a build with the rest of the team? Here are some ideas to make it work: - -* Put wheels on a shared network drive -* Run a private package index server (maybe with pypiserver_) -* Expose CI built wheels using Apache, Nginx, or SimpleHTTPServer_ - -.. _pypiserver: https://pypi.python.org/pypi/pypiserver -.. _SimpleHTTPServer: https://docs.python.org/2.7/library/simplehttpserver.html#module-SimpleHTTPServer - - -Issues ------- - -Here are some of the issues/drawbacks to the methods described above and some possible solutions. - -* Configuring pip using environment variables - - No worries. Pip uses configuration files too. And a benefit to using them is that it makes all these wheels available to other Python projects you may be working on, and vise versa. - - .. code:: - - # /home/volttron/.config/pip/pip.conf - [global] - wheel-dir = /home/volttron/.cache/pip/wheelhouse - find-links = file:///home/volttron/.cache/pip/wheelhouse - - Find more on configuring pip here_. - - .. _here: https://pip.pypa.io/en/latest/user_guide.html#configuration - -* pip does not clean the wheelhouse - - This is not a deal-breaker. The wheel directory can just be removed and it will be recreated. Or a script can be used to remove all but the latest versions of packages. - -* Requires an additional step or two - - That's the price for speed. But it can be mitigated by writing a script or bash alias to perform the steps. - - -Conclusion ----------- - -Here is a quick summary of the build times executed above: - -======================= ====== ====== - Method Time (minutes) ------------------------ -------------- -Each builds on previous CPU Total -======================= ====== ====== -baseline 8:07 9:02 -with download cache 8:05 8:35 -ccache, first run 5:09 6:03 -ccache, subsequent runs 2:32 3:16 -wheel, first run 2:35 3:30 -wheel, subsequent runs 0:12 0:15 -======================= ====== ====== - -Not everyone cares about build times, but for those who do, pre-building Python wheels is a great way to improve install times. At a very minimum, every Python developer installing compiled packages will benefit from using ccache. - -The techniques used in this document aren't just for VOLTTRON, either. They are generally useful for all moderately sized Python projects. - -If you haven't installed ccache yet, go do it. There is no excuse. - -.. vim: ft=rst spell wrap: diff --git a/docs/source/setup/VOLTTRON-Prerequisites.rst b/docs/source/setup/VOLTTRON-Prerequisites.rst deleted file mode 100644 index 5818bf0c7c..0000000000 --- a/docs/source/setup/VOLTTRON-Prerequisites.rst +++ /dev/null @@ -1,92 +0,0 @@ -.. _VOLTTRON-Prerequisites: - -Required Software: Linux -======================== - -The following packages will need to be installed on the system: - -- git -- build-essential -- python3.6-dev -- python3.6-venv -- openssl -- libssl-dev -- libevent-dev - -On **Debian-based systems**, these can all be installed with the following -command: - -.. code-block:: bash - - sudo apt-get update - sudo apt-get install build-essential python3-dev python3-venv openssl libssl-dev libevent-dev git - -On Ubuntu-based systems, available packages allow you to specify the python3 version, 3.6 or greater is required (Debian itself does not provide those packages). - -On arm-based systems (including, but not limited to, Raspbian), you must also install libffi-dev, you can do this with: - -.. code-block:: bash - - sudo apt-get install libffi-dev - -On **Redhat or CENTOS systems**, these can all be installed with the following -command: - -.. code-block:: bash - - sudo yum update - sudo yum install make automake gcc gcc-c++ kernel-devel python3-devel openssl openssl-devel libevent-devel git - -.. note:: - Python 3.6 or greater is required. - -If you have an agent which requires the pyodbc package, install the -following: - -- freetds-bin -- unixodbc-dev - -On **Debian-based systems** these can be installed with the following command: - -.. code-block:: bash - - sudo apt-get install freetds-bin unixodbc-dev - -On **Redhat or CentOS systems**, these can be installed from the Extra Packages for Enterprise Linux (EPEL) repository: - -.. code-block:: bash - - sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm - sudo yum install freetds unixODBC-devel - -.. note:: - The above command to install the EPEL repository is for Centos/Redhat 8. Change the number to match your OS version. - - EPEL packages are included in Fedora repositories, so installing EPEL is not required on Fedora. - - -Possible issues -~~~~~~~~~~~~~~~ - -The /tmp directory must allow exec. This error could manifest itself -during the building of gevent. - -:: - - # Executing mount should have an entry like the following - mount - - tmpfs on /tmp type tmpfs (rw,nosuid,nodev) - -To change the mount you can use the following code - -:: - - # remount /tmp to allow exec - sudo mount -o remount,exec /tmp - -:: - - # remount /tmp to disallow exec - sudo mount -o remount,noexec /tmp - diff --git a/docs/source/setup/VOLTTRON-Source-Options.rst b/docs/source/setup/VOLTTRON-Source-Options.rst deleted file mode 100644 index 74879fd3c0..0000000000 --- a/docs/source/setup/VOLTTRON-Source-Options.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. _Repository-Structure: - -Repository Structure -==================== - -There are several options for using the VOLTTRON code depending on -whether you require the most stable version of the code or want the -latest updates as they happen. In order of decreasing stability and -increasing currency: - -For most stable, download the source code for the latest release at: -https://github.com/VOLTTRON/volttron/releases These are purely source -code and are not tied to the git repository. To update them will require -downloading the newest source code and re-installing. - -The master branch is now the default branch for VOLTTRON (meaning this -is what you clone if you do not use the “-b” option). This branch will -get the latest stable features as they are pushed. The current major -version is 7.x. - -The “develop” branch contains the latest features as they are developed. -Once a feature is considered “finished” it is merged back into develop. -Develop will be merged into master once it is considered stable and -ready for release. This branch can be cloned by those wanting to work -from the latest version of the platform but should not be used in -deployments. - -Features are developed on “feature” branches or developers' forks of -the main repository. It is not -recommended to clone these branches except for exploring a new -feature. diff --git a/docs/source/setup/images/VOLTTRON_User_Guide.pdf b/docs/source/setup/images/VOLTTRON_User_Guide.pdf deleted file mode 100755 index 4c8b77a890..0000000000 Binary files a/docs/source/setup/images/VOLTTRON_User_Guide.pdf and /dev/null differ diff --git a/docs/source/setup/images/add-chart.png b/docs/source/setup/images/add-chart.png deleted file mode 100755 index 6d30604c8f..0000000000 Binary files a/docs/source/setup/images/add-chart.png and /dev/null differ diff --git a/docs/source/setup/images/clone-existing.png b/docs/source/setup/images/clone-existing.png deleted file mode 100755 index 465f351bf0..0000000000 Binary files a/docs/source/setup/images/clone-existing.png and /dev/null differ diff --git a/docs/source/setup/images/dashboard-blank.png b/docs/source/setup/images/dashboard-blank.png deleted file mode 100755 index 2e62a2331d..0000000000 Binary files a/docs/source/setup/images/dashboard-blank.png and /dev/null differ diff --git a/docs/source/setup/images/eclipse-marketplace.png b/docs/source/setup/images/eclipse-marketplace.png deleted file mode 100755 index cf6a0136b2..0000000000 Binary files a/docs/source/setup/images/eclipse-marketplace.png and /dev/null differ diff --git a/docs/source/setup/images/eclipse-marketplace2.png b/docs/source/setup/images/eclipse-marketplace2.png deleted file mode 100755 index 91b4b5f706..0000000000 Binary files a/docs/source/setup/images/eclipse-marketplace2.png and /dev/null differ diff --git a/docs/source/setup/images/edit-chart.png b/docs/source/setup/images/edit-chart.png deleted file mode 100755 index 771bcff359..0000000000 Binary files a/docs/source/setup/images/edit-chart.png and /dev/null differ diff --git a/docs/source/setup/images/example_market.png b/docs/source/setup/images/example_market.png deleted file mode 100644 index ba953dcb7e..0000000000 Binary files a/docs/source/setup/images/example_market.png and /dev/null differ diff --git a/docs/source/setup/images/finish-import.png b/docs/source/setup/images/finish-import.png deleted file mode 100755 index 5a32e44151..0000000000 Binary files a/docs/source/setup/images/finish-import.png and /dev/null differ diff --git a/docs/source/setup/images/general-project.png b/docs/source/setup/images/general-project.png deleted file mode 100755 index 106228a41d..0000000000 Binary files a/docs/source/setup/images/general-project.png and /dev/null differ diff --git a/docs/source/setup/images/git-view.png b/docs/source/setup/images/git-view.png deleted file mode 100755 index 0b59e8fcc3..0000000000 Binary files a/docs/source/setup/images/git-view.png and /dev/null differ diff --git a/docs/source/setup/images/import-project.png b/docs/source/setup/images/import-project.png deleted file mode 100755 index 5ac524f41d..0000000000 Binary files a/docs/source/setup/images/import-project.png and /dev/null differ diff --git a/docs/source/setup/images/install-volttron-restricted.png b/docs/source/setup/images/install-volttron-restricted.png deleted file mode 100755 index 73465e547e..0000000000 Binary files a/docs/source/setup/images/install-volttron-restricted.png and /dev/null differ diff --git a/docs/source/setup/images/linux-mint.png b/docs/source/setup/images/linux-mint.png deleted file mode 100644 index 1697fb8522..0000000000 Binary files a/docs/source/setup/images/linux-mint.png and /dev/null differ diff --git a/docs/source/setup/images/listener-all-vars.png b/docs/source/setup/images/listener-all-vars.png deleted file mode 100755 index f4fdde5296..0000000000 Binary files a/docs/source/setup/images/listener-all-vars.png and /dev/null differ diff --git a/docs/source/setup/images/login-screen.png b/docs/source/setup/images/login-screen.png deleted file mode 100755 index a9a9d18efb..0000000000 Binary files a/docs/source/setup/images/login-screen.png and /dev/null differ diff --git a/docs/source/setup/images/logout-button.png b/docs/source/setup/images/logout-button.png deleted file mode 100755 index 81fab0beff..0000000000 Binary files a/docs/source/setup/images/logout-button.png and /dev/null differ diff --git a/docs/source/setup/images/new-python-run.png b/docs/source/setup/images/new-python-run.png deleted file mode 100755 index 580b462c8f..0000000000 Binary files a/docs/source/setup/images/new-python-run.png and /dev/null differ diff --git a/docs/source/setup/images/overview.png b/docs/source/setup/images/overview.png deleted file mode 100755 index aa6a136158..0000000000 Binary files a/docs/source/setup/images/overview.png and /dev/null differ diff --git a/docs/source/setup/images/pick-python.png b/docs/source/setup/images/pick-python.png deleted file mode 100755 index 2d3e8eafaf..0000000000 Binary files a/docs/source/setup/images/pick-python.png and /dev/null differ diff --git a/docs/source/setup/images/pin-to-dashboard.png b/docs/source/setup/images/pin-to-dashboard.png deleted file mode 100755 index b4041b15df..0000000000 Binary files a/docs/source/setup/images/pin-to-dashboard.png and /dev/null differ diff --git a/docs/source/setup/images/platform-default.png b/docs/source/setup/images/platform-default.png deleted file mode 100755 index b224d4d817..0000000000 Binary files a/docs/source/setup/images/platform-default.png and /dev/null differ diff --git a/docs/source/setup/images/platform-run-config.png b/docs/source/setup/images/platform-run-config.png deleted file mode 100755 index 083e157d83..0000000000 Binary files a/docs/source/setup/images/platform-run-config.png and /dev/null differ diff --git a/docs/source/setup/images/platforms.png b/docs/source/setup/images/platforms.png deleted file mode 100755 index 6845265011..0000000000 Binary files a/docs/source/setup/images/platforms.png and /dev/null differ diff --git a/docs/source/setup/images/pydev-python.png b/docs/source/setup/images/pydev-python.png deleted file mode 100755 index e20f45bdb2..0000000000 Binary files a/docs/source/setup/images/pydev-python.png and /dev/null differ diff --git a/docs/source/setup/images/register-new-platform-authorization.png b/docs/source/setup/images/register-new-platform-authorization.png deleted file mode 100755 index 5a03488d0b..0000000000 Binary files a/docs/source/setup/images/register-new-platform-authorization.png and /dev/null differ diff --git a/docs/source/setup/images/register-new-platform.png b/docs/source/setup/images/register-new-platform.png deleted file mode 100755 index 796c1029f6..0000000000 Binary files a/docs/source/setup/images/register-new-platform.png and /dev/null differ diff --git a/docs/source/setup/images/run-results.png b/docs/source/setup/images/run-results.png deleted file mode 100755 index 5568a59585..0000000000 Binary files a/docs/source/setup/images/run-results.png and /dev/null differ diff --git a/docs/source/setup/images/select-branch.png b/docs/source/setup/images/select-branch.png deleted file mode 100755 index b26ba74292..0000000000 Binary files a/docs/source/setup/images/select-branch.png and /dev/null differ diff --git a/docs/source/setup/images/select-path.png b/docs/source/setup/images/select-path.png deleted file mode 100755 index 7f690f5696..0000000000 Binary files a/docs/source/setup/images/select-path.png and /dev/null differ diff --git a/docs/source/setup/images/select-repo.png b/docs/source/setup/images/select-repo.png deleted file mode 100755 index d74fd20b45..0000000000 Binary files a/docs/source/setup/images/select-repo.png and /dev/null differ diff --git a/docs/source/setup/images/set-as-pydev.png b/docs/source/setup/images/set-as-pydev.png deleted file mode 100755 index 681eeefd60..0000000000 Binary files a/docs/source/setup/images/set-as-pydev.png and /dev/null differ diff --git a/docs/source/setup/images/setup-python.png b/docs/source/setup/images/setup-python.png deleted file mode 100755 index 6e90e4537c..0000000000 Binary files a/docs/source/setup/images/setup-python.png and /dev/null differ diff --git a/docs/source/setup/images/start-agent.png b/docs/source/setup/images/start-agent.png deleted file mode 100755 index 57b3b0006a..0000000000 Binary files a/docs/source/setup/images/start-agent.png and /dev/null differ diff --git a/docs/source/setup/images/transport-payload.png b/docs/source/setup/images/transport-payload.png deleted file mode 100755 index 9826244157..0000000000 Binary files a/docs/source/setup/images/transport-payload.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-bidirectional.png b/docs/source/setup/images/vbox-bidirectional.png deleted file mode 100644 index 7a52548b5b..0000000000 Binary files a/docs/source/setup/images/vbox-bidirectional.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-controller.png b/docs/source/setup/images/vbox-controller.png deleted file mode 100644 index 3339221522..0000000000 Binary files a/docs/source/setup/images/vbox-controller.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-credentials.png b/docs/source/setup/images/vbox-credentials.png deleted file mode 100644 index 6e14d378f2..0000000000 Binary files a/docs/source/setup/images/vbox-credentials.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-download.png b/docs/source/setup/images/vbox-download.png deleted file mode 100644 index 90ca3d7059..0000000000 Binary files a/docs/source/setup/images/vbox-download.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-hard-disk-xfce.png b/docs/source/setup/images/vbox-hard-disk-xfce.png deleted file mode 100644 index 74ec8167ff..0000000000 Binary files a/docs/source/setup/images/vbox-hard-disk-xfce.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-memory-size.png b/docs/source/setup/images/vbox-memory-size.png deleted file mode 100644 index 1b8c0ca542..0000000000 Binary files a/docs/source/setup/images/vbox-memory-size.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-naming.png b/docs/source/setup/images/vbox-naming.png deleted file mode 100644 index b539b82a7e..0000000000 Binary files a/docs/source/setup/images/vbox-naming.png and /dev/null differ diff --git a/docs/source/setup/images/vbox-proc-settings.png b/docs/source/setup/images/vbox-proc-settings.png deleted file mode 100644 index aec8b53e0b..0000000000 Binary files a/docs/source/setup/images/vbox-proc-settings.png and /dev/null differ diff --git a/docs/source/setup/images/vc-run-demo.png b/docs/source/setup/images/vc-run-demo.png deleted file mode 100755 index af2159e55b..0000000000 Binary files a/docs/source/setup/images/vc-run-demo.png and /dev/null differ diff --git a/docs/source/setup/images/volttron-console.png b/docs/source/setup/images/volttron-console.png deleted file mode 100755 index 1a63411b9a..0000000000 Binary files a/docs/source/setup/images/volttron-console.png and /dev/null differ diff --git a/docs/source/setup/images/volttron-main-args.png b/docs/source/setup/images/volttron-main-args.png deleted file mode 100755 index fa42572509..0000000000 Binary files a/docs/source/setup/images/volttron-main-args.png and /dev/null differ diff --git a/docs/source/setup/images/volttron-main.png b/docs/source/setup/images/volttron-main.png deleted file mode 100755 index 2443671f96..0000000000 Binary files a/docs/source/setup/images/volttron-main.png and /dev/null differ diff --git a/docs/source/setup/images/volttron-pick-main.png b/docs/source/setup/images/volttron-pick-main.png deleted file mode 100755 index 13168ba066..0000000000 Binary files a/docs/source/setup/images/volttron-pick-main.png and /dev/null differ diff --git a/docs/source/setup/images/volttron-webimage.jpg b/docs/source/setup/images/volttron-webimage.jpg deleted file mode 100644 index 12dd1a4739..0000000000 Binary files a/docs/source/setup/images/volttron-webimage.jpg and /dev/null differ diff --git a/docs/source/setup/index.rst b/docs/source/setup/index.rst deleted file mode 100644 index a29b03b90b..0000000000 --- a/docs/source/setup/index.rst +++ /dev/null @@ -1,374 +0,0 @@ -.. _setup: - -.. _Building-VOLTTRON: - -Installing VOLTTRON -=================== - -.. note:: Volttron version 7.0rc1 is currently tested for Ubuntu versions 18.04 and - 18.10 as well as Linux Mint version 19.3. Version 6.x is tested for Ubuntu - versions 16.04 and 18.04 as well as Linux Mint version 19.1. - - -Install Required Software -------------------------- -Ensure that all the -:ref:`required packages ` are installed. - - -Clone VOLTTRON source code --------------------------- -From version 6.0 VOLTTRON supports two message bus - ZMQ and RabbitMQ. For the latest -build use the develop branch. For a more conservative branch -please use the master branch. - -:: - - git clone https://github.com/VOLTTRON/volttron --branch - -For other options see: :ref:`Getting VOLTTRON ` - - -Setup virtual environment -------------------------- - -The VOLTTRON project includes a bootstrap script which automatically -downloads dependencies and builds VOLTTRON. The script also creates a -Python virtual environment for use by the project which can be activated -after bootstrapping with `. env/bin/activate`. This activated Python -virtual environment should be used for subsequent bootstraps whenever -there are significant changes. The system's Python need only be used on -the initial bootstrap. - -Steps for ZMQ -~~~~~~~~~~~~~ - -:: - - cd - python bootstrap.py - source env/bin/activate - -Proceed to `Testing the Installation`_. - - -Steps for RabbitMQ -~~~~~~~~~~~~~~~~~~ - -1. Install Erlang version >= 21 -############################### - - For RabbitMQ based VOLTTRON, some of the RabbitMQ specific software packages have to be installed. - If you are running an **Debian or CentOS system**, you can install the RabbitMQ dependencies by running the - rabbit dependencies script, passing in the os name and approriate distribution as a parameter. - The following are supported - - * debian bionic (for Ubuntu 18.04) - * debian xenial (for Ubuntu 16.04) - * debian xenial (for Linux Mint 18.04) - * debian stretch (for Debian Stretch) - * centos 7 (for CentOS 7) - * centos 6 (for CentOS 6) - - Example command - - :: - - ./scripts/rabbit_dependencies.sh debian xenial - - **Alternatively** - - You can download and install Erlang from `Erlang Solution `_ - Please include OTP/components - ssl, public_key, asn1, and crypto. - Also lock version of Erlang using the `yum-plugin-versionlock `_ - -2. Configure hostname -###################### - - Rabbitmq requires a valid hostname to start. Use the command hostname on your linux machine to verify if a valid - hostname is set. If not add a valid hostname to the file /etc/hostname. You would need sudo access to edit this file - If you want your rabbitmq instance to be reachable externally, then a hostname should be resolvable to a valid ip. - In order to do this you need to have a entry in /etc/hosts file. For example, the below shows a valid /etc/hosts file - - .. code:: - - 127.0.0.1 localhost - 127.0.0.1 myhost - - 192.34.44.101 externally_visible_hostname - - After the edit, logout and log back in for the changes to take effect. - - If you are testing with VMs make please make sure to provide unique host names for each of the VM you are using. - - .. note:: - - If you change /etc/hostname after setting up rabbitmq (/rabbitmq_server-3.7.7) - -3. Bootstrap -############ - - Install the required software by running the bootstrap script with --rabbitmq option - - :: - - cd volttron - - # bootstrap.py --help will show you all of the "package options" such as - # installing required packages for volttron central or the platform agent. - - python bootstrap.py --rabbitmq [optional install directory defaults to - /rabbitmq_server] - - .. note:: If your PYTHON_PATH is configured for Python 2.7, you'll need to use - ``python3 bootstrap.py ..`` - - This will build the platform and create a virtual Python environment and - dependencies for RabbitMQ. It also installs RabbitMQ server as the current user. - If an install path is provided, path should exists and be writeable. RabbitMQ - will be installed under /rabbitmq_server-3.7.7 Rest of the - documentation refers to the directory /rabbitmq_server-3.7.7 as - $RABBITMQ_HOME - - You can check if RabbitMQ server is installed by checking it's status. - - :: - - $RABBITMQ_HOME/sbin/rabbitmqctl status - - - Please note, RABBITMQ_HOME environment variable can be set in ~/.bashrc. If doing so, - it needs to be set to RabbitMQ installation directory (default path is - /rabbitmq_server/rabbitmq_server-3.7.7) - - :: - - echo 'export RABBITMQ_HOME=$HOME/rabbitmq_server/rabbitmq_server-3.7.7'|tee --append ~/.bashrc | source ~/.bashrc - # Reload the environment variables in the current shell - source ~/.bashrc - - -4. Activate the environment -########################### - - :: - - source env/bin/activate - -5. Create RabbitMQ setup for VOLTTRON -###################################### - - :: - - vcfg --rabbitmq single [optional path to rabbitmq_config.yml] - - Refer to examples/configurations/rabbitmq/rabbitmq_config.yml for a sample configuration file. At a minimum you would - need to provide the host name and a unique common-name (under certificate-data) in the - configuration file. Note. common-name must be unique and the general conventions is to use -root-ca. - - Running the above command without the optional configuration file parameter will prompt user for all the - needed data at the command prompt and use that to generate a rabbitmq_config.yml file in VOLTTRON_HOME - directory. - - This scripts creates a new virtual host and creates SSL certificates needed for this VOLTTRON instance. - These certificates get created under the sub directory "certificates" in your VOLTTRON home - (typically in ~/.volttron). It then creates the main VIP exchange named "volttron" to route message - between platform and agents and alternate exchange to capture unrouteable messages. - - NOTE: We configure RabbitMQ instance for a single volttron_home and volttron_instance. This script will - confirm with the user the volttron_home to be configured. volttron instance name will be read from - volttron_home/config if available, if not user will be prompted for volttron instance name. To run the - scripts without any prompts, save the volttron instance name in volttron_home/config file and pass the - volttron home directory as command line argument For example: "vcfg --vhome /home/vdev/.new_vhome --rabbitmq single" - - Following is the example inputs for "vcfg --rabbitmq single" command. Since no config file is passed the - script prompts for necessary details. - - :: - - Your VOLTTRON_HOME currently set to: /home/vdev/new_vhome2 - - Is this the volttron you are attempting to setup? [Y]: - Creating rmq config yml - RabbitMQ server home: [/home/vdev/rabbitmq_server/rabbitmq_server-3.7.7]: - Fully qualified domain name of the system: [cs_cbox.pnl.gov]: - - Enable SSL Authentication: [Y]: - - Please enter the following details for root CA certificates - Country: [US]: - State: Washington - Location: Richland - Organization: PNNL - Organization Unit: Volttron-Team - Common Name: [volttron1-root-ca]: - Do you want to use default values for RabbitMQ home, ports, and virtual host: [Y]: N - Name of the virtual host under which RabbitMQ VOLTTRON will be running: [volttron]: - AMQP port for RabbitMQ: [5672]: - http port for the RabbitMQ management plugin: [15672]: - AMQPS (SSL) port RabbitMQ address: [5671]: - https port for the RabbitMQ management plugin: [15671]: - INFO:rmq_setup.pyc:Starting rabbitmq server - Warning: PID file not written; -detached was passed. - INFO:rmq_setup.pyc:**Started rmq server at /home/vdev/rabbitmq_server/rabbitmq_server-3.7.7 - INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost - INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost - INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost - INFO:rmq_setup.pyc: - Checking for CA certificate - - INFO:rmq_setup.pyc: - Root CA (/home/vdev/new_vhome2/certificates/certs/volttron1-root-ca.crt) NOT Found. Creating root ca for volttron instance - Created CA cert - INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost - INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost - INFO:rmq_setup.pyc:**Stopped rmq server - Warning: PID file not written; -detached was passed. - INFO:rmq_setup.pyc:**Started rmq server at /home/vdev/rabbitmq_server/rabbitmq_server-3.7.7 - INFO:rmq_setup.pyc: - - ####################### - - Setup complete for volttron home /home/vdev/new_vhome2 with instance name=volttron1 - Notes: - - Please set environment variable VOLTTRON_HOME to /home/vdev/new_vhome2 before starting volttron - - On production environments, restrict write access to - /home/vdev/new_vhome2/certificates/certs/volttron1-root-ca.crt to only admin user. For example: sudo chown root /home/vdev/new_vhome2/certificates/certs/volttron1-root-ca.crt - - A new admin user was created with user name: volttron1-admin and password=default_passwd. - You could change this user's password by logging into https://cs_cbox.pnl.gov:15671/ Please update /home/vdev/new_vhome2/rabbitmq_config.yml if you change password - - ####################### - - -Testing the Installation ------------------------- - -We are now ready to start VOLTTRON instance. If configured with RabbitMQ message bus a config file would have been -generated in $VOLTTRON_HOME/config with the entry message-bus=rmq. If you need to revert back to ZeroMQ based VOLTTRON, -you will have to either remove "message-bus" parameter or set it to default "zmq" in $VOLTTRON_HOME/config. -The following command starts volttron process in the background - -:: - - volttron -vv -l volttron.log& - -This enters the virtual Python environment and then starts the platform in debug (vv) mode with a log file -named volttron.log. Alternatively you can use the utility script start-volttron script that does the same. To stop -stop volttron you can use the stop-volttron script. - -:: - - ./start-volttron - - -.. warning:: - If you plan on running VOLTTRON in the background and detaching it from the - terminal with the ``disown`` command be sure to redirect stderr and stdout to ``/dev/null``. - Some libraries which VOLTTRON relies on output directly to stdout and stderr. - This will cause problems if those file descriptors are not redirected to ``/dev/null`` - - :: - - #To start the platform in the background and redirect stderr and stdout - #to /dev/null - volttron -vv -l volttron.log > /dev/null 2>&1& - - - -Installing and Running Agents ------------------------------ - -VOLTTRON platform comes with several built in services and example agents out of the box. To install a agent -use the script install-agent.py - -:: - - python scripts/install-agent.py -s [-c ] - - -For example, we can use the command to install and start the Listener Agent - a simple agent that periodically publishes -heartbeat message and listens to everything on the message bus. Install and start the Listener agent using the -following command. - -:: - - python scripts/install-agent.py -s examples/ListenerAgent --start - - -Check volttron.log to ensure that the listener agent is publishing heartbeat messages. - -:: - - tail volttron.log - -:: - - 2016-10-17 18:17:52,245 (listeneragent-3.2 11367) listener.agent INFO: Peer: 'pubsub', Sender: 'listeneragent-3.2_1':, Bus: u'', Topic: 'heartbeat/listeneragent-3.2_1', Headers: {'Date': '2016-10-18T01:17:52.239724+00:00', 'max_compatible_version': u'', 'min_compatible_version': '3.0'}, Message: {'status': 'GOOD', 'last_updated': '2016-10-18T01:17:47.232972+00:00', 'context': 'hello'} - - -You can also use the vctl or volttron-ctl command to start, stop or check the status of an agent - -:: - - (volttron)volttron@volttron1:~/git/rmq_volttron$ vctl status - AGENT IDENTITY TAG STATUS HEALTH - 6 listeneragent-3.2 listeneragent-3.2_1 running [13125] GOOD - f master_driveragent-3.2 platform.driver master_driver - -:: - - vctl stop - - -To stop the platform: - -:: - - volttron-ctl shutdown --platform - -or - -:: - - ./stop-volttron - -**Note:** The default working directory is ~/.volttron. The default -directory for creation of agent packages is ~/.volttron/packaged - - - -Next Steps ----------- - -Now that the project is configured correctly: - -See the following links for core services and volttron features: - - * :ref:`Core Services` - * :ref:`Platform Specifications` - -See the following links for agent development: - - * :ref:`Agent Development ` - * :ref:`VOLTTRON Development in Eclipse ` - * :ref:`VOLTTRON Development in PyCharm ` - - -Please refer to related topics to for advanced setup instructions - -Related Topics --------------- - -.. toctree:: - :glob: - :maxdepth: 2 - - RabbitMQ/index - * - diff --git a/docs/source/setup/planning-install.rst b/docs/source/setup/planning-install.rst deleted file mode 100644 index 6acb8882b4..0000000000 --- a/docs/source/setup/planning-install.rst +++ /dev/null @@ -1,97 +0,0 @@ -.. _planning-install: - -=========================== -Planning a VOLTTRON Install -=========================== - -The 3 major installation types for VOLTTRON are doing development, doing research using VOLTTRON, and -collecting and managing physical devices. - -Development and Research installation tend to be smaller footprint installations. For development, the -data is usually synthetic or copied from another source. The existing documentation covers development -installs in significant detail. - -Other deployments will have a better installation experience if they consider certain kinds of questions -while they plan their installation. - -Questions -========= - - * Do you want to send commands to the machines ? - * Do you want to store the data centrally ? - * How many machines do you expect to collect data from on each "collector" ? - * How often will the machines collect data ? - * Are all the devices visible to the same network ? - * What types of VOLTTRON applications do you want to run ? - - -Commands --------- - -If you wish to send commands to the devices, you will want to install and configure the Volttron Central -agent. If you are only using VOLTTRON to securely collect the data, you can turn off the extra agents -to reduce the footprint. - -Storing Data ------------- - -VOLTTRON supports multiple historians. mySQL and MongoDB are the most commonly used. As you plan your -installation, you should consider how quickly you need access to the data and where. If you are looking -at the health and well-being of an entire suite of devices, its likely that you want to do that from a -central location. Analytics can be performed at the edge by VOLTTRON applications or can be performed -across the data usually from a central data repository. The latency that you can tolerate in your data -being available will also determine choices in different agents (ForwardHistorian versus Data Mover) - - -How Many --------- - -The ratio of how many devices-to-collector machine is based on several factors. These include: - - * how much memory and network bandwidth the collection machine has. More = More devices - * how fast the local storage is can affect how fast the data cache can be written. Very slow - storage devices can fall behind - -The second half of the "how many" question is how many collector paltforms are writing to a single -VOLTTRON platform to store data - and whether that storage is local, remote, big enough, etc. - -If you are storing more than moderate amount of data, you will probably benefit from installing -your database on a different machine than your concreate historian machine. Note: This is -contra-indicated if you have a slow network connection between you concrete historian and your database machine. - -In synthetic testing up to 6 virtual machines hosting 500 devices each ( 18 points) were easily -supported by a single centralized platform writing to a Mongo database - using a high speed network. -That central platform experienced very little CPU or memory load when the VOLTTRON Central agent was disabled. - - -How Often ---------- - -This question is closely related to the last. A higher sampling frequency will create more data. This -wil place more work in the storage phase. - - -Networks --------- - -In many cases, there are constraints on how networks can interact with each other. In many cases, -these include security considerations. On some sites, the primary network will be protected from less -secure networks and may require different installation considerations. For example, if a data collector -machine and the database machine are on the same network with sufficient security, you may choose -to have the data collector write directly to the database. If the collector is on an isolated building -network then you will likely need to use the ForwardHistorian to bridge the two networks. - - -Other Considerations --------------------- - -Physical location and maintenance of collector machines must be considered in all live deployments. -Although the number of data points may imply a heavy load on a data collection box, the physical constraints -may limit the practicality of having more than a single box. The other side of that discussion is deploying -many collector boxes may be simpler initially, but may create a maintenance challenge if you don't -plan ahead on how you apply patches, etc. - -Naming conventions should also be considered. The ability to trace data through the system and identify -the collector machine and device can be invaluable in debugging and analysis. - - diff --git a/docs/source/specifications/agent-vip-id.rst b/docs/source/specifications/agent-vip-id.rst deleted file mode 100644 index 67b4127197..0000000000 --- a/docs/source/specifications/agent-vip-id.rst +++ /dev/null @@ -1,98 +0,0 @@ -.. _vip-identity-assignment: - -=========================================== -Agent VIP IDENTITY Assignment Specification -=========================================== - -This document explains how an agent obtains it's VIP IDENTITY, how the platform sets an agent's VIP IDENTITY at startup, and what mechanisms are available to the user to set the VIP IDENTITY for any agent. - -What is a VIP IDENTITY ----------------------- - -A VIP IDENTITY is a platform instance unique identifier for agents. The IDENTITY is used to route messages from one Agent through the VOLTTRON router to the recipiant Agent. The VIP IDENTITY provides a consistant, user defined, and human readable character set to build a VIP IDENTITY. VIP IDENTITIES should be composed of both upper and lowercase lettters, numbers and the following special caracters _.-. - - -Runtime -------- - -The primary interface for obtaining a VIP IDENTITY *at runtime* is via the runtime environment of the agent. At startup the utility function vip_main shall check for the environment variable **AGENT_VIP_IDENTITY**. If the **AGENT_VIP_IDENTITY** environment variable is not set then the vip_main function will fall back to a supplied identity argument. vip_main will pass the appropriate identity argument to the agent constructor. If no identity is set the Agent class will create a random VIP IDENTITY using python's uuid4 function. - -An agent that inherits from the platform's base Agent class can get it's current VIP IDENTITY by retrieving the value of self.core.identity. - -The primary use of the 'identity' argument to vip_main is for agent development. For development it allows agents to specify a default VIP IDENTITY when run outside the platform. As platform Agents are not started via vip_main they will simply receive their VIP IDENTITY via the identity argument when they are instantiated. Using the identity argument of the Agent constructor to set the VIP IDENTITY via agent configuration is no longer supported. - -At runtime the platform will set the environment variable **AGENT_VIP_IDENTITY** to the value set at installation time. - -Agents not based on the platform's base Agent should set their VIP IDENTITY by setting the identity of the ZMQ socket before the socket connects to the platform. If the agent fails to set it's VIP IDENTITY via the ZMQ socket it will be selected automatically by the platform. This platform chosen ID is currently not discoverable to the agent. - -Agent Implementation --------------------- - -If an Agent has a preferred VIP IDENTITY (for example the MasterDriverAgent prefers to use "platform.driver") it may specify this as a default packed value. This is done by including a file named IDENTITY containing only the desired VIP IDENTITY in ASCII plain text in the same directory at the setup.py file for the Agent. This will cause the packaged agent wheel to include an instruction to set the VIP IDENTITY at installation time. - -This value may be overridden at packaging or installation time. - -Packaging ---------- - -An Agent may have it's VIP IDENTITY configured when it is packaged. The packaged value may be used by the platform to set the **AGENT_VIP_IDENTITY** environment variable for the agent process. - -The packaged VIP IDENTITY may be overridden at installation time. This overrides any preferred VIP IDENTITY of the agent. This will cause the packaged agent wheel to include an instruction to set the VIP IDENTITY at installation time. - -To specify the VIP IDENTITY when packaging use the *--vip-identity* option when running "volttron-pkg package". - -Installation ------------- - -An agent may have it's VIP IDENTITY configured when it is installed. This overrides any VIP IDENTITY specified when the agent was packaged. - -To specify the VIP IDENTITY when packaging use the *--vip-identity* option when running "volttron-ctl install". - -Installation Default VIP IDENTITY -********************************* - -If no VIP IDENTITY has been specified by installation time the platform will assign one automatically. - -The platform uses the following template to generate a VIP IDENTITY: - -.. code-block:: python - - "{agent_name}_{n}" - -{agent_name} is substituted with the name of the actual agent such as "listeneragent-0.1" - -{n} is a number to make VIP IDENTITY unique. {n} is set to the first unused number (starting from 1) for all installed instances of an agent. e.g. If there are 2 listener agents installed and the first (VIP IDENTITY listeneragent-0.1_1) is uninstalled leaving the second (VIP IDENTITY "listeneragent-0.1_2") a new listener agent will receive the VIP IDENTITY "listeneragent-0.1_1" when installed. The next installed listener will receive a VIP IDENTITY of "listeneragent-0.1_3". - -The # sign is used to prevent confusing the agent version number with the installed instance number. - -If an agent is repackaged with a new version number it is treated as a new agent and the number will start again from 1. - -VIP IDENTITY Conflicts During Installation -****************************************** - -If an agent is assigned a VIP IDENTITY besides the default value given to it by the platform it is possible for VIP IDENTITY conflicts to exist between installed agents. In this case the platform rejects the installation of an agent with a conflicting VIP IDENTITY and reports an error to the user. - -VIP IDENTITY Conflicts During Runtime -************************************* - -In the case where agents are not started through the platform (usually during development or when running standalone agents) it is possible to encounter a VIP IDENTITY conflict during runtime. In this case the first agent to use a VIP IDENTITY will function as normal. Subsequent agents will still connect to the ZMQ socket but will be silently rejected by the platform router. The router will not route any message to that Agent. Agents using the platforms base Agent class will detect this automatically during the initial handshake with the platform. This condition will shutdown the Agent with an error indicating a VIP IDENTITY conflict as the most likely cause of the problem. - -Auto Numbering With Non-Default VIP IDENTITYs ---------------------------------------------- - -It is possible to use the auto numbering mechanism that the default VIP IDENTITY scheme uses. Simply include the string "{n}" somewhere in the requested VIP IDENTITY and it will be replaced with a number in the same manner as the default VIP IDENTITY is. Python string.format() escaping rules apply. `See this question on StackOverflow. `__ - -Script Features ---------------- - -The scripts/install-agent.py script supports specifying the desired VIP IDENTITY using the -i (or --vip-identity) option - -Security/Privacy ----------------- - -Currently, much like the TAG file in an installed agent, there is nothing to stop someone from modifying the IDENTITY file in the installed agent. - -Constraints and Limitations ---------------------------- - -Currently there is no way for an agent based on the platform base Agent class to recover from a VIP IDENTITY conflict. As that is case only affects developers and a very tiny minority of users and is reported via an error message, there are no plans to fix it. diff --git a/docs/source/specifications/chargepoint_driver.rst b/docs/source/specifications/chargepoint_driver.rst deleted file mode 100644 index 6ff018b9aa..0000000000 --- a/docs/source/specifications/chargepoint_driver.rst +++ /dev/null @@ -1,431 +0,0 @@ -.. _Chargepoint-Driver: - -Chargepoint API Driver -====================== - -Spec Version 1.1 - -`ChargePoint `_ operates the largest independently owned EV charging network in the US. -It sells charge stations to businesses and provides a web application to manage and report on these chargestations. -Chargepoint offers a `Web Services API `_ -that its customers may use to develop applications that integrate with the chargepoint network devices. - -The Chargepoint API Driver for VOLTTRON will enable real-time monitoring and control of Chargepoint EVSEs within -the VOLTTRON platform by creating a standard VOLTTRON device driver on top of the Chargepoint Web Services API. -Each port on each managed chargestation will look like a standard VOLTTRON device, monitored and controlled through -the VOLTTRON device driver interface. - - -Driver Scope & Functions ------------------------- - -This driver will enable VOLTTRON to support the following use cases with Chargepoint EVSEs: - - - Monitoring of chargestation status, load and energy consumption - - Demand charge reduction - - Time shifted charging - - Demand response program participation - -The data and functionality to be made available through the driver interface will be implemented using the -following Chargepoint web services: - - -================================ ==================================================================== -API Method Name Key Data/Function Provided -================================ ==================================================================== -getStationStatus Port status: AVAILABLE, INUSE, UNREACHABLE, UNKNOWN -shedLoad Limit station power by percent or max load for some time period. -clearShedState Clear all shed state and allow normal charging -getLoad Port load in Kw, shedState, allowedLoad, percentShed -getAlarms Only the last alarm will be available. -clearAlarms Clear all alarms. -getStationRights Name of station rights profile, eg. 'network_manager' -getChargingSessionData Energy used in last session, start/end timestamps -getStations Returns description/address/nameplate of chargestation. -================================ ==================================================================== - -The Chargepoint Driver will implement version 5.0 Rev 7 of the Chargepoint API. While the developer's guide -is not yet publicly available, the WSDL Schema is. -*Note: Station Reservation API has been removed from the 5.0 version of the API.* - -WSDL for this API is located here: - - https://webservices.chargepoint.com/cp_api_5.0.wsdl - - -Mapping VOLTTRON Device Interface to Chargepoint APIs ------------------------------------------------------ - -The VOLTTRON driver interface represents a single device as a list of registers accessed through a simple get_point/ -set_point API. In contrast, the Chargepoint web services for real-time monitoring and control are spread across -eight distinct APIs that return hierarchical XML. The Chargepoint driver is the adaptor that will make a suite -of web services look like a single VOLTTRON device. - - - -Device Mapping -^^^^^^^^^^^^^^ - -The chargepoint driver will map a single VOLTTRON device (a driver instance) to one chargestation. Since -a chargestation can have multiple ports, each with their own set of telemetry, the registry will include a port -index column on attributes that are specific to a port. This will allow deployments to use an indexing convention -that has been followed with other drivers. (See Registry Configuration for more details) - -Requirements ------------- - -The chargepoint driver requires at least one additional python library and has its own ``requirements.txt``. -Make sure to run - -:: - - pip install -r /requirements.txt - -before using this driver. - -Driver Configuration --------------------- - -Each device must be configured with its own Driver Configuration File. The Driver Configuration must reference -the Registry Configuration File, defining the set of points that will be available from the device. For -chargestation devices, the ``driver_config`` entry of the Driver Configuration file will need to contain all -parameters required by the web service API: - - -======================= ========================================================================== -Parameter Purpose -======================= ========================================================================== -username Credentials established through Chargepoint account -password -stationID Unique station ID assigned by chargepoint -======================= ========================================================================== - -The ``driver_type`` must be ``chargepoint`` - - A sample driver configuration file for a single device, looks like this: - -.. code-block:: json - - { - "driver_config": { - "username" : "1b905c936af141b98f9b0f816087f3605a30c1df1d07f146281b151", - "password" : "**Put your chargepoint API passqword here**", - "stationID" : "1:34003", - }, - "driver_type": "chargepoint", - "registry_config":"config://chargepoint.csv", - "interval": 60, - "heart_beat_point": "heartbeat" - } - - - -API Plans & Access Rights -^^^^^^^^^^^^^^^^^^^^^^^^^ - -Chargepoint offers API plans that vary in available features and access rights. Some of the API calls -to be implemented here are not available across all plans. Furthermore, the attributes returned in response -to an API call may be limited by the API plan and access rights associated with the userid. Runtime -exceptions related to plans and access rights will generate DriverInterfaceError exceptions. These can be -avoided by using a registry configuration that does not include APIs or attributes that are not -available to the . - - -Registry Configuration ----------------------- - -The registry file defines the individual points that will be exposed by the Chargepoint driver. It should only -reference points that will actually be used since each point is potentially an additional web service call. The driver -will be smart and limit API calls to those that are required to satisfy the points found in the CSV. - -Naming of points will conform to the conventions established by the Chargepoint Web services API whenever possible. -Note that Chargepoint naming conventions are camel-cased with no spaces or hyphens. Multi-word names start -with a lowercase letter. Single word names start uppercase. - -The available registry entries for each API method name are shown below along with a description of any -notable behavior associated with that register. Following that is a sample of the -associated XML returned by the API. - - -getStationStatus -^^^^^^^^^^^^^^^^ - -The getStationStatus query returns information for all ports on the chargestation. - -.. note:: - - In all the registry entries shown below, the **Attribute Name** column defines the unique name within the - chargepoint driver that must be used to reference this particular attribute and associated API. The - **VOLTTRON point name** usually matches the **Attribute Name** in these examples but may be changed during deployment. - - -.. csv-table:: getStationStatus - :header: Volttron Point Name,Attribute Name,Register Name,Port #,Type,Units,Starting Value,Writable,Notes - - Status,Status,StationStatusRegister,1,string,,,FALSE,"AVAILABLE, INUSE, UNREACHABLE, UNKNOWN " - Status.TimeStamp,TimeStamp,StationStatusRegister,1,datetime,,,FALSE,Timestamp of the last communication between the station and ChargePoint - -Sample XML returned by getStationStatus. - -.. code-block:: xml - - - 100 - API input request executed successfully. - - 1:33923 - - 1 - AVAILABLE - 2016-11-07T19:19:19Z - - - 2 - INUSE - 2016-11-07T19:19:19Z - - - 0 - - - -getLoad, shedLoad, clearShedState -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Reading any of these values will return the result of a call to getLoad. Writing shedState=True will call -shedLoad and pass the last written value of allowedLoad or percentShed. The API allows only one of these -two values to be provided. Writing to allowedLoad will simultaneously set percentShed to None and vice -versa. - -.. csv-table:: getLoad, shedLoad, clearShedState - :header: Volttron Point Name,Attribute Name,Register Name,Port #,Type,Units,Starting Value,Writable,Notes - - shedState,shedState,LoadRegister,1,integer,0 or 1,0,TRUE,True when load shed limits are in place - portLoad,portLoad,LoadRegister,1,float,kw,,FALSE,Load in kw - allowedLoad,allowedLoad,LoadRegister,1,float,kw,,TRUE,Allowed load in kw when shedState is True - percentShed,percentShed,LoadRegister,1,integer,percent,,TRUE,Percent of max power shed when shedState is True - -Sample XML returned by getLoad - -.. code-block:: xml - - - 100 - API input request executed successfully. - - - - - 1:33923 - ALCOGARSTATIONS / ALCOPARK 8 -005
165 13th St, Oakland, California, 94612, United States
- 3.314 - - 1 - - - 0 - 0.000 - 0.000 - 0 - - - 2 - 664719 - CNCP0000481668 - 0 - 3.314 - 0.000 - 0 - - - - -Sample shedLoad XML query to set the allowed load on a port to 3.0kw. - -.. code-block:: xml - - - - - 1:123456 - - - 1 - 3.0 - - - - - - - - -getAlarms, clearAlarms -^^^^^^^^^^^^^^^^^^^^^^ - -The getAlarms query returns a list of all alarms since last cleared. The driver interface will only return -data for the most recent alarm, if present. While the getAlarm query provides various station identifying -attributes, these will be made available through registers associated with the getStations API. If an alarm is -not specific to a particular port, it will be associated with all chargestation ports and available through any -of its device instances. - -Write ``True`` to clearAlarms to submit the clearAlarms query to the **chargestation**. It will clear alarms -across all ports on that chargestation. - - -.. csv-table:: getAlarms, clearAlarms - :header: Volttron Point Name,Attribute Name,Register Name,Port #,Type,Units,Starting Value,Writable,Notes - - alarmType,alarmType,AlarmRegister,,string,,,FALSE,eg. 'GFCI Trip' - alarmTime,alarmTime,AlarmRegister,,datetime,,,FALSE, - clearAlarms,clearAlarms,AlarmRegister,,int,,0,TRUE,Sends the clearAlarms query when set to True - - -.. code-block:: xml - - - 1:33973 - ALCOGARSTATIONS / ALCOPARK 8 -003 - CT2100-HD-CCR - 1:ORG07225 - Alameda County - - 115110013418 - - Reachable - 2016-09-26T12:19:16Z - 1 - - - -getStationRights -^^^^^^^^^^^^^^^^ - -Returns the name of the stations rights profile. A station may have multiple station rights profiles, each associated -with a different station group ID. For this reason, the stationRightsProfile register will return a dictionary of -(sgID, name) pairs. Since this is a chargestation level attribute, it will be returned for all ports. - - -.. csv-table:: getStationRights - :header: Volttron Point Name,Attribute Name,Register Name,Port #,Type,Units,Starting Value,Writable,Notes - - stationRightsProfile,stationRightsProfile,StationRightsRegister,,dictionary,,,FALSE,"Dictionary of sgID, rights name tuples." - - - -.. code-block:: xml - - - 39491 - AlcoPark 8 - network_manager - - 1:34003 - ALCOGARSTATIONS / ALCOPARK 8 -004 - 115110013369 - 000D:6F00:0154:F1FC - - - - 58279 - AlcoGarageStations - network_manager - - 1:34003 - ALCOGARSTATIONS / ALCOPARK 8 -004 - 115110013369 - 000D:6F00:0154:F1FC - - - - -getChargingSessionData -^^^^^^^^^^^^^^^^^^^^^^ - -Like getAlarms, this query returns a list of session data. The driver interface implementation will make the -last session data available. - -.. csv-table:: getChargingSessionData - :header: Volttron Point Name,Attribute Name,Register Name,Port #,Type,Units,Starting Value,Writable,Notes - - sessionID,sessionID,ChargingSessionRegister,1,string,,,FALSE, - startTime,startTime,ChargingSessionRegister,1,datetime,,,FALSE, - endTime,endTime,ChargingSessionRegister,1,datetime,,,FALSE, - Energy,Energy,ChargingSessionRegister,1,float,,,FALSE, - rfidSerialNumber,rfidSerialNumber,ChargingSessionRegister,1,string,,,FALSE, - driverAccountNumber,driverAccountNumber,ChargingSessionRegister,1,string,,,FALSE, - driverName,driverName,ChargingSessionRegister,1,string,,,FALSE, - -.. code-block:: xml - - - 1:34003 - ALCOGARSTATIONS / ALCOPARK 8 -004 - 2 -
165 13th St, Oakland, California, 94612, United States
- Oakland - California - United States - 94612 - 53068029 - 12.120572 - 2016-10-25T15:53:35Z - 2016-10-25T20:14:46Z - 452777 - 1 - 490178743 - - - -getStations -^^^^^^^^^^^ - -This API call returns a complete description of the chargestation in 40 fields. This information is essentially -static and will change infrequently. It should not be scraped on a regular basis. The list of attributes will be -included in the registry CSV but are only listed here: - -.. code-block:: text - - stationID, stationManufacturer, stationModel, portNUmber, stationName, stationMacAddr, stationSerialNum, Address, City, - State, Country, postalCode, Lat, Long, Reservable, Level, Mode, Connector, Voltage, Current, Power, numPorts, Type, - startTime, endTime, minPrice, maxPrice, unitPricePerHour, unitPricePerSession, unitPricePerKWh, unitPricePerHourThereafter, - sessionTime, Description, mainPhone, orgID, organizationName, sgID, sgName, currencyCode - - -Engineering Discussion ----------------------- - - -Questions -^^^^^^^^^ - - - **Allowed python-type** - We propose a register with a `python-type` of dictionary. Is this OK? - - **Scrape Interval** - Scrape all should not return all registers defined in the CSV, we propose fine grained control with a scrape-interval on each register. Response: ok to add extra settings to registry but don't worry about pubishing static data with every scrape - - **Data currency** - Since devices are likely to share api calls, at least across ports, we need to think about the currency of the data and possibly allowing this to be a configurable parameter or derviced from the scrape interval. Response: add to CSV with default values if not present - - - -Performance -^^^^^^^^^^^ -Web service calls across the internet will be significantly slower than typical VOLTTRON Bacnet or Modbus devices. It -may be prohibitively expensive for each chargepoint sub-agent instance to make individual requests on behalf of -its own EVSE+port. We will need to examine the possibility of making a single request for all active chargestations -and sharing that information across driver instances. This could be done through a separate agent that regularly -queries the chargepoint network and makes the data available to each sub-agent via an RPC call. - - -3rd Party Library Dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The chargepoint driver implementation will depend on one additional 3rd part library that is not part of a standard -VOLTTRON installation: - -.. - - https://bitbucket.org/jurko/suds - - -Is there a mechanism for drivers to specify their own requirements.txt ? - -Driver installation and configuration documentation can reference requirement.txt - - diff --git a/docs/source/specifications/configuration-store.rst b/docs/source/specifications/configuration-store.rst deleted file mode 100644 index 5807ea8faa..0000000000 --- a/docs/source/specifications/configuration-store.rst +++ /dev/null @@ -1,286 +0,0 @@ -.. _ConfigurationStore: - -Agent Configuration Store -========================= - -This document describes the configuration store feature and explains how an agent uses it. - -The configuration store enables users to store agent configurations on the platform and allows the agent to automatically retrieve them during runtime. Users may update the configurations and the agent will automatically be informed of the changes. - -Compatibility -------------- - -Supporting the configuration store will *not* be required by Agents, however the usage will be strongly encouraged as it should substantially improve user experience. - -The previous method for configuring an agent will still be available to agents (and in some cases required). However agents can be created to only work with the configuration store and not support the old method at all. - -It will be possible to create an agent to use the traditional method for configuration to establish defaults if no configuration exist in the platform configuration store. - - -Configuration Names and Paths ------------------------------ - -Any valid OS file path name is a valid configuration name. Any leading or trailing "/", "\" and whitespace is removed by the store. - -The canonical name for the main agent configuration is "config". - -The configuration subsystem remembers the case of configuration names. Name matching is case insensitive both on the Agent and platform side. Configuration names are reported to agent callbacks in the original case used when adding them to the configuration. If a new configuration is store with a different case of an existing name the new name case is used. - -Configuration Ownership ------------------------ - -Each configuration belongs to one agent and one agent only. When an agent refers to a configuration file via it's path it does not need to supply any information about its identity to the platform in the file path. The only configurations an agent has direct access to are it's own. The platform will only inform the owning agent configuration changes. - - -Configuration File Types ------------------------- - -Configurations files come in three types: json, csv, and raw. The type of a configuration file is declared when it is added to or changed in the store. - -The parser assumes the first row of every CSV file is a header. - -Invalid json or csv files are rejected at the time they are added to the store. - -Raw files are unparsed and accepted as is. - -Other parsed types may be added in the future. - -Configuration File Representation to Agents -------------------------------------------- - -JSON -**** - -A json file is parsed and represented as appropriate data types to the requester. - -Consider a file with the following contents: - -.. code-block:: json - - { - "result": "PREEMPTED", - "info": null, - "data": { - "agentID": "my_agent", - "taskID": "my_task" - } - } - -The file will be parsed and presented as a dictionary with 3 values to the requester. - -CSV -*** - -A CSV file is represented as a list of objects. Each object represents a row in the CSV file. - -For instance this (simplified) CSV file: - -.. csv-table:: Example CSV - :header: Volttron Point Name,Modbus Register,Writable,Point Address - - ReturnAirCO2,>f,FALSE,1001 - ReturnAirCO2Stpt,>f,TRUE,1011 - -Will be represented like this: - -.. code-block:: json - - [ - { - "Volttron Point Name": "ReturnAirCO2", - "Modbus Register": ">f", - "Writable": "FALSE", - "Point Address": "1001" - }, - { - "Volttron Point Name": "ReturnAirCO2Stpt", - "Modbus Register": ">f", - "Writable": "TRUE", - "Point Address": "1011" - } - ] - -Raw -*** - -Raw files are represented as a string containing the contents of the file. - -File references ---------------- - -The Platform Configuration Store supports referencing one configuration file from another. If a referenced file exists the contents of that file will replace the file reference when the file is sent to the owning agent. Otherwise the reference will be replaced with None. - -Only configurations that are parsed by the platform (currently "json" or "csv") will be examined for references. If the file referenced is another parsed file type (json or csv, currently) then the replacement will be the parsed contents of the file. - -In a json object the name of a value will never be considered a reference. - -A file reference is any value string that starts with "config://". The rest of the string is the path in the config store to that configuration. The config store path is converted to lower case for comparison purposes. - -Consider the following configuration files named "devices/vav1.config" and "registries/vav.csv", respectively: - -.. code-block:: json - - { - "driver_config": {"device_address": "10.1.1.5", - "device_id": 500}, - - "driver_type": "bacnet", - "registry_config":"config://registries/vav.csv", - "campus": "pnnl", - "building": "isb1", - "unit": "vav1" - } - -.. csv-table:: vav.csv - :header: Volttron Point Name,Modbus Register,Writable,Point Address - - ReturnAirCO2,>f,FALSE,1001 - ReturnAirCO2Stpt,>f,TRUE,1011 - -The resulting configuration returns when an agent asks for "devices/vav1.config". The python object will have the following configuration: - -.. code-block:: python - - { - "driver_config": {"device_address": "10.1.1.5", - "device_id": 500}, - - "driver_type": "bacnet", - "registry_config":[ - { - "Volttron Point Name": "ReturnAirCO2", - "Modbus Register": ">f", - "Writable": "FALSE", - "Point Address": "1001" - }, - { - "Volttron Point Name": "ReturnAirCO2Stpt", - "Modbus Register": ">f", - "Writable": "TRUE", - "Point Address": "1011" - } - ], - "campus": "pnnl", - "building": "isb1", - "unit": "vav1" - } - -Circular references are not allowed. Adding a file that creates a circular reference will cause that file to be rejected by the platform. - -If a file is changed in anyway ("NEW", "UPDATE", or "DELETE") and that file is referred to by another file then the platform considers the referring configuration as changed. The configuration subsystem on the Agent will call every callback listening to a file or any file referring to that file either directly or indirectly. - -Agent Configuration Sub System ------------------------------- - -The configuration store shall be implemented on the Agent(client) side in the form of a new subsystem called config. - -The subsystem caches configurations as the platform updates the state to the agent. Changes to the cache triggered by an RPC call from the platform will trigger callbacks in the agent. - -No callback methods are called until the "onconfig" phase of agent startup. A new phase to agent startup called "onconfig" will be added to the Core class. Originally it was planned to have this run after the "onstart" phase has completed but that is currently not possible. Ideally if an agent is using the config store feature it will not need any "onstart" methods. - -When the "onconfig" phase is triggered the subsystem will retrieve the current configuration state from the platform and call all callbacks registered to a configuration in the store to the "NEW" action. No callbacks are called before this point in agent startup. - -The first time callbacks are called at agent startup any callbacks subscribed to a configuration called "config" are called first. - -Configuration Sub System Agent Methods -************************************** - -These methods are part of the interface available to the Agent. - -config.get( config_name="config" ) - Get the contents of a configuration. If no name is provided the contents of the main agent configuration "config" is returned. This may not be called before "ONSTART" methods are called. If called during "ONSTART" phase it will trigger the subsystem to initialize early but will not trigger any callbacks. - -config.subscribe(callback, action=("NEW", "UPDATE", "DELETE"), pattern="*") - Sets up a callback for handling a configuration change. The platform will automatically update the agent when a configuration changes ultimately triggering all callbacks that match the pattern specified. The action argument describes the types of configuration change action that will trigger the callback. Possible actions are "NEW", "UPDATE", and "DELETE" or a tuple of any combination of actions. If no action is supplied the callback happens for all changes. A list of actions can be supplied if desired. If no file name pattern is supplied then the callback is called for all configurations. The pattern is an regex used match the configuration name. - -The callback will also be called if any file referenced by a configuration file is changed. - - The signature of the callback method is callback(config_name, action, contents) where file_name is the file that triggered the callback, action is the action that triggered the callback, and contents are the new contents of the configuration. Contents will be None on a "DELETE" action. All callbacks registered for "NEW" events will be called at agent startup after all "ONSTART" methods have been called. Unlike pubsub subscriptions, this may be called at any point in an agent's lifetime. - -config.unsubscribe(callback=None, config_name_pattern=None) - Unsubscribe from configuration changes. Specifying a callback only will unsubscribe that callback from all config name patterns they have been bound to. If a pattern only is specified then all callbacks bound to that pattern will be removed. Specifying both will remove that callback from that pattern. Calling with no arguments will remove all subscriptions. This will not be available in the first version of config store. - -config.unsubscribe_all() - Unsubscribe from all configuration changes. - -config.set( config_name, contents, trigger_callback=False ) - Set the contents of a configuration. This may not be called before "ONSTART" methods are called. This can be used by an agent to store agent state across agent installations. This will *not* trigger any callbacks unless trigger_callback is set to True. To prevent deadlock with the platform this method may not be called from a configuration callback function. Doing so will raise a RuntimeError exception. - - This will not modify the local configuration cache the Agent maintains. It will send the configuration change to the platform and rely on the subsequent update_config call. - -config.delete( config_name, trigger_callback=False ) - Remove the configuration from the store. This will *not* trigger any callbacks unless trigger_callback is True. To prevent deadlock with the platform this method may not be called from a configuration callback function. Doing so will raise a RuntimeError exception. - -config.list( ) - Returns a list of configuration names. - -config.set_default(config_name, contents, trigger_callback=False) - Set a default value for a configuration. DOES NOT modify the platform's configuration store but creates a default configuration that is used for agent configuration callbacks if the configuration does not exist in the store or the configuration is deleted from the store. The callback will only be triggered if trigger_callback is true and the configuration store subsystem on the agent is not aware of a configuration with that name from the platform store. - - Typically this will be called in the __init__ method of an agent with the parsed contents of the packaged configuration file. This may not be called from a configuration callback. Doing so will raise a RuntimeError. - -config.delete_default(config_name, trigger_callback=False) - Delete a default value for a configuration. I have no idea why you would ever call this. It is here for completeness. This may not be called from a configuration callback. Doing so will raise a RuntimeError. - - -Configuration Sub System RPC Methods -************************************ - -These methods are made available on each agent to allow the platform to communicate changes to a configuration to the affected agent. - -As these methods are not part of the exposed interface they are subject to change. - -config.update( config_name, action, contents=None, trigger_callback=True ) - called by the platform when a configuration was changed by some method other than the Agent changing the configuration itself. Trigger callback tells the agent whether or not to call any callbacks associate with the configuration. - -Notes on trigger_callback -************************* - -As the configuration subsystem calls all callbacks in the "onconfig" phase and none are called beforehand the trigger_callback setting is effectively ignored if an agent sets a configuration or default configuration before the end of the "onstart" phase. - -Platform Configuration Store ----------------------------- - -The platform configuration store handles the storage and maintenance of configuration states on the platform. - -As these methods are not part of the exposed interface they are subject to change. - -Platform RPC Methods -******************** - -Methods for Agents -++++++++++++++++++ - -Agent methods that change configurations do not trigger any callbacks unless trigger_callback is True. - -set_config( config_name, contents, trigger_callback=False ) - Change/create a configuration file on the platform. - -get_configs( ) - Get all of the configurations for an Agent. - -delete_config( config_name, trigger_callback=False ) - Delete a configuration. - -Methods for Management -++++++++++++++++++++++ - -manage_store_config( identity, config_name, contents, config_type="raw" ) - Change/create a configuration on the platform for an agent with the specified identity - -manage_delete_config( identity, config_name ) - Delete a configuration for an agent with the specified identity. Calls the agent's update_config with the action "DELETE_ALL" and no configuration name. - -manage_delete_store( identity ) - Delete all configurations for a VIP IDENTITY. - -manage_list_config( identity ) - Get a list of configurations for an agent with the specified identity. - -manage_get_config( identity, config_name, raw=True ) - Get the contents of a configuration file. If raw is set to True this function will return the original file, otherwise it will return the parsed representation of the file. - -manage_list_stores( ) - Get a list of all the agents with configurations. - -Direct Call Methods -+++++++++++++++++++ - -Services local to the platform who wish to use the configuration store may use two helper methods on the agent class created for this purpose. This allows the auth service to use the config store before the router is started. - -delete(self, identity, config_name, trigger_callback=False) - Same as functionality as delete_config, but the caller must specify the indentity of the config store. - -store(self, identity, config_name, contents, trigger_callback=False) - Same functionality as set_config, but the caller must specify the indentity of the config store. - -Command Line Interface -********************** - -The command line interface will consist of a new commands for the volttron-ctl program called "config" with four sub-commands called "store", "delete", "list", "get". These commands will map directly to the management RPC functions in the previous section. - - -Disabling the Configuration Store -********************************* - -Agents may optionally disable support for the configuration store by passing enable_store=False to the __init__ method of the Agent class. This allows temporary agents to not spin up the subsystem when it is not needed. Platform service agents that do not yet support the configuration store and the temporary agents used by volttron-ctl will set this value. - diff --git a/docs/source/specifications/dnp3_agent.rst b/docs/source/specifications/dnp3_agent.rst deleted file mode 100644 index 27dc528160..0000000000 --- a/docs/source/specifications/dnp3_agent.rst +++ /dev/null @@ -1,222 +0,0 @@ -.. _DNP3-Agent: - -DNP3 -==== - -`DNP3 `_ (Distributed Network Protocol) is -a set of communications protocols that are widely used by utilities such as -electric power companies, primarily for `SCADA `_ purposes. -It was adopted in 2010 -as `IEEE Std 1815-2010 `_, -later updated to `1815-2012 `_. - -VOLTTRON's DNP3Agent is an implementation of a DNP3 Outstation as specified in -IEEE Std 1815-2012. It engages in bidirectional network communications with a DNP3 Master, -which might be located at a power utility. - -Like some other VOLTTRON protocol agents (e.g. IEEE2030_5Agent), DNP3Agent can optionally be -front-ended by a DNP3 device driver running under VOLTTRON's MasterDriverAgent. This -allows a DNP3 Master to be treated like any other device in VOLTTRON's ecosystem. - -The VOLTTRON DNP3Agent implementation of an Outstation is built on PyDNP3, -an open-source library from Kisensum containing Python language -bindings for Automatak's C++ `opendnp3 `_ -library, the de facto reference implementation of DNP3. - -DNP3Agent exposes DNP3 application-layer functionality, creating an extensible -base from which specific custom behavior can be designed and supported. By default, DNP3Agent -acts as a simple transfer agent, publishing data received from the Master on -the VOLTTRON Message Bus, and responding to RPCs from other VOLTTRON agents -by sending data to the Master. - -Requirements ------------- - -PyDNP3 can be installed in an activated environment with: - -:: - - pip install pydnp3 - -RPC Calls -~~~~~~~~~ - -DNP3Agent exposes the following VOLTTRON RPC calls: - -.. code-block:: python - - def get_point(self, point_name): - """ - Look up the most-recently-received value for a given output point. - - @param point_name: The point name of a DNP3 PointDefinition. - @return: The (unwrapped) value of a received point. - """ - - def get_point_by_index(self, group, index): - """ - Look up the most-recently-received value for a given point. - - @param group: The group number of a DNP3 point. - @param index: The index of a DNP3 point. - @return: The (unwrapped) value of a received point. - """ - - def get_points(self): - """ - Look up the most-recently-received value of each configured output point. - - @return: A dictionary of point values, indexed by their VOLTTRON point names. - """ - - def set_point(self, point_name, value): - """ - Set the value of a given input point. - - @param point_name: The point name of a DNP3 PointDefinition. - @param value: The value to set. The value's data type must match the one in the DNP3 PointDefinition. - """ - - def set_points(self, point_list): - """ - Set point values for a dictionary of points. - - @param point_list: A dictionary of {point_name: value} for a list of DNP3 points to set. - """ - - def config_points(self, point_map): - """ - For each of the agent's points, map its VOLTTRON point name to its DNP3 group and index. - - @param point_map: A dictionary that maps a point's VOLTTRON point name to its DNP3 group and index. - """ - - def get_point_definitions(self, point_name_list): - """ - For each DNP3 point name in point_name_list, return a dictionary with each of the point definitions. - - The returned dictionary looks like this: - - { - "point_name1": { - "property1": "property1_value", - "property2": "property2_value", - ... - }, - "point_name2": { - "property1": "property1_value", - "property2": "property2_value", - ... - } - } - - If a definition cannot be found for a point name, it is omitted from the returned dictionary. - - :param point_name_list: A list of point names. - :return: A dictionary of point definitions. - """ - -Pub/Sub Calls -~~~~~~~~~~~~~ - -DNP3Agent uses two topics when publishing data to the VOLTTRON message bus: - - * **Point Values (default topic: dnp3/point)**: As DNP3Agent communicates with the Master, - it publishes received point values on the VOLTTRON message bus. - - * **Outstation status (default topic: dnp3/status)**: If the status of the DNP3Agent outstation - changes, for example if it is restarted, it publishes its new status on the VOLTTRON message bus. - -Data Dictionary of Point Definitions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -DNP3Agent loads and uses a data dictionary of point definitions, which are maintained by -agreement between the (DNP3Agent) Outstation and the DNP3 Master. -The data dictionary is stored in the agent's registry. - -Current Point Values -~~~~~~~~~~~~~~~~~~~~ - -DNP3Agent tracks the most-recently-received value for each point definition in its -data dictionary, regardless of whether the point value's source is a VOLTTRON RPC call or -a message from the DNP3 Master. - -Agent Configuration -~~~~~~~~~~~~~~~~~~~ - -The DNP3Agent configuration file specifies the following fields: - - - **local_ip**: (string) - Outstation's host address (DNS resolved). - Default: 0.0.0.0. - - **port**: (integer) - Outstation's port number - the port that the remote endpoint (Master) is listening on. - Default: 20000. - - **point_topic**: (string) - VOLTTRON message bus topic to use when publishing DNP3 point values. - Default: dnp3/point. - - **outstation_status_topic**: (string) - Message bus topic to use when publishing outstation status. - Default: dnp3/outstation_status. - - **outstation_config**: (dictionary) - Outstation configuration parameters. All are optional. Parameters include: - - -- **database_sizes**: (integer) - Size of each outstation database buffer. - Default: 10. - -- **event_buffers**: (integer) - Size of the database event buffers. - Default: 10. - -- **allow_unsolicited**: (boolean) - Whether to allow unsolicited requests. - Default: True. - -- **link_local_addr**: (integer) - Link layer local address. - Default: 10. - -- **link_remote_addr**: (integer) - Link layer remote address. - Default: 1. - -- **log_levels**: (list) - List of bit field names (OR'd together) that filter what gets logged by DNP3. - Default: [NORMAL]. Possible values: ALL, ALL_APP_COMMS, ALL_COMMS, NORMAL, NOTHING. - -- **threads_to_allocate**: (integer) - Threads to allocate in the manager's thread pool. - Default: 1. - -A sample DNP3Agent configuration file is available in `services/core/DNP3Agent/dnp3agent.config`. - -VOLTTRON DNP3 Device Driver ---------------------------- - -VOLTTRON's DNP3 device driver exposes get_point/set_point calls, and scrapes, for DNP3 points. - -The driver periodically issues DNP3Agent RPC calls to refresh its cached -representation of DNP3 data. It issues RPC calls to DNP3Agent as needed when -responding to get_point, set_point and scrape_all calls. - -For information about the DNP3 driver, see :ref:`DNP3 Driver Configuration `. - -Installing DNP3Agent --------------------- - -To install DNP3Agent, please consult the installation advice in `services/core/DNP3Agent/README.md`. -README.md specifies a default agent configuration, which can be overridden as needed. - -An agent installation script is available: - -.. code-block:: python - - $ export VOLTTRON_ROOT= - $ cd $VOLTTRON_ROOT - $ source services/core/DNP3Agent/install_dnp3_agent.sh - -When installing MesaAgent, please note that the agent's point definitions must be -loaded into the agent's config store. See install_dnp3_agent.sh for -an example of how to load them. - -For Further Information ------------------------ - -Questions? Please contact: - - - Rob Calvert (rob@kisensum.com) diff --git a/docs/source/specifications/driver-override.rst b/docs/source/specifications/driver-override.rst deleted file mode 100644 index fce4b90c75..0000000000 --- a/docs/source/specifications/driver-override.rst +++ /dev/null @@ -1,54 +0,0 @@ -.. _DriverOverride: - -Driver Override Specification -============================== -This document describes the specification for the global override feature. -By default, every user is allowed write access to the devices by the master driver. The override feature will allow the user (for example, building administrator) to override this default behavior and enable the user to lock the write access on the devices for a specified duration of time or indefinitely. - -Functional Capabilities ------------------------------ - -1. User shall be able to specify the following when turning on the override behavior on the devices. - - * Override pattern, for example, - - If pattern is campus/building1/* - Override condition is turned on for all the devices under campus/building1/. - - If pattern is campus/building1/ahu1 - Override condition is turned on for only campus/building1/ahu1 - - The pattern matching shall use bash style filename matching semantics. - - * Time duration over which override behavior is applicable. If the time duration is negative, then override condition is applied indefinitely. - - * Optional revert-to-fail-safe-state flag. If the flag is set, master driver shall set all the set points falling under the override condition to its default state/value immediately. This is to ensure that the devices are in fail-safe state when the override/lock feature is removed. If the flag is not set, the device state/value is untouched. - - * Optional staggered revert flag. If this flag is set, reverting of devices will be staggered. - -2. User shall be able to disable/turn off the override behavior on devices by specifying: - - * Pattern on which the override/lock feature has be disabled. (example: campus/building/\*) - -3. User shall be able to get a list of all the devices with the override condition set. - -4. User shall be able to get a list of all the override patterns that are currently active. - -5. User shall be able to clear all the overrides. - -6. Any changes to override patterns list shall be stored in the config store. On startup, list of override patterns and corresponding end times are retrieved from the config store. If the end time is indefinite or greater than current time for any pattern, then override is set on the matching devices for remaining duration of time. - -7. Whenever a device is newly configured, a check is made to see if it is part of the overridden patterns. If yes, it is added to list of overridden devices. - -8. When a device is being removed, a check is made to see if it is part of the overridden devices. If yes, it is removed from the list of overridden devices. - -Driver RPC Methods -******************** -set_override_on( pattern, duration=0.0, failsafe_revert=True, staggered_revert=True ) - Turn on override condition on all the devices matching the pattern. Time duration for the override condition has to be in seconds. For indefinite duration, the time duration has to be <= 0.0. - -set_override_off( pattern ) - Turn off override condition on all the devices matching the pattern. The specified pattern will be removed from the override patterns list. All the devices falling under the given pattern will be removed from the list of overridden devices. - -get_override_devices( ) - Get a list of all the devices with override condition. - -get_override_patterns( ) - Get a list of override patterns that are currently active. - -clear_overrides( ) - Clear all the overrides. - diff --git a/docs/source/specifications/external-rpc-enhancement.rst b/docs/source/specifications/external-rpc-enhancement.rst deleted file mode 100644 index 350a02f209..0000000000 --- a/docs/source/specifications/external-rpc-enhancement.rst +++ /dev/null @@ -1,157 +0,0 @@ -.. _ExternalRPCEnhancement: - -RPC Communication Between Remote Platforms -========================================== - -This document describes RPC communication between different platforms. In the current setup of VOLTTRON, if an agent in -one platform wants to make a RPC method call on an agent in a different platform, it responsible for establishing and -managing the connection with the target platform. Instead, if allow the VIP routers of each platform to make the -connection and manage the RPC communication internally, this will reduce the burden on the agents and enable a more -seamless RPC communication between agents on different platforms. - - -VIP Router -********** - -The VIP Router on each platform is responsible for establishing and maintaining the connection with remote platforms. - - -Router Functional Capabilities -****************************** - -1. Each VOLTTRON platform shall have a list of other VOLTTRON platforms that it has to establish connection in a config -file. - -2. The VIP router of each platform connects to other platforms on startup. It is responsible for maintaining the -connection (detects disconnects and intiate reconnects etc). - -3. The VIP router routes the external RPC message as described in "Messages for External RPC communication" section. - - -External RPC Subsystem -********************** - -External RPC subsystem allows an agent to make RPC method calls on agents running in remote platforms. - - -External RPC Functional Capabilities -************************************ -1. The agent needs to specify the remote platform name as an additional argument in the original RPC call or notify -method. - -2. The external RPC subsystem on the agent side adds the remote platform name into its VIP frame and sends to the -VIP router for routing to correct destination platform. It is described in detail in the next section. - - -Messages for External RPC communication -*************************************** - -The VIP router and external RPC subsystem on the agent side will be using VIP protocol for communication. The -communication between the VIP routers and the external RPC susbsytem on the agent side can be best explained with an -example. Suppose an agent 1 on platform V1 wants to make RPC method call on agent 2 in platform V2. Then the underlying -messages exchanged between the two platforms will look like below. - -Message format for external RPC subsystem of agent 1 on platform V1 to send to its VIP router. -:: - - +-+ - | | Empty recipient frame (implies VIP router is the destination) - +-+----+ - | VIP1 | Signature frame - +-+---------+ - |V1 user id | Empty user ID frame - +-+---------+ - | 0001 | Method request ID, for example "0001" - +-------------++ - | external_rpc | Subsystem, "external_rpc" - +-----------------------------+ - | external RPC request message| Dictionary containing destination platform name, destination agent identity, - | | source agent identity, method name and method arguments - +-----------------------------+ - - -Message sent by VIP router on platform V1 to VIP router of platform V2. - -:: - - +-----+ - | V2 | Destination platform ID, "V2" in this case - +-+---+ - | | Empty recipient frame - +-+----+ - | VIP1 | Signature frame - +-+---------+ - |V1 user id | Empty user ID frame - +-+---------+ - | 0001 | Method Request ID, for example "0001" - +--------------+ - | external_rpc | Subsystem, "external_rpc" - +------------------------------+ - | external RPC request message | Dictionary containing destination platform name, destination agent identity, - | | source platform name, source agent identity, method and arguments - +------------------------------+ - - -When the VIP router of platform V2 receives the message, it extracts the destination agent identity from the external -RPC request message frame and routes it to the intended agent. - -The result of the RPC method execution needs to be returned back to the calling agent. So the messages for the return -path are as follows. The source and destination platforms and agents are interchanged in the reply message. - -Message sent by external RPC subsystem of agent 2 on platform V2 to its VIP router. - -:: - - +-+ - | | Empty recipient frame (implies destination is VIP router) - +-+----+ - | VIP1 | Signature frame - +-+---------+ - |V2 user id | Empty user ID frame - +-+---------+ - | 0001 | Method Request ID, for example "0001" - +--------------+ - | external_rpc | Subsystem, "external_rpc" - +------------------------------+ - | external rpc reply message | Dictionary containing destination platform name, destination agent identity - | | source platform name, source agent identity and method result - +------------------------------+ - - -Message sent by VIP router of platform V2 to VIP router of platform V1. -:: - - +-----+ - | V1 | Source platform ID frame, "V1" in this case - +-+---+ - | | Empty recipient frame - +-+----+ - | VIP1 | Signature frame - +-+---------+ - |V1 user id | Empty user ID frame - +-+---------+ - | 0001 | Method Request ID, for example "0001" - +--------------+ - | external_rpc | Subsystem, "external_rpc" - +------------------------------+ - | external rpc reply message | Dictionary containing destination platform name, destination agent identity - | | source platform name, source agent identity and method result - +------------------------------+ - -The VIP router of platform V1 extracts the destination agent identity from the external RPC reply message frame and -routes it to the calling agent. - - -Methods for External RPC Subsystem -********************************** - -call(peer, method, \*args, \**kwargs) - New 'external_platform' parameter need to be added in kwargs to the -original RPC subsystem call. If the platform name of the target platform is passed into the 'external_platform' -parameter, the RPC method on the target platform gets executed. - -notify(peer, method, \*args, \**kwargs) - New 'external_platform' parameter need to be added in kwargs to the -original RPC subsystem notify method. If the platform name of the target platform is passed into the 'external_platform' -parameter, the RPC method on the target platform gets executed. - -handle_external_rpc_subsystem(message) - Handler for the external RPC subsystem messages. It executes the requested RPC -method and returns the result to the calling platform. diff --git a/docs/source/specifications/ieee2030_5_agent.rst b/docs/source/specifications/ieee2030_5_agent.rst deleted file mode 100644 index 1471549c3b..0000000000 --- a/docs/source/specifications/ieee2030_5_agent.rst +++ /dev/null @@ -1,229 +0,0 @@ -.. _IEEE2030_5: - -IEEE 2030.5 DER Support -======================= - -Version 1.0 - -Smart Energy Profile 2.0 (SEP2, IEEE 2030.5) specifies a REST architecture built -around the core HTTP verbs: GET, HEAD, PUT, POST and DELETE. -A specification for the IEEE 2030.5 protocol can be found -`here `_. - -IEEE 2030.5 EndDevices (clients) POST XML resources representing their state, -and GET XML resources containing command and control information from the server. -The server never reaches out to the client unless a "subscription" is -registered and supported for a particular resource type. This implementation -does not use IEEE 2030.5 registered subscriptions. - -The IEEE 2030.5 specification requires HTTP headers, and it explicitly requires RESTful -response codes, for example: - - - 201 - "Created" - - 204 - "No Content" - - 301 - "Moved Permanently" - - etc. - -IEEE 2030.5 message encoding may be either XML or EXI. -Only XML is supported in this implementation. - -IEEE 2030.5 requires HTTPS/TLS version 1.2 along with support for the -cipher suite TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8. -Production installation requires a certificate issued by a IEEE 2030.5 CA. -The encryption requirement can be met by using a web server such as -Apache to proxy the HTTPs traffic. - -IEEE 2030.5 discovery, if supported, must be implemented by an xmDNS server. -Avahi can be modified to perform this function. - -Function Sets -------------- - -IEEE 2030.5 groups XML resources into "Function Sets." Some of these function sets -provide a core set of functionality used across higher-level function sets. -This implementation implements resources from the following function sets: - - - Time - - Device Information - - Device Capabilities - - End Device - - Function Set Assignments - - Power Status - - Distributed Energy Resources - -Distributed Energy Resources ----------------------------- - -Distributed energy resources (DERs) are devices that generate energy, e.g., solar inverters, -or store energy, e.g., battery storage systems, electric vehicle supply equipment (EVSEs). -These devices are managed by a IEEE 2030.5 DER server using DERPrograms which are described by -the IEEE 2030.5 specification as follows: - - Servers host one or more DERPrograms, which in turn expose DERControl events to DER clients. - DERControl instances contain attributes that allow DER clients to respond to events - that are targeted to their device type. A DERControl instance also includes scheduling - attributes that allow DER clients to store and process future events. These attributes - include start time and duration, as well an indication of the need for randomization of - the start and / or duration of the event. The IEEE 2030.5 DER client model is based on the - SunSpec Alliance Inverter Control Model [SunSpec] which is derived from - IEC 61850-90-7 [61850] and [EPRI]. - -EndDevices post multiple IEEE 2030.5 resources describing their status. The following is an -example of a Power Status resource that might be posted by an EVSE (vehicle charging -station): - -.. code-block:: xml - - - 4 - 1487812095 - 1 - 9300 - - - 3 - -5 - - - 3 - 22 - - - 3 - 7 - - 11280 - 10000 - 9223372036854775807 - 1487812095 - - - -Design Details --------------- - -.. image:: files/volttron_ieee2030_5.jpg - -VOLTTRON's IEEE 2030.5 implementation includes a IEEE2030_5Agent and a IEEE 2030.5 device driver, -as described below. - -VOLTTRON IEEE2030_5Agent -~~~~~~~~~~~~~~~~~~ - -IEEE2030_5Agent implements a IEEE 2030.5 server that receives HTTP POST/PUT -requests from IEEE 2030.5 devices. The requests are routed to IEEE2030_5Agent over the VOLTTRON -message bus by VOLTTRON's MasterWebService. IEEE2030_5Agent returns an appropriate HTTP -response. In some cases (e.g., DERControl requests), this response includes a data -payload. - -IEEE2030_5Agent maps IEEE 2030.5 resource data to a VOLTTRON IEEE 2030.5 data model based on SunSpec, -using block numbers and point names as defined in the SunSpec Information Model, -which in turn is harmonized with 61850. The data model is given in detail below. - -Each device's data is stored by IEEE2030_5Agent in an EndDevice memory structure. This -structure is not persisted to a database. Each EndDevice retains only the most -recently received value for each field. - -IEEE2030_5Agent exposes RPC calls for getting and setting EndDevice data. - -VOLTTRON IEEE 2030.5 Device Driver -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The IEEE 2030.5 device driver is a new addition to VOLTTRON MasterDriverAgent's family of -standard device drivers. It exposes get_point/set_point calls for IEEE 2030.5 EndDevice fields. - -The IEEE 2030.5 device driver periodically issues IEEE2030_5Agent RPC calls to refresh its cached -representation of EndDevice data. It issues RPC calls to IEEE2030_5Agent as needed when -responding to get_point, set_point and scrape_all calls. - -Field Definitions -~~~~~~~~~~~~~~~~~ - -These field IDs correspond to the ones in the IEEE 2030.5 device driver's configuration file, ieee2030_5.csv. -They have been used in that file's "Volttron Point Name" column and also in its "Point Name" column. - -================= ======================== ==================================================== ======= ====== -Field ID IEEE 2030.5 Resource/Property Description Units Type -================= ======================== ==================================================== ======= ====== -b1_Md device_information Model (32 char lim). string - mfModel -b1_Opt device_information Long-form device identifier (32 char lim). string - lfdi -b1_SN abstract_device Short-form device identifier (32 char lim). string - sfdi -b1_Vr device_information Version (16 char lim). string - mfHwVer -b113_A mirror_meter_reading AC current. A float - PhaseCurrentAvg -b113_DCA mirror_meter_reading DC current. A float - InstantPackCurrent -b113_DCV mirror_meter_reading DC voltage. V float - LineVoltageAvg -b113_DCW mirror_meter_reading DC power. W float - PhasePowerAvg -b113_PF mirror_meter_reading AC power factor. % float - PhasePFA -b113_WH mirror_meter_reading AC energy. Wh float - EnergyIMP -b120_AhrRtg der_capability Usable capacity of the battery. Ah float - rtgAh Maximum charge minus minimum charge. -b120_ARtg der_capability Maximum RMS AC current level capability of the A float - rtgA inverter. -b120_MaxChaRte der_capability Maximum rate of energy transfer into the device. W float - rtgMaxChargeRate -b120_MaxDisChaRte der_capability Maximum rate of energy transfer out of the device. W float - rtgMaxDischargeRate -b120_WHRtg der_capability Nominal energy rating of the storage device. Wh float - rtgWh -b120_WRtg der_capability Continuous power output capability of the inverter. W float - rtgW -b121_WMax der_settings Maximum power output. Default to WRtg. W float - setMaxChargeRate -b122_ActWh mirror_meter_reading AC lifetime active (real) energy output. Wh float - EnergyEXP -b122_StorConn der_status CONNECTED=0, AVAILABLE=1, OPERATING=2, TEST=3. enum - storConnectStatus -b124_WChaMax der_control Setpoint for maximum charge. This is the only W float - opModFixedFlow field that is writable with a set_point call. -b403_Tmp mirror_meter_reading Pack temperature. C float - InstantPackTemp -b404_DCW PEVInfo Power flow in or out of the inverter. W float - chargingPowerNow -b404_DCWh der_availability Output energy (absolute SOC). Wh float - availabilityDuration Calculated as (availabilityDuration / 3600) * WMax. -b802_LocRemCtl der_status Control Mode: REMOTE=0, LOCAL=1. enum - localControlModeStatus -b802_SoC der_status State of Charge %. % WHRtg float - stateOfChargeStatus -b802_State der_status DISCONNECTED=1, INITIALIZING=2, CONNECTED=3, enum - inverterStatus STANDBY=4, SOC PROTECTION=5, FAULT=99. -================= ======================== ==================================================== ======= ====== - -Revising and Expanding the Field Definitions --------------------------------------------- - -The IEEE 2030.5-to-SunSpec field mappings in this implementation are a relatively thin subset of all possible -field definitions. Developers are encouraged to expand the definitions. - -The procedure for expanding the field mappings requires you to make changes in two places: - -1. Update the driver's point definitions in services/core/MasterDriverAgent/master_driver/ieee2030_5.csv -2. Update the IEEE 2030.5-to-SunSpec field mappings in services/core/IEEE2030_5Agent/ieee2030_5/end_device.py and __init__.py - -When updating VOLTTRON's IEEE 2030.5 data model, please use field IDs that conform to the SunSpec -block-number-and-field-name model outlined in the SunSpec Information Model Reference -(see the link below). - -For Further Information ------------------------ - -SunSpec References: - - - Information model specification: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Information-Models-12041.pdf - - Information model reference spreadsheet: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Information-Model-Reference.xlsx - - Inverter models: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Inverter-Models-12020.pdf - - Energy storage models: http://sunspec.org/wp-content/uploads/2015/06/SunSpec-Energy-Storage-Models-12032.pdf - -Questions? Please contact: - - - Rob Calvert (rob@kisensum.com) or James Sheridan (james@kisensum.com) diff --git a/docs/source/specifications/index.rst b/docs/source/specifications/index.rst deleted file mode 100644 index ce6c403d3b..0000000000 --- a/docs/source/specifications/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. _platform-specifications: - -======================= -Platform Specifications -======================= - -.. toctree:: - :glob: - :maxdepth: 1 - - * diff --git a/docs/source/specifications/mesa_agent.rst b/docs/source/specifications/mesa_agent.rst deleted file mode 100644 index 32ca1b3ed9..0000000000 --- a/docs/source/specifications/mesa_agent.rst +++ /dev/null @@ -1,263 +0,0 @@ -.. _MESA: - -MesaAgent ---------- - -MesaAgent is a VOLTTRON agent that handles MESA-ESS DNP3 outstation communications. -It subclasses and extends the functionality of VOLTTRON's DNP3Agent. Like DNP3Agent, -MesaAgent models a DNP3 outstation, communicating with a DNP3 master. - -`DNP3 `_ (Distributed Network Protocol) is -a set of communications protocols that are widely used by utilities such as -electric power companies, primarily for `SCADA `_ purposes. -It was adopted in 2010 -as `IEEE Std 1815-2010 `_, -later updated to `1815-2012 `_. - -VOLTTRON's MesaAgent and DNP3Agent are implementations of a DNP3 Outstation as specified in -IEEE Std 1815-2012. They engage in bidirectional network communications with a DNP3 Master, -which might be located at a power utility. - -MESA-ESS is an extension and enhancement to DNP3. It builds on the basic DNP3 communications -protocol, adding support for more complex structures, including functions, arrays, curves and schedules. -The draft specification for MESA-ESS, as well as a spreadsheet of point definitions, can be -found at **http://mesastandards.org/mesa-ess-2016/**. - -VOLTTRON's DNP3Agent and MesaAgents implementations of an Outstation are built on pydnp3, -an open-source library from Kisensum containing Python language -bindings for Automatak's C++ `opendnp3 `_ -library, the de facto reference implementation of DNP3. - -MesaAgent exposes DNP3 application-layer functionality, creating an extensible -base from which specific custom behavior can be designed and supported, including support -for MESA functions, arrays and selector blocks. By default, MesaAgent -acts as a simple transfer agent, publishing data received from the Master on -the VOLTTRON Message Bus, and responding to RPCs from other VOLTTRON agents -by sending data to the Master. Properties of the point and function definitions also enable -the use of more complex controls for point data capture and publication. - -MesaAgent was developed by Kisensum for use by 8minutenergy, which provided generous -financial support for the open-source contribution to the VOLTTRON platform, along with -valuable feedback based on experience with the agent in a production context. - -RPC Calls -~~~~~~~~~ - -MesaAgent exposes the following VOLTTRON RPC calls: - -.. code-block:: python - - def get_point(self, point_name): - """ - Look up the most-recently-received value for a given output point. - - @param point_name: The point name of a DNP3 PointDefinition. - @return: The (unwrapped) value of a received point. - """ - - def get_point_by_index(self, data_type, index): - """ - Look up the most-recently-received value for a given point. - - @param data_type: The data_type of a DNP3 point. - @param index: The index of a DNP3 point. - @return: The (unwrapped) value of a received point. - """ - - def get_points(self): - """ - Look up the most-recently-received value of each configured output point. - - @return: A dictionary of point values, indexed by their point names. - """ - - def get_configured_points(self): - """ - Look up the most-recently-received value of each configured point. - - @return: A dictionary of point values, indexed by their point names. - """ - - def set_point(self, point_name, value): - """ - Set the value of a given input point. - - @param point_name: The point name of a DNP3 PointDefinition. - @param value: The value to set. The value's data type must match the one in the DNP3 PointDefinition. - """ - - def set_points(self, point_dict): - """ - Set point values for a dictionary of points. - - @param point_dict: A dictionary of {point_name: value} for a list of DNP3 points to set. - """ - - def config_points(self, point_map): - """ - For each of the agent's points, map its VOLTTRON point name to its DNP3 group and index. - - @param point_map: A dictionary that maps a point's VOLTTRON point name to its DNP3 group and index. - """ - - def get_point_definitions(self, point_name_list): - """ - For each DNP3 point name in point_name_list, return a dictionary with each of the point definitions. - - The returned dictionary looks like this: - - { - "point_name1": { - "property1": "property1_value", - "property2": "property2_value", - ... - }, - "point_name2": { - "property1": "property1_value", - "property2": "property2_value", - ... - } - } - - If a definition cannot be found for a point name, it is omitted from the returned dictionary. - - :param point_name_list: A list of point names. - :return: A dictionary of point definitions. - """ - - def get_selector_block(self, point_name, edit_selector): - """ - Return a dictionary of point values for a given selector block. - - :param point_name: Name of the first point in the selector block. - :param edit_selector: The index (edit selector) of the block. - :return: A dictionary of point values. - """ - - def reset(self): - """ - Reset the agent's internal state, emptying point value caches. Used during iterative testing. - """ - -Pub/Sub Calls -~~~~~~~~~~~~~ - -MesaAgent uses three topics when publishing data to the VOLTTRON message bus: - - * **Point Values (default topic: dnp3/point)**: As MesaAgent communicates with the Master, - it publishes received point values on the VOLTTRON message bus. - - * **Functions (default topic: mesa/function)**: When MesaAgent receives a function step - with a "publish" action value, it publishes the current state of the function (all - steps received to date) on the VOLTTRON message bus. - - * **Outstation status (default topic: mesa/status)**: If the status of the MesaAgent outstation - changes, for example if it is restarted, it publishes its new status on the VOLTTRON message bus. - -Data Dictionaries of Point and Function Definitions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -MesaAgent loads and uses data dictionaries of point and function definitions, -which are maintained by agreement between the (MesaAgent) Outstation and the DNP3 Master. -The data dictionaries are stored in the agent's registry. - -Current Point Values -~~~~~~~~~~~~~~~~~~~~ - -MesaAgent tracks the most-recently-received value for each point definition in its -data dictionary, regardless of whether the point value's source is a VOLTTRON RPC call or -a message from the DNP3 Master. - -Agent Configuration -~~~~~~~~~~~~~~~~~~~ - -The MesaAgent configuration specifies the following fields: - - - **local_ip**: (string) - Outstation's host address (DNS resolved). - Default: 0.0.0.0. - - **port**: (integer) - Outstation's port number - the port that the remote endpoint (Master) is listening on. - Default: 20000. - - **point_topic**: (string) - VOLTTRON message bus topic to use when publishing DNP3 point values. - Default: dnp3/point. - - **function_topic**: (string) - Message bus topic to use when publishing MESA-ESS functions. - Default: mesa/function. - - **outstation_status_topic**: (string) - Message bus topic to use when publishing outstation status. - Default: mesa/outstation_status. - - **all_functions_supported_by_default**: (boolean) - When deciding whether to reject points for unsupported - functions, ignore the values of their 'supported' points: simply treat all functions as - supported. Used primarily during testing. - Default: False. - - **function_validation**: (boolean) - When deciding whether to support sending single points to MesaAgent. - If function_validation is True, MesaAgent will raise an exception when receiving any - invalid point in current function. If function_validation is False, MesaAgent will - reset current function to None instead of raising the exception. - Default: False. - - **outstation_config**: (dictionary) - Outstation configuration parameters. All are optional. Parameters include: - - -- **database_sizes**: (integer) - Size of each outstation database buffer. - Default: 10. - -- **event_buffers**: (integer) - Size of the database event buffers. - Default: 10. - -- **allow_unsolicited**: (boolean) - Whether to allow unsolicited requests. - Default: True. - -- **link_local_addr**: (integer) - Link layer local address. - Default: 10. - -- **link_remote_addr**: (integer) - Link layer remote address. - Default: 1. - -- **log_levels**: (list) - List of bit field names (OR'd together) that filter what gets logged by DNP3. - Default: [NORMAL]. Possible values: ALL, ALL_APP_COMMS, ALL_COMMS, NORMAL, NOTHING. - -- **threads_to_allocate**: (integer) - Threads to allocate in the manager's thread pool. - Default: 1. - -A sample MesaAgent configuration file is available in **services/core/DNP3Agent/mesaagent.config**. - -Installing MesaAgent --------------------- - -To install MesaAgent, please consult the installation advice in **services/core/DNP3Agent/README.md**, -which includes advice on installing **pydnp3**, a library upon which DNP3Agent depends. - -After installing libraries as described in README.md, -the agent can be installed from a command-line shell as follows: - -.. code-block:: python - - $ export VOLTTRON_ROOT= - $ cd $VOLTTRON_ROOT - $ source services/core/DNP3Agent/install_mesa_agent.sh - -README.md specifies a default agent configuration, which can be overridden as needed. - -Here are some things to note when installing MesaAgent: - - - MesaAgent source code resides in, and is installed from, a dnp3 subdirectory, thus allowing it - to be implemented as a subclass of the base DNP3 agent class. - When installing MesaAgent, inform the install script that it should build from the - mesa subdirectory by exporting the following environment variable: - - -- $ export AGENT_MODULE=dnp3.mesa.agent - - - The agent's point and function definitions must be loaded into the agent's config store. See the - install_mesa_agent.sh script for an example of how to load them. - -For Further Information ------------------------ - -Questions? Please contact: - - - Anh Nguyen at ChargePoint (anh.nguyen@chargepoint.com) diff --git a/docs/source/specifications/message-debugging.rst b/docs/source/specifications/message-debugging.rst deleted file mode 100644 index 9796963369..0000000000 --- a/docs/source/specifications/message-debugging.rst +++ /dev/null @@ -1,348 +0,0 @@ -.. _MessageDebugging: - -Message Bus Visualization and Debugging - Specification -======================================================= - -NOTE: This is a planning document, created prior to implementation of the -VOLTTRON Message Debugger. It describes the tool's general goals, but it's not -always accurate about specifics of the ultimate implementation. For a description -of Message Debugging as implemented, with advice on how to configure and -use it, please see :doc:`Message-Debugging <../devguides/agent_development/Message-Debugging>`. - -Description ------------ - -VOLTTRON agents send messages to each other on the VOLTTRON message bus. -It can be useful to examine the contents of this message stream -while debugging and troubleshooting agents and drivers. - -In satisfaction of this specification, a new Message Monitor capability will be implemented -allowing VOLTTRON agent/driver developers to monitor the message stream, -filter it for an interesting set of messages, and display the -contents and characteristics of each message. - -Some elements below are central to this effort (required), -while others are useful improvements (optional) that may be -implemented if time permits. - -Feature: Capture Messages and Display a Message Summary -------------------------------------------------------- - -When enabled, the Message Monitor will capture details about a stream of routed messages. -On demand, it will display a message summary, either in real time as the messages are routed, -or retrospectively. - -A summary view will convey the high level interactions occurring between VOLTTRON agents -as conversations that may be expanded for more detail. A simple RPC call that involves -4 message send/recv segments will be displayed as a single object that can be expanded. -In this way, the message viewer will provide a higher-level view of -message bus activity than might be gleaned from verbose logs using grep. - -Pub/sub interactions will be summarized at the topic level with high-level statistics -such as the number of subscribers, # of messages published during the capture period, etc. -Drilling into the interaction might show the last message published with the ability to -drill deeper into individual messages. A diff display would show how the published -data is changing. - - -Summary view -:: - - - 11:09:31.0831 RPC set_point charge.control platform.driver - | - params: ('set_load', 10) return: True - - 11:09:31.5235 Pub/Sub devices/my_device platform.driver 2 subscribers - | - Subscriber: charge.control - | - Last message 11:09:31.1104: - [ - { - 'Heartbeat': True, - 'PowerState': 0, - 'temperature': 50.0, - 'ValveState': 0 - }, - ... - ] - | - Diff to 11:09:21.5431: - 'temperature': 48.7, - -The summary's contents and format will vary by message subsystem. - -RPC request/response pairs will be displayed on a single line: -:: - - (volttron) d1:volttron myname$ msmon —agent='(Agent1,Agent2)' - - Agent1 Agent2 - 2016-11-22T11:09:31.083121+00:00 rpc: devices/my_topic; 2340972387; sent 2016-11-22T11:09:31.277933+00:00 responded: 0.194 sec - 2016-11-22T11:09:32.005938+00:00 rpc: devices/my_topic; 2340972388; sent 2016-11-22T11:09:32.282193+00:00 responded: 0.277 sec - 2016-11-22T11:09:33.081873+00:00 rpc: devices/my_topic; 2340972389; sent 2016-11-22T11:09:33.271199+00:00 responded: 0.190 sec - 2016-11-22T11:09:34.049139+00:00 rpc: devices/my_topic; 2340972390; sent 2016-11-22T11:09:34.285393+00:00 responded: 0.236 sec - 2016-11-22T11:09:35.053183+00:00 rpc: devices/my_topic; 2340972391; sent 2016-11-22T11:09:35.279317+00:00 responded: 0.226 sec - 2016-11-22T11:09:36.133948+00:00 rpc: devices/my_topic; 2340972392; sent 2016-11-22T11:09:36.133003+00:00 dequeued - -When PubSub messages are displayed, each message's summary will include its count of subscribers: -:: - - (volttron) d1:volttron myname$ msmon —agent=(Agent1) - - Agent1 - 2016-11-22T11:09:31.083121+00:00 pubsub: devices/my_topic; 2340972487; sent; 2 subs - 2016-11-22T11:09:32.005938+00:00 pubsub: devices/my_topic; 2340972488; sent; 2 subs - 2016-11-22T11:09:33.081873+00:00 pubsub: devices/my_topic; 2340972489; sent; 2 subs - 2016-11-22T11:09:34.049139+00:00 pubsub: devices/my_topic; 2340972490; sent; 2 subs - 2016-11-22T11:09:35.053183+00:00 pubsub: devices/my_topic; 2340972491; sent; 2 subs - -While streaming output of a message summary, a defined keystroke sequence will "pause" the output, -and another keystroke sequence will "resume" displaying the stream. - -Feature: Capture and Display Message Details --------------------------------------------- - -The Message Monitor will capture a variety of details about each message, including: - - 1. Sending agent ID - 2. Receiving agent ID - 3. User ID - 4. Message ID - 5. Subsystem - 6. Topic - 7. Message data - 8. Message lifecycle timestamps, in UTC (when sent, dequeued, responded) - 9. Message status (sent, responded, error, timeout) - 10. Message size - 11. Other message properties TBD (e.g., queue depth?) - -On demand, it will display these details for a single message ID: -:: - - (volttron)d1:volttron myname$ msmon --id='2340972390' - - 2016-11-22T11:09:31.053183+00:00 (Agent1) - INFO: - Subsystem: 'pubsub', - Sender: 'Agent1', - Topic: 'devices/my_topic', - ID: '2340972390', - Sent: '2016-11-22T11:09:31.004986+00:00', - Message: - [ - { - 'Heartbeat': True, - 'PowerState': 0, - 'temperature': 50.0, - 'ValveState': 0 - }, - { - 'Heartbeat': - { - 'units': 'On/Off', - 'type': 'integer' - }, - 'PowerState': - { - 'units': '1/0', - 'type': 'integer' - }, - 'temperature': - { - 'units': 'Fahrenheit', - 'type': 'integer' - }, - 'ValveState': - { - 'units': '1/0', - 'type': 'integer' - } - } - ] - -A VOLTTRON message ID is not unique to a single message. A group of messages in a "conversation" -may share a common ID, for instance during RPC request/response exchanges. -When detailed display of all messages for a single message ID is requested, they will be displayed -in chronological order. - -Feature: Display Message Statistics ------------------------------------ - -Statistics about the message stream will also be available on demand: - - 1. Number of messages sent, by agent, subsystem, topic - 2. Number of messages received, by agent, subsystem, topic - -Feature: Filter the Message Stream ----------------------------------- - -The Message Monitor will be able to filter the message stream display -to show only those messages that match a given set of criteria: - - 1. Sending agent ID(s) - 2. Receiving agent ID(s) - 3. User ID(s) - 4. Subsystem(s) - 5. Topic - Specific topic(s) - 6. Topic - Prefix(es) - 7. Specific data value(s) - 8. Sampling start/stop time - 9. Other filters TBD - -User Interface: Linux Command Line ----------------------------------- - -A Linux command-line interface will enable the following user actions: - - 1. Enable message tracing - 2. Disable message tracing - 3. Define message filters - 4. Define verbosity of displayed-message output - 5. Display message stream - 6. Begin recording messages - 7. Stop recording messages - 8. Display recorded messages - 9. Play back (re-send) recorded messages - -Feature (not implemented): Watch Most Recent --------------------------------------------- - -Optionally, the Message Monitor can be asked to "watch" a specific data element. -In that case, it will display the value of that element in the most recent message -matching the filters currently in effect. As the data to be displayed changes, -the display will be updated in place without scrolling (similar to "top" output): - -:: - - (volttron) d1:volttron myname$ msmon —agent='(Agent1)' --watch='temperature' - - Agent1 - 2016-11-22T11:09:31.053183+00:00 pubsub: my_topic; 2340972487; sent; 2 subs; temperature=50 - -Feature (not implemented): Regular Expression Support ------------------------------------------------------ - -It could help for the Message Monitor's filtering logic to support regular expressions. -Regex support has also been requested (Issue #207) when identifying a subscribed pub/sub topic -during VOLTTRON message routing. - -Optionally, regex support will be implemented in Message Monitor filtering criteria, -and also (configurably) during VOLTTRON topic matching. - -Feature (not implemented): Message Stream Record and Playback -------------------------------------------------------------- - -The Message Monitor will be able to "record" and "play back" a message sequence: - - 1. Capture a set of messages as a single "recording" - 2. Inspect the contents of the "recording" - 3. "Play back" the recording -- re-send the recording's messsage sequence in VOLTTRON - -Feature (not implemented): On-the-fly Message Inspection and Modification -------------------------------------------------------------------------- - -VOLTTRON message inspection and modification, on-the-fly, may be supported from the command line. -The syntax and implementation would be similar to pdb (Python Debugger), and might -be written as an extension to pdb. - -Capabilities: - - 1. Drill-down inspection of message contents. - 2. Set a breakpoint based on message properties, halting upon routing a matching message. - 3. While halted on a breakpoint, alter a message's contents. - -Feature (not implemented): PyCharm Debugging Plugin ---------------------------------------------------- - -VOLTTRON message debugging may also be published as a PyCharm plugin. -The plugin would form a more user-friendly interface for the same set of capabilities -described above -- on-the-fly message inspection and modification, with the ability to -set a breakpoint based on message properties. - -User Interface (not implemented): PCAP/Wireshark ------------------------------------------------- - -Optionally, we may elect to render the message trace as a stream of PCAP data, -thereby exploiting Wireshark's filtering and display capabilities. -This would be in accord with the enhancement suggested in VOLTTRON Issue #260. - -User Interface (not implemented): Volttron Central Dashboard Widget -------------------------------------------------------------------- - -Optionally, the Message Monitor will be integrated as a new Volttron Central dashboard widget, -supporting each of the following: - - 1. Enable/Disable the monitor - 2. Filter messages - 3. Configure message display details - 4. Record/playback messages - -User Interface (not implemented): Graphical Display of Message Sequence ------------------------------------------------------------------------ - -Optionally, the Volttron Central dashboard widget will provide graphical display -of message sequences, allowing enhanced visualization of request/response patterns. - -Related Development: PyCharm Documentation ------------------------------------------- - -Also included in this effort will be a contribution to VOLTTRON documentation about installing -and configuring a PyCharm environment for developing, debugging and testing VOLTTRON -agents and drivers. - -Engineering Design Notes -======================== - -Grabbing Messages Off the Bus ------------------------------ - -This tool depends on reading and storing all messages that pass through the VIP router. The Router class -already has hooks that allow for the capturing of messages at various points in the routing workflow. The -BaseRouter abstract class defines ``issue(self, topic, frames, extra)``. This method is called from ``BaseRouter.route`` -and ``BaseRouter._send`` during the routing of messasges. The ``topic`` parameter (not to be confused with a -message topic found in ``frames``) identifies the point or state in the routing worflow at which the issue was called. - -The defined ``topics`` are: INCOMING, OUTGOING, ERROR and UNROUTABLE. Most messages will result in two calls, one -with the INCOMING topic as the message enters the router and one with the OUTGOING topic as the message is -sent on to its destination. Messages without a recipient are intended for the router itself and do not result -in an OUTGOING call to ``issue``. - -``Router.issue`` contains the concrete implementation of the method. It does two things: - -1. It writes the topic, frames and optional extra parameters to the logger using the FramesFormatter. -2. It invokes ``self._tracker.hit(topic, frames, extra)``. The Tracker class collects statistics by topic and counts the messages within a topic by peer, user and subsystem. - -The issue method can be modified to optionally publish the ``issue`` messages to an in-process ZMQ address -that the message-viewing tool will subscribe to. This will minimize changes to core VOLTTRON code and minimize -the impact of processing these messages for debugging. - -Message Processor ------------------ - -The message processor will subscribe to messages coming out of the Router.issue() method and process these -messages based on the current message viewer configuration. Messages will be written to a SQLite db since this -is packaged with Python and currently used by other VOLTTRON agents. - -Message Viewer --------------- - -The message viewer will display messages from the SQLite db. We need to consider whether it should also subscribe -to receiving messages in real-time. The viewer will be responsible for displaying message statistics and will provide -a command line interface to filter and display messages. - -Message Db Schema ------------------ - -:: - - message(id, created_on, issue_topic, extras, sender, recipient, user_id, msg_id, subsystem, data) - -msg_id will be used to associate pairs of incoming/outgoing messages. - -.. note:: data will be a jsonified list of frames, alternatively we could add a message_data table with one - row per frame. - -A session table will track the start and end of a debug session and, at the end of a session, record statistics -on the messages in the session. - -:: - - session(id, created_on, name, start_time, end_time, num_messages) - -The command line tool will allow users to delete old sessions and select a session for review/playback. diff --git a/docs/source/styleguide.rst b/docs/source/styleguide.rst deleted file mode 100644 index e1ff2c775b..0000000000 --- a/docs/source/styleguide.rst +++ /dev/null @@ -1,66 +0,0 @@ -.. _styleguide: -.. Reference anchor should be the same as the filename - - -================================== -This is the main title of the page -================================== - -.. _code blocks: - -Example Code Blocks --------------------- - -Use bash for commands or user actions - -.. code-block:: bash - - ls -al - - -Use this for the results of a command - -.. code-block:: console - - total 5277200 - drwxr-xr-x 22 volttron volttron 4096 Oct 20 09:44 . - drwxr-xr-x 23 volttron volttron 4096 Oct 19 18:39 .. - -rwxr-xr-x 1 volttron volttron 164 Sep 29 17:08 agent-setup.sh - drwxr-xr-x 3 volttron volttron 4096 Sep 29 17:13 applications - - -Use this when Python source code is displayed - -.. code-block:: python - - @RPC.export - def status_agents(self): - return self._aip.status_agents() - - -Directives ----------- - -Taken from this `reference `_ - -.. DANGER:: - - Something very bad! - -.. tip:: - - This is something good to know - -Some other directives -~~~~~~~~~~~~~~~~~~~~~ - -"attention", "caution", "danger", "error", "hint", "important", "note", "tip", "warning", "admonition" - -You can use anchors for internal :ref:`references ` too - -Other resources ---------------- - -- http://pygments.org/docs/lexers/ -- http://documentation-style-guide-sphinx.readthedocs.io/en/latest/style-guide.html -- http://www.sphinx-doc.org/en/stable/markup/code.html diff --git a/docs/source/supporting/examples/ConfigActuation.rst b/docs/source/supporting/examples/ConfigActuation.rst deleted file mode 100644 index 6949b19f7a..0000000000 --- a/docs/source/supporting/examples/ConfigActuation.rst +++ /dev/null @@ -1,22 +0,0 @@ -.. _ConfigActuation: - -Config Actuation Example -========================= - -The ConfigActuation example attempts to set points on a device when files -are added or updated in its :ref:`configuration store `. - - -Configuration -------------- - -The name of a configuration file must match the name of the device to -be actuated. The configuration file is a json dictionary of point name and value -pairs. Any number of points on the device can be listed in the config. - -.. code-block:: python - - { - "point0": value, - "point1": value - } diff --git a/docs/source/unused/deprecated/Logging.rst b/docs/source/unused/deprecated/Logging.rst deleted file mode 100644 index e3c1b2a96f..0000000000 --- a/docs/source/unused/deprecated/Logging.rst +++ /dev/null @@ -1,64 +0,0 @@ -Data Logging ------------- - -A mechanism allowing agents to store timeseries data has been provided. -In VOLTTRON 2.0 this facility was provided by an sMAP agent but it has -now been folded into the new Historians. This service still uses the old -format to maintain compatibility. - -Data Logging Format -~~~~~~~~~~~~~~~~~~~ - -Data sent to the data logger should be sent as a JSON object that -consists of a dictionary of dictionaries. The keys of the outer -dictionary are used as the points to store the data items. The inner -dictionary consists of 2 required fields and 1 optional. The required -fields are "Readings" and "Units". Readings contains the data that will -be written. It may contain either a single value, or a list of lists -which consists of timestamp/value pairs. Units is a string that -identifies the meaning of the scale values of the data. The optional -entry is data\_type, which indicates the type of the data to be stored. -This may be either long or double. - -:: - - { - "test3": { - "Readings": [[1377788595, 1.1],[1377788605,2.0]], - "Units": "KwH", - "data_type": "double" - }, - "test4": { - "Readings": [[1377788595, 1.1],[1377788605,2.0]], - "Units": "TU", - "data_type": "double" - } - } - -Example Code -~~~~~~~~~~~~ - -:: - - headers[headers_mod.FROM] = self._agent_id - headers[headers_mod.CONTENT_TYPE] = headers_mod.CONTENT_TYPE.JSON - - mytime = int(time.time()) - - content = { - "listener": { - "Readings": [[mytime, 1.0]], - "Units": "TU", - "data_type": "double" - }, - "hearbeat": { - "Readings": [[mytime, 1.0]], - "Units": "TU", - "data_type": "double" - } - } - - - - self.publish('datalogger/log/', headers, json.dumps(content)) - diff --git a/docs/source/unused/deprecated/MultiBuildingMessaging.rst b/docs/source/unused/deprecated/MultiBuildingMessaging.rst deleted file mode 100644 index 6aa877c75e..0000000000 --- a/docs/source/unused/deprecated/MultiBuildingMessaging.rst +++ /dev/null @@ -1,88 +0,0 @@ -MultiBuilding Agent -=================== - -This agent has been superseded by the VIP functionality introduced in 3.0 and should be considered deprecated. However it is still a usable agent. - -Multi-building (or multi-node) messaging is implemented as a -service-style agent. Its use is optional and it can be enabled/disabled -by simply enabling/disabling the multibuilding service agent. It is -easily configured using the service configuration file and provides -several new topics for use in the local agent exchange bus. - -Configuration -~~~~~~~~~~~~~ - -The service configuration file may contain the declarations below: - -- | *building-publish-address*: - | A ØMQ address on which to listen for messages published by other - nodes. Defaults to 'tcp://0.0.0.0:9161'. - -- | *building-subscribe-address*: - | A ØMQ address on which to listen for messages subscribed to by - other nodes. Defaults to 'tcp://0.0.0.0:9160'. - -- | *public-key*, *secret-key*: - | Curve keypair (create with zmq.curve\_keypair()) to use for - authentication and encryption. If not provided, all communications - will be unauthenticated and unencrypted. - -- | *hosts*: - | A mapping (dictionary) of building names to publish/subscribe - addresses. Each entry is of the form: - - :: - - "CAMPUS/BUILDING": {"pub": "PUB_ADDRESS", "sub": "SUB_ADDRESS", "public-key": "PUBKEY", "allow": "PUB_OR_SUB"} - -- CAMPUS/BUILDING: building for which the given parameters apply -- PUB\_ADDRESS: ØMQ address used to connect to the building for - publishing -- SUB\_ADDRESS: ØMQ address used to connect to the building - subscriptions -- PUBKEY: curve public key of the host used to authenticate incoming - connections -- PUB\_OR\_SUB: the string "pub" to allow publishing only or "sub" to - allow both publish and subscribe - -- *cleanup-period*: Frequency, in seconds, to check for and close stale - connections. Defaults to 600 seconds (10 minutes). - -- *uuid*: A UUID to use in the Cookie header. If not given, one will be - automatically generated. - -Sending and Receiving Inter-building Messages -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Three topics are provided for inter-building messaging: - -- | building/recv/\ ``CAMPUS/BUILDING/TOPIC``: - | Agents can subscribe to to this topic to receive messages sent to - ``TOPIC`` at the building specified by ``CAMPUS``/``BUILDING``. - -- | building/send/\ ``CAMPUS/BUILDING/TOPIC``: - | Agents can send messages to this topic to have them forwarded to - ``TOPIC`` at the building specified by ``CAMPUS``/``BUILDING``. - -- | building/error/\ ``CAMPUS/BUILDING/TOPIC``: - | Errors encountered during sending/receiving to/from the above two - topics will be sent over this topic. - -Limitations and Future Work -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- | Requires opening multiple listening ports: - | It would be nice to multiplex all inter-building communications - over a single port to decrease the attack footprint and ease firewall - administration. - -- | There is limited authorization: - | a remote host can either publish or publish and subscribe. Perhaps - a filter list can be included to limit which topics a host may - subscribe to or publish on. - -- | Remote host lookup is kept in a static file: - | Ideally, host lookup would be done through some central directory - service, but that is not currently implemented. - - diff --git a/docs/source/unused/deprecated/MultiNodeExample.rst b/docs/source/unused/deprecated/MultiNodeExample.rst deleted file mode 100644 index 77a48eab0d..0000000000 --- a/docs/source/unused/deprecated/MultiNodeExample.rst +++ /dev/null @@ -1,141 +0,0 @@ -Multinode Example -================= - -The MultiNode example agent demonstrates how to setup and make use of -the [[MultiBuildingMessaging]] agent. - -For convenience, this example is setup to be run from a single machine -but could be easily modified to run off multiple systems. Multiple -instances of VOLTTRON can be run on a single machine with the proper -configuration. For this example, two separate VOLTTRON homes are setup -and the MultiBuilding service binds to different local addresses. -[[PlatformConfiguration]] shows how the VOLTTRON\_HOME is used for -platform directories. - -The example agent directory contains the config files for the example -agents and the multibuilding agents. Each is setup to know about the -other platform instance and contains its own pub and sub addresses. -Please see [[MultiBuildingMessaging]] for details on the configuration -file. - -MultiBuilding config: - -:: - - { - "building-publish-address": "tcp://127.0.0.1:12201", - "building-subscribe-address": "tcp://127.0.0.1:12202", - "uuid": "MultiBuildingService", - "hosts": { - "campus/platform1": { - "pub": "tcp://127.0.0.1:12201", - "sub": "tcp://127.0.0.1:12202" - }, - "campus/platform2": { - "pub": "tcp://127.0.1.1:12201", - "sub": "tcp://127.0.1.1:12202" - } - } - } - -Each GreeterAgent is setup with the other hosts it will be publishing to -in the publish\_heartbeat method. - -GreeterAgent config: - -:: - - { - "agentid": "Greeter1", - "receiving_platforms": ["platform2"] - } - -In order to run this example: - -- First activate the platform: - - :: - - . env/bin/activate - -- Then, create the directories which will be used by each platform as - its VOLTTRON\_HOME: - - :: - - mkdir ~/.platform1 - mkdir ~/.platform2 - -- Start the first platform: - - :: - - VOLTTRON_HOME=~/.platform1 volttron -vv -l platform1.log& - -- Build, configure, and install the multibuilding platform agent. The - Agent is installed with "multinode=" to tag the agent at the same - time it is installed. This is a convenient way to refer to the agent - later. - - :: - - VOLTTRON_HOME=~/.platform1 volttron-pkg package Agents/MultiBuilding - VOLTTRON_HOME=~/.platform1 volttron-pkg configure ~/.volttron/packaged/multibuildingagent-0.1-py2-none-any.whl Agents/MultiNodeExample/multicomm.service - VOLTTRON_HOME=~/.platform1 volttron-ctl install multinode=~/.volttron/packaged/multibuildingagent-0.1-py2-none-any.whl - -- Build, configure, and install the GreeterAgent - - :: - - VOLTTRON_HOME=~/.platform1 volttron-pkg package Agents/MultiNodeExample - VOLTTRON_HOME=~/.platform1 volttron-pkg configure ~/.volttron/packaged/greeteragent-0.1-py2-none-any.whl Agents/MultiNodeExample/agent1.config - VOLTTRON_HOME=~/.platform1 volttron-ctl install greeter=~/.volttron/packaged/greeteragent-0.1-py2-none-any.whl - -- Start the second platform: - - :: - - VOLTTRON_HOME=~/.platform2 volttron -vv -l platform2.log& - -- Build, configure, and install the MultiBuilding service for the - second platform - - :: - - VOLTTRON_HOME=~/.platform2 volttron-pkg package Agents/MultiBuilding - VOLTTRON_HOME=~/.platform2 volttron-pkg configure ~/.volttron/packaged/multibuildingagent-0.1-py2-none-any.whl Agents/MultiNodeExample/multicomm2.service - VOLTTRON_HOME=~/.platform2 volttron-ctl install multinode=~/.volttron/packaged/multibuildingagent-0.1-py2-none-any.whl - -- Build, configure, and install the GreeterAgent for the second - platform - - :: - - VOLTTRON_HOME=~/.platform2 volttron-pkg package Agents/MultiNodeExample - VOLTTRON_HOME=~/.platform2 volttron-pkg configure ~/.volttron/packaged/greeteragent-0.1-py2-none-any.whl Agents/MultiNodeExample/agent2.config - VOLTTRON_HOME=~/.platform2 volttron-ctl install greeter=~/.volttron/packaged/greeteragent-0.1-py2-none-any.whl - -- Start up the agents on both platforms by referring to them by the tag - they were installed with - - :: - - VOLTTRON_HOME=~/.platform1 volttron-ctl start --tag multinode - VOLTTRON_HOME=~/.platform1 volttron-ctl start --tag greeter - VOLTTRON_HOME=~/.platform2 volttron-ctl start --tag multinode - VOLTTRON_HOME=~/.platform2 volttron-ctl start --tag greeter - -- Check the logs for each platform for the presence of messages from - the other platform's GreeterAgent - - :: - - grep Greeter2 platform1.log -a - - ``2014-09-30 17:13:41,840 (greeteragent-0.1 13878) greeter.agent DEBUG: Topic: greetings/hello, Headers: Headers({u'Date': u'2014-10-01 00:13:41.831539Z', u'Cookie': u'Greeter2', u'AgentID': u'Greeter2', u'Content-Type': [u'application/json']}), Message: ['{"message":"HELLO from Greeter2!"}']`` - - :: - - grep Greeter1 platform2.log -a - - diff --git a/docs/source/volttron-topics/change-log/index.rst b/docs/source/volttron-topics/change-log/index.rst new file mode 100644 index 0000000000..e680a53f10 --- /dev/null +++ b/docs/source/volttron-topics/change-log/index.rst @@ -0,0 +1,17 @@ +.. _Change-Log: + +========== +Change Log +========== + +This section includes individual documents describing important changes to platform components, such as the RabbitMQ +message bus implementation. For information on specific changes, please refer to the corresponding document. + + +.. toctree:: + + message-bus-refactor/index + scalability/scalability + pubsub-enhancement + version-history + upgrading-versions diff --git a/docs/source/setup/RabbitMQ/remote_agent.rst b/docs/source/volttron-topics/change-log/message-bus-refactor/agent-communication-rabbitmq.rst similarity index 90% rename from docs/source/setup/RabbitMQ/remote_agent.rst rename to docs/source/volttron-topics/change-log/message-bus-refactor/agent-communication-rabbitmq.rst index 15480ed1e7..248663f224 100644 --- a/docs/source/setup/RabbitMQ/remote_agent.rst +++ b/docs/source/volttron-topics/change-log/message-bus-refactor/agent-communication-rabbitmq.rst @@ -1,4 +1,4 @@ -.. _Connecting_to_remote_RMQ: +.. _Agent-Communication-to-Remote-RabbitMQ: =============================================== Agent communication to Remote RabbitMQ instance @@ -25,7 +25,7 @@ Configuration ------------- Both volttron-server and volttron-client must be configured for RabbitMQ message bus with SSL using the step described -at :ref:`Installing Volttron`. +at :ref:`Installing Volttron `. In addition the remote-volttron-instance configuration file must have a https bind-web-address specified in the instance config file. Below is an example config file with bind-web-address. Restart volttron after editing the config @@ -73,9 +73,10 @@ The following is a code snippet from the remote-agent to connect to the remote v value = self.vip.auth.connect_remote_platform(address) The above function call will return an agent that connects to the remote instance only after the request is approved -by an adminstrator of the remote instance. It is up to the agent to repeat calling `connect_remote_platform` +by an administrator of the remote instance. It is up to the agent to repeat calling `connect_remote_platform` periodically until an agent object is obtained. + Approving a CSR Request ~~~~~~~~~~~~~~~~~~~~~~~ @@ -86,8 +87,9 @@ is not None. |CSR Approval| + Denying a CSR Request -~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~ The following diagram shows the sequence of events when an access request is denied by the administrator. The client agent repeats the call to connect_remote_platform until the return value is not None. When the remote instance's administrator denies a access request, the auth subsystem will raise an alert and shutdown the agent. @@ -95,9 +97,9 @@ administrator denies a access request, the auth subsystem will raise an alert an |CSR Denied| -.. |CSR Approval| image:: images/csr-sequence-approval.png -.. |CSR Denied| image:: images/csr-sequence-deny.png +.. |CSR Approval| image:: files/csr-sequence-approval.png +.. |CSR Denied| image:: files/csr-sequence-deny.png -Follow walk-through in :ref:`Multi-Platform Multi-Bus Walk-through <_Multi_Platform_Walkthrough>` for setting up different -combinations of multi-bus multi-platform setup using CSR. +Follow walk-through in :ref:`Multi-Platform Multi-Bus Walk-through ` for setting up +different combinations of multi-bus multi-platform setup using CSR. diff --git a/docs/source/volttron-topics/change-log/message-bus-refactor/files/csr-sequence-approval.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/csr-sequence-approval.png new file mode 100644 index 0000000000..cf32895d02 Binary files /dev/null and b/docs/source/volttron-topics/change-log/message-bus-refactor/files/csr-sequence-approval.png differ diff --git a/docs/source/volttron-topics/change-log/message-bus-refactor/files/csr-sequence-deny.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/csr-sequence-deny.png new file mode 100644 index 0000000000..6f7a5d4396 Binary files /dev/null and b/docs/source/volttron-topics/change-log/message-bus-refactor/files/csr-sequence-deny.png differ diff --git a/docs/source/core_services/messagebus_refactor/files/federation.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/federation.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/federation.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/federation.png diff --git a/docs/source/core_services/messagebus_refactor/files/multiplatform_pubsub.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_pubsub.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/multiplatform_pubsub.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_pubsub.png diff --git a/docs/source/core_services/messagebus_refactor/files/multiplatform_rpc.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_rpc.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/multiplatform_rpc.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_rpc.png diff --git a/docs/source/core_services/messagebus_refactor/files/multiplatform_shovel_pubsub.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_shovel_pubsub.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/multiplatform_shovel_pubsub.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_shovel_pubsub.png diff --git a/docs/source/core_services/messagebus_refactor/files/multiplatform_shovel_rpc.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_shovel_rpc.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/multiplatform_shovel_rpc.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_shovel_rpc.png diff --git a/docs/source/core_services/messagebus_refactor/files/multiplatform_ssl.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_ssl.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/multiplatform_ssl.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/multiplatform_ssl.png diff --git a/docs/source/core_services/messagebus_refactor/files/proxy_router.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/proxy_router.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/proxy_router.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/proxy_router.png diff --git a/docs/source/core_services/messagebus_refactor/files/pubsub.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/pubsub.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/pubsub.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/pubsub.png diff --git a/docs/source/core_services/messagebus_refactor/files/rabbitmq_exchange.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/rabbitmq_exchange.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/rabbitmq_exchange.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/rabbitmq_exchange.png diff --git a/docs/source/core_services/messagebus_refactor/files/rmq_server_ssl_certs.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/rmq_server_ssl_certs.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/rmq_server_ssl_certs.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/rmq_server_ssl_certs.png diff --git a/docs/source/core_services/messagebus_refactor/files/rpc.png b/docs/source/volttron-topics/change-log/message-bus-refactor/files/rpc.png similarity index 100% rename from docs/source/core_services/messagebus_refactor/files/rpc.png rename to docs/source/volttron-topics/change-log/message-bus-refactor/files/rpc.png diff --git a/docs/source/core_services/messagebus_refactor/index.rst b/docs/source/volttron-topics/change-log/message-bus-refactor/index.rst similarity index 90% rename from docs/source/core_services/messagebus_refactor/index.rst rename to docs/source/volttron-topics/change-log/message-bus-refactor/index.rst index 559a351142..1effc75f19 100644 --- a/docs/source/core_services/messagebus_refactor/index.rst +++ b/docs/source/volttron-topics/change-log/message-bus-refactor/index.rst @@ -22,7 +22,9 @@ The goal of the message bus refactor task is to .. toctree:: - :glob: - :maxdepth: 2 - * + rabbitmq-overview + message-bus-plugin + rabbitmq-refactor + agent-communication-rabbitmq + rabbitmq-ssl-auth diff --git a/docs/source/core_services/messagebus_refactor/Messagebus-Plugin.rst b/docs/source/volttron-topics/change-log/message-bus-refactor/message-bus-plugin.rst similarity index 99% rename from docs/source/core_services/messagebus_refactor/Messagebus-Plugin.rst rename to docs/source/volttron-topics/change-log/message-bus-refactor/message-bus-plugin.rst index b6a191ee18..461313bad5 100644 --- a/docs/source/core_services/messagebus_refactor/Messagebus-Plugin.rst +++ b/docs/source/volttron-topics/change-log/message-bus-refactor/message-bus-plugin.rst @@ -1,8 +1,9 @@ - .. _Messagebus-Plugin: + .. _Message-Bus-Plugin: ============================ Message Bus Plugin Framework ============================ + The message bus plugin framework aims to decouple the VOLTTRON specific code from the message bus implementation without compromising the existing features of the platform. The concept of the plugin framework is similar to that used in historian diff --git a/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-overview.rst b/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-overview.rst new file mode 100644 index 0000000000..79935b2733 --- /dev/null +++ b/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-overview.rst @@ -0,0 +1,267 @@ + .. _RabbitMQ-Overview: + +================= +RabbitMQ Overview +================= + +.. NOTE:: + + Some of the RabbitMQ summary/overview documentation and supporting images added here are taken from the + `RabbitMQ official documentation `_. + +RabbitMQ is the most popular messaging library with over 35,000 production deployments. It is highly scalable, easy to +deploy, runs on many operating systems and cloud environments. It supports many kinds of distributed deployment +methodologies such as clusters, federation and shovels. + + +RabbitMQ uses `Advanced Message Queueing Protocol` (AMQP) and works on the basic producer consumer model. A consumer is +a program that consumes/receives messages and producer is a program that sends the messages. Following are some +important definitions that we need to know before we proceed. + +* Queue - Queues can be considered like a post box that stores messages until consumed by the consumer. Each consumer + must create a queue to receives messages that it is interested in receiving. We can set properties to the queue + during it's declaration. The queue properties are: + + * Name - Name of the queue + * Durable - Flag to indicate if the queue should survive broker restart. + * Exclusive - Used only for one connection and it will be removed when connection is closed. + * Auto-queue - Flag to indicate if auto-delete is needed. The queue is deleted when last consumer un-subscribes from + it. + * Arguments - Optional, can be used to set message TTL (Time To Live), queue limit etc. + +* Bindings - Consumers bind the queue to an exchange with binding keys or routing patterns. Producers send messages and + associate them with a routing key. Messages are routed to one or many queues based on a pattern matching between a + message routing key and binding key. + +* Exchanges - Exchanges are entities that are responsible for routing messages to the queues based on the routing + pattern/binding key used. They look at the routing key in the message when deciding how to route messages to queues. + There are different types of exchanges and one must choose the type of exchange depending on the application design + requirements + + #. Fanout - It blindly broadcasts the message it receives to all the queues it knows. + + #. Direct - Here, the message is routed to a queue if the routing key of the message exactly matches the binding key + of the queue. + + #. Topic - Here, the message is routed to a queue based on pattern matching of the routing key with the binding key. + The binding key and the routing key pattern must be a list of words delimited by dots, for example, + "car.subaru.outback" or "car.subaru.*", "car.#". A message sent with a particular routing key will be delivered + to all the queues that are bound with a matching binding key with some special rules as + + '*' (star) - can match exactly one word in that position. + '#' (hash) - can match zero or more words + + #. Headers - If we need more complex matching then we can add a header to the message with all the attributes set to + the values that need to be matched. The message is considered matching if the values of the attributes in the + header is equal to that of the binding. The Header exchange ignores the routing key. + + We can set some properties of the exchange during it's declaration. + + * Name - Name of the exchange + * Durable - Flag to indicate if the exchange should survive broker restart. + * Auto-delete - Flag indicates if auto-delete is needed. If set to true, the exchange is deleted when the last queue + is unbound from it. + * Arguments - Optional, used by plugins and broker-specific features + +Lets use an example to understand how they all fit together. Consider an example where there are four consumers +(Consumer 1 - 4) interested in receiving messages matching the pattern "green", "red" or "yellow". In this example, we +are using a direct exchange that will route the messages to the queues only when there is an exact match of the routing +key of the message with the binding key of the queues. Each of the consumers declare a queue and bind the queue to the +exchange with a binding key of interest. Lastly, we have a producer that is continuously sending messages to exchange +with routing key "green". The exchange will check for an exact match and route the messages to only Consumer 1 and +Consumer 3. + +.. image:: files/rabbitmq_exchange.png + + +For more information about queues, bindings, exchanges, please refer to the +`RabbitMQ tutorial `_. + + +Distributed RabbitMQ Brokers +============================ + +RabbitMQ allows multiple distributed RabbitMQ brokers to be connected in three different ways - with clustering, with +federation and using shovel. We take advantage of these built-in plugins for multi-platform VOLTTRON communication. For +more information about the differences between clustering, federation, and shovel, please refer to the RabbitMQ +documentation on `Distributed RabbitMQ brokers `_. + +Clustering +---------- + +Clustering connects multiple brokers residing in multiple machines to form a single logical broker. It is used in +applications where tight coupling is necessary i.e, where each node shares the data and knows the state of all other +nodes in the cluster. A new node can connect to the cluster through a peer discovery mechanism if configured to do so +in the RabbitMQ config file. For all the nodes to be connected together in a cluster, it is necessary for them to share +the same Erlang cookie and be reachable through it's DNS hostname. A client can connect to any one of the nodes in the +cluster and perform any operation (to send/receive messages from other nodes etc.), the nodes will route the operation +internally. In case of a node failure, clients should be able to reconnect to a different node, recover their topology +and continue operation. + +.. note:: + + This feature is not integrated into VOLTTRON, but we hope to support it in the future. For more detailed + information about clustering, please refer to RabbitMQ documentation on the + `Clustering plugin `_. + + +.. _RabbitMQ-Federation: + +Federation +---------- +Federation plugin is used in applications that does not require as much of tight coupling as clustering. Federation has +several useful features: + +* Loose coupling - The federation plugin can transmit messages between brokers (or clusters) in different administrative + domains: + + * they may have different users and virtual hosts; + * they may run on different versions of RabbitMQ and Erlang. + +* WAN friendliness - They can tolerate network intermittent connectivity. + +* Specificity - Not everything needs to be federated ( made available to other brokers ); There can be local-only + components. + +* Scalability - Federation does not require O(n2) connections for *n* brokers, so it scales better. + +The federation plugin allows you to make exchanges and queues *federated*. A federated exchange or queue can receive +messages from one or more upstreams (remote exchanges and queues on other brokers). A federated exchange can route +messages published upstream to a local queue. A federated queue lets a local consumer receive messages from an upstream +queue. + +Before we move forward, let's define upstream and downstream servers. + +* Upstream server - The node that is publishing some message of interest +* Downstream server - The node connected to a different broker that wants to receive messages from the upstream server + +A federation link needs to be established from downstream server to the upstream server. The data flows in single +direction from upstream server to downstream server. For bi-directional data flow, we would need to create federation +links on both the nodes. + +We can receive messages from upstream server to downstream server by either making an exchange or a queue *federated*. + +For more detailed information about federation, please refer to RabbitMQ documentation +`Federation plugin `_. + + +Federated Exchange +------------------ + +When we make an exchange on the downstream server *federated*, the messages published to the upstream exchanges are +copied to the federated exchange, as though they were published directly to it. + +.. image:: files/federation.png + +The above figure explains message transfer using federated exchange. The box on the right acts as the downstream server +and the box on the left acts as the upstream server. A federation/upstream link is established between the downstream +server and the upstream server by using the federation management plugin. + +An exchange on the downstream server is made *federated* using federation policy configuration. The federated exchange +only receives the messages for which it has subscribed. An upstream queue is created on the upstream server with a +binding key same as subscription made on the federated exchange. For example, if an upstream server is publishing +messages with binding key "foo" and a client on the downstream server is interested in receiving messages of the +binding key "foo", then it creates a queue and binds the queue to the federated with the same binding key. This binding +is sent to the upstream and the upstream queue binds to the upstream exchange with that key. + +Publications to either exchange may be received by queues bound to the federated exchange, but publications +directly to the federated exchange cannot be received by queues bound to the upstream exchange. + +For more information about federated exchanges and different federation topologies, please read about +`Federated Exchanges `_. + + +Federated Queue +--------------- + +Federated queue provides a way of balancing load of a single queue across nodes or clusters. A federated queue lets a +local consumer receive messages from an upstream queue. A typical use would be to have the same "logical" queue +distributed over many brokers. Such a logical distributed queue is capable of having higher capacity than a single +queue. A federated queue links to other upstream queues. + +A federation or upstream link needs to be created like before and a federated queue needs to be setup on the downstream +server using federation policy configuration. The federated queue will only retrieve messages when it has run out of +messages locally, it has consumers that need messages, and the upstream queue has "spare" messages that are not being +consumed. + +For more information about federated queues, please read about +`Federated Queues `_. + + +.. _RabbitMQ-Shovel: + +Shovel +------ +The Shovel plugin allows you to reliably and continually move messages from a source in one +broker to destination in another broker. A shovel behaves like a well-written client application in that it: + +* connects to it's source and destination broker +* consumes messages from the source queue +* re-publishes messages to the destination if the messages match the routing key. + +The Shovel plugin uses an Erlang client under the hood. In the case of shovel, apart from configuring the hostname, +port and virtual host of the remote node, we will also have to provide a list of routing keys that we want to forward to +the remote node. The primary advantages of shovels are: + +* Loose coupling - A shovel can move messages between brokers (or clusters) in different administrative domains: + * they may have different users and virtual hosts; + * they may run on different versions of RabbitMQ and Erlang. +* WAN friendliness - They can tolerate network intermittent connectivity. + +Shovels are also useful in cases where one of the nodes is behind NAT. We can setup shovel on the node behind NAT to +forward messages to the node outside NAT. Shovels do not allow you to adapt to subscriptions like a federation link and +we need to a create a new shovel per subscription. + +For more detailed information about shovel, please refer to RabbitMQ documentation on the +`Shovel plugin `_. + + +Authentication in RabbitMQ +========================== + +By default RabbitMQ supports SASL PLAIN authentication with username and password. RabbitMQ supports other SASL +authentication mechanisms using plugins. In VOLTTRON we use one such external plugin based on x509 certificates +(``_). This authentication is based on a technique called +public key cryptography which consists of a key pair - a public key and a private key. Data that has been encrypted +with a public key can only be decrypted with the corresponding private key and vice versa. The owner of key pair makes +the public key available and keeps the private confidential. To send a secure data to a receiver, a sender encrypts the +data with the receiver's public key. Since only the receiver has access to his own private key only the receiver can +decrypted. This ensures that others, even if they can get access to the encrypted data, cannot decrypt it. This is how +public key cryptography achieves confidentiality. + +A digital certificate is a digital file that is used to prove ownership of a public key. Certificates act like +identification cards for the owner/entity. Certificates are therefore crucial to determine that a sender is using the +right public key to encrypt the data in the first place. Digital Certificates are issued by Certification +Authorities(CA). Certification Authorities fulfill the role of the `Trusted Third Party` by accepting Certificate +applications from entities, authenticating applications, issuing Certificates and maintaining status information about +the Certificates issued. Each CA has its own public private key pair and its public key certificate is called a root CA +certificate. The CA attests to the identity of a Certificate applicant when it signs the Digital Certificate using its +private key. + +In x509 based authentication, a signed certificate is presented instead of username/password for authentication and if +the server recognizes the the signer of the certificate as a trusted CA, accepts and allows the connection. Each +server/system can maintain its own list of trusted CAs (i.e. list of public certificates of CAs). Certificates signed +by any of the trusted CA would be considered trusted. Certificates can also be signed by intermediate CAs that are in +turn signed by a trusted. + +This section only provides a brief overview about the SSL based authentication. Please refer to the vast material +available online for detailed description. Some useful links to start: + + * ``_ + * ``_ + + +Management Plugin +================= + +The RabbitMQ-management plugin provides an HTTP-based API for management and monitoring of RabbitMQ nodes and clusters, +along with a browser-based UI and a command line tool, *rabbitmqadmin*. The management interface allows you to: + +* Create, Monitor the status and delete resources such as virtual hosts, users, exchanges, queues etc. +* Monitor queue length, message rates and connection information and more +* Manage users and add permissions (read, write and configure) to use the resources +* Manage policies and runtime parameters +* Send and receive messages (for trouble shooting) + +For more detailed information about the management plugin, please refer to RabbitMQ documentation on the +`Management Plugin `_. diff --git a/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-refactor.rst b/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-refactor.rst new file mode 100644 index 0000000000..886b10f64a --- /dev/null +++ b/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-refactor.rst @@ -0,0 +1,457 @@ +.. _RabbitMQ-VOLTTRON: + +======================= +RabbitMQ Based VOLTTRON +======================= + +RabbitMQ VOLTTRON uses the `Pika` library for the RabbitMQ message bus implementation. To install Pika, it is +recommended to use the VOLTTRON :ref:`bootstrap.py ` script: + +.. code-block:: bash + + python3 bootstrap.py --rabbitmq + + +Configuration +============= + +To setup a VOLTTRON instance to use the RabbitMQ message bus, we need to first configure VOLTTRON to use the RabbitMQ +message library. The contents of the RabbitMQ configuration file should follow the pattern below. + +Path: `$VOLTTRON_HOME/rabbitmq_config.yml` + +.. code-block:: yaml + + #host parameter is mandatory parameter. fully qualified domain name + host: mymachine.pnl.gov + + # mandatory. certificate data used to create root ca certificate. Each volttron + # instance must have unique common-name for root ca certificate + certificate-data: + country: 'US' + state: 'Washington' + location: 'Richland' + organization: 'PNNL' + organization-unit: 'VOLTTRON Team' + # volttron1 has to be replaced with actual instance name of the VOLTTRON + common-name: 'volttron1_root_ca' + # + # optional parameters for single instance setup + # + virtual-host: 'volttron' # defaults to volttron + + # use the below four port variables if using custom rabbitmq ports + # defaults to 5672 + amqp-port: '5672' + + # defaults to 5671 + amqp-port-ssl: '5671' + + # defaults to 15672 + mgmt-port: '15672' + + # defaults to 15671 + mgmt-port-ssl: '15671' + + # defaults to true + ssl: 'true' + + # defaults to ~/rabbitmq_server/rabbbitmq_server-3.7.7 + rmq-home: "~/rabbitmq_server/rabbitmq_server-3.7.7" + +Each VOLTTRON instance resides within a RabbitMQ virtual host. The name of the virtual host needs to be unique per +VOLTTRON instance if there are multiple virtual instances within a single host/machine. The hostname needs to be able +to resolve to a valid IP. The default port of an AMQP port without authentication is `5672` and with authentication +it is `5671`. The default management HTTP port without authentication is `15672` and with authentication is `15671`. +These needs to be set appropriately if the default ports are not used. + +The 'ssl' flag indicates if SSL based authentication is required or not. If set to `True`, information regarding SSL +certificates needs to be also provided. SSL based authentication is described in detail in +`Authentication And Authorization With RabbitMQ Message Bus `_. + +To configure the VOLTTRON instance to use RabbitMQ message bus, run the following command: + +.. code-block:: bash + + vcfg --rabbitmq single [optional path to rabbitmq_config.yml] + +At the end of the setup process, a RabbitMQ broker is setup to use the configuration provided. A new topic exchange for +the VOLTTRON instance is created within the configured virtual host. + +On platform startup, VOLTTRON checks for the type of message bus to be used. If using the RabbitMQ message bus, the +RabbitMQ platform router is instantiated. The RabbitMQ platform router: + +* Connects to RabbitMQ broker (with or without authentication) +* Creates a VIP queue and binds itself to the "VOLTTRON" exchange with binding key `.router`. This + binding key makes it unique across multiple VOLTTRON instances in a single machine as long as each instance has a + unique instance name. +* Handles messages intended for router module such as `hello`, `peerlist`, `query` etc. +* Handles "unrouteable" messages - Messages which cannot be routed to any destination agent are captured and an error + message indicating "Host Unreachable" error is sent back to the caller. +* Disconnects from the broker when the platform shuts down. + +When any agent is installed and started, the Agent Core checks for the type of message bus used. If it is RabbitMQ +message bus then: + +* It creates a RabbitMQ user for the agent +* If SSL based authentication is enabled, client certificates for the agent is created +* Connect to the RabbitQM broker with appropriate connection parameters +* Creates a VIP queue and binds itself to the "VOLTTRON" exchange with binding key `.` +* Sends and receives messages using Pika library methods. +* Checks for the type of subsystem in the message packet that it receives and calls the appropriate subsystem message + handler. +* Disconnects from the broker when the agent stops or platform shuts down. + + +RPC In RabbitMQ VOLTTRON +======================== + +The agent functionality remain unchanged regardless of the underlying message bus used, meaning they can continue to use +the same RPC interfaces without any change. + +.. image:: files/rpc.png + +Consider two agents with VIP identities "agent_a" and "agent_b" connected to VOLTTRON platform +with instance name "volttron1". Agent A and B each have a VIP queue with binding key volttron1.agent_a" +and "volttron1.agent_b". Following is the sequence of operation when Agent A wants to make RPC +call to Agent B: + +1. Agent A makes a RPC call to Agent B. + +.. code-block:: python + + agent_a.vip.rpc.call("agent_b", set_point, "point_name", 2.5) + +2. RPC subsystem wraps this call into a VIP message object and sends it to Agent B. +3. The VOLTTRON exchange routes the message to Agent B as the destination routing in the VIP message object matches with + the binding key of Agent B. +4. Agent Core on Agent B receives the message, unwraps the message to find the subsystem type and calls the RPC + subsystem handler. +5. RPC subsystem makes the actual RPC call `set_point()` and gets the result. It then wraps into VIP message object and + sends it back to the caller. +6. The VOLTTRON exchange routes it to back to Agent A. +7. Agent Core on Agent A calls the RPC subsystem handler which in turn hands over the RPC result to Agent A application. + + +PUBSUB In RabbitMQ VOLTTRON +=========================== + +The agent functionality remains unchanged irrespective of the platform using ZeroMQ based pubsub or +RabbitMQ based pubsub, i.e. agents continue to use the same PubSub interfaces and use the same topic +format delimited by “/”. Since RabbitMQ expects binding key to be delimited by '.', RabbitMQ PUBSUB +internally replaces '/' with ".". Additionally, all agent topics are converted to +``_pubsub__..`` to differentiate them from the main Agent VIP queue binding. + +.. image:: files/pubsub.png + +Consider two agents with VIP identities "agent_a" and "agent_b" connected to VOLTTRON platform +with instance name "volttron1". Agent A and B each have a VIP queue with binding key "volttron1.agent_a" +and "volttron1.agent_b". Following is the sequence of operation when Agent A subscribes to a topic and Agent B +publishes to same the topic: + +1. Agent B makes subscribe call for topic "devices". + +.. code-block:: python + + agent_b.vip.pubsub.subscribe("pubsub", prefix="devices", callback=self.onmessage) + +2. Pubsub subsystem creates binding key from the topic ``__pubsub__.volttron1.devices.#`` + +3. It creates a queue internally and binds the queue to the VOLTTRON exchange with the above binding key. + +4. Agent B is publishing messages with topic: "devices/hvac1". + +.. code-block:: python + + agent_b.vip.pubsub.publish("pubsub", topic="devices/hvac1", headers={}, message="foo"). + +5. PubSub subsystem internally creates a VIP message object and publishes on the VOLTTRON exchange. + +6. RabbitMQ broker routes the message to Agent B as routing key in the message matches with the binding key of the topic + subscription. + +7. The pubsub subsystem unwraps the message and calls the appropriate callback method of Agent A. + +If agent wants to subscribe to topic from remote instances, it uses: + +.. code-block:: python + + agent.vip.subscribe('pubsub', 'devices.hvac1', all_platforms=True) + +It is internally set to ``__pubsub__.*.`` + + +Further Work +------------ + +The Pubsub subsystem for the ZeroMQ message bus performs O(N) comparisons where N is the number of unique subscriptions. +The RabbitMQ Topic Exchange was enhanced in version 2.6.0 to reduce the overhead of additional unique subscriptions to +almost nothing in most cases. We speculate they are using a tree structure to store the binding keys which would reduce +the search time to O(1) in most cases and O(ln) in the worst case. The VOLTTRON PubSub with ZeroMQ could be updated to +match this performance scalability with some effort. + + +Multi-Platform Communication In RabbitMQ VOLTTRON +================================================= + +With ZeroMQ based VOLTTRON, multi-platform communication was accomplished in three different ways: + +1. Direct connection to remote instance - Write an agent that would connect to a remote instance directly. + +2. Special agents - Use special agents such as forward historian/data puller agents that would forward/receive messages + to/from remote instances. In RabbitMQ-VOLTTRON, we make use of the `shovel` plugin to achieve this behavior. Please + refer to :ref:`Shovel Plugin ` to get an overview of shovels. + +3. Multi-Platform RPC and PubSub - Configure VIP address of all remote instances that an instance has to connect to in + it's `$VOLTTRON_HOME/external_discovery.json` and let the router module in each instance manage the connection and + take care of the message routing for us. In RabbitMQ-VOLTTRON, we make use of the `federation` plugin to achieve + this behavior. Please refer to :ref:`Federation Plugin ` get an overview of federation. + + +Using the Federation Plugin +--------------------------- + +We can connect multiple VOLTTRON instances using the federation plugin. Before setting up federation links, we need to +first identify the upstream server and downstream server. The upstream server is the node that is publishing some +message of interest and downstream server is the node that wants to receive messages from the upstream server. A +federation link needs to be established from the downstream VOLTTRON instance to the upstream VOLTTRON instance. To +setup a federation link, we will need to add upstream server information in a RabbitMQ federation configuration file: + +Path: `$VOLTTRON_HOME/rabbitmq_federation_config.yml` + +.. code-block:: yaml + + # Mandatory parameters for federation setup + federation-upstream: + rabbit-4: + port: '5671' + virtual-host: volttron4 + rabbit-5: + port: '5671' + virtual-host: volttron5 + +To configure the VOLTTRON instance to setup federation, run the following command: + +.. code-block:: bash + + vcfg --rabbitmq federation [optional path to rabbitmq_federation_config.yml] + +This will setup federation links to upstream servers and sets policy to make the VOLTTRON exchange *federated*. Once a +federation link is established to remote instance, the messages published on the remote instance become available to +local instance as if it were published on the local instance. + +For detailed instructions to setup federation, please refer to the +:ref:`platform installation docs `. + + +Multi-Platform RPC With Federation +---------------------------------- + +For multi-platform RPC communication, federation links need to be established on both the VOLTTRON +nodes. Once the federation links are established, RPC communication becomes fairly simple. + +.. image:: files/multiplatform_rpc.png + +Consider Agent A on VOLTTRON instance "volttron1" on host "host_A" wants to make RPC call to Agent B +on VOLTTRON instance "volttron2" on host "host_B". + +1. Agent A makes RPC call. + +.. code-block:: Python + + kwargs = {"external_platform": self.destination_instance_name} + agent_a.vip.rpc.call("agent_b", set_point, "point_name", 2.5, \**kwargs) + +2. The message is transferred over federation link to VOLTTRON instance "volttron2" as both the exchanges are made + *federated*. + +3. The RPC subsystem of Agent B calls the actual RPC method and gets the result. It encapsulates the message result + into a VIP message object and sends it back to Agent A on VOLTTRON instance "volttron1". + +4. The RPC subsystem on Agent A receives the message result and gives it to the Agent A application. + + +Multi-Platform PubSub With Federation +------------------------------------- + +For multi-platform PubSub communication, it is sufficient to have federation link from the downstream server +to the upstream server. In case of bi-directional data flow, links have to established in both the directions. + +.. image:: files/multiplatform_pubsub.png + +Consider Agent B on VOLTTRON instance "volttron2" on host "host_B" which wants to subscribe to messages from +VOLTTRON instance "volttron2" on host "host_B". First, a federation link needs to be established from +"volttron2" to "volttron1". + +1. Agent B makes a subscribe call: + +.. code-block:: python + + agent_b.vip.subscribe.call("pubsub", prefix="devices", all_platforms=True) + +2. The PubSub subsystem converts the prefix to ``__pubsub__.*.devices.#``. Here, "*" indicates that agent is subscribing + to the "devices" topic from all VOLTTRON platforms. + +3. A new queue is created and bound to VOLTTRON exchange with the above binding key. Since the VOLTTRON exchange is a + *federated exchange*, any subscribed message on the upstream server becomes available on the federated exchange and + Agent B will be able to receive it. + +4. Agent A publishes message to topic `devices/pnnl/isb1/hvac1` + +5. The PubSub subsystem publishes this message on it's VOLTTRON exchange. + +6. Due to the federation link, message is received by the Pubsub subsystem of Agent A. + + +Using the Shovel Plugin +----------------------- + +Shovels act as well written client applications which move messages from a source to a destination broker. +The below configuration shows how to setup a shovel to forward PubSub messages or perform multi-platform RPC +communication from local to a remote instance. It expects `hostname`, `port` and `virtual host` configuration vlues for +the remote instance. + +Path: `$VOLTTRON_HOME/rabbitmq_shovel_config.yml` + +.. code-block:: yaml + + # Mandatory parameters for shovel setup + shovel: + rabbit-2: + port: '5671' + virtual-host: volttron + # Configuration to forward pubsub topics + pubsub: + # Identity of agent that is publishing the topic + platform.driver: + - devices + # Configuration to make remote RPC calls + rpc: + # Remote instance name + volttron2: + # List of pair of agent identities (local caller, remote callee) + - [scheduler, platform.actuator] + +To forward PubSub messages, the topic and agent identity of the publisher agent is needed. To perform RPC, the instance +name of the remote instance and agent identities of the local agent and remote agent are needed. + +To configure the VOLTTRON instance to setup shovel, run the following command. + +.. code-block:: bash + + vcfg --rabbitmq shovel [optional path to rabbitmq_shovel_config.yml] + +This setups up a shovel that forwards messages (either PubSub or RPC) from local exchange to remote exchange. + + +Multi-Platform PubSub With Shovel +--------------------------------- +After the shovel link is established for Pubsub, the below figure shows how the communication happens. + +.. note:: + + For bi-directional pubsub communication, shovel links need to be created on both the nodes. The "blue" arrows show + the shovel binding key. The pubsub topic configuration in `$VOLTTRON_HOME/rabbitmq_shovel_config.yml` gets + internally converted to the shovel binding key: `"__pubsub__.."`. + +.. image:: files/multiplatform_shovel_pubsub.png + +Now consider a case where shovels are setup in both the directions for forwarding "devices" topic. + +1. Agent B makes a subscribe call to receive messages with topic "devices" from all connected platforms. + +.. code-block:: python + + agent_b.vip.subscribe.call("pubsub", prefix="devices", all_platforms=True) + +2. The PubSub subsystem converts the prefix to ``__pubsub__.*.devices.#`` "*" indicates that agent is subscribing to + the "devices" topic from all the VOLTTRON platforms. + +3. A new queue is created and bound to VOLTTRON exchange with above binding key. + +4. Agent A publishes message to topic `devices/pnnl/isb1/hvac1` + +5. PubSub subsystem publishes this message on it's VOLTTRON exchange. + +6. Due to a shovel link from VOLTTRON instance "volttron1" to "volttron2", the message is forwarded from VOLTTRON + exchange "volttron1" to "volttron2" and is picked up by Agent A on "volttron2". + + +Multi-Platform RPC With Shovel +------------------------------ + +After the shovel link is established for multi-platform RPC, the below figure shows how the RPC communication happens. + +.. note:: + + It is mandatory to have shovel links on both directions as it is request-response type of communication. We will + need to set the agent identities for caller and callee in the `$VOLTTRON_HOME/rabbitmq_shovel_config.yml`. The + "blue" arrows show the resulting the shovel binding key. + +.. image:: files/multiplatform_shovel_rpc.png + +Consider Agent A on VOLTTRON instance "volttron1" on host "host_A" wants to make RPC call on Agent B +on VOLTTRON instance "volttron2" on host "host_B". + +1. Agent A makes RPC call: + +.. code-block:: Python + + kwargs = {"external_platform": self.destination_instance_name} + agent_a.vip.rpc.call("agent_b", set_point, "point_name", 2.5, \**kwargs) + +2. The message is transferred over shovel link to VOLTTRON instance "volttron2". + +3. The RPC subsystem of Agent B calls the actual RPC method and gets the result. It encapsulates the message result + into a VIP message object and sends it back to Agent A on VOLTTRON instance "volttron1". + +4. The RPC subsystem on Agent A receives the message result and gives it to Agent A's application. + + +RabbitMQ Management Tool Integrated Into VOLTTRON +================================================= + +Some of the important native RabbitMQ control and management commands are now integrated with the +:ref`volttron-ctl ` (vctl) utility. Using `volttron-ctl`'s RabbitMQ management utility, we can +control and monitor the status of RabbitMQ message bus: + +.. code-block:: console + + vctl rabbitmq --help + usage: vctl command [OPTIONS] ... rabbitmq [-h] [-c FILE] [--debug] + [-t SECS] + [--msgdebug MSGDEBUG] + [--vip-address ZMQADDR] + ... + subcommands: + + add-vhost add a new virtual host + add-user Add a new user. User will have admin privileges + i.e,configure, read and write + add-exchange add a new exchange + add-queue add a new queue + list-vhosts List virtual hosts + list-users List users + list-user-properties + List users + list-exchanges add a new user + list-exchange-properties + list exchanges with properties + list-queues list all queues + list-queue-properties + list queues with properties + list-bindings list all bindings with exchange + list-federation-parameters + list all federation parameters + list-shovel-parameters + list all shovel parameters + list-policies list all policies + remove-vhosts Remove virtual host/s + remove-users Remove virtual user/s + remove-exchanges Remove exchange/s + remove-queues Remove queue/s + remove-federation-parameters + Remove federation parameter + remove-shovel-parameters + Remove shovel parameter + remove-policies Remove policy diff --git a/docs/source/core_services/messagebus_refactor/RabbitMQ-SSL-Auth.rst b/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-ssl-auth.rst similarity index 97% rename from docs/source/core_services/messagebus_refactor/RabbitMQ-SSL-Auth.rst rename to docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-ssl-auth.rst index 3e684d06e8..125004fb1f 100644 --- a/docs/source/core_services/messagebus_refactor/RabbitMQ-SSL-Auth.rst +++ b/docs/source/volttron-topics/change-log/message-bus-refactor/rabbitmq-ssl-auth.rst @@ -11,7 +11,9 @@ RabbitMQ VOLTTRON uses SSL based authentication, rather than the default usernam adds SSL based configuration entries into the 'rabbitmq.conf' file during the setup process. The necessary SSL configurations can be seen by running the following command: -``cat ~/rabbitmq_server/rabbitmq_server-3.7.7/etc/rabbitmq/rabbitmq.conf`` +.. code-block:: bash + + cat ~/rabbitmq_server/rabbitmq_server-3.7.7/etc/rabbitmq/rabbitmq.conf The configurations required to enable SSL: @@ -119,6 +121,7 @@ key to the server and the server validates if it is signed by a root CA it trust started with. Since there is only a single root CA for one VOLTTRON instance, all the agents in this instance can communicate with the message bus over SSL. + Multi-Platform Communication With RabbitMQ SSL ============================================== @@ -136,7 +139,7 @@ know it is safe to talk to VOLTTRON2. Agents trying to connect to remote instance directly, need to have a public certificate signed by the remote instance for authenticated SSL based connection. To facilitate this process, the VOLTTRON platform exposes a web based server api for requesting, listing, approving and denying certificate requests. For more detailed description, refer to -:ref:`Agent communication to Remote RabbitMQ instance<_Connecting_to_remote_RMQ>` +:ref:`Agent communication to Remote RabbitMQ instance ` Authorization in RabbitMQ VOLTTRON diff --git a/docs/source/specifications/pubsub-enhancement.rst b/docs/source/volttron-topics/change-log/pubsub-enhancement.rst similarity index 96% rename from docs/source/specifications/pubsub-enhancement.rst rename to docs/source/volttron-topics/change-log/pubsub-enhancement.rst index 210b192e55..585fa4f894 100644 --- a/docs/source/specifications/pubsub-enhancement.rst +++ b/docs/source/volttron-topics/change-log/pubsub-enhancement.rst @@ -7,12 +7,14 @@ This document describes pubsub communication between different platforms. The go the current setup of having a forward historian to forward local pubsub messages to remote platforms. So the agents interested in receiving PubSub messages from external platforms will not need to have a forward historian running in source platform to forward pubsub messages to the interested destination platforms. The VIP router will now do all the -work; it shall use Routing Service to internally manage connections with external VOLTTRON platforms and use PubSubService -for the actual inter platform PubSub communication. +work; it shall use the Routing Service to internally manage connections with external VOLTTRON platforms and use the +PubSubService for the actual inter platform PubSub communication. + For Future: -The specification will need to be extended to support PubSub communication between platforms that are multiple hops away. -The VIP router of each platform shall need to maintain a routing table and use it to forward pubsub messages to subscribed -platforms that are multiple hops away. The routing table shall contain shortest path to each destination platform. +The specification will need to be extended to support PubSub communication between platforms that are multiple hops +away. The VIP router of each platform shall need to maintain a routing table and use it to forward pubsub messages to +subscribed platforms that are multiple hops away. The routing table shall contain shortest path to each destination +platform. Functional Capabilities diff --git a/docs/source/devguides/scalability/Scalability-Improvements.rst b/docs/source/volttron-topics/change-log/scalability/scalability-improvements.rst similarity index 100% rename from docs/source/devguides/scalability/Scalability-Improvements.rst rename to docs/source/volttron-topics/change-log/scalability/scalability-improvements.rst diff --git a/docs/source/devguides/scalability/Scalability.rst b/docs/source/volttron-topics/change-log/scalability/scalability.rst similarity index 98% rename from docs/source/devguides/scalability/Scalability.rst rename to docs/source/volttron-topics/change-log/scalability/scalability.rst index cbfeafbe01..f7c09d62ee 100644 --- a/docs/source/devguides/scalability/Scalability.rst +++ b/docs/source/volttron-topics/change-log/scalability/scalability.rst @@ -126,3 +126,8 @@ results - Go straight to db to see how far behind other historians + +.. toctree:: + + scalability-improvements + testing-driver-scalability diff --git a/docs/source/devguides/scalability/Testing-Driver-Scalability.rst b/docs/source/volttron-topics/change-log/scalability/testing-driver-scalability.rst similarity index 100% rename from docs/source/devguides/scalability/Testing-Driver-Scalability.rst rename to docs/source/volttron-topics/change-log/scalability/testing-driver-scalability.rst diff --git a/docs/source/volttron-topics/change-log/upgrading-versions.rst b/docs/source/volttron-topics/change-log/upgrading-versions.rst new file mode 100644 index 0000000000..7fbe36ded2 --- /dev/null +++ b/docs/source/volttron-topics/change-log/upgrading-versions.rst @@ -0,0 +1,127 @@ +.. _Upgrading-Versions: + +============================== +Upgrading Existing Deployments +============================== + +It is often recommended that users upgrade to the latest stable release of VOLTTRON for their deployments. Major +releases include helpful new features, bug fixes, and other improvements. Please see the guides below for upgrading +your existing deployment to the latest version. + + +VOLTTRON 7 +========== + +VOLTTRON 7 includes a migration from Python 2.7 to Python 3.6, as well as security features, new agents, and more. + +From 6.x +-------- + +From version 6.x to 7.x important changes have been made to the virtual environment as well as `VOLTTRON_HOME`. Take +the following steps to upgrade: + +.. note:: + + The following instructions are for debian based Linux distributions (including Ubuntu and Linux Mint). For Red Hat, + Arch or other distributions, please use the corresponding package manager and commands. + +#. Install the VOLTTRON dependencies using the following command: + + .. code-block:: bash + + sudo apt install python3-dev python3-venv libffi-dev + + .. note:: + + This assumes you have existing 6.x dependencies installed. If you're unsure, refer to the + :ref:`platform installation ` instructions. + +#. Remove your existing virtual environment and run the bootstrap process. + + To remove the virtual environment, change directory to the VOLTTRON project root and run the `rm` command with the + ``-r`` option. + + .. code-block:: bash + + cd $VOLTTRON_ROOT/ + rm -r env + + Now you can use the included `bootstrap.py` script to set up the new virtual environment. For information on how + to install dependencies for VOLTTRON integrations, run the script with the ``--help`` option. + + .. code-block:: bash + + python3 bootstrap.py + + .. note:: + + Because the new environment uses a different version of Python, using the ``--force`` option with bootstrap will + throw errors. Please follow the above instructions when upgrading. + +#. Make necessary VOLTTRON_HOME changes + + + .. warning:: + + It is possible that some existing agents may continue to operate after the platform upgrade, however this is not + true for most agents, and it is recommended to reinstall the agent to ensure the agent wheel is compatible and + that there are no side-effects. + + A. Reinstall Agents + + It is recommended to reinstall all agents that exist on the platform to ensure the agent wheel is compatible with + Python3 VOLTTRON. In many cases, the configurations for version 7.x are backwards compatible with 6.x, requiring no + additional changes from the user. For information on individual agent configs, please read through that agent's + documentation. + + B. Modify Agent Directories + + .. note:: + + Modifying the agent directories is only necessary if not reinstalling agents. + + To satisfy the security requirements of the secure agents feature included with VOLTTRON 7, changes have been made + to the agent directory structure. + + 1. Keystore.json + + The agent keystore file has been moved from the agent's `agent-data` directory to the agent's `dist-info` + directory. To move the file, change directory to the agents install directory and use the `mv` command. + + .. code-block:: bash + + cd $VOLTTRON_HOME/agents// + mv agent.agent-data/keystore.json agent.dist-info/ + + 2. Historian Database + + Historians with a local database file have had their default location change do the `data` directory inside of + the agent's install directory. It is recommended to relocate the file from $VOLTTRON_HOME/data to the agent's + data directory. Alternatively, a path can be used if the user the agent is run as (the VOLTTRON user for + deployments not using the secure agents feature) has read-write permissions for the file. + + .. code-block:: bash + + mv $VOLTTRON_HOME/data/historian.sqlite $VOLTTRON_HOME/agents///data + + .. warning:: + + If not specifying a path to the database, the database will be created in the agent's data directory. This + is important if removing or uninstalling the historian as the database file will be removed when the agent + dir is cleaned up. Copy the database file to a temporary directory, reinstall the agent, and move the + database file back to the agent's data directory + +#. Forward Historian + + For deployments which are passing data from 6.x VOLTTRON to the latest 7.x release, some users will experience + timeout issues with the Forward Historian. By updating the 6.x deployment to the latest from the releases/6.x + branch, and restarting the platform and forwarder, this issue can be resolved. + + .. code-block:: bash + + . env/bin/activate + ./stop-volttron + git pull + git checkout releases/6.x + ./start-volttron + vctl start diff --git a/docs/source/overview/version-history.rst b/docs/source/volttron-topics/change-log/version-history.rst similarity index 93% rename from docs/source/overview/version-history.rst rename to docs/source/volttron-topics/change-log/version-history.rst index db0ce7f199..448c4d945a 100644 --- a/docs/source/overview/version-history.rst +++ b/docs/source/volttron-topics/change-log/version-history.rst @@ -5,6 +5,7 @@ Version History =============== VOLTTRON 1.0 – 1.2 +================== - Agent execution platform - Message bus @@ -17,6 +18,7 @@ VOLTTRON 1.0 – 1.2 - Weather service VOLTTRON 2.0 +============ - Advanced Security Features - Guaranteed resource allocation to agents using execution contracts @@ -27,6 +29,7 @@ VOLTTRON 2.0 - Enhanced command framework VOLTTRON 3.0 +============ - Modularize Data Historian - Modularize Device Drivers @@ -34,6 +37,7 @@ VOLTTRON 3.0 - Web Console for Monitoring and Administering VOLTTRON Deployments VOLTTRON 4.0 +============ - Documentation moved to ReadTheDocs - VOLTTRON Configuration Wizard @@ -45,6 +49,7 @@ VOLTTRON 4.0 VOLTTRON 5.0 +============ - Tagging service for attaching metadata to topics for simpler retrieval - Message bus performance improvement @@ -52,6 +57,7 @@ VOLTTRON 5.0 - Drivers contributed back for SEP 2.0 and ChargePoint EV VOLTTRON 6.0 +============ - Maintained backward compatibility with communication between zmq and rmq deployments. - Added DarkSky Weather Agent @@ -67,8 +73,10 @@ VOLTTRON 6.0 - Integrated RabbitMQ documentation into the core documentation. VOLTTRON 7.0rc1 +=============== Python3 Upgrade +--------------- - Update libraries to appropriate and compatible versions - String handling efficiency @@ -79,6 +87,7 @@ Python3 Upgrade - Resolved gevent monkey patch issues when using third party libraries RabbitMQ Message Bus +-------------------- - Client code for integrating non-VOLTTRON applications with the message bus available at: https://github.com/VOLTTRON/external-clients-for-rabbitmq @@ -86,10 +95,12 @@ RabbitMQ Message Bus clients Config store secured +-------------------- - Agents can prevent other agents from modifying their configuration store entry Known Issues which will be dealt with for the final release: +------------------------------------------------------------ - Python 3.7 has conflicts with some libraries such as gevent - The VOLTTRON Central agent is not fully integrated into Python3 diff --git a/docs/source/volttron-topics/troubleshooting/index.rst b/docs/source/volttron-topics/troubleshooting/index.rst new file mode 100644 index 0000000000..ecbc727fa0 --- /dev/null +++ b/docs/source/volttron-topics/troubleshooting/index.rst @@ -0,0 +1,13 @@ +.. _Troubleshooting: + +=============== +Troubleshooting +=============== + +This section contains individual documents intended to help the user troubleshoot various platform components. For +troubleshooting of individual agents and drivers please refer to the corresponding document for each. + + +.. toctree:: + + troubleshooting-rmq diff --git a/docs/source/volttron-topics/troubleshooting/troubleshooting-rmq.rst b/docs/source/volttron-topics/troubleshooting/troubleshooting-rmq.rst new file mode 100644 index 0000000000..a471678a93 --- /dev/null +++ b/docs/source/volttron-topics/troubleshooting/troubleshooting-rmq.rst @@ -0,0 +1,93 @@ +.. _Troubleshooting-RMQ: + +======================== +RabbitMQ Troubleshooting +======================== + + +Check the status of the federation connection +--------------------------------------------- + +.. code-block:: bash + + $RABBITMQ_HOME/sbin/rabbitmqctl eval 'rabbit_federation_status:status().' + +If everything is properly configured, then the status is set to `running`. If not look for the error status. Some of +the typical errors are: + +a. **failed_to_connect_using_provided_uris** - Check if RabbitMQ user is created in downstream server node. Refer to + step 3-b of federation setup + +b. **unknown ca** - Check if the root CAs are copied to all the nodes correctly. Refer to step 2 of federation setup + +c. **no_suitable_auth_mechanism** - Check if the AMPQ/S ports are correctly configured. + + +Check the status of the shovel connection +----------------------------------------- + +.. code-block:: bash + + RABBITMQ_HOME/sbin/rabbitmqctl eval 'rabbit_shovel_status:status().' + +If everything is properly configured, then the status is set to `running`. If not look for the error status. Some of +the typical errors are: + +a. **failed_to_connect_using_provided_uris** - Check if RabbitMQ user is created in subscriber node. Refer to step 3-b + of shovel setup + +b. **unknown ca** - Check if the root CAs are copied to remote servers correctly. Refer to step 2 of shovel setup + +c. **no_suitable_auth_mechanism** - Check if the AMPQ/S ports are correctly configured. + + +Check the RabbitMQ logs for any errors +--------------------------------------- + +.. code-block:: bash + + tail -f /rabbitmq.log + + +Rabbitmq startup hangs +---------------------- + +a. Check for errors in the RabbitMQ log. There is a `rabbitmq.log` file in your VOLTTRON source directory that is a + symbolic link to the RabbitMQ server logs. + +b. Check for errors in syslog (`/var/log/syslog` or `/var/log/messages`) + +c. If there are no errors in either of the logs, restart the RabbitMQ server in foreground and see if there are any + errors written on the console. Once you find the error you can kill the process by entering `Ctl+C`, fix the error + and start rabbitmq again using ``./start-rabbitmq`` from VOLTTRON source directory. + + .. code-block:: bash + + ./stop-volttron + ./stop-rabbitmq + @RABBITMQ_HOME/sbin/rabbitmq-server + + +SSL trouble shooting +-------------------- +There are few things that are essential for SSL certificates to work right. + +a. Please use a unique common-name for CA certificate for each VOLTTRON instance. This is configured under + `certificate-data` in the `rabbitmq_config.yml` or if no yml file is used while configuring a VOLTTRON single + instance (using ``vcfg --rabbitmq single``). Certificate generated for agent will automatically get agent's VIP + identity as the certificate's common-name + +b. The host name in the SSL certificate should match hostname used to access the server. For example, if the fully + qualified domain name was configured in the `certificate-data`, you should use the fully qualified domain name to + access RabbitMQ's management url. + +c. Check if your system time is correct especially if you are running virtual machines. If the system clock is not + right, it could lead to SSL certificate errors + + +DataMover troubleshooting +------------------------- + +If output from `volttron.log` is not as expected check for ``{'alert_key': 'historian_not_publishing'}`` in the callee +node's `volttron.log`. Most likely cause is the historian is not running properly or credentials between caller and +callee nodes was not set properly. diff --git a/docs/source/volttron_applications/files/1-simulation-out.jpg b/docs/source/volttron-topics/volttron-applications/files/1-simulation-out.jpg similarity index 100% rename from docs/source/volttron_applications/files/1-simulation-out.jpg rename to docs/source/volttron-topics/volttron-applications/files/1-simulation-out.jpg diff --git a/docs/source/devguides/supporting/applications/files/1_Example_Passive_AFDD_Agent_Configuration_file.jpg b/docs/source/volttron-topics/volttron-applications/files/1_Example_Passive_AFDD_Agent_Configuration_file.jpg similarity index 100% rename from docs/source/devguides/supporting/applications/files/1_Example_Passive_AFDD_Agent_Configuration_file.jpg rename to docs/source/volttron-topics/volttron-applications/files/1_Example_Passive_AFDD_Agent_Configuration_file.jpg diff --git a/docs/source/volttron_applications/files/2-simulation-out.png b/docs/source/volttron-topics/volttron-applications/files/2-simulation-out.png similarity index 100% rename from docs/source/volttron_applications/files/2-simulation-out.png rename to docs/source/volttron-topics/volttron-applications/files/2-simulation-out.png diff --git a/docs/source/devguides/supporting/applications/files/2_File_Selection_Dialog_Box.jpg b/docs/source/volttron-topics/volttron-applications/files/2_File_Selection_Dialog_Box.jpg similarity index 100% rename from docs/source/devguides/supporting/applications/files/2_File_Selection_Dialog_Box.jpg rename to docs/source/volttron-topics/volttron-applications/files/2_File_Selection_Dialog_Box.jpg diff --git a/docs/source/volttron_applications/files/3-simulation-out.png b/docs/source/volttron-topics/volttron-applications/files/3-simulation-out.png similarity index 100% rename from docs/source/volttron_applications/files/3-simulation-out.png rename to docs/source/volttron-topics/volttron-applications/files/3-simulation-out.png diff --git a/docs/source/devguides/supporting/applications/files/3_Sample_of_CSV_Data.jpg b/docs/source/volttron-topics/volttron-applications/files/3_Sample_of_CSV_Data.jpg similarity index 100% rename from docs/source/devguides/supporting/applications/files/3_Sample_of_CSV_Data.jpg rename to docs/source/volttron-topics/volttron-applications/files/3_Sample_of_CSV_Data.jpg diff --git a/docs/source/devguides/supporting/applications/files/4-1_Example_DR_Agent_Configuration_File.jpg b/docs/source/volttron-topics/volttron-applications/files/4-1_Example_DR_Agent_Configuration_File.jpg similarity index 100% rename from docs/source/devguides/supporting/applications/files/4-1_Example_DR_Agent_Configuration_File.jpg rename to docs/source/volttron-topics/volttron-applications/files/4-1_Example_DR_Agent_Configuration_File.jpg diff --git a/docs/source/devguides/supporting/applications/files/4-2_Example_DR_Agent_Configuration_File.jpg b/docs/source/volttron-topics/volttron-applications/files/4-2_Example_DR_Agent_Configuration_File.jpg similarity index 100% rename from docs/source/devguides/supporting/applications/files/4-2_Example_DR_Agent_Configuration_File.jpg rename to docs/source/volttron-topics/volttron-applications/files/4-2_Example_DR_Agent_Configuration_File.jpg diff --git a/docs/source/volttron_applications/files/4-simulation-out.png b/docs/source/volttron-topics/volttron-applications/files/4-simulation-out.png similarity index 100% rename from docs/source/volttron_applications/files/4-simulation-out.png rename to docs/source/volttron-topics/volttron-applications/files/4-simulation-out.png diff --git a/docs/source/volttron-topics/volttron-applications/index.rst b/docs/source/volttron-topics/volttron-applications/index.rst new file mode 100755 index 0000000000..41cc71b982 --- /dev/null +++ b/docs/source/volttron-topics/volttron-applications/index.rst @@ -0,0 +1,20 @@ +.. _Volttron-Applications: + +============ +Applications +============ + +These resources summarize the use of the sample applications that have been created by VOLTTRON users. For detailed +information on these applications, refer to the report +`Transactional Network Platform `_. + +Note, as of VOLTTRON 4.0, applications are now in their own repository at: +https://github.com/VOLTTRON/volttron-applications + +.. toctree:: + :maxdepth: 2 + + sample-applications + simulated-drivers + openadr-vtn/index + matlab/driven-matlab-agent-walk-through diff --git a/docs/source/devguides/walkthroughs/DrivenMatlabAgent-Walkthrough.rst b/docs/source/volttron-topics/volttron-applications/matlab/driven-matlab-agent-walk-through.rst similarity index 100% rename from docs/source/devguides/walkthroughs/DrivenMatlabAgent-Walkthrough.rst rename to docs/source/volttron-topics/volttron-applications/matlab/driven-matlab-agent-walk-through.rst diff --git a/docs/source/devguides/walkthroughs/files/matlab-archi.png b/docs/source/volttron-topics/volttron-applications/matlab/files/matlab-archi.png similarity index 100% rename from docs/source/devguides/walkthroughs/files/matlab-archi.png rename to docs/source/volttron-topics/volttron-applications/matlab/files/matlab-archi.png diff --git a/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/1-simulation-out.jpg b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/1-simulation-out.jpg new file mode 100644 index 0000000000..784186670a Binary files /dev/null and b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/1-simulation-out.jpg differ diff --git a/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/2-simulation-out.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/2-simulation-out.png new file mode 100644 index 0000000000..d36dbeb827 Binary files /dev/null and b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/2-simulation-out.png differ diff --git a/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/3-simulation-out.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/3-simulation-out.png new file mode 100644 index 0000000000..abc59f1d1d Binary files /dev/null and b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/3-simulation-out.png differ diff --git a/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/4-simulation-out.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/4-simulation-out.png new file mode 100644 index 0000000000..06a6c83749 Binary files /dev/null and b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/4-simulation-out.png differ diff --git a/docs/source/core_services/openadr/files/vtn_add_customer_screen.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_add_customer_screen.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_add_customer_screen.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_add_customer_screen.png diff --git a/docs/source/core_services/openadr/files/vtn_create_event.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_create_event.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_create_event.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_create_event.png diff --git a/docs/source/core_services/openadr/files/vtn_create_new_site.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_create_new_site.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_create_new_site.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_create_new_site.png diff --git a/docs/source/core_services/openadr/files/vtn_create_program.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_create_program.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_create_program.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_create_program.png diff --git a/docs/source/core_services/openadr/files/vtn_event_overview.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_event_overview.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_event_overview.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_event_overview.png diff --git a/docs/source/core_services/openadr/files/vtn_export_report_data.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_export_report_data.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_export_report_data.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_export_report_data.png diff --git a/docs/source/core_services/openadr/files/vtn_login_screen.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_login_screen.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_login_screen.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_login_screen.png diff --git a/docs/source/core_services/openadr/files/vtn_offline_site.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_offline_site.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_offline_site.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_offline_site.png diff --git a/docs/source/core_services/openadr/files/vtn_overview_screen.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_overview_screen.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_overview_screen.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_overview_screen.png diff --git a/docs/source/core_services/openadr/files/vtn_overview_screen_with_customers.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_overview_screen_with_customers.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_overview_screen_with_customers.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_overview_screen_with_customers.png diff --git a/docs/source/core_services/openadr/files/vtn_site_detail_screen.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_site_detail_screen.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_site_detail_screen.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_site_detail_screen.png diff --git a/docs/source/core_services/openadr/files/vtn_site_with_ven_id.png b/docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_site_with_ven_id.png similarity index 100% rename from docs/source/core_services/openadr/files/vtn_site_with_ven_id.png rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/files/vtn_site_with_ven_id.png diff --git a/docs/source/core_services/openadr/index.rst b/docs/source/volttron-topics/volttron-applications/openadr-vtn/index.rst similarity index 95% rename from docs/source/core_services/openadr/index.rst rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/index.rst index 2036bc63d8..d5b52f58b7 100644 --- a/docs/source/core_services/openadr/index.rst +++ b/docs/source/volttron-topics/volttron-applications/openadr-vtn/index.rst @@ -32,7 +32,8 @@ pre-committed, an “optOut” may incur penalties.) .. toctree:: - :glob: - :maxdepth: 2 + :maxdepth: 2 - * + openadr-reference-app + vtn-server-guide + vtn-server-config diff --git a/docs/source/volttron_applications/Reference-App.rst b/docs/source/volttron-topics/volttron-applications/openadr-vtn/openadr-reference-app.rst similarity index 99% rename from docs/source/volttron_applications/Reference-App.rst rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/openadr-reference-app.rst index 5b56d17692..84fa3ead45 100644 --- a/docs/source/volttron_applications/Reference-App.rst +++ b/docs/source/volttron-topics/volttron-applications/openadr-vtn/openadr-reference-app.rst @@ -1,7 +1,8 @@ -.. _Reference-App: +.. _OpenADR-Reference-App: +===================== Reference Application -~~~~~~~~~~~~~~~~~~~~~ +===================== This reference application for VOLTTRON's OpenADR Virtual End Node (VEN) and its Simulation Subsystem demonstrates interactions between the VOLTTRON VEN agent and simulated devices. @@ -222,7 +223,7 @@ Point last_timestamp datetime VEN Configuration ================= -The VEN may be configured according to its documentation :ref:`here `. +The VEN may be configured according to its documentation :ref:`here `. Running the Simulation ====================== @@ -230,6 +231,7 @@ Running the Simulation There are three main ways to monitor the ReferenceApp simulation's progress. One way is to look at debug trace in VOLTTRON’s log output, for example: + :: 2018-01-08 17:41:30,333 (referenceappagent-1.0 23842) referenceapp.agent DEBUG: 2018-01-08 17:41:30.333260 Initializing drivers diff --git a/docs/source/core_services/openadr/VtnServerConfig.rst b/docs/source/volttron-topics/volttron-applications/openadr-vtn/vtn-server-config.rst similarity index 98% rename from docs/source/core_services/openadr/VtnServerConfig.rst rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/vtn-server-config.rst index 1992efdaec..3808be8d6d 100644 --- a/docs/source/core_services/openadr/VtnServerConfig.rst +++ b/docs/source/volttron-topics/volttron-applications/openadr-vtn/vtn-server-config.rst @@ -1,5 +1,6 @@ -.. _VtnServerConfig: +.. _VTN-Server-Config: +================================================== OpenADR VTN Server: Installation and Configuration ================================================== diff --git a/docs/source/core_services/openadr/VtnServerGuide.rst b/docs/source/volttron-topics/volttron-applications/openadr-vtn/vtn-server-guide.rst similarity index 96% rename from docs/source/core_services/openadr/VtnServerGuide.rst rename to docs/source/volttron-topics/volttron-applications/openadr-vtn/vtn-server-guide.rst index f99c059034..4f2b80c69a 100644 --- a/docs/source/core_services/openadr/VtnServerGuide.rst +++ b/docs/source/volttron-topics/volttron-applications/openadr-vtn/vtn-server-guide.rst @@ -1,18 +1,18 @@ -.. _VtnServerGuide: +.. _VTN-Server-Guide: OpenADR VTN Server: User Guide ============================== .. Warning:: This VTN server implementation is incomplete, and is not supported by the VOLTTRON core team. For information about its status including known issues, refer to the :ref:`VTN Server Configuration docs - `. + `. This guide assumes that you have a valid user account to access and log in to the VTN application website. Login Screen ------------ -In order to begin using the VTN application, navigate to \http://yourhostname**ip:8000/vtn. +In order to begin using the VTN application, navigate to ``\http://yourhostname**ip:8000/vtn``. .. image:: files/vtn_login_screen.png diff --git a/docs/source/devguides/supporting/applications/sample-applications.rst b/docs/source/volttron-topics/volttron-applications/sample-applications.rst similarity index 100% rename from docs/source/devguides/supporting/applications/sample-applications.rst rename to docs/source/volttron-topics/volttron-applications/sample-applications.rst diff --git a/docs/source/volttron_applications/Simulated-Drivers.rst b/docs/source/volttron-topics/volttron-applications/simulated-drivers.rst similarity index 99% rename from docs/source/volttron_applications/Simulated-Drivers.rst rename to docs/source/volttron-topics/volttron-applications/simulated-drivers.rst index f87e933af0..bb57eed291 100644 --- a/docs/source/volttron_applications/Simulated-Drivers.rst +++ b/docs/source/volttron-topics/volttron-applications/simulated-drivers.rst @@ -23,7 +23,7 @@ Three agents work together to run a simulation: and it has been asked to start a simulation, it provides the current simulated time in response to requests. If no stop time has been provided, the SimulationClockAgent continues to manage the simulation clock until the agent is stopped. If no clock-speed - multiplier has been provided, the simulation clock runs at normal wallclock speed. + multiplier has been provided, the simulation clock runs at normal wall-clock speed. 2. **SimulationDriverAgent.** Like MasterDriverAgent, this agent is a front-end manager for device drivers. It handles get_point/set_point requests from other agents, and it periodically "scrapes" and publishes each driver's points. If a device driver has been diff --git a/docs/source/volttron.rst b/docs/source/volttron.rst deleted file mode 100644 index 9908c17f7f..0000000000 --- a/docs/source/volttron.rst +++ /dev/null @@ -1,8 +0,0 @@ -Volttron -======== - -.. toctree:: - :maxdepth: 4 - - volttron_api/modules - diff --git a/docs/source/volttron_applications/index.rst b/docs/source/volttron_applications/index.rst deleted file mode 100644 index 3179b92e08..0000000000 --- a/docs/source/volttron_applications/index.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. _applications: - -============ -Applications -============ - -Community-contributed applications, agents and drivers that are not directly -integrated into the VOLTTRON core platform reside in a separate github -repository, https://github.com/VOLTTRON/volttron-applications. This section -provides user guides and other documents for those contributions. - -.. toctree:: - :glob: - :maxdepth: 1 - - Simulated-Drivers - ven_agent - Reference-App diff --git a/docs/source/volttron_applications/ven_agent.rst b/docs/source/volttron_applications/ven_agent.rst deleted file mode 100644 index 14f8371681..0000000000 --- a/docs/source/volttron_applications/ven_agent.rst +++ /dev/null @@ -1,232 +0,0 @@ -.. _ven_agent: - -VEN Agent: OpenADR 2.0b Interface Specification -=============================================== - -OpenADR (Automated Demand Response) is a standard for alerting and responding -to the need to adjust electric power consumption in response to fluctuations in -grid demand. - -OpenADR communications are conducted between Virtual Top Nodes (VTNs) and -Virtual End Nodes (VENs). In this implementation a VOLTTRON agent, VEN agent, -acts as a VEN, communicating with its VTN by means of EIEvent and EIReport services -in conformance with a subset of the OpenADR 2.0b specification. This document's -"VOLTTRON Interface" section defines how the VEN agent relays information to, -and receives data from, other VOLTTRON agents. - -The OpenADR 2.0b specification (http://www.openadr.org/specification) is available -from the OpenADR Alliance. This implementation also generally follows the DR program -characteristics of the Capacity Program described in Section 9.2 of the OpenADR Program Guide -(http://www.openadr.org/assets/openadr_drprogramguide_v1.0.pdf). - -DR Capacity Bidding and Events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The OpenADR Capacity Bidding program relies on a pre-committed agreement about the -VEN’s load shed capacity. This agreement is reached in a bidding process -transacted outside of the OpenADR interaction, typically with a long-term scope, -perhaps a month or longer. The VTN can “call an event,” indicating that a load-shed event should occur -in conformance with this agreement. The VTN indicates the level of load shedding -desired, when the event should occur, and for how long. The VEN responds with an -"optIn” acknowledgment. (It can also “optOut,” but since it has been pre-committed, -an “optOut” may incur penalties.) - -Reporting -~~~~~~~~~ - -The VEN agent reports device status and usage telemetry to the VTN, relying on -information received periodically from other VOLTTRON agents. - -General Approach -~~~~~~~~~~~~~~~~ - -Events: - -- The VEN agent maintains a persistent record of DR events. -- Event updates (including creation) trigger publication of event JSON on the VOLTTRON message bus. -- Other VOLTTRON agents can also call a get_events() RPC to retrieve the current status of - particular events, or of all active events. - -Reporting: - -- The VEN agent configuration defines telemetry values (data points) that can be reported to the VTN. -- The VEN agent maintains a persistent record of telemetry values over time. -- Other VOLTTRON agents are expected to call report_telemetry() to supply the VEN agent - with a regular stream of telemetry values for reporting. -- Other VOLTTRON agents can receive notification of changes in telemetry reporting - requirements by subscribing to publication of telemetry parameters. - -VEN Agent VOLTTRON Interface -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The VEN agent implements the following VOLTTRON PubSub and RPC calls. - -PubSub: event update -:: - -.. code-block:: python - - def publish_event(self, an_event): - """ - Publish an event. - - When an event is created/updated, it is published to the VOLTTRON bus - with a topic that includes 'openadr/event_update'. - - Event JSON structure: - { - "event_id" : String, - "creation_time" : DateTime, - "start_time" : DateTime, - "end_time" : DateTime or None, - "signals" : String, # Values: json string describing one or more signals. - "status" : String, # Values: unresponded, far, near, active, - # completed, canceled. - "opt_type" : String # Values: optIn, optOut, none. - } - - If an event status is 'unresponded', the VEN agent is awaiting a decision on - whether to optIn or optOut. The downstream agent that subscribes to this PubSub - message should communicate that choice to the VEN agent by calling respond_to_event() - (see below). The VEN agent then relays the choice to the VTN. - - @param an_event: an EiEvent. - """ - -PubSub: telemetry parameters update -:: - -.. code-block:: python - - def publish_telemetry_parameters_for_report(self, report): - """ - Publish telemetry parameters. - - When the VEN agent telemetry reporting parameters have been updated (by the VTN), - they are published with a topic that includes 'openadr/telemetry_parameters'. - If a particular report has been updated, the reported parameters are for that report. - - Telemetry parameters JSON example: - { - "telemetry": { - "baseline_power_kw": { - "r_id": "baseline_power", - "frequency": "30", - "report_type": "baseline", - "reading_type": "Mean", - "method_name": "get_baseline_power" - } - "current_power_kw": { - "r_id": "actual_power", - "frequency": "30", - "report_type": "reading", - "reading_type": "Mean", - "method_name": "get_current_power" - } - "manual_override": "False", - "report_status": "active", - "online": "False", - } - } - - The above example indicates that, for reporting purposes, telemetry values - for baseline_power and actual_power should be updated -- via report_telemetry() -- at - least once every 30 seconds. - - Telemetry value definitions such as baseline_power and actual_power come from the - agent configuration. - - @param report: (EiReport) The report whose parameters should be published. - """ - -RPC calls: - -.. code-block:: python - - @RPC.export - def respond_to_event(self, event_id, opt_in_choice=None): - """ - Respond to an event, opting in or opting out. - - If an event's status=unresponded, it is awaiting this call. - When this RPC is received, the VENAgent sends an eventResponse to - the VTN, indicating whether optIn or optOut has been chosen. - If an event remains unresponded for a set period of time, - it times out and automatically optsIn to the event. - - Since this call causes a change in the event's status, it triggers - a PubSub call for the event update, as described above. - - @param event_id: (String) ID of an event. - @param opt_in_choice: (String) 'OptIn' to opt into the event, anything else is treated as 'OptOut'. - """ - -.. code-block:: python - - @RPC.export - def get_events(self, event_id=None, in_progress_only=True, started_after=None, end_time_before=None): - """ - Return a list of events as a JSON string. - - Sample request: - self.get_events(started_after=utils.get_aware_utc_now() - timedelta(hours=1), - end_time_before=utils.get_aware_utc_now()) - - Return a list of events. - - By default, return only event requests with status=active or status=unresponded. - - If an event's status=active, a DR event is currently in progress. - - @param event_id: (String) Default None. - @param in_progress_only: (Boolean) Default True. - @param started_after: (DateTime) Default None. - @param end_time_before: (DateTime) Default None. - @return: (JSON) A list of events -- see 'PubSub: event update'. - """ - -.. code-block:: python - - @RPC.export - def get_telemetry_parameters(self): - """ - Return the VEN agent's current set of telemetry parameters. - - @return: (JSON) Current telemetry parameters -- see 'PubSub: telemetry parameters update'. - """ - -.. code-block:: python - - @RPC.export - def set_telemetry_status(self, online, manual_override): - """ - Update the VEN agent's reporting status. - - Set these properties to either 'TRUE' or 'FALSE'. - - @param online: (Boolean) Whether the VEN agent's resource is online. - @param manual_override: (Boolean) Whether resource control has been overridden. - """ - -.. code-block:: python - - @RPC.export - def report_telemetry(self, telemetry): - """ - Receive an update of the VENAgent's report metrics, and store them in the agent's database. - - Examples of telemetry are: - { - 'baseline_power_kw': '15.2', - 'current_power_kw': '371.1', - 'start_time': '2017-11-21T23:41:46.051405', - 'end_time': '2017-11-21T23:42:45.951405' - } - - @param telemetry_values: (JSON) Current value of each report metric, with reporting-interval start/end. - """ - -For Further Information -~~~~~~~~~~~~~~~~~~~~~~~ - -Please contact Rob Calvert at Kisensum, rob@kisensum.com diff --git a/requirements.py b/requirements.py index 0791185e5e..637e1887a5 100644 --- a/requirements.py +++ b/requirements.py @@ -85,7 +85,11 @@ 'pymongo', 'Sphinx', 'recommonmark', - 'sphinx-rtd-theme' + 'sphinx-rtd-theme', + 'werkzeug' + 'pint', + 'jwt', + 'passlib' ], 'drivers': [ 'pymodbus',