This guide explains the usage of AR-CVI component within the SHOP4CF EU project.

AR-CVI is a component to display instructions for a human worker. The display can be either on a screen or by projecting to a surface. The communication is via ROS messages or via FIWARE Orion Context Broker (with Linked Data Extensions). Please check our short introduction video:

.

Please also check a short demonstration showing how to run AR-CVI here.

Content

• Installation of Orion-LD
• Setting Config Files
• Docker
  • Ubuntu
  • Windows WSL 2
    • Windows 10 WSL 2 Installation
    • Docker Desktop WSL 2 Backend Installation
    • VcXsrv Windows X Server Installation and Configuration
    • Running AR-CVI on Windows WSL 2 for Ubuntu
• Message Format
  • Instruction types
  • Add message example
  • Reset message examples
  • AddTemplate message example
  • ResetTemplate message example
  • Template explanation
• Docker Images
• Release Notes

Installation of Orion-LD

In order to communicate using FIWARE messages, Orion-LD should be installed. The RAMP-IoT-LD folder contains install.sh script to install requirements and to start necessary containers with necessary networks set up.

The uninstall.sh script would remove all docker related files including images previously generated. Please check and modify it before using.

sh RAMP-IoU-LD/install.sh

Setting Config Files

AR-CVI reads configuration information for FIWARE communication from a configuration file. The configuration file should be named as arcvi-fiware-config.json. The folder that contains the config file should be mounted to the /configs in the docker container.

The configuration information is given below:

{
    "projectName": "bos-uc1",
    "IP_fiware-orion-ld": "172.19.1.1",
    "checkStatus": false,
    "checkRelationship": false,
    "relationshipObject": "urn:ngsi-ld:Device:siemens:projector",
    "defaultButtons": true
}

• projectName is a use-case specific name that will be read from the context id. The id is in urn:ngsi-ld:Task:<projectName>:task<task_number> format (An example: urn:ngsi-ld:Task:bos-uc1:task0013 or urn:ngsi-ld:Task:bos-uc1:task0123xTyaw987).

• <task_number> is not specifically read by AR-CVI, but it is to avoid any conflicts due to tasks published with the same id.

• IP_fiware-orion-ld can be also specified as auto. In this case, the component will check FIWARE_ORION_ID environment variable, which is passed to the container with the arcvi_run.sh script. That environment variable is filled by the IP Address of the fiware-orion-ld container to communicate with the context broker. It is recommended to set this as auto if the install.sh is used to start Fiware containers.

• checkStatus is to check the status of the Fiware message. If true, AR-CVI displays only the Tasks with the inProgress status. Otherwise, the component displays the published tasks with the given <project_name> tag in the context id. When the status is changed to complete, the display will stop.

• checkRelationship is to check the Relationship attribute of involves in the Data Model. If checkRelationship is true and the attribute in the fiware message matches with the config's relationshipObject, the content of the message will be displayed. This is to ensure that the fiware message aims the AR-CVI component.

• relationshipObject defines the Relationship attribute's value.

• defaultButtons defines whether to show two additional buttons on each new display by default. These buttons are Show Grid and Reset. Show Grid prints the coordinates of grid points as a guidance for determining object locations. Reset button cleans the display.

Docker image

Ubuntu

arcvi_run.sh starts the container and the AR-CVI component. AR-CVI is installed in emecercelik/ar-cvi:ar-cvi_v1 image, which can be downloaded from docker hub with docker pull emecercelik/ar-cvi:ar-cvi_v1.

arcvi_run.sh starts the container ensuring the following setup:

configs=<Path to the folder where configuration files are kept in HOST>
templates=<Path to the folder where AR-CVI projection templates are kept in HOST>
FIWARE_ORION_ID=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' fiware-orion-ld)

xhost local:root
docker run -it --rm 	-v $configs:/configs \
			-v $templates:/templates\
			--name ar-cvi \
			--net=ramp-iot-ld_default \
			--privileged \
			--env="DISPLAY" \
			--env="QT_X11_NO_MITSHM=1" \
			--env="FIWARE_ORION_ID=${FIWARE_ORION_ID}" \
			--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
			emecercelik/ar-cvi:arcvi-1.3

xhost -local:root

configs host path should contain arcvi-fiware-config.json configuration file. The $configs should be mounted in /configs inside the container as indicated. $templates contains the json templates for AR-CVI to project pre-defined displays or instructions. The $templates host folder should be mounted in /templates inside the container as shown. The network bridge that the container connects to is indicated with --net flag. The RAMP-IoU-LD/install.sh script sets up ramp-iot-ld_default network. The AR-CVI can communicate using FIWARE messages through this network. FIWARE_ORION_ID environment variable is passed to the docker container to get fiware-orion-ld IP address and use it automatically while making GET requests.

--privileged, --env="DISPLAY", --env="QT_X11_NO_MITSHM=1", --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" tags and xhost commands are necessary for using screen with the docker container.

Windows WSL 2

Please refer to ref1, ref2, and ref3 for the solutions explained below.

Please follow steps below to run AR-CVI on Windows WSL 2.

• Windows 10 WSL 2 Installation
• Docker Desktop WSL 2 Backend Installation
• VcXsrv Windows X Server Installation and Configuration
• Running AR-CVI on Windows WSL 2 for Ubuntu

Windows 10 WSL 2 Installation

You can install WSL 2 by following the documentation for the default Linux distribution, which is Ubuntu. Running AR-CVI is only tested with Ubuntu running under WSL 2.

Docker Desktop WSL 2 Backend Installation

Docker Desktop WSL 2 Backend can be installed by following this documentation. The prerequisites in the document should be followed and the WSL 2 support must be checked when asked during the installation.

VcXsrv Windows X Server Installation and Configuration

The Windows X Server is required to be able to connect to the display. VcXsrv can be installed here following the images below for the configuration. Also, the installation steps are similar to the steps given here with a few vital differences.

At the last step please save the configuration file in one of the following paths:

%appdata%\Xming
%userprofile%\Desktop
%userprofile%

Running AR-CVI on Windows WSL 2 for Ubuntu

Similar to the Ubuntu version, you can create the necessary folders: configs and templates.

Please run the following script or arcvi_run_win.sh to start AR-CVI.

configs=<Path to the folder where configuration files are kept in HOST>
templates=<Path to the folder where AR-CVI projection templates are kept in HOST>
images=<Path to the folder where images given in templates are kept in the HOST machine>

FIWARE_ORION_ID=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' fiware-orion-ld)

docker run -it --rm     -v $configs:/configs \
                        -v $templates:/templates\
                        -v $images:/images \
                        --name ar-cvi \
                        --net=ramp-iot-ld_default \
                        --privileged \
                        -e DISPLAY=host.docker.internal:0.0 -e LIBGL_ALWAYS_INDIRECT= \
                        --env="FIWARE_ORION_ID=${FIWARE_ORION_ID}" \
                        emecercelik/ar-cvi:arcvi-1.3

Message Format

AR-CVI message format is built on top of SHOP4CF Task data model. All data models defined for SHOP4CF can be found here.

There are 4 types of commands for AR-CVI:

1. Add: This command adds the given instructions to the displayed instructions on the screen.
2. Reset: This command resets the screen and displays the given instructions in a Reset message on a clear screen.
3. AddTemplate: Similar to Add, this commands adds the given instructions provided with a pre-defined json template.
4. ResetTemplate: Similar to Reset, this command clears the screen and displays the instructions provided with a pre-defined json template.

An example of AddTemplate message type can be seen in ar-cvi_addtemplate_example.pdf pdf document. In this pdf, examples of AddTemplate message and the template are shown with their effects on the screen.

Instruction types

The supported instruction types are listed below:

1. Written instructions
Written instruction is displayed at the bottom of the screen. Defined with`"instruction":"Written instruction to be displayed"`.
2. Boxes
Rectangular shaped boxes defined with the properties below:

{
"position":["0.15","0.2","0.0"], 
"width":"0.01",
"height":"0.01",
"label":"Box3",
"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}
}

position defines where the center of the box will be by x, y, and z values. Here z is invalid. width and height define size of the box. label provides a displayed label for the box. fill_color and border_color are the colors inside the box and at the border of the box. These are defined with r, g, b, a values as shown.

It is possible to to define more than one box at a message. The dictionaries defining multiple boxes can be appended to a list.

3. Images

Images can be also displayed. The image extension can be .svg, .png, and .jpg. As shown below, position defines its center in x and y directions, width and height define its size, and path shows where the image is.

{
"position":["-0.45","0.25","0.0"],
"width":"750.0",
"height":"1000.0",
"path":"/ros_files/coord.png"
}

More than one image can be defined at a time. Image dictionaries should be appended to a list in that case.

4. Points

Points are circles displayed on the screen defined as seen below. position indicates center of the circle in 2D. radius defines the radius of the circle. label is the label of the circle that will be displayed as well. fill_color and border_color define colors inside the box and at the border.

{
"position":["0.3","0.2","0.0"],
"radius":"0.03",
"label":"Circle3",
"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}
}

Multiple points can be defined by appending multiple point dictionaries to a list.

5. Outcome buttons (User input)
Outcome buttons are information buttons shown at the bottom of the screen.

{
"name": "0-default", 
"type": "0", 
"id": "0"
}

name is the label shown on the button. type indicates the button type, which are default (gray color, type:0), info (blue color, type:1), success (green color, type:2), warn (orange color, type:3), and error (red color, type:4) with indicated colors. id is the id given to that button.

When the button is pressed, a fiware message containing name, type, and id will be sent to the context broker. An example is given below:

curl -iX POST 'http://<fiware_ip>:1026/ngsi-ld/v1/entities' 
    -H 'Content-Type: application/ld+json' 
    -d '{"id": "urn:ngsi-ld:Task:arcvi_ui<task_id>,
         "type": "Task",
         "workParameters": {
             "type":"Property",
             "value": {"uiName":<button.name>,
                       "uiType":<button.type>, 
                       "uiId":<button.id>
                      }
                      },
         "outputParameters": {"type": "Property",
                              "value": {"materialsAlreadyTransported": 0,"percentageCompleted": 0},
                              "observedAt": "2020-12-01T11:23:19Z"},
         "@context": []
         }'

Subscribers to this Task model can read the message and use the message as a trigger checking the uiName attribute, which is defined while preparing the display Task messages above.

Add message example

curl -iX POST \
  'http://localhost:1026/ngsi-ld/v1/entities' \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:Task:bos-uc1:task0017",
    "type": "Task",
    "workParameters": {
        "type": "Property",
        "value": {"type":"Add",
                   "displayStatus":"inProgress",
                   "instruction":"Written Instruction",
                            "boxes":[{
                            	"position":["0.35","0.2","0.0"], 
                            	"width":"0.01",
                            	"height":"0.01",
                            	"label":"Box1",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}},
                            
                            	{
                            	"position":["0.15","0.4","0.0"],
                            	"width":"0.02",
                            	"height":"0.05",
                            	"label":"Box2",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}}
                            	],
                            	
                            "images":[
                            	{
                            	"position":["-0.65","0.25","0.0"],
                            	"width":"750.0",
                            	"height":"1000.0",
                            	"path":"/ros_files/coord.png"},
                            	
                            	{
                            	"position":["-0.1","0.23","0.0"],
                            	"width":"100.0",
                            	"height":"100.0",
                            	"path":"/catkin_ws/src/tum_ros/tum_ar/tum_ar_window/images/shop4cf.svg"}
                            	],
                            	
                            "pois":[
                            	{
                            	"position":["0.6","0.5","0.0"],
                            	"radius":"0.02",
                            	"label":"Circle1",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}},
                            	
                            	{
                            	"position":["0.4","-0.1","0.0"],
                            	"radius":"0.02",
                            	"label":"circle2",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}}
                            	],
                            	
                            "outcomes":[
                            	{"name": "0-default", 
                            	"type": "0", 
                            	"id": "0"}, 
                            	{"name": "1-info",
                            	 "type": "1",
                            	  "id": "1"}
                            	]
             }
    },
    "outputParameters": {
        "type": "Property",
        "value": {
            "materialsAlreadyTransported": 0,
            "percentageCompleted": 0
        },
        "observedAt": "2020-12-01T11:23:19Z"
    },
    "@context": [
    ]
}'

Reset message examples

curl -iX POST \
  'http://localhost:1026/ngsi-ld/v1/entities' \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:Task:bos-uc1:task0028",
    "type": "Task",
    "workParameters": {
        "type": "Property",
        "value": {"type":"Reset",
                   "displayStatus":"inProgress",
                   "instruction":"Written Instruction 3",
                            "boxes":[
                            	{
                            	"position":["0.10","0.15","0.0"], 
                            	"width":"0.01",
                            	"height":"0.01",
                            	"label":"Box3",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}},
                            
                            	{
                            	"position":["0.15","0.2","0.0"],
                            	"width":"0.02",
                            	"height":"0.05",
                            	"label":"Box4",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}}
                            	],
                            	
                            "images":[
                            	{
                            	"position":["-0.4","0.2","0.0"],
                            	"width":"750.0",
                            	"height":"1000.0",
                            	"path":"/ros_files/coord.png"},
                            	
                            	{
                            	"position":["0.12","0.2","0.0"],
                            	"width":"100.0",
                            	"height":"100.0",
                            	"path":"/catkin_ws/src/tum_ros/tum_ar/tum_ar_window/images/shop4cf.svg"}
                            	],
                            	
                            "pois":[
                            	{
                            	"position":["0.3","0.2","0.0"],
                            	"radius":"0.03",
                            	"label":"Circle3",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}},
                            	{
                            	"position":["0.5","-0.2","0.0"],
                            	"radius":"0.01",
                            	"label":"circle4",
                            	"fill_color":{"r":"0.7","g":"0.7","b":"0.7","a":"0.7"},
                            	"border_color":{"r":"1.0","g":"1.0","b":"1.0","a":"1.0"}}
                            	],
                            	
                            "outcomes":[
                            	{"name": "2-success", 
                            	"type": "2", 
                            	"id": "2"},
                            	{"name": "3-warn", 
                            	"type": "3", 
                            	"id": "3"}, 
                            	{"name": "4-error", 
                            	"type": "4", 
                            	"id": "4"}
                            	]
             }
    },
    "outputParameters": {
        "type": "Property",
        "value": {
            "materialsAlreadyTransported": 0,
            "percentageCompleted": 0
        },
        "observedAt": "2020-12-01T11:23:19Z"
    },
    "@context": [
    ]
}

curl -iX POST \
  'http://localhost:1026/ngsi-ld/v1/entities' \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:Task:bos-uc1:task0003",
    "type": "Task",
    "workParameters": {
        "type": "Property",
        "value": {"type":"Reset",
                   "displayStatus":"inProgress"
                   
             }
    },
    "outputParameters": {
        "type": "Property",
        "value": {
            "materialsAlreadyTransported": 0,
            "percentageCompleted": 0
        },
        "observedAt": "2020-12-01T11:23:19Z"
    },
    "@context": [
    ]
}'

AddTemplate message example

curl -iX POST \
  'http://localhost:1026/ngsi-ld/v1/entities' \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:Task:bos-uc1:task0012",
    "type": "Task",
    "workParameters": {
        "type": "Property",
        "value": {"type":"AddTemplate",
                   "path":"/templates/temp3.json",
                   "displayStatus":"inProgress"
             }
    },
    "outputParameters": {
        "type": "Property",
        "value": {
            "materialsAlreadyTransported": 0,
            "percentageCompleted": 0
        },
        "observedAt": "2020-12-01T11:23:19Z"
    },
    "@context": [
    ]
}'

ResetTemplate message example

curl -iX POST \
  'http://localhost:1026/ngsi-ld/v1/entities' \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:Task:bos-uc1:task0011",
    "type": "Task",
    "workParameters": {
        "type": "Property",
        "value": {"type":"ResetTemplate",
                   "path":"/templates/temp2.json",
                   "displayStatus":"inProgress"
             }
    },
    "outputParameters": {
        "type": "Property",
        "value": {
            "materialsAlreadyTransported": 0,
            "percentageCompleted": 0
        },
        "observedAt": "2020-12-01T11:23:19Z"
    },
    "@context": [
        "https://smartdatamodels.org/context.jsonld",
        "https://raw.githubusercontent.com/shop4cf/data-models/master/docs/shop4cfcontext.jsonld"
    ]
}'

Template explanation

Template is a json file that contains display instructions in a formatted way. An example can be seen below:

{
    "instruction": "Template Instruction 1",
    "boxes": [
        {
            "position": [
                "0.15",
                "0.2",
                "0.0"
            ],
            "width": "0.01",
            "height": "0.01",
            "label": "TempBox1",
            "fill_color": {
                "r": "0.7",
                "g": "0.7",
                "b": "0.7",
                "a": "0.7"
            },
            "border_color": {
                "r": "1.0",
                "g": "1.0",
                "b": "1.0",
                "a": "1.0"
            }
        },
        {
            "position": [
                "0.15",
                "0.2",
                "0.0"
            ],
            "width": "0.02",
            "height": "0.05",
            "label": "TempBox2",
            "fill_color": {
                "r": "0.7",
                "g": "0.7",
                "b": "0.7",
                "a": "0.7"
            },
            "border_color": {
                "r": "1.0",
                "g": "1.0",
                "b": "1.0",
                "a": "1.0"
            }
        }
    ],
    "images": [
        {
            "position": [
                "-0.45",
                "0.25",
                "0.0"
            ],
            "width": "750.0",
            "height": "1000.0",
            "path": "/ros_files/coord.png"
        },
        {
            "position": [
                "0.1",
                "0.23",
                "0.0"
            ],
            "width": "100.0",
            "height": "100.0",
            "path": "/catkin_ws/src/tum_ros/tum_ar/tum_ar_window/images/shop4cf.svg"
        }
    ],
    "pois": [
        {
            "position": [
                "0.3",
                "0.2",
                "0.0"
            ],
            "radius": "0.03",
            "label": "Circle3",
            "fill_color": {
                "r": "0.7",
                "g": "0.7",
                "b": "0.7",
                "a": "0.7"
            },
            "border_color": {
                "r": "1.0",
                "g": "1.0",
                "b": "1.0",
                "a": "1.0"
            }
        },
        {
            "position": [
                "0.5",
                "-0.2",
                "0.0"
            ],
            "radius": "0.01",
            "label": "circle4",
            "fill_color": {
                "r": "0.7",
                "g": "0.7",
                "b": "0.7",
                "a": "0.7"
            },
            "border_color": {
                "r": "1.0",
                "g": "1.0",
                "b": "1.0",
                "a": "1.0"
            }
        }
    ],
    "outcomes": [
        {
            "name": "2-success",
            "type": "2",
            "id": "2"
        },
        {
            "name": "3-warn",
            "type": "3",
            "id": "3"
        },
        {
            "name": "4-error",
            "type": "4",
            "id": "4"
        }
    ]
}

Docker Images

Please check Docker Hub for AR-CVI Docker Images.

Release Notes

• [27.04.2022] arcvi-1.3 docker pull emecercelik/ar-cvi:arcvi-1.3
  • Objects are drawn before images that images are behind all other objects except the background.
  • Welcome page is generated.
  • Grid template is generated as a guidance for coordinates of the display.
  • The option of adding Show Grid and Reset buttons to each slide is added.
• [20.04.2022] arcvi-1.2 docker pull emecercelik/ar-cvi:arcvi-1.2
  • Change user input (press button) Fiware urn IDs. 8 byte hexadecimal token is used (Ex: a23d8aad20332582) instead of incremental integers (Ex: 000001, 0000002, ...).
  • Update observation time for the Task data model.
  • Change POST method to Python request.
  • Check number of UIs in the context broker and prune if above threshold.
• [19.04.2022] arcvi-1.1 docker pull emecercelik/ar-cvi:arcvi-1.1
  • Fix overloaded context returns. Filtering context information is updated.
  • displayStatus flag is added to the messages. After a Task is displayed its displayStatus is updated to completed. If this flag is updated to inProgress, the updated Task message is re-displayed.
• [Initial] ar-cvi_v1 docker pull emecercelik/ar-cvi:ar-cvi_v1