diff --git a/README.md b/README.md index 947ca11..5dc7381 100644 --- a/README.md +++ b/README.md @@ -1,76 +1,150 @@ -### STREET-SIGNS DETECTION in VIDEO STREAM OVER SPARK-HADOOP CLUSTER +### STREET-SIGNS DETECTION in VIDEO over SPARK CLUSTER #### STEP 1: Clone or download cloudmesh.street repository to local machine git clone https://github.com/cloudmesh/cloudmesh.street.git -#### STEP 2: After getting local copy of the git repository, Go to directory './code/scripts' +#### STEP 2.1: UPDATE THE USER PREFERENCES +After getting local copy of the git repository, Go to directory "./ansible" + UPDATE THE FOLLOWING VARIABLES IN "user_vars.yaml" + + --- + ############################################################ + #Variables for execution of complete package + + #EDIT FOLLOWING DETAILS AS PER REQUIREMENT + + ##############CLOUDMESH SETTINGS############################ + #cloud: "chameleon" or "jetstream" + cloud: + #username: "cloudmesh username as key_name" + username: + ############HADOOP CLUSTER SETTINGS######################### + #Chameleon image_name: CC-Ubuntu14.04 + #jetstream image_name: ubuntu-14.04-trusty-server-cloudimg + #flavor: m1.small, m1.medium, m1.large[preferred: m1.medium] + #addons: spark pig hive + + image_name: + count: + flavor: + addons: + + ########################################################## + +#### For e.g.: + + --- + ############################################################ + #Variables for execution of complete package + + #EDIT FOLLOWING DETAILS AS PER REQUIREMENT + + ##############CLOUDMESH SETTINGS############################ + #cloud: "chameleon" or "jetstream" + cloud: "chameleon" + #username: "cloudmesh username as key_name" + username: "rraghata" + ############HADOOP CLUSTER SETTINGS######################### + #Chameleon image_name: CC-Ubuntu14.04 + #jetstream image_name: ubuntu-14.04-trusty-server-cloudimg + #flavor: m1.small, m1.medium, m1.large [preferred: m1.medium] + #addons: spark pig hive -#### STEP 3: To install ansible, cloudmesh client for the first-time: - (Note: Skip 3.1 if already installed ) -##### 3.1 Run the script local_setup.sh from ./code/scripts folder: + + image_name: "CC-Ubuntu14.04" + count: "6" + flavor: "m1.medium" + addons: "spark" + + ############################################################ + +#### STEP 2.2: CHANGE WORKING DIRECTORY +Go to directory '/cloudmesh.street/code/scripts' + +#### STEP 3: To install ansible, cloudmesh client for the first-time and run complete package run following script: + + . run_all.sh + +#### STEP 3: ANSIBLE PLAYBOOK EXECUTION BREAKDOWN +{Note: Skip 3.1 if already installed ) + +##### 3.1 Run the script local_setup.sh for ansible and cloudmesh setup on local machine: . setup.sh The above script when runs uses playbook--> ansible/local_setup.yaml +Edit the cloudmesh.yaml file as per requirements [DETAILS GIVEN IN APPENDIX] -##### 3.2 Run the script configure.sh from ./code/scripts folder: +##### 3.2 Run the script configure.sh for cloudmesh and cloud configuration: . configure.sh -The above script when runs uses playbook--> ansible/cloud_config.yaml and ask for user input for following fields: - (Note: Values in [] are default values just press ENTER if requires default configuration) - - enter the cloudname[chameleon]: - enter the key_name,preferably cloud_username[TBD]: - +The above script when runs uses playbook--> ansible/cloud_config.yaml -##### 3.3 Run the script deploy.sh from ./code/scripts folder: +##### 3.3 Run the script deploy.sh to hadoop-spark cluster: . deploy.sh -The above script when runs ,uses playbook--> ansible/hadoop_deploy.yaml and ask for user input for following fields: - (Note: Values in [] are default values just press ENTER if requires default configuration) - enter image_name[CC-Ubuntu14.04]: - enter the no. of nodes[3]: - enter the flavor requried[m1.medium]: - enter the addons required[spark pig]: - -##### 3.4 Run the script opencv_setup.sh from ./code/scripts folder: +The above script when runs ,uses playbook--> ansible/hadoop_deploy.yaml + +##### 3.4 Run the script opencv_setup.sh for environment setup over cloud cluster for opencv and pyspark: . opencv_setup.sh The above script when runs ,uses playbook--> ansible/opencv_setup.yaml -##### 3.5 Run the script sifn_detection.sh from ./code/scripts folder: +##### 3.5 Run the script sign_detection.sh to perform the sign detection analysis over cloud spark cluster: . sign_detection.sh The above script when runs ,uses playbook--> ansible/sign_detection.yaml -##### 3.6 Run the script transfer.sh from ./code/scripts folder: +##DEFAULT: +As the images dataset as well as sample video is present in project directory[DETAILS IN APPENDIX], The default program performs sign detection on images. + +To perform video analysis, Update the following file for last task: + + cloudmesh.street/ansible/roles/analysis/tasks/mail.yml + +with + + su - hadoop -c "spark-submit --master yarn --deploy-mode client --executor-memory 1g --driver-memory 2g --name signdetection --conf "spark.app.id=signdetection" /opencv_workspace/code/signdetectionbyspark.py /opencv_workspace/test_data/videos/ /opencv_workspace/output/" + +##### 3.6 Run the script transfer.sh to get the output from remote vms(cloud) to local machine for visual confirmation: . transfer.sh The above script when runs ,uses playbook--> ansible/transfer_output_to_local.yaml - -##### 3.7 Run the script clean.sh from ./code/scripts folder: + +The output gets stored at "cloudmesh.street/ansible/output" + +##### 3.7 Run the script clean.sh to clean the environment: . clean.sh The above script when runs ,uses playbook--> env_clean.yaml - +It deletes all the VMS, undefines all the clusters, delete the output directory and deletes the stacks. + +##### 4. BENCHMARK +##### 4.1 UPDATE the ./ansible/user_vars.yaml with preferences +##### 4.2 GO TO Directory ./benchmark and run below script: + + . benchmark.sh + +The above script runs all the scripts[from ../code/scripts] with output in "./benchmark/benchmark_time" file for every script + ## Appendix: #### A.1 Use of Cloudmesh_client ##### A.1.1 Edit ~/.cloudmesh/cloudmesh.yaml for following sections, edit <''>/ in the file correct credentials: - profile: - firstname: - lastname: - email: - user: + profile: + firstname: + lastname: + email: + user: ##### A.1.2 Change the entry of active cloud for the one you need,For e.g. chameleon cloud as shown below @@ -79,7 +153,7 @@ The above script when runs ,uses playbook--> env_clean.yaml clouds: ... -##### A.1.3 Edit the configuration for the active cloud below it from the list(kilo/chameleon/jestream/..), the entry with < > should be customized as per your credentials. +##### A.1.3 Edit the configuration for the active cloud below it from the list(kilo/chameleon/jestream/..), the entry with should be customized as per your credentials. Chameleon Example: @@ -95,3 +169,17 @@ Chameleon Example: default: flavor: m1.medium image: CC-Ubuntu14.04 +#### A.2 DATA +##### A.2.1 Test Data has been provided in "ansible/roles/analysis/files/test_data/" +Following directories are included as sample test-data: + + ./images/ - 50 images dataset + ./videos/ - 1 video stop_video_1.mp4 (2 sec) + +#### A.3 CLASSIFIER +"STOP SIGN CLASSIFIER" has been provided in "ansible/roles/analysis/files/classifier/" +##### A.3.1 For testing,only one classifier-"STOP SIGN" has been used. +Multiple classifiers can be added to the directory if you have. + +### A.3.2 Above classifier was taken from . + diff --git a/ansible/benchmark_time b/ansible/benchmark_time deleted file mode 100644 index 6662622..0000000 --- a/ansible/benchmark_time +++ /dev/null @@ -1,7 +0,0 @@ -time_for_cloud_setup 37 -time_for_hadoop_deploy 16303 -time_for_opencv_setup 2 -time_for_sign_detection 0 -time_for_data_transfer 1 -time_for_env_cleanup 0 -time_for_cloud_setup 47 diff --git a/ansible/inventory.txt b/ansible/inventory.txt index 8b13789..f8be2ac 100644 --- a/ansible/inventory.txt +++ b/ansible/inventory.txt @@ -1 +1,11 @@ - +[localhost] +localhost ansible_connection=local +[remote] +rraghata-055 ansible_ssh_host="129.114.33.8" ansible_ssh_user="cc" +rraghata-056 ansible_ssh_host="129.114.33.98" ansible_ssh_user="cc" +rraghata-057 ansible_ssh_host="129.114.32.247" ansible_ssh_user="cc" +rraghata-058 ansible_ssh_host="129.114.33.95" ansible_ssh_user="cc" +rraghata-059 ansible_ssh_host="129.114.110.127" ansible_ssh_user="cc" +rraghata-060 ansible_ssh_host="129.114.33.229" ansible_ssh_user="cc" +[master] +rraghata-055 ansible_ssh_host="129.114.33.8" ansible_ssh_user="cc" diff --git a/ansible/output/output/1.jpg b/ansible/output/output/1.jpg new file mode 100644 index 0000000..47a8f3d Binary files /dev/null and b/ansible/output/output/1.jpg differ diff --git a/ansible/output/output/10.jpg b/ansible/output/output/10.jpg new file mode 100644 index 0000000..a5df735 Binary files /dev/null and b/ansible/output/output/10.jpg differ diff --git a/ansible/output/output/100.jpg b/ansible/output/output/100.jpg new file mode 100644 index 0000000..6b4a3f7 Binary files /dev/null and b/ansible/output/output/100.jpg differ diff --git a/ansible/output/output/101.jpg b/ansible/output/output/101.jpg new file mode 100644 index 0000000..a722941 Binary files /dev/null and b/ansible/output/output/101.jpg differ diff --git a/ansible/output/output/102.jpg b/ansible/output/output/102.jpg new file mode 100644 index 0000000..0885f88 Binary files /dev/null and b/ansible/output/output/102.jpg differ diff --git a/ansible/output/output/103.jpg b/ansible/output/output/103.jpg new file mode 100644 index 0000000..7195883 Binary files /dev/null and b/ansible/output/output/103.jpg differ diff --git a/ansible/output/output/104.jpg b/ansible/output/output/104.jpg new file mode 100644 index 0000000..8a32af1 Binary files /dev/null and b/ansible/output/output/104.jpg differ diff --git a/ansible/output/output/105.jpg b/ansible/output/output/105.jpg new file mode 100644 index 0000000..02b0247 Binary files /dev/null and b/ansible/output/output/105.jpg differ diff --git a/ansible/output/output/106.jpg b/ansible/output/output/106.jpg new file mode 100644 index 0000000..9de4a7f Binary files /dev/null and b/ansible/output/output/106.jpg differ diff --git a/ansible/output/output/107.jpg b/ansible/output/output/107.jpg new file mode 100644 index 0000000..ad4e8ad Binary files /dev/null and b/ansible/output/output/107.jpg differ diff --git a/ansible/output/output/108.jpg b/ansible/output/output/108.jpg new file mode 100644 index 0000000..47ddf5e Binary files /dev/null and b/ansible/output/output/108.jpg differ diff --git a/ansible/output/output/109.jpg b/ansible/output/output/109.jpg new file mode 100644 index 0000000..3c008ea Binary files /dev/null and b/ansible/output/output/109.jpg differ diff --git a/ansible/output/output/11.jpg b/ansible/output/output/11.jpg new file mode 100644 index 0000000..ac1586c Binary files /dev/null and b/ansible/output/output/11.jpg differ diff --git a/ansible/output/output/110.jpg b/ansible/output/output/110.jpg new file mode 100644 index 0000000..0b79ea7 Binary files /dev/null and b/ansible/output/output/110.jpg differ diff --git a/ansible/output/output/111.jpg b/ansible/output/output/111.jpg new file mode 100644 index 0000000..4c1ccd3 Binary files /dev/null and b/ansible/output/output/111.jpg differ diff --git a/ansible/output/output/112.jpg b/ansible/output/output/112.jpg new file mode 100644 index 0000000..a6084eb Binary files /dev/null and b/ansible/output/output/112.jpg differ diff --git a/ansible/output/output/113.jpg b/ansible/output/output/113.jpg new file mode 100644 index 0000000..63cde5b Binary files /dev/null and b/ansible/output/output/113.jpg differ diff --git a/ansible/output/output/114.jpg b/ansible/output/output/114.jpg new file mode 100644 index 0000000..efc384e Binary files /dev/null and b/ansible/output/output/114.jpg differ diff --git a/ansible/output/output/115.jpg b/ansible/output/output/115.jpg new file mode 100644 index 0000000..ff63b07 Binary files /dev/null and b/ansible/output/output/115.jpg differ diff --git a/ansible/output/output/116.jpg b/ansible/output/output/116.jpg new file mode 100644 index 0000000..f8c75bb Binary files /dev/null and b/ansible/output/output/116.jpg differ diff --git a/ansible/output/output/117.jpg b/ansible/output/output/117.jpg new file mode 100644 index 0000000..914bd7a Binary files /dev/null and b/ansible/output/output/117.jpg differ diff --git a/ansible/output/output/118.jpg b/ansible/output/output/118.jpg new file mode 100644 index 0000000..09b0457 Binary files /dev/null and b/ansible/output/output/118.jpg differ diff --git a/ansible/output/output/119.jpg b/ansible/output/output/119.jpg new file mode 100644 index 0000000..5c85d29 Binary files /dev/null and b/ansible/output/output/119.jpg differ diff --git a/ansible/output/output/12.jpg b/ansible/output/output/12.jpg new file mode 100644 index 0000000..de3fcac Binary files /dev/null and b/ansible/output/output/12.jpg differ diff --git a/ansible/output/output/120.jpg b/ansible/output/output/120.jpg new file mode 100644 index 0000000..2c9dba7 Binary files /dev/null and b/ansible/output/output/120.jpg differ diff --git a/ansible/output/output/13.jpg b/ansible/output/output/13.jpg new file mode 100644 index 0000000..1756e31 Binary files /dev/null and b/ansible/output/output/13.jpg differ diff --git a/ansible/output/output/14.jpg b/ansible/output/output/14.jpg new file mode 100644 index 0000000..c9b9ce4 Binary files /dev/null and b/ansible/output/output/14.jpg differ diff --git a/ansible/output/output/15.jpg b/ansible/output/output/15.jpg new file mode 100644 index 0000000..fb1bd54 Binary files /dev/null and b/ansible/output/output/15.jpg differ diff --git a/ansible/output/output/16.jpg b/ansible/output/output/16.jpg new file mode 100644 index 0000000..7915027 Binary files /dev/null and b/ansible/output/output/16.jpg differ diff --git a/ansible/output/output/17.jpg b/ansible/output/output/17.jpg new file mode 100644 index 0000000..d3f0414 Binary files /dev/null and b/ansible/output/output/17.jpg differ diff --git a/ansible/output/output/18.jpg b/ansible/output/output/18.jpg new file mode 100644 index 0000000..6606949 Binary files /dev/null and b/ansible/output/output/18.jpg differ diff --git a/ansible/output/output/19.jpg b/ansible/output/output/19.jpg new file mode 100644 index 0000000..6c4e7ef Binary files /dev/null and b/ansible/output/output/19.jpg differ diff --git a/ansible/output/output/2.jpg b/ansible/output/output/2.jpg new file mode 100644 index 0000000..ded26e0 Binary files /dev/null and b/ansible/output/output/2.jpg differ diff --git a/ansible/output/output/20.jpg b/ansible/output/output/20.jpg new file mode 100644 index 0000000..79f8dae Binary files /dev/null and b/ansible/output/output/20.jpg differ diff --git a/ansible/output/output/21.jpg b/ansible/output/output/21.jpg new file mode 100644 index 0000000..8584d70 Binary files /dev/null and b/ansible/output/output/21.jpg differ diff --git a/ansible/output/output/22.jpg b/ansible/output/output/22.jpg new file mode 100644 index 0000000..df443ea Binary files /dev/null and b/ansible/output/output/22.jpg differ diff --git a/ansible/output/output/23.jpg b/ansible/output/output/23.jpg new file mode 100644 index 0000000..a63c3d3 Binary files /dev/null and b/ansible/output/output/23.jpg differ diff --git a/ansible/output/output/24.jpg b/ansible/output/output/24.jpg new file mode 100644 index 0000000..5991e86 Binary files /dev/null and b/ansible/output/output/24.jpg differ diff --git a/ansible/output/output/25.jpg b/ansible/output/output/25.jpg new file mode 100644 index 0000000..42afbd1 Binary files /dev/null and b/ansible/output/output/25.jpg differ diff --git a/ansible/output/output/26.jpg b/ansible/output/output/26.jpg new file mode 100644 index 0000000..00bc553 Binary files /dev/null and b/ansible/output/output/26.jpg differ diff --git a/ansible/output/output/27.jpg b/ansible/output/output/27.jpg new file mode 100644 index 0000000..fa3fe12 Binary files /dev/null and b/ansible/output/output/27.jpg differ diff --git a/ansible/output/output/28.jpg b/ansible/output/output/28.jpg new file mode 100644 index 0000000..d0a2f11 Binary files /dev/null and b/ansible/output/output/28.jpg differ diff --git a/ansible/output/output/29.jpg b/ansible/output/output/29.jpg new file mode 100644 index 0000000..5db2328 Binary files /dev/null and b/ansible/output/output/29.jpg differ diff --git a/ansible/output/output/3.jpg b/ansible/output/output/3.jpg new file mode 100644 index 0000000..ba44bc8 Binary files /dev/null and b/ansible/output/output/3.jpg differ diff --git a/ansible/output/output/30.jpg b/ansible/output/output/30.jpg new file mode 100644 index 0000000..aebe047 Binary files /dev/null and b/ansible/output/output/30.jpg differ diff --git a/ansible/output/output/31.jpg b/ansible/output/output/31.jpg new file mode 100644 index 0000000..fedf2ed Binary files /dev/null and b/ansible/output/output/31.jpg differ diff --git a/ansible/output/output/32.jpg b/ansible/output/output/32.jpg new file mode 100644 index 0000000..7366669 Binary files /dev/null and b/ansible/output/output/32.jpg differ diff --git a/ansible/output/output/33.jpg b/ansible/output/output/33.jpg new file mode 100644 index 0000000..78359c3 Binary files /dev/null and b/ansible/output/output/33.jpg differ diff --git a/ansible/output/output/34.jpg b/ansible/output/output/34.jpg new file mode 100644 index 0000000..fa0a8b3 Binary files /dev/null and b/ansible/output/output/34.jpg differ diff --git a/ansible/output/output/35.jpg b/ansible/output/output/35.jpg new file mode 100644 index 0000000..7ad8384 Binary files /dev/null and b/ansible/output/output/35.jpg differ diff --git a/ansible/output/output/36.jpg b/ansible/output/output/36.jpg new file mode 100644 index 0000000..db4c092 Binary files /dev/null and b/ansible/output/output/36.jpg differ diff --git a/ansible/output/output/37.jpg b/ansible/output/output/37.jpg new file mode 100644 index 0000000..4d5bba1 Binary files /dev/null and b/ansible/output/output/37.jpg differ diff --git a/ansible/output/output/38.jpg b/ansible/output/output/38.jpg new file mode 100644 index 0000000..df49bd1 Binary files /dev/null and b/ansible/output/output/38.jpg differ diff --git a/ansible/output/output/39.jpg b/ansible/output/output/39.jpg new file mode 100644 index 0000000..9adb948 Binary files /dev/null and b/ansible/output/output/39.jpg differ diff --git a/ansible/output/output/4.jpg b/ansible/output/output/4.jpg new file mode 100644 index 0000000..034e8a2 Binary files /dev/null and b/ansible/output/output/4.jpg differ diff --git a/ansible/output/output/40.jpg b/ansible/output/output/40.jpg new file mode 100644 index 0000000..1f3c8cd Binary files /dev/null and b/ansible/output/output/40.jpg differ diff --git a/ansible/output/output/41.jpg b/ansible/output/output/41.jpg new file mode 100644 index 0000000..1f19714 Binary files /dev/null and b/ansible/output/output/41.jpg differ diff --git a/ansible/output/output/42.jpg b/ansible/output/output/42.jpg new file mode 100644 index 0000000..d6fa64d Binary files /dev/null and b/ansible/output/output/42.jpg differ diff --git a/ansible/output/output/43.jpg b/ansible/output/output/43.jpg new file mode 100644 index 0000000..ecd4c2c Binary files /dev/null and b/ansible/output/output/43.jpg differ diff --git a/ansible/output/output/44.jpg b/ansible/output/output/44.jpg new file mode 100644 index 0000000..a663c0b Binary files /dev/null and b/ansible/output/output/44.jpg differ diff --git a/ansible/output/output/45.jpg b/ansible/output/output/45.jpg new file mode 100644 index 0000000..a0b6d00 Binary files /dev/null and b/ansible/output/output/45.jpg differ diff --git a/ansible/output/output/46.jpg b/ansible/output/output/46.jpg new file mode 100644 index 0000000..042f5ca Binary files /dev/null and b/ansible/output/output/46.jpg differ diff --git a/ansible/output/output/47.jpg b/ansible/output/output/47.jpg new file mode 100644 index 0000000..a7004d2 Binary files /dev/null and b/ansible/output/output/47.jpg differ diff --git a/ansible/output/output/48.jpg b/ansible/output/output/48.jpg new file mode 100644 index 0000000..79c5db3 Binary files /dev/null and b/ansible/output/output/48.jpg differ diff --git a/ansible/output/output/49.jpg b/ansible/output/output/49.jpg new file mode 100644 index 0000000..fd4b8e0 Binary files /dev/null and b/ansible/output/output/49.jpg differ diff --git a/ansible/output/output/5.jpg b/ansible/output/output/5.jpg new file mode 100644 index 0000000..95ac3c6 Binary files /dev/null and b/ansible/output/output/5.jpg differ diff --git a/ansible/output/output/50.jpg b/ansible/output/output/50.jpg new file mode 100644 index 0000000..0c96430 Binary files /dev/null and b/ansible/output/output/50.jpg differ diff --git a/ansible/output/output/51.jpg b/ansible/output/output/51.jpg new file mode 100644 index 0000000..bface54 Binary files /dev/null and b/ansible/output/output/51.jpg differ diff --git a/ansible/output/output/52.jpg b/ansible/output/output/52.jpg new file mode 100644 index 0000000..c3d23ef Binary files /dev/null and b/ansible/output/output/52.jpg differ diff --git a/ansible/output/output/53.jpg b/ansible/output/output/53.jpg new file mode 100644 index 0000000..107fbe3 Binary files /dev/null and b/ansible/output/output/53.jpg differ diff --git a/ansible/output/output/54.jpg b/ansible/output/output/54.jpg new file mode 100644 index 0000000..d59e96d Binary files /dev/null and b/ansible/output/output/54.jpg differ diff --git a/ansible/output/output/55.jpg b/ansible/output/output/55.jpg new file mode 100644 index 0000000..40966fc Binary files /dev/null and b/ansible/output/output/55.jpg differ diff --git a/ansible/output/output/56.jpg b/ansible/output/output/56.jpg new file mode 100644 index 0000000..89a8559 Binary files /dev/null and b/ansible/output/output/56.jpg differ diff --git a/ansible/output/output/57.jpg b/ansible/output/output/57.jpg new file mode 100644 index 0000000..134806d Binary files /dev/null and b/ansible/output/output/57.jpg differ diff --git a/ansible/output/output/58.jpg b/ansible/output/output/58.jpg new file mode 100644 index 0000000..58a7a1c Binary files /dev/null and b/ansible/output/output/58.jpg differ diff --git a/ansible/output/output/59.jpg b/ansible/output/output/59.jpg new file mode 100644 index 0000000..12ec791 Binary files /dev/null and b/ansible/output/output/59.jpg differ diff --git a/ansible/output/output/6.jpg b/ansible/output/output/6.jpg new file mode 100644 index 0000000..7540a76 Binary files /dev/null and b/ansible/output/output/6.jpg differ diff --git a/ansible/output/output/60.jpg b/ansible/output/output/60.jpg new file mode 100644 index 0000000..85563eb Binary files /dev/null and b/ansible/output/output/60.jpg differ diff --git a/ansible/output/output/61.jpg b/ansible/output/output/61.jpg new file mode 100644 index 0000000..fe4ed53 Binary files /dev/null and b/ansible/output/output/61.jpg differ diff --git a/ansible/output/output/62.jpg b/ansible/output/output/62.jpg new file mode 100644 index 0000000..eb42f45 Binary files /dev/null and b/ansible/output/output/62.jpg differ diff --git a/ansible/output/output/63.jpg b/ansible/output/output/63.jpg new file mode 100644 index 0000000..fc392c4 Binary files /dev/null and b/ansible/output/output/63.jpg differ diff --git a/ansible/output/output/64.jpg b/ansible/output/output/64.jpg new file mode 100644 index 0000000..f6b3ad4 Binary files /dev/null and b/ansible/output/output/64.jpg differ diff --git a/ansible/output/output/65.jpg b/ansible/output/output/65.jpg new file mode 100644 index 0000000..6369f71 Binary files /dev/null and b/ansible/output/output/65.jpg differ diff --git a/ansible/output/output/66.jpg b/ansible/output/output/66.jpg new file mode 100644 index 0000000..f77f959 Binary files /dev/null and b/ansible/output/output/66.jpg differ diff --git a/ansible/output/output/67.jpg b/ansible/output/output/67.jpg new file mode 100644 index 0000000..b180e54 Binary files /dev/null and b/ansible/output/output/67.jpg differ diff --git a/ansible/output/output/68.jpg b/ansible/output/output/68.jpg new file mode 100644 index 0000000..600cc17 Binary files /dev/null and b/ansible/output/output/68.jpg differ diff --git a/ansible/output/output/69.jpg b/ansible/output/output/69.jpg new file mode 100644 index 0000000..3b2a8be Binary files /dev/null and b/ansible/output/output/69.jpg differ diff --git a/ansible/output/output/7.jpg b/ansible/output/output/7.jpg new file mode 100644 index 0000000..64544c9 Binary files /dev/null and b/ansible/output/output/7.jpg differ diff --git a/ansible/output/output/70.jpg b/ansible/output/output/70.jpg new file mode 100644 index 0000000..d80a38b Binary files /dev/null and b/ansible/output/output/70.jpg differ diff --git a/ansible/output/output/71.jpg b/ansible/output/output/71.jpg new file mode 100644 index 0000000..5b913c8 Binary files /dev/null and b/ansible/output/output/71.jpg differ diff --git a/ansible/output/output/72.jpg b/ansible/output/output/72.jpg new file mode 100644 index 0000000..b2898f6 Binary files /dev/null and b/ansible/output/output/72.jpg differ diff --git a/ansible/output/output/73.jpg b/ansible/output/output/73.jpg new file mode 100644 index 0000000..9a8c971 Binary files /dev/null and b/ansible/output/output/73.jpg differ diff --git a/ansible/output/output/74.jpg b/ansible/output/output/74.jpg new file mode 100644 index 0000000..ebcb109 Binary files /dev/null and b/ansible/output/output/74.jpg differ diff --git a/ansible/output/output/75.jpg b/ansible/output/output/75.jpg new file mode 100644 index 0000000..61e24b9 Binary files /dev/null and b/ansible/output/output/75.jpg differ diff --git a/ansible/output/output/76.jpg b/ansible/output/output/76.jpg new file mode 100644 index 0000000..50585b9 Binary files /dev/null and b/ansible/output/output/76.jpg differ diff --git a/ansible/output/output/77.jpg b/ansible/output/output/77.jpg new file mode 100644 index 0000000..82c6661 Binary files /dev/null and b/ansible/output/output/77.jpg differ diff --git a/ansible/output/output/78.jpg b/ansible/output/output/78.jpg new file mode 100644 index 0000000..b126e26 Binary files /dev/null and b/ansible/output/output/78.jpg differ diff --git a/ansible/output/output/79.jpg b/ansible/output/output/79.jpg new file mode 100644 index 0000000..140dd19 Binary files /dev/null and b/ansible/output/output/79.jpg differ diff --git a/ansible/output/output/8.jpg b/ansible/output/output/8.jpg new file mode 100644 index 0000000..3424458 Binary files /dev/null and b/ansible/output/output/8.jpg differ diff --git a/ansible/output/output/80.jpg b/ansible/output/output/80.jpg new file mode 100644 index 0000000..53245b8 Binary files /dev/null and b/ansible/output/output/80.jpg differ diff --git a/ansible/output/output/81.jpg b/ansible/output/output/81.jpg new file mode 100644 index 0000000..2dc7dcd Binary files /dev/null and b/ansible/output/output/81.jpg differ diff --git a/ansible/output/output/82.jpg b/ansible/output/output/82.jpg new file mode 100644 index 0000000..6ad6a59 Binary files /dev/null and b/ansible/output/output/82.jpg differ diff --git a/ansible/output/output/83.jpg b/ansible/output/output/83.jpg new file mode 100644 index 0000000..89fb85e Binary files /dev/null and b/ansible/output/output/83.jpg differ diff --git a/ansible/output/output/84.jpg b/ansible/output/output/84.jpg new file mode 100644 index 0000000..4796e6b Binary files /dev/null and b/ansible/output/output/84.jpg differ diff --git a/ansible/output/output/85.jpg b/ansible/output/output/85.jpg new file mode 100644 index 0000000..aa71aa7 Binary files /dev/null and b/ansible/output/output/85.jpg differ diff --git a/ansible/output/output/86.jpg b/ansible/output/output/86.jpg new file mode 100644 index 0000000..f5f84cc Binary files /dev/null and b/ansible/output/output/86.jpg differ diff --git a/ansible/output/output/87.jpg b/ansible/output/output/87.jpg new file mode 100644 index 0000000..9e95585 Binary files /dev/null and b/ansible/output/output/87.jpg differ diff --git a/ansible/output/output/88.jpg b/ansible/output/output/88.jpg new file mode 100644 index 0000000..0801926 Binary files /dev/null and b/ansible/output/output/88.jpg differ diff --git a/ansible/output/output/89.jpg b/ansible/output/output/89.jpg new file mode 100644 index 0000000..3d02029 Binary files /dev/null and b/ansible/output/output/89.jpg differ diff --git a/ansible/output/output/9.jpg b/ansible/output/output/9.jpg new file mode 100644 index 0000000..39a37a0 Binary files /dev/null and b/ansible/output/output/9.jpg differ diff --git a/ansible/output/output/90.jpg b/ansible/output/output/90.jpg new file mode 100644 index 0000000..35e335d Binary files /dev/null and b/ansible/output/output/90.jpg differ diff --git a/ansible/output/output/91.jpg b/ansible/output/output/91.jpg new file mode 100644 index 0000000..4282f65 Binary files /dev/null and b/ansible/output/output/91.jpg differ diff --git a/ansible/output/output/92.jpg b/ansible/output/output/92.jpg new file mode 100644 index 0000000..2e2f041 Binary files /dev/null and b/ansible/output/output/92.jpg differ diff --git a/ansible/output/output/93.jpg b/ansible/output/output/93.jpg new file mode 100644 index 0000000..d1820d7 Binary files /dev/null and b/ansible/output/output/93.jpg differ diff --git a/ansible/output/output/94.jpg b/ansible/output/output/94.jpg new file mode 100644 index 0000000..801412c Binary files /dev/null and b/ansible/output/output/94.jpg differ diff --git a/ansible/output/output/95.jpg b/ansible/output/output/95.jpg new file mode 100644 index 0000000..20773a9 Binary files /dev/null and b/ansible/output/output/95.jpg differ diff --git a/ansible/output/output/96.jpg b/ansible/output/output/96.jpg new file mode 100644 index 0000000..fae85a3 Binary files /dev/null and b/ansible/output/output/96.jpg differ diff --git a/ansible/output/output/97.jpg b/ansible/output/output/97.jpg new file mode 100644 index 0000000..1e24248 Binary files /dev/null and b/ansible/output/output/97.jpg differ diff --git a/ansible/output/output/98.jpg b/ansible/output/output/98.jpg new file mode 100644 index 0000000..9c0f05f Binary files /dev/null and b/ansible/output/output/98.jpg differ diff --git a/ansible/output/output/99.jpg b/ansible/output/output/99.jpg new file mode 100644 index 0000000..51e4fd7 Binary files /dev/null and b/ansible/output/output/99.jpg differ diff --git a/ansible/roles/analysis/files/code/signdetectionbyspark.py b/ansible/roles/analysis/files/code/signdetectionbyspark.py index dfd8d8a..8107976 100644 --- a/ansible/roles/analysis/files/code/signdetectionbyspark.py +++ b/ansible/roles/analysis/files/code/signdetectionbyspark.py @@ -33,7 +33,7 @@ def process_images(image): if (imageDir != '/opencv_workspace/test_data/images/'): imageDir = video2image(imageDir) print(imageDir) - pd = sc.parallelize(paths.list_images(imageDir),100) + pd = sc.parallelize(paths.list_images(imageDir),60) pdc = pd.map(process_images) result = pdc.collect() diff --git a/ansible/roles/analysis/files/code/signdetectionbyspark_images.py b/ansible/roles/analysis/files/code/signdetectionbyspark_images.py deleted file mode 100644 index 1977a4a..0000000 --- a/ansible/roles/analysis/files/code/signdetectionbyspark_images.py +++ /dev/null @@ -1,45 +0,0 @@ -from __future__ import print_function -import imutils -from imutils import paths -from pyspark import SparkContext, SparkConf -import cv2 -import matplotlib.pyplot as plt -import numpy -import sys - -def process_images(image): - import cv2 - stop_cascade = cv2.CascadeClassifier('/opencv_workspace/classifier/stopsign_classifier.xml') - test = cv2.imread(image) - gray = cv2.cvtColor(test,cv2.COLOR_BGR2GRAY) - stops = stop_cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=2) - for (x,y,w,h) in stops: - cv2.rectangle(test,(x,y),(x+w,y+h),(255,0,0),2) - return test - -if __name__== "__main__": - if(len(sys.argv)!=3): - print("Usage: streetsign input output") - exit(-1) - - conf = SparkConf().setAppName("Street Sign") - - sc = SparkContext(conf = conf) - imageDir = sys.argv[1] - outputDir = sys.argv[2] - #print(imageDir) - #print(outputDir) - #print(paths.list_images(imageDir)) - pd = sc.parallelize(paths.list_images(imageDir)) - pdc = pd.map(process_images) - result = pdc.collect() - count = 1 - for img in result: - # show objects on street image - #plt.figure() - #plt.axis("off") - #plt.imsave(outputDir+str(count)+'.jpg',cv2.cvtColor(img, cv2.COLOR_BGR2RGB)) - cv2.imwrite(outputDir+str(count)+'.jpg',img) - count = count+1 - if (count == 51): - break diff --git a/ansible/roles/analysis/files/code/test_video.py b/ansible/roles/analysis/files/code/test_video.py deleted file mode 100644 index 0151dac..0000000 --- a/ansible/roles/analysis/files/code/test_video.py +++ /dev/null @@ -1,25 +0,0 @@ -import numpy as np -import cv2 - -stop_cascade = cv2.CascadeClassifier('stopsign_classifier.xml') - -cap = cv2.VideoCapture("stop_video.mp4") -pic_num=1 -while (cap.isOpened()): - ret, img = cap.read() - gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) - stops = stop_cascade.detectMultiScale(gray,scaleFactor=1.4, minNeighbors=3) - - # add this - for (x,y,w,h) in stops: - cv2.rectangle(img,(x,y),(x+w,y+h),(255,255,0),2) - - #cv2.imshow('img',img) - cv2.imwrite('output'+str(pic_num)+'.jpg',img) - pic_num+=1 - k = cv2.waitKey(30) & 0xff - if k == 27: - break - -cap.release() -cv2.destroyAllWindows() diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323803184.avi_image26.png b/ansible/roles/analysis/files/test_data/images/stop_1323803184.avi_image26.png new file mode 100644 index 0000000..31d2d82 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323803184.avi_image26.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812801.avi_image6.png b/ansible/roles/analysis/files/test_data/images/stop_1323812801.avi_image6.png new file mode 100644 index 0000000..c1fe648 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812801.avi_image6.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812801.avi_image8.png b/ansible/roles/analysis/files/test_data/images/stop_1323812801.avi_image8.png new file mode 100644 index 0000000..7e3e937 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812801.avi_image8.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image16.png b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image16.png new file mode 100644 index 0000000..26840f4 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image16.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image17.png b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image17.png new file mode 100644 index 0000000..ad86450 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image17.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image18.png b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image18.png new file mode 100644 index 0000000..eeb0089 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image18.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image19.png b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image19.png new file mode 100644 index 0000000..fc4958f Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image19.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image20.png b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image20.png new file mode 100644 index 0000000..688afab Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image20.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image21.png b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image21.png new file mode 100644 index 0000000..d4d9e42 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323812975.avi_image21.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image0.png b/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image0.png new file mode 100644 index 0000000..fcaa13b Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image0.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image1.png b/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image1.png new file mode 100644 index 0000000..326ca4d Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image1.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image16.png b/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image16.png new file mode 100644 index 0000000..8b4868c Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323816428.avi_image16.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323816448.avi_image4.png b/ansible/roles/analysis/files/test_data/images/stop_1323816448.avi_image4.png new file mode 100644 index 0000000..8e58a76 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323816448.avi_image4.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323816448.avi_image5.png b/ansible/roles/analysis/files/test_data/images/stop_1323816448.avi_image5.png new file mode 100644 index 0000000..de847a7 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323816448.avi_image5.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323817816.avi_image23.png b/ansible/roles/analysis/files/test_data/images/stop_1323817816.avi_image23.png new file mode 100644 index 0000000..8fe6773 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323817816.avi_image23.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323817842.avi_image1.png b/ansible/roles/analysis/files/test_data/images/stop_1323817842.avi_image1.png new file mode 100644 index 0000000..283eb6b Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323817842.avi_image1.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image12.png b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image12.png new file mode 100644 index 0000000..5dfe804 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image12.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image13.png b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image13.png new file mode 100644 index 0000000..8ffffe0 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image13.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image14.png b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image14.png new file mode 100644 index 0000000..988eed1 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image14.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image15.png b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image15.png new file mode 100644 index 0000000..c6396da Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820728.avi_image15.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image10.png b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image10.png new file mode 100644 index 0000000..8a3c48e Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image10.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image11.png b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image11.png new file mode 100644 index 0000000..479f6e6 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image11.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image19.png b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image19.png new file mode 100644 index 0000000..eac216a Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image19.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image8.png b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image8.png new file mode 100644 index 0000000..99100fb Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image8.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image9.png b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image9.png new file mode 100644 index 0000000..f717ca5 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323820809.avi_image9.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image24.png b/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image24.png new file mode 100644 index 0000000..1d60d49 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image24.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image26.png b/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image26.png new file mode 100644 index 0000000..527d00c Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image26.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image29.png b/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image29.png new file mode 100644 index 0000000..10105bb Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896588.avi_image29.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image0.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image0.png new file mode 100644 index 0000000..b0ffb4e Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image0.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image1.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image1.png new file mode 100644 index 0000000..ebd456a Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image1.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image2.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image2.png new file mode 100644 index 0000000..cf513f5 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image2.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image3.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image3.png new file mode 100644 index 0000000..2db550b Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image3.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image4.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image4.png new file mode 100644 index 0000000..2a9ecc8 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image4.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image5.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image5.png new file mode 100644 index 0000000..9505fb9 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image5.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image6.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image6.png new file mode 100644 index 0000000..46d920f Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image6.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image7.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image7.png new file mode 100644 index 0000000..8cc819f Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image7.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image8.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image8.png new file mode 100644 index 0000000..73fb9f7 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image8.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image9.png b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image9.png new file mode 100644 index 0000000..928aac9 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896696.avi_image9.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image20.png b/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image20.png new file mode 100644 index 0000000..8752277 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image20.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image22.png b/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image22.png new file mode 100644 index 0000000..eab509f Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image22.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image27.png b/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image27.png new file mode 100644 index 0000000..b11637b Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896809.avi_image27.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image12.png b/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image12.png new file mode 100644 index 0000000..d91971d Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image12.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image20.png b/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image20.png new file mode 100644 index 0000000..e9c7a63 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image20.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image23.png b/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image23.png new file mode 100644 index 0000000..a8f2f71 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896859.avi_image23.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896977.avi_image14.png b/ansible/roles/analysis/files/test_data/images/stop_1323896977.avi_image14.png new file mode 100644 index 0000000..d49a7fc Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896977.avi_image14.png differ diff --git a/ansible/roles/analysis/files/test_data/images/stop_1323896977.avi_image15.png b/ansible/roles/analysis/files/test_data/images/stop_1323896977.avi_image15.png new file mode 100644 index 0000000..1b2f18b Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/stop_1323896977.avi_image15.png differ diff --git a/ansible/roles/analysis/files/test_data/images/turnRight_1323820783.avi_image13.png b/ansible/roles/analysis/files/test_data/images/turnRight_1323820783.avi_image13.png new file mode 100644 index 0000000..b3153f1 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/turnRight_1323820783.avi_image13.png differ diff --git a/ansible/roles/analysis/files/test_data/images/turnRight_1323820783.avi_image24.png b/ansible/roles/analysis/files/test_data/images/turnRight_1323820783.avi_image24.png new file mode 100644 index 0000000..fd3bb71 Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/turnRight_1323820783.avi_image24.png differ diff --git a/ansible/roles/analysis/files/test_data/images/yield_1323813350.avi_image4.png b/ansible/roles/analysis/files/test_data/images/yield_1323813350.avi_image4.png new file mode 100644 index 0000000..1f9306d Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/yield_1323813350.avi_image4.png differ diff --git a/ansible/roles/analysis/files/test_data/images/yield_1323813350.avi_image5.png b/ansible/roles/analysis/files/test_data/images/yield_1323813350.avi_image5.png new file mode 100644 index 0000000..e01873e Binary files /dev/null and b/ansible/roles/analysis/files/test_data/images/yield_1323813350.avi_image5.png differ diff --git a/ansible/roles/analysis/files/test_data/videos/stop_video_1.mp4 b/ansible/roles/analysis/files/test_data/videos/stop_video_1.mp4 new file mode 100644 index 0000000..e8fa1ed Binary files /dev/null and b/ansible/roles/analysis/files/test_data/videos/stop_video_1.mp4 differ diff --git a/ansible/roles/analysis/files/test_data/videos/stop_video_2.mp4 b/ansible/roles/analysis/files/test_data/videos/stop_video_2.mp4 deleted file mode 100644 index 5f0fa01..0000000 Binary files a/ansible/roles/analysis/files/test_data/videos/stop_video_2.mp4 and /dev/null differ diff --git a/ansible/roles/analysis/main.yml b/ansible/roles/analysis/main.yml deleted file mode 100644 index 45c7933..0000000 --- a/ansible/roles/analysis/main.yml +++ /dev/null @@ -1,51 +0,0 @@ ---- -# tasks file for analysis -- name: Check if the folder exists - stat: path={{workspace}} - register: dir_flag - -- name: Delete opencv_workspace if exists - become: yes - file: path={{workspace}} state=absent - when: dir_flag.stat.exists == true - -- name: Copy code files from local to master - copy: src={{ item }} dest={{code_dir}} - with_fileglob: - - files/code/*.py -# - files/code/sign*.py - when: dir_flag.stat.exists == false - -- name: Copy input files images from local to master - copy: src={{ item }} dest={{test_images}} - with_fileglob: - - files/test_data/images/* - when: dir_flag.stat.exists == false - -- name: Copy video from local to master - copy: src={{ item }} dest={{test_videos}} - with_fileglob: - - files/test_data/videos/* - when: dir_flag.stat.exists == false - -- name: Copy stop sign classifier from local to master - copy: src={{ item }} dest={{classifier_dir}} - with_fileglob: - - files/classifier/*.* - when: dir_flag.stat.exists == false - -- name: Create directory for output files - become: yes - file: path={{output_dir}} state=directory mode=0777 - when: dir_flag.stat.exists == false - -- name: Create directory for video2image files - become: yes - file: path={{video2image_dir}} state=directory mode=0777 - when: dir_flag.stat.exists == false - -- name: Run the spark-job - become: true - shell: | - chown -R 777 {{workspace}} - su - hadoop -c "spark-submit --master yarn --deploy-mode client --driver-memory 512 --executor-memory 3g --name signdetection --conf "spark.app.id=signdetection" /opencv_workspace/code/signdetectionbyspark.py /opencv_workspace/test_data/videos/ /opencv_workspace/output/" diff --git a/ansible/roles/analysis/tasks/main.yml b/ansible/roles/analysis/tasks/main.yml index 21fb668..4830e68 100644 --- a/ansible/roles/analysis/tasks/main.yml +++ b/ansible/roles/analysis/tasks/main.yml @@ -13,39 +13,32 @@ copy: src={{ item }} dest={{code_dir}} with_fileglob: - files/code/*.py -# - files/code/sign*.py - when: dir_flag.stat.exists == false - name: Copy input files images from local to master copy: src={{ item }} dest={{test_images}} with_fileglob: - files/test_data/images/* - when: dir_flag.stat.exists == false - name: Copy video from local to master copy: src={{ item }} dest={{test_videos}} with_fileglob: - files/test_data/videos/* - when: dir_flag.stat.exists == false - name: Copy stop sign classifier from local to master copy: src={{ item }} dest={{classifier_dir}} with_fileglob: - files/classifier/*.* - when: dir_flag.stat.exists == false - name: Create directory for output files become: yes file: path={{output_dir}} state=directory mode=0777 - when: dir_flag.stat.exists == false - name: Create directory for video2image files become: yes file: path={{video2image_dir}} state=directory mode=0777 - when: dir_flag.stat.exists == false - name: Run the spark-job become: true shell: | chown -R 777 {{workspace}} - su - hadoop -c "spark-submit --master yarn --deploy-mode client --executor-memory 1g --driver-memory 2g --name signdetection --conf "spark.app.id=signdetection" /opencv_workspace/code/signdetectionbyspark.py /opencv_workspace/test_data/videos/ /opencv_workspace/output/" + su - hadoop -c "spark-submit --master yarn --deploy-mode client --executor-memory 1g --name signdetection --conf "spark.app.id=signdetection" /opencv_workspace/code/signdetectionbyspark.py /opencv_workspace/test_data/images/ /opencv_workspace/output/" diff --git a/ansible/roles/output/.travis.yml b/ansible/roles/output/.travis.yml deleted file mode 100644 index 36bbf62..0000000 --- a/ansible/roles/output/.travis.yml +++ /dev/null @@ -1,29 +0,0 @@ ---- -language: python -python: "2.7" - -# Use the new container infrastructure -sudo: false - -# Install ansible -addons: - apt: - packages: - - python-pip - -install: - # Install ansible - - pip install ansible - - # Check ansible version - - ansible --version - - # Create ansible.cfg with correct roles_path - - printf '[defaults]\nroles_path=../' >ansible.cfg - -script: - # Basic role syntax check - - ansible-playbook tests/test.yml -i tests/inventory --syntax-check - -notifications: - webhooks: https://galaxy.ansible.com/api/v1/notifications/ \ No newline at end of file diff --git a/ansible/roles/output/README.md b/ansible/roles/output/README.md deleted file mode 100644 index 225dd44..0000000 --- a/ansible/roles/output/README.md +++ /dev/null @@ -1,38 +0,0 @@ -Role Name -========= - -A brief description of the role goes here. - -Requirements ------------- - -Any pre-requisites that may not be covered by Ansible itself or the role should be mentioned here. For instance, if the role uses the EC2 module, it may be a good idea to mention in this section that the boto package is required. - -Role Variables --------------- - -A description of the settable variables for this role should go here, including any variables that are in defaults/main.yml, vars/main.yml, and any variables that can/should be set via parameters to the role. Any variables that are read from other roles and/or the global scope (ie. hostvars, group vars, etc.) should be mentioned here as well. - -Dependencies ------------- - -A list of other roles hosted on Galaxy should go here, plus any details in regards to parameters that may need to be set for other roles, or variables that are used from other roles. - -Example Playbook ----------------- - -Including an example of how to use your role (for instance, with variables passed in as parameters) is always nice for users too: - - - hosts: servers - roles: - - { role: username.rolename, x: 42 } - -License -------- - -BSD - -Author Information ------------------- - -An optional section for the role authors to include contact information, or a website (HTML is not allowed). diff --git a/ansible/roles/output/defaults/main.yml b/ansible/roles/output/defaults/main.yml deleted file mode 100644 index 4f800df..0000000 --- a/ansible/roles/output/defaults/main.yml +++ /dev/null @@ -1,2 +0,0 @@ ---- -# defaults file for output diff --git a/ansible/roles/output/tasks/main.yml b/ansible/roles/output/tasks/main.yml deleted file mode 100644 index 5bd8e23..0000000 --- a/ansible/roles/output/tasks/main.yml +++ /dev/null @@ -1,2 +0,0 @@ ---- -# tasks file for output diff --git a/ansible/user_vars.yaml b/ansible/user_vars.yaml index d9dee4c..0e9a78e 100644 --- a/ansible/user_vars.yaml +++ b/ansible/user_vars.yaml @@ -1,16 +1,26 @@ --- -############################################### +############################################################ #Variables for execution of complete package #EDIT FOLLOWING DETAILS AS PER REQUIREMENT -##############CLOUDMESH SETTINGS############### -cloud: "chameleon" -username: "rraghata" # key_name to be uploaded -############HADOOP CLUSTER SETTINGS############ -image_name: "CC-Ubuntu14.04" -count: "3" -flavor: "m1.large" +##############CLOUDMESH SETTINGS############################ + +#cloud: "chameleon" or "jetstream" +cloud: +#username: "cloudmesh username as key_name" +username: + +############HADOOP CLUSTER SETTINGS######################### +#Chameleon image_name: CC-Ubuntu14.04 +#jetstream image_name: ubuntu-14.04-trusty-server-cloudimg +#flavor: m1.small, m1.medium, m1.large[preferred: m1.medium] +#addons: spark pig hive + + +image_name: +count: +flavor: addons: "spark" -############################################### +############################################################ diff --git a/benchmark/benchmark.sh b/benchmark/benchmark.sh index 4ef7f93..129878b 100755 --- a/benchmark/benchmark.sh +++ b/benchmark/benchmark.sh @@ -1,8 +1,13 @@ #!/bin/bash echo ' ' >> benchmark_time echo ' ' >> benchmark_time +cat ../ansible/user_vars.yaml >> benchmark_time +echo ' ' >> benchmark_time +echo ' ' >> benchmark_time echo 'Benchmark' >> benchmark_time +echo -e "Host github.com\n\tStrictHostKeyChecking no\n" > ~/.ssh/config + echo Benchmark 1 echo "Time for local machine setup(ansible,cloudmesh,packages)" start=`date +%s` diff --git a/benchmark/benchmark_chameleon b/benchmark/benchmark_chameleon_images similarity index 77% rename from benchmark/benchmark_chameleon rename to benchmark/benchmark_chameleon_images index a3a4e6f..ce76f22 100755 --- a/benchmark/benchmark_chameleon +++ b/benchmark/benchmark_chameleon_images @@ -66,6 +66,24 @@ time_for_hadoop_deploy 677 time_for_opencv_setup 262 time_for_sign_detection 442 time_for_data_transfer 24 +time_for_env_cleanup 0 + + +Benchmark chameleon 4vms m1.medium 50images +time_for_local_setup 0 +time_for_cloud_setup 39 +time_for_hadoop_deploy 691 +time_for_opencv_setup 240 +time_for_sign_detection 383 +time_for_data_transfer 637 time_for_env_cleanup 0 - - + + +Benchmark chameleon 6vms m1.medium 50images +time_for_local_setup 0 +time_for_cloud_setup 95 +time_for_hadoop_deploy 1201 +time_for_opencv_setup 445 +time_for_sign_detection 311 +time_for_data_transfer 15 +time_for_env_cleanup 0 diff --git a/benchmark/benchmark_chameleon_video b/benchmark/benchmark_chameleon_video new file mode 100755 index 0000000..b61e523 --- /dev/null +++ b/benchmark/benchmark_chameleon_video @@ -0,0 +1,288 @@ + + +--- +############################################### +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############### +cloud: "chameleon" +username: "snehalc" # key_name to be uploaded +############HADOOP CLUSTER SETTINGS############ +image_name: "CC-Ubuntu14.04" +count: "1" +flavor: "m1.large" +addons: "spark" + +############################################### + +Benchmark +time_for_local_setup 0 +time_for_cloud_setup 41 +time_for_hadoop_deploy 530 +time_for_opencv_setup 215 +time_for_sign_detection 107 +time_for_data_transfer 35 +time_for_env_cleanup 0 + + +--- +############################################### +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############### +cloud: "chameleon" +username: "snehalc" # key_name to be uploaded +############HADOOP CLUSTER SETTINGS############ +image_name: "CC-Ubuntu14.04" +count: "2" +flavor: "m1.large" +addons: "spark" + +############################################### + +Benchmark +time_for_local_setup 0 +time_for_cloud_setup 38 +time_for_hadoop_deploy 551 +time_for_opencv_setup 237 +time_for_sign_detection 108 +time_for_data_transfer 34 +time_for_env_cleanup 0 + + +--- +############################################################ +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############################ +#cloud: "chameleon" or "jetstream" +cloud: "chameleon" +#username: "cloudmesh username as key_name" +username: "rraghata" +############HADOOP CLUSTER SETTINGS######################### +#Chameleon image_name: CC-Ubuntu14.04 +#jetstream image_name: ubuntu-14.04-trusty-server-cloudimg +#flavor: m1.small, m1.medium, m1.large[preferred: m1.medium] +#addons: spark pig hive + + +image_name: "CC-Ubuntu14.04" +count: "3" +flavor: "m1.large" +addons: "spark" + +############################################################ + + +Benchmark +time_for_cloud_setup 47 +time_for_hadoop_deploy 980 +time_for_opencv_setup 262 +time_for_sign_detection 129 +time_for_data_transfer 31 +time_for_env_cleanup 0 + + +--- +############################################### +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############### +cloud: "chameleon" +username: "snehalc" # key_name to be uploaded +############HADOOP CLUSTER SETTINGS############ +image_name: "CC-Ubuntu14.04" +count: "4" +flavor: "m1.large" +addons: "spark" + +############################################### +time_for_local_setup 0 +time_for_cloud_setup 39 +time_for_hadoop_deploy 664 +time_for_opencv_setup 248 +time_for_sign_detection 104 +time_for_data_transfer 18 +time_for_env_cleanup 0 + + +--- +############################################################ +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############################ +#cloud: "chameleon" or "jetstream" +cloud: "chameleon" +#username: "cloudmesh username as key_name" +username: "rraghata" +############HADOOP CLUSTER SETTINGS######################### +#Chameleon image_name: CC-Ubuntu14.04 +#jetstream image_name: ubuntu-14.04-trusty-server-cloudimg +#flavor: m1.small, m1.medium, m1.large[preferred: m1.medium] +#addons: spark pig hive + + +image_name: "CC-Ubuntu14.04" +count: "3" +flavor: "m1.medium" +addons: "spark" + +############################################################ + + +Benchmark +time_for_local_setup 0 +time_for_cloud_setup 27 +time_for_hadoop_deploy 614 +time_for_opencv_setup 235 +time_for_sign_detection 125 +time_for_data_transfer 16 +time_for_env_cleanup 0 + + +--- +############################################################ +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############################ +#cloud: "chameleon" or "jetstream" +cloud: "chameleon" +#username: "cloudmesh username as key_name" +username: "rraghata" +############HADOOP CLUSTER SETTINGS######################### +#Chameleon image_name: CC-Ubuntu14.04 +#jetstream image_name: ubuntu-14.04-trusty-server-cloudimg +#flavor: m1.small, m1.medium, m1.large[preferred: m1.medium] +#addons: spark pig hive + + +image_name: "CC-Ubuntu14.04" +count: "4" +flavor: "m1.medium" +addons: "spark" + +############################################################ + + +Benchmark +time_for_local_setup 0 +time_for_cloud_setup 29 +time_for_hadoop_deploy 691 +time_for_opencv_setup 247 +time_for_sign_detection 123 +time_for_data_transfer 16 +time_for_env_cleanup 0 + + +--- +############################################### +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############### +cloud: "chameleon" +username: "rraghata" # key_name to be uploaded +############HADOOP CLUSTER SETTINGS############ +image_name: "CC-Ubuntu14.04" +count: "5" +flavor: "m1.medium" +addons: "spark" + +############################################### + + +Benchmark +time_for_local_setup 0 +time_for_cloud_setup 31 +time_for_hadoop_deploy 776 +time_for_opencv_setup 248 +time_for_sign_detection 121 +time_for_data_transfer 21 +time_for_env_cleanup 0 + + +--- +############################################################ +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############################ +#cloud: "chameleon" or "jetstream" +cloud: "chameleon" +#username: "cloudmesh username as key_name" +username: "rraghata" +############HADOOP CLUSTER SETTINGS######################### +#Chameleon image_name: CC-Ubuntu14.04 +#jetstream image_name: ubuntu-14.04-trusty-server-cloudimg +#flavor: m1.small, m1.medium, m1.large[preferred: m1.medium] +#addons: spark pig hive + + +image_name: "CC-Ubuntu14.04" +count: "6" +flavor: "m1.medium" +addons: "spark" + +############################################################ + + +Benchmark +time_for_local_setup 0 +time_for_cloud_setup 27 +time_for_hadoop_deploy 1041 +time_for_opencv_setup 444 +time_for_sign_detection 125 +time_for_data_transfer 19 +time_for_env_cleanup 0 + + +--- +############################################################ +#Variables for execution of complete package + +#EDIT FOLLOWING DETAILS AS PER REQUIREMENT + +##############CLOUDMESH SETTINGS############################ +#cloud: "chameleon" or "jetstream" +cloud: "chameleon" +#username: "cloudmesh username as key_name" +username: "rraghata" +############HADOOP CLUSTER SETTINGS######################### +#Chameleon image_name: CC-Ubuntu14.04 +#jetstream image_name: ubuntu-14.04-trusty-server-cloudimg +#flavor: m1.small, m1.medium, m1.large[preferred: m1.medium] +#addons: spark pig hive + + +image_name: "CC-Ubuntu14.04" +count: "7" +flavor: "m1.medium" +addons: "spark" + +############################################################ + + +Benchmark +time_for_local_setup 0 +time_for_cloud_setup 47 +time_for_hadoop_deploy 1157 +time_for_opencv_setup 457 +time_for_sign_detection 115 +time_for_data_transfer 19 +time_for_env_cleanup 0 + + diff --git a/benchmark/benchmark_jetstream b/benchmark/benchmark_jetstream_images similarity index 100% rename from benchmark/benchmark_jetstream rename to benchmark/benchmark_jetstream_images diff --git a/benchmark/benchmark_time b/benchmark/benchmark_time deleted file mode 100644 index b07bdba..0000000 --- a/benchmark/benchmark_time +++ /dev/null @@ -1,26 +0,0 @@ -#Sample Benchmark - -Benchmark -time_for_local_setup 0 -time_for_cloud_setup 0 -time_for_hadoop_deploy 0 -time_for_opencv_setup 0 -time_for_sign_detection 0 -time_for_data_transfer 0 -time_for_env_cleanup 0 - - - - -Benchmark -time_for_local_setup 0 -time_for_cloud_setup 0 -time_for_hadoop_deploy 0 -time_for_opencv_setup 0 -time_for_sign_detection 0 -time_for_data_transfer 0 -time_for_env_cleanup 0 - - -Benchmark -time_for_local_setup 0 diff --git a/code/scripts/deploy.sh b/code/scripts/deploy.sh index 90daf25..80e1de2 100755 --- a/code/scripts/deploy.sh +++ b/code/scripts/deploy.sh @@ -17,5 +17,6 @@ echo '' > inventory.txt #Run ansible playbook: ansible-playbook -i inventory.txt hadoop_deploy.yaml -v + rm $HOME/.ssh/known_hosts touch $HOME/.ssh/known_hosts