updates main readme

rodekruis · Jan 24, 2022 · 4cb4573 · 4cb4573
1 parent f88face
commit 4cb4573
Show file tree

Hide file tree

Showing 4 changed files with 138 additions and 137 deletions.
diff --git a/README.md b/README.md
@@ -8,21 +8,23 @@
 
 ## Introduction
 
-**NOTE**: For now some background on IBF-terminology (e.g. triggers) is
+**NOTE**: Some background on IBF-terminology (e.g. triggers) is
 expected. This can be expanded on later.
 
-This is the repository for the IBF-system. It includes 2 main folders.
+This is the repository for the IBF-system. It includes a.o.:
 
-1. [Services (backend)](./services/)
+1. An [API-service (backend)](./services/API-service/)
 
-- API-service
 - which accepts input from various IBF-pipelines, that upload impact forecast data to the IBF-system on regular intervals. (See 'Dependencies' below.)
 - and which lets the IBF-dashboard - or other authorized accounts - retrieve data from the IBF-database.
 
-2. [Interfaces (frontend)](./interfaces/)
+2. An [IBF-dashboard (frontend)](./interfaces/IBF-dashboard/)
 
-- IBF-dashboard
-- showing all impact forecast data - either leading to a trigger or not - in a visual portal
+- showing all impact forecast data - either leading to a trigger or not - in the IBF-portal
+
+3. A [features](./features/) folder describing all feature scenarios in the IBF-system, using Gherking language.
+
+4. A [docs](./docs) folder with further documentation beyond this readme.
 
 ## Dependencies
 
@@ -31,7 +33,7 @@ This is the repository for the IBF-system. It includes 2 main folders.
 - See the 510 IBF Project Document for more info and links to the 510-instances of these pipelines per disaster-type.
 - For development/testing purposes, there are mock-endpoints and mock-data available to replace the need for these pipelines. (See 'Load local database with data' below.)
 
-## System design (draft)
+## System design
 
 ![IBF-system design (draft)](./system-design/ibf-system-design.PNG)
 
@@ -54,11 +56,11 @@ This is the repository for the IBF-system. It includes 2 main folders.
 ### Using Docker
 
 ```
-docker-compose -f docker-compose.yml up # for production
+docker-compose -f docker-compose.yml up -d # for production
 
-docker-compose up # for development
+docker-compose up -d # for development
 
-docker-compose -f docker-compose.yml -f docker-compose.override.yml up # for development (explicit)
+docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d # for development (same as previous line)
 
 ```
 
@@ -92,7 +94,7 @@ are connected to a database-server). For setting up a fully working version of t
 
 1. Seed database with initial static data
 
-- docker-compose exec ibf-api-service npm run seed
+- `docker-compose exec ibf-api-service npm run seed`
 
 2. Load initial raster data
 
@@ -110,14 +112,6 @@ are connected to a database-server). For setting up a fully working version of t
   - or run for 1 country and 1 disaster-type: http://localhost:3000/docs/#/scripts/ScriptsController_mockDynamic
 - or by having external pipeline make a call to IBF-system
 
-Adding new data
-
-- Any new static data needs to be imported using a seed-script + corresponding
-  TypeORM entity
-- This includes e.g. geojson data
-- The only exception are raster-files, which need to be included in data.zip [here](https://rodekruis.sharepoint.com/sites/510-CRAVK-510/Gedeelde%20%20documenten/Forms/AllItems.aspx?id=%2Fsites%2F510%2DCRAVK%2D510%2FGedeelde%20%20documenten%2F%5BRD%5D%20Impact%2Dbased%20forecasting%2FGeneral%5FData%2FProduction%20Data&p=true&originalPath=aHR0cHM6Ly9yb2Rla3J1aXMuc2hhcmVwb2ludC5jb20vc2l0ZXMvNTEwLUNSQVZLLTUxMC9fbGF5b3V0cy8xNS9ndWVzdGFjY2Vzcy5hc3B4P2ZvbGRlcmlkPTBmYTQ1NGU2ZGMwMDI0ZGJkYmE3YTE3ODY1NWJkYzIxNiZhdXRoa2V5PUFjcWhNODVKSFpZOGNjNkg3QlRLZ08wJmV4cGlyYXRpb249MjAyMS0xMS0yOVQyMyUzYTAwJTNhMDAuMDAwWiZydGltZT1zekJQVnJfSjJFZw)
-  and transfered to all relevant servers.
-
 ### Installation result
 
 These commands will install the IBF-system with listeners at,
@@ -213,123 +207,6 @@ published release. This takes a while (approx 20 mins) to update.
 Please read the contributing guidlelines in the
 [CONTRIBUTING.md](docs/CONTRIBUTING.md)
 
-## Adding a new country
-
-For adding a new country to the IBF-system, a lot of components are already
-generic, and thus automized. But also quit some manual steps are needed at the
-moment. This is intended to be improved in the future. The list below is
-intended to give a full overview. It is not however meant to be detailed enough
-to execute each step, without further knowledge. Ask a developer who knows more.
-
-### Adding country with disaster type _Floods_
-
-NOTE: outdated!! Check with developers first.
-
-1. IBF-API-Service
-   - Users:
-     - Add user for country to src/scripts/users.json
-     - Add country to admin-user
-   - Country
-     - Add country in src/scripts/countries.json
-   - Upload through 'npm run seed' from API-service
-2. Data for database (look at existing countries and files for examples in terms
-   of format)
-   - Save admin-area-boundary file (.shp) for agreed upon admin-level as
-     geojson (with extension .json) with the right column names in
-     `services/API-service/src/scripts/git-lfs/`
-   - Save Glofas*stations_locations_with_trigger_levels*<countryCodeISO3>.csv
-     in the same folder
-   - Save Glofas*station_per_admin_area*<countryCodeISO3>.csv in the same
-     folder
-     - which admin-areas are triggered if station X is triggered?
-     - note: this should include all admin-areas. If not mapped to any
-       station, use 'no_station'
-   - Potentially add extra code in seed-scripts (seed-amin-area.ts /
-     seed-glofas-station.ts / etc.) to process new data correctly.
-   - Run seed script of IBF-API-service
-   - NOTE: we are in a migration, where we want to move new data as much as
-     possible to this new seed-script set up. So also for other data, not
-     mentioned here, the goal is to upload this via seed-scripts as well. Some
-     other data that is not yet included in seed-script
-     - COVID risk data (.csv) > uploaded through specifically created
-       endpoint
-3. Geodata for IBF-pipeline and IBF-geoserver (look at existing countries and
-   files for examples in terms of format)
-   - Save in `services/IBF-pipeline/pipeline/data` in the right subfolder ..
-     - Flood extent raster (for at least 1 return period) + an 'empty' raster
-       of the same exten/pixel size. (.tif)
-     - Population (.tif)
-     - Grassland + cropland (.tif)
-   - When deploying to other environments (local/remote) this data needs to be
-     transfered (e.g. as data.zip through WinSCP or similar)
-4. IBF-pipeline
-   - add countryCodeISO3 to .env (for development settings, replace by ONLY
-     that code)
-   - add country-specific settings to settings.py (e.g. right links to
-     abovementioned data)
-     - with model = 'glofas'
-   - add country-specific settings to secrets.py
-   - add dummy-trigger-station to glofasdata.py with forecast-value that
-     exceeds trigger-value
-   - Run runPipeline.py (`python3 runPipeline.py`) to test pipeline.
-5. Geoserver
-   - Manually create new stores+layers in
-     [Geoserver interface of test-vm](https://ibf-test.510.global/geoserver/web)
-     - flood*extent*<lead-time>\_<countryCodeISO3> for each lead-time
-     - population\_<countryCodeISO3>
-     - grassland\_<countryCodeISO3>
-     - cropland\_<countryCodeISO3>
-   - Test that the specifics layers are viewable in the dashboard now
-   - When done, commit the (automatically) generated content in
-     IBF-pipeline/geoserver-workspaces to Github
-   - This will prevent you from having to do the same for another server, or if
-     your content is lost somehow
-6. IBF-dashboard
-   - Test dashboard by logging in through admin-user or country-specific user
-7. Specifics/Extras
-   - Whatsapp:
-     - create whatsapp group
-     - paste link in IBF-pipeline/pipeline/lib/notifications/formatInfo.py
-   - EAP-link
-     - create bookmark in Google Docs at place where Trigger Model section
-       starts
-     - paste link (incl bookmark) in countries seed-script
-     - paste link (excl bookmark) in
-       IBF-pipeline/pipeline/lib/notifications/formatInfo.py
-   - Logo's
-     - Get logo(s) (.png)
-     - Paste in IBF-dashboard/app/assets/logos + add reference to each logo
-       in countries seed-script
-     - Paste in
-       IBF-pipeline/pipeline/lib/notifications/logos/email-logo-<countryCodeISO3>.png
-     - Upload logo to mailchimp + retrieve shareable link + copy this in
-       IBF-pipeline/pipeline/lib/notifications/formatInfo.py
-   - Mailchimp segment
-     - Add new tag '<countryCodeISO3>' to at least 1 user
-     - Create new segment '<countryCodeISO3>' defined as users with tag
-       '<countryCodeISO3>'.
-     - Get segmentId of new segment
-     - Paste this in IBF-pipeline/pipeline/secrets.py
-   - EAP-actions
-     - Summarize actions from EAP-document + define 1 Area of Focus per
-       EAP-action
-     - Add to API-service/seed-data/EAP-actions.json
-     - run 'npm run seed' from API-service
-
-### Adding country with disaster type _Heavy rainfall_
-
-NOTE: outdated!! Check with developers first.
-
-- Follow the 'flood' manual above as much as possible, with notable exceptions
-- Input data database
-  - Rainfall_station_locations_with_trigger_levels.csv > currently not
-    included in seed-script yet, but manually uploaded (through runSetup.py)
-- Input data pipeline
-  - There is no equivalent input to the flood extent raster. This is created
-    in the pipeline.
-- Add country in IBF-pipeline settings.py with model = 'rainfall'
-- Save geoserver output as rainfall*extent*<leadTime>\_<countrYCode>
-
 ## Glossary
 
 | Term          | Definition (_we_ use)                                                                                     |

diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md
@@ -126,6 +126,130 @@ NOTE: if you're making many datamodel changes at once, or are doing a lot of tri
     - generate migration-script (see above)
     - restart ibf-api-service (like above, to run the new migration-script)
 
+## Adding new data
+
+- Any new static data needs to be imported using a seed-script + corresponding
+  TypeORM entity
+- This includes e.g. csv, json, geojson data
+- The only exception are raster-files, which need to be included in data.zip [here](https://rodekruis.sharepoint.com/sites/510-CRAVK-510/Gedeelde%20%20documenten/Forms/AllItems.aspx?id=%2Fsites%2F510%2DCRAVK%2D510%2FGedeelde%20%20documenten%2F%5BRD%5D%20Impact%2Dbased%20forecasting%2FGeneral%5FData%2FProduction%20Data&p=true&originalPath=aHR0cHM6Ly9yb2Rla3J1aXMuc2hhcmVwb2ludC5jb20vc2l0ZXMvNTEwLUNSQVZLLTUxMC9fbGF5b3V0cy8xNS9ndWVzdGFjY2Vzcy5hc3B4P2ZvbGRlcmlkPTBmYTQ1NGU2ZGMwMDI0ZGJkYmE3YTE3ODY1NWJkYzIxNiZhdXRoa2V5PUFjcWhNODVKSFpZOGNjNkg3QlRLZ08wJmV4cGlyYXRpb249MjAyMS0xMS0yOVQyMyUzYTAwJTNhMDAuMDAwWiZydGltZT1zekJQVnJfSjJFZw)
+  and transfered to all relevant servers.
+
+## Adding a new country (OLD - NEEDS UPDATE)
+
+For adding a new country to the IBF-system, a lot of components are already
+generic, and thus automized. But also quit some manual steps are needed at the
+moment. This is intended to be improved in the future. The list below is
+intended to give a full overview. It is not however meant to be detailed enough
+to execute each step, without further knowledge. Ask a developer who knows more.
+
+### Adding country with disaster type _Floods_
+
+NOTE: outdated!! Check with developers first.
+
+1. IBF-API-Service
+   - Users:
+     - Add user for country to src/scripts/users.json
+     - Add country to admin-user
+   - Country
+     - Add country in src/scripts/countries.json
+   - Upload through 'npm run seed' from API-service
+2. Data for database (look at existing countries and files for examples in terms
+   of format)
+   - Save admin-area-boundary file (.shp) for agreed upon admin-level as
+     geojson (with extension .json) with the right column names in
+     `services/API-service/src/scripts/git-lfs/`
+   - Save Glofas*stations_locations_with_trigger_levels*<countryCodeISO3>.csv
+     in the same folder
+   - Save Glofas*station_per_admin_area*<countryCodeISO3>.csv in the same
+     folder
+     - which admin-areas are triggered if station X is triggered?
+     - note: this should include all admin-areas. If not mapped to any
+       station, use 'no_station'
+   - Potentially add extra code in seed-scripts (seed-amin-area.ts /
+     seed-glofas-station.ts / etc.) to process new data correctly.
+   - Run seed script of IBF-API-service
+   - NOTE: we are in a migration, where we want to move new data as much as
+     possible to this new seed-script set up. So also for other data, not
+     mentioned here, the goal is to upload this via seed-scripts as well. Some
+     other data that is not yet included in seed-script
+     - COVID risk data (.csv) > uploaded through specifically created
+       endpoint
+3. Geodata for IBF-pipeline and IBF-geoserver (look at existing countries and
+   files for examples in terms of format)
+   - Save in `services/IBF-pipeline/pipeline/data` in the right subfolder ..
+     - Flood extent raster (for at least 1 return period) + an 'empty' raster
+       of the same exten/pixel size. (.tif)
+     - Population (.tif)
+     - Grassland + cropland (.tif)
+   - When deploying to other environments (local/remote) this data needs to be
+     transfered (e.g. as data.zip through WinSCP or similar)
+4. IBF-pipeline
+   - add countryCodeISO3 to .env (for development settings, replace by ONLY
+     that code)
+   - add country-specific settings to settings.py (e.g. right links to
+     abovementioned data)
+     - with model = 'glofas'
+   - add country-specific settings to secrets.py
+   - add dummy-trigger-station to glofasdata.py with forecast-value that
+     exceeds trigger-value
+   - Run runPipeline.py (`python3 runPipeline.py`) to test pipeline.
+5. Geoserver
+   - Manually create new stores+layers in
+     [Geoserver interface of test-vm](https://ibf-test.510.global/geoserver/web)
+     - flood*extent*<lead-time>\_<countryCodeISO3> for each lead-time
+     - population\_<countryCodeISO3>
+     - grassland\_<countryCodeISO3>
+     - cropland\_<countryCodeISO3>
+   - Test that the specifics layers are viewable in the dashboard now
+   - When done, commit the (automatically) generated content in
+     IBF-pipeline/geoserver-workspaces to Github
+   - This will prevent you from having to do the same for another server, or if
+     your content is lost somehow
+6. IBF-dashboard
+   - Test dashboard by logging in through admin-user or country-specific user
+7. Specifics/Extras
+   - Whatsapp:
+     - create whatsapp group
+     - paste link in IBF-pipeline/pipeline/lib/notifications/formatInfo.py
+   - EAP-link
+     - create bookmark in Google Docs at place where Trigger Model section
+       starts
+     - paste link (incl bookmark) in countries seed-script
+     - paste link (excl bookmark) in
+       IBF-pipeline/pipeline/lib/notifications/formatInfo.py
+   - Logo's
+     - Get logo(s) (.png)
+     - Paste in IBF-dashboard/app/assets/logos + add reference to each logo
+       in countries seed-script
+     - Paste in
+       IBF-pipeline/pipeline/lib/notifications/logos/email-logo-<countryCodeISO3>.png
+     - Upload logo to mailchimp + retrieve shareable link + copy this in
+       IBF-pipeline/pipeline/lib/notifications/formatInfo.py
+   - Mailchimp segment
+     - Add new tag '<countryCodeISO3>' to at least 1 user
+     - Create new segment '<countryCodeISO3>' defined as users with tag
+       '<countryCodeISO3>'.
+     - Get segmentId of new segment
+     - Paste this in IBF-pipeline/pipeline/secrets.py
+   - EAP-actions
+     - Summarize actions from EAP-document + define 1 Area of Focus per
+       EAP-action
+     - Add to API-service/seed-data/EAP-actions.json
+     - run 'npm run seed' from API-service
+
+### Adding country with disaster type _Heavy rainfall_
+
+NOTE: outdated!! Check with developers first.
+
+- Follow the 'flood' manual above as much as possible, with notable exceptions
+- Input data database
+  - Rainfall_station_locations_with_trigger_levels.csv > currently not
+    included in seed-script yet, but manually uploaded (through runSetup.py)
+- Input data pipeline
+  - There is no equivalent input to the flood extent raster. This is created
+    in the pipeline.
+- Add country in IBF-pipeline settings.py with model = 'rainfall'
+- Save geoserver output as rainfall*extent*<leadTime>\_<countrYCode>
 
 ## ! ! ! DO NOT ! ! !
 

diff --git a/system-design/IBF-system-design.pptx b/system-design/IBF-system-design.pptx
diff --git a/system-design/ibf-system-design.PNG b/system-design/ibf-system-design.PNG