diff --git a/README.md b/README.md index 7a7e8cc742..2c67837cd4 100644 --- a/README.md +++ b/README.md @@ -8,21 +8,23 @@ ## Introduction -**NOTE**: For now some background on IBF-terminology (e.g. triggers) is +**NOTE**: Some background on IBF-terminology (e.g. triggers) is expected. This can be expanded on later. -This is the repository for the IBF-system. It includes 2 main folders. +This is the repository for the IBF-system. It includes a.o.: -1. [Services (backend)](./services/) +1. An [API-service (backend)](./services/API-service/) -- API-service - which accepts input from various IBF-pipelines, that upload impact forecast data to the IBF-system on regular intervals. (See 'Dependencies' below.) - and which lets the IBF-dashboard - or other authorized accounts - retrieve data from the IBF-database. -2. [Interfaces (frontend)](./interfaces/) +2. An [IBF-dashboard (frontend)](./interfaces/IBF-dashboard/) -- IBF-dashboard -- showing all impact forecast data - either leading to a trigger or not - in a visual portal +- showing all impact forecast data - either leading to a trigger or not - in the IBF-portal + +3. A [features](./features/) folder describing all feature scenarios in the IBF-system, using Gherking language. + +4. A [docs](./docs) folder with further documentation beyond this readme. ## Dependencies @@ -31,7 +33,7 @@ This is the repository for the IBF-system. It includes 2 main folders. - See the 510 IBF Project Document for more info and links to the 510-instances of these pipelines per disaster-type. - For development/testing purposes, there are mock-endpoints and mock-data available to replace the need for these pipelines. (See 'Load local database with data' below.) -## System design (draft) +## System design ![IBF-system design (draft)](./system-design/ibf-system-design.PNG) @@ -54,11 +56,11 @@ This is the repository for the IBF-system. It includes 2 main folders. ### Using Docker ``` -docker-compose -f docker-compose.yml up # for production +docker-compose -f docker-compose.yml up -d # for production -docker-compose up # for development +docker-compose up -d # for development -docker-compose -f docker-compose.yml -f docker-compose.override.yml up # for development (explicit) +docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d # for development (same as previous line) ``` @@ -92,7 +94,7 @@ are connected to a database-server). For setting up a fully working version of t 1. Seed database with initial static data -- docker-compose exec ibf-api-service npm run seed +- `docker-compose exec ibf-api-service npm run seed` 2. Load initial raster data @@ -110,14 +112,6 @@ are connected to a database-server). For setting up a fully working version of t - or run for 1 country and 1 disaster-type: http://localhost:3000/docs/#/scripts/ScriptsController_mockDynamic - or by having external pipeline make a call to IBF-system -Adding new data - -- Any new static data needs to be imported using a seed-script + corresponding - TypeORM entity -- This includes e.g. geojson data -- The only exception are raster-files, which need to be included in data.zip [here](https://rodekruis.sharepoint.com/sites/510-CRAVK-510/Gedeelde%20%20documenten/Forms/AllItems.aspx?id=%2Fsites%2F510%2DCRAVK%2D510%2FGedeelde%20%20documenten%2F%5BRD%5D%20Impact%2Dbased%20forecasting%2FGeneral%5FData%2FProduction%20Data&p=true&originalPath=aHR0cHM6Ly9yb2Rla3J1aXMuc2hhcmVwb2ludC5jb20vc2l0ZXMvNTEwLUNSQVZLLTUxMC9fbGF5b3V0cy8xNS9ndWVzdGFjY2Vzcy5hc3B4P2ZvbGRlcmlkPTBmYTQ1NGU2ZGMwMDI0ZGJkYmE3YTE3ODY1NWJkYzIxNiZhdXRoa2V5PUFjcWhNODVKSFpZOGNjNkg3QlRLZ08wJmV4cGlyYXRpb249MjAyMS0xMS0yOVQyMyUzYTAwJTNhMDAuMDAwWiZydGltZT1zekJQVnJfSjJFZw) - and transfered to all relevant servers. - ### Installation result These commands will install the IBF-system with listeners at, @@ -213,123 +207,6 @@ published release. This takes a while (approx 20 mins) to update. Please read the contributing guidlelines in the [CONTRIBUTING.md](docs/CONTRIBUTING.md) -## Adding a new country - -For adding a new country to the IBF-system, a lot of components are already -generic, and thus automized. But also quit some manual steps are needed at the -moment. This is intended to be improved in the future. The list below is -intended to give a full overview. It is not however meant to be detailed enough -to execute each step, without further knowledge. Ask a developer who knows more. - -### Adding country with disaster type _Floods_ - -NOTE: outdated!! Check with developers first. - -1. IBF-API-Service - - Users: - - Add user for country to src/scripts/users.json - - Add country to admin-user - - Country - - Add country in src/scripts/countries.json - - Upload through 'npm run seed' from API-service -2. Data for database (look at existing countries and files for examples in terms - of format) - - Save admin-area-boundary file (.shp) for agreed upon admin-level as - geojson (with extension .json) with the right column names in - `services/API-service/src/scripts/git-lfs/` - - Save Glofas*stations_locations_with_trigger_levels*.csv - in the same folder - - Save Glofas*station_per_admin_area*.csv in the same - folder - - which admin-areas are triggered if station X is triggered? - - note: this should include all admin-areas. If not mapped to any - station, use 'no_station' - - Potentially add extra code in seed-scripts (seed-amin-area.ts / - seed-glofas-station.ts / etc.) to process new data correctly. - - Run seed script of IBF-API-service - - NOTE: we are in a migration, where we want to move new data as much as - possible to this new seed-script set up. So also for other data, not - mentioned here, the goal is to upload this via seed-scripts as well. Some - other data that is not yet included in seed-script - - COVID risk data (.csv) > uploaded through specifically created - endpoint -3. Geodata for IBF-pipeline and IBF-geoserver (look at existing countries and - files for examples in terms of format) - - Save in `services/IBF-pipeline/pipeline/data` in the right subfolder .. - - Flood extent raster (for at least 1 return period) + an 'empty' raster - of the same exten/pixel size. (.tif) - - Population (.tif) - - Grassland + cropland (.tif) - - When deploying to other environments (local/remote) this data needs to be - transfered (e.g. as data.zip through WinSCP or similar) -4. IBF-pipeline - - add countryCodeISO3 to .env (for development settings, replace by ONLY - that code) - - add country-specific settings to settings.py (e.g. right links to - abovementioned data) - - with model = 'glofas' - - add country-specific settings to secrets.py - - add dummy-trigger-station to glofasdata.py with forecast-value that - exceeds trigger-value - - Run runPipeline.py (`python3 runPipeline.py`) to test pipeline. -5. Geoserver - - Manually create new stores+layers in - [Geoserver interface of test-vm](https://ibf-test.510.global/geoserver/web) - - flood*extent*\_ for each lead-time - - population\_ - - grassland\_ - - cropland\_ - - Test that the specifics layers are viewable in the dashboard now - - When done, commit the (automatically) generated content in - IBF-pipeline/geoserver-workspaces to Github - - This will prevent you from having to do the same for another server, or if - your content is lost somehow -6. IBF-dashboard - - Test dashboard by logging in through admin-user or country-specific user -7. Specifics/Extras - - Whatsapp: - - create whatsapp group - - paste link in IBF-pipeline/pipeline/lib/notifications/formatInfo.py - - EAP-link - - create bookmark in Google Docs at place where Trigger Model section - starts - - paste link (incl bookmark) in countries seed-script - - paste link (excl bookmark) in - IBF-pipeline/pipeline/lib/notifications/formatInfo.py - - Logo's - - Get logo(s) (.png) - - Paste in IBF-dashboard/app/assets/logos + add reference to each logo - in countries seed-script - - Paste in - IBF-pipeline/pipeline/lib/notifications/logos/email-logo-.png - - Upload logo to mailchimp + retrieve shareable link + copy this in - IBF-pipeline/pipeline/lib/notifications/formatInfo.py - - Mailchimp segment - - Add new tag '' to at least 1 user - - Create new segment '' defined as users with tag - ''. - - Get segmentId of new segment - - Paste this in IBF-pipeline/pipeline/secrets.py - - EAP-actions - - Summarize actions from EAP-document + define 1 Area of Focus per - EAP-action - - Add to API-service/seed-data/EAP-actions.json - - run 'npm run seed' from API-service - -### Adding country with disaster type _Heavy rainfall_ - -NOTE: outdated!! Check with developers first. - -- Follow the 'flood' manual above as much as possible, with notable exceptions -- Input data database - - Rainfall_station_locations_with_trigger_levels.csv > currently not - included in seed-script yet, but manually uploaded (through runSetup.py) -- Input data pipeline - - There is no equivalent input to the flood extent raster. This is created - in the pipeline. -- Add country in IBF-pipeline settings.py with model = 'rainfall' -- Save geoserver output as rainfall*extent*\_ - ## Glossary | Term | Definition (_we_ use) | diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 35189d9336..2ab05c115d 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -126,6 +126,130 @@ NOTE: if you're making many datamodel changes at once, or are doing a lot of tri - generate migration-script (see above) - restart ibf-api-service (like above, to run the new migration-script) +## Adding new data + +- Any new static data needs to be imported using a seed-script + corresponding + TypeORM entity +- This includes e.g. csv, json, geojson data +- The only exception are raster-files, which need to be included in data.zip [here](https://rodekruis.sharepoint.com/sites/510-CRAVK-510/Gedeelde%20%20documenten/Forms/AllItems.aspx?id=%2Fsites%2F510%2DCRAVK%2D510%2FGedeelde%20%20documenten%2F%5BRD%5D%20Impact%2Dbased%20forecasting%2FGeneral%5FData%2FProduction%20Data&p=true&originalPath=aHR0cHM6Ly9yb2Rla3J1aXMuc2hhcmVwb2ludC5jb20vc2l0ZXMvNTEwLUNSQVZLLTUxMC9fbGF5b3V0cy8xNS9ndWVzdGFjY2Vzcy5hc3B4P2ZvbGRlcmlkPTBmYTQ1NGU2ZGMwMDI0ZGJkYmE3YTE3ODY1NWJkYzIxNiZhdXRoa2V5PUFjcWhNODVKSFpZOGNjNkg3QlRLZ08wJmV4cGlyYXRpb249MjAyMS0xMS0yOVQyMyUzYTAwJTNhMDAuMDAwWiZydGltZT1zekJQVnJfSjJFZw) + and transfered to all relevant servers. + +## Adding a new country (OLD - NEEDS UPDATE) + +For adding a new country to the IBF-system, a lot of components are already +generic, and thus automized. But also quit some manual steps are needed at the +moment. This is intended to be improved in the future. The list below is +intended to give a full overview. It is not however meant to be detailed enough +to execute each step, without further knowledge. Ask a developer who knows more. + +### Adding country with disaster type _Floods_ + +NOTE: outdated!! Check with developers first. + +1. IBF-API-Service + - Users: + - Add user for country to src/scripts/users.json + - Add country to admin-user + - Country + - Add country in src/scripts/countries.json + - Upload through 'npm run seed' from API-service +2. Data for database (look at existing countries and files for examples in terms + of format) + - Save admin-area-boundary file (.shp) for agreed upon admin-level as + geojson (with extension .json) with the right column names in + `services/API-service/src/scripts/git-lfs/` + - Save Glofas*stations_locations_with_trigger_levels*.csv + in the same folder + - Save Glofas*station_per_admin_area*.csv in the same + folder + - which admin-areas are triggered if station X is triggered? + - note: this should include all admin-areas. If not mapped to any + station, use 'no_station' + - Potentially add extra code in seed-scripts (seed-amin-area.ts / + seed-glofas-station.ts / etc.) to process new data correctly. + - Run seed script of IBF-API-service + - NOTE: we are in a migration, where we want to move new data as much as + possible to this new seed-script set up. So also for other data, not + mentioned here, the goal is to upload this via seed-scripts as well. Some + other data that is not yet included in seed-script + - COVID risk data (.csv) > uploaded through specifically created + endpoint +3. Geodata for IBF-pipeline and IBF-geoserver (look at existing countries and + files for examples in terms of format) + - Save in `services/IBF-pipeline/pipeline/data` in the right subfolder .. + - Flood extent raster (for at least 1 return period) + an 'empty' raster + of the same exten/pixel size. (.tif) + - Population (.tif) + - Grassland + cropland (.tif) + - When deploying to other environments (local/remote) this data needs to be + transfered (e.g. as data.zip through WinSCP or similar) +4. IBF-pipeline + - add countryCodeISO3 to .env (for development settings, replace by ONLY + that code) + - add country-specific settings to settings.py (e.g. right links to + abovementioned data) + - with model = 'glofas' + - add country-specific settings to secrets.py + - add dummy-trigger-station to glofasdata.py with forecast-value that + exceeds trigger-value + - Run runPipeline.py (`python3 runPipeline.py`) to test pipeline. +5. Geoserver + - Manually create new stores+layers in + [Geoserver interface of test-vm](https://ibf-test.510.global/geoserver/web) + - flood*extent*\_ for each lead-time + - population\_ + - grassland\_ + - cropland\_ + - Test that the specifics layers are viewable in the dashboard now + - When done, commit the (automatically) generated content in + IBF-pipeline/geoserver-workspaces to Github + - This will prevent you from having to do the same for another server, or if + your content is lost somehow +6. IBF-dashboard + - Test dashboard by logging in through admin-user or country-specific user +7. Specifics/Extras + - Whatsapp: + - create whatsapp group + - paste link in IBF-pipeline/pipeline/lib/notifications/formatInfo.py + - EAP-link + - create bookmark in Google Docs at place where Trigger Model section + starts + - paste link (incl bookmark) in countries seed-script + - paste link (excl bookmark) in + IBF-pipeline/pipeline/lib/notifications/formatInfo.py + - Logo's + - Get logo(s) (.png) + - Paste in IBF-dashboard/app/assets/logos + add reference to each logo + in countries seed-script + - Paste in + IBF-pipeline/pipeline/lib/notifications/logos/email-logo-.png + - Upload logo to mailchimp + retrieve shareable link + copy this in + IBF-pipeline/pipeline/lib/notifications/formatInfo.py + - Mailchimp segment + - Add new tag '' to at least 1 user + - Create new segment '' defined as users with tag + ''. + - Get segmentId of new segment + - Paste this in IBF-pipeline/pipeline/secrets.py + - EAP-actions + - Summarize actions from EAP-document + define 1 Area of Focus per + EAP-action + - Add to API-service/seed-data/EAP-actions.json + - run 'npm run seed' from API-service + +### Adding country with disaster type _Heavy rainfall_ + +NOTE: outdated!! Check with developers first. + +- Follow the 'flood' manual above as much as possible, with notable exceptions +- Input data database + - Rainfall_station_locations_with_trigger_levels.csv > currently not + included in seed-script yet, but manually uploaded (through runSetup.py) +- Input data pipeline + - There is no equivalent input to the flood extent raster. This is created + in the pipeline. +- Add country in IBF-pipeline settings.py with model = 'rainfall' +- Save geoserver output as rainfall*extent*\_ ## ! ! ! DO NOT ! ! ! diff --git a/system-design/IBF-system-design.pptx b/system-design/IBF-system-design.pptx index 584231f60c..d1df1adbd7 100644 Binary files a/system-design/IBF-system-design.pptx and b/system-design/IBF-system-design.pptx differ diff --git a/system-design/ibf-system-design.PNG b/system-design/ibf-system-design.PNG index 1ec522042b..db17ddc060 100644 Binary files a/system-design/ibf-system-design.PNG and b/system-design/ibf-system-design.PNG differ