Skip to content

Commit

Permalink
Update notebooks to use Google-hosted assets so nightly builds will s…
Browse files Browse the repository at this point in the history 
…ucceed and readers can run demo without exporting their own assets

PiperOrigin-RevId: 601113331
  • Loading branch information
@jdbcode @copybara-github
jdbcode authored and copybara-github committed Jan 24, 2024
1 parent 446543a commit 08e7b8c
Show file tree
Hide file tree
Showing 3 changed files with 79 additions and 29 deletions.
3 changes: 2 additions & 1 deletion tutorials/imad-tutorial-pt1/index.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,8 @@
},
"outputs": [],
"source": [
"aoi = ee.FeatureCollection('projects/google/imad_tutorial_aoi').geometry()"
"aoi = ee.FeatureCollection(\n",
" 'projects/google/imad_tutorial/landkreis_olpe_aoi').geometry()"
]
},
{
Expand Down
35 changes: 26 additions & 9 deletions tutorials/imad-tutorial-pt2/index.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,16 @@
"## Preliminaries"
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"This tutorial will optionally export assets. Edit the following\n",
"`EXPORT_PATH` variable to specify the location to store the assets.\n",
"All assets that are needed to complete the tutorial are hosted by Earth Engine,\n",
"but if you'd like to display assets that you export, replace paths as needed."
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -365,7 +375,8 @@
"outputs": [],
"source": [
"# Landkreis Olpe.\n",
"aoi = ee.FeatureCollection('projects/google/imad_tutorial_aoi').geometry()\n",
"aoi = ee.FeatureCollection(\n",
" 'projects/google/imad_tutorial/landkreis_olpe_aoi').geometry()\n",
"\n",
"visirbands = ['B2', 'B3', 'B4', 'B8', 'B11', 'B12']\n",
"visbands = ['B2', 'B3', 'B4']\n",
Expand Down Expand Up @@ -529,7 +540,9 @@
"The following code cell is a routine to run the iMAD algorithm as an export task,\n",
"avoiding memory and time limitations in the active runtime. The asset will be exported\n",
"to the location specified by the `EXPORT_PATH` variable defined earlier. It requires\n",
"about 130 MB of space and can take 15 to 20 minutes to complete."
"about 130 MB of space and can take 15 to 20 minutes to complete. Earth Engine\n",
"provides the asset for use in this demo, so it is not required that you run the\n",
"export cell to complete the tutorial."
]
},
{
Expand All @@ -542,7 +555,7 @@
"outputs": [],
"source": [
"#@title Run iMAD algorithm as export task\n",
"def run_imad(aoi, image1, image2, assetFN, scale=20, maxiter=100):\n",
"def run_imad(aoi, image1, image2, assetId, scale=20, maxiter=100):\n",
" try:\n",
" N = image1.bandNames().length().getInfo()\n",
" imadnames = ['iMAD'+str(i+1) for i in range(N)]\n",
Expand All @@ -565,7 +578,6 @@
" niter = result.getNumber('niter')\n",
" # Export iMAD and Z as a singe image, including rhos and number of iterations in properties.\n",
" iMAD_export = ee.Image.cat(iMAD, Z).rename(imadnames).set('rhos', rhos, 'niter', niter)\n",
" assetId = EXPORT_PATH + assetFN\n",
" assexport = ee.batch.Export.image.toAsset(iMAD_export,\n",
" description='assetExportTask',\n",
" assetId=assetId, scale=scale, maxPixels=1e10)\n",
Expand Down Expand Up @@ -594,7 +606,8 @@
},
"outputs": [],
"source": [
"run_imad(aoi, im1.select(visirbands), im2.select(visirbands), 'LandkreisOlpe')"
"asset_path = f'{EXPORT_PATH}LandkreisOlpe'\n",
"run_imad(aoi, im1.select(visirbands), im2.select(visirbands), asset_path)"
]
},
{
Expand All @@ -604,8 +617,10 @@
"id": "21061d67"
},
"source": [
"After the export finishes, the number of iterations and the final canonical correlations\n",
"can be read from properties of the exported image:"
"After the export finishes (if you ran it), the number of iterations and the\n",
"final canonical correlations can be read from properties of the exported image.\n",
"(Earth Engine-hosted assets are imported here, edit the paths if you'd like to\n",
"use your copies)"
]
},
{
Expand All @@ -617,8 +632,10 @@
},
"outputs": [],
"source": [
"im_imad = ee.Image(EXPORT_PATH + 'LandkreisOlpe').select(0, 1, 2, 3, 4, 5)\n",
"im_z = ee.Image(EXPORT_PATH + 'LandkreisOlpe').select(6).rename('Z')\n",
"im_imad = ee.Image(\n",
" 'projects/google/imad_tutorial/LandkreisOlpe').select(0, 1, 2, 3, 4, 5)\n",
"im_z = ee.Image(\n",
" 'projects/google/imad_tutorial/LandkreisOlpe').select(6).rename('Z')\n",
"niter = im_imad.get('niter').getInfo()\n",
"rhos = ee.List(im_imad.get('rhos')).getInfo()\n",
"print('iteratons: %i'%niter)\n",
Expand Down
70 changes: 51 additions & 19 deletions tutorials/imad-tutorial-pt3/index.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,16 @@
"## Preliminaries"
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"This tutorial will optionally export assets. Edit the following\n",
"`EXPORT_PATH` variable to specify the location to store the assets.\n",
"All assets that are needed to complete the tutorial are hosted by Earth Engine,\n",
"but if you'd like to display assets that you export, replace paths as needed."
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -325,7 +335,7 @@
" 'image': image, 'allrhos': allrhos, 'Z': Z, 'iMAD': iMAD})\n",
"\n",
"#@title Run iMAD algorithm as export task\n",
"def run_imad(aoi, image1, image2, assetFN, scale=10, maxiter=100):\n",
"def run_imad(aoi, image1, image2, assetId, scale=10, maxiter=100):\n",
" try:\n",
" N = image1.bandNames().length().getInfo()\n",
" imadnames = ['iMAD'+str(i+1) for i in range(N)]\n",
Expand All @@ -348,7 +358,6 @@
" niter = result.getNumber('niter')\n",
" # Export iMAD and Z as a singe image, including rhos and number of iterations in properties.\n",
" iMAD_export = ee.Image.cat(iMAD, Z).rename(imadnames).set('rhos', rhos, 'niter', niter)\n",
" assetId = EXPORT_PATH + assetFN\n",
" assexport = ee.batch.Export.image.toAsset(iMAD_export,\n",
" description='assetExportTask',\n",
" assetId=assetId, scale=scale, maxPixels=1e10)\n",
Expand Down Expand Up @@ -443,15 +452,15 @@
" coeffs = coeffs.add(ee.List([b, a, rho]))\n",
" return ee.Dictionary({'image': image, 'coeffs': coeffs})\n",
"\n",
"def normalize(ref, target, imadID, pmin=0.9, bandNames=['B2', 'B3', 'B4', 'B8', 'B11', 'B12']):\n",
"def normalize(ref, target, imad_path, pmin=0.9, bandNames=['B2', 'B3', 'B4', 'B8', 'B11', 'B12']):\n",
" '''\n",
" Relative radiometric normalization of target to reference\n",
" using iMAD for 6-band visual/infrared images\n",
"\n",
" Args:\n",
" - ref: Reference image\n",
" - target: Target image\n",
" - imadID: Asset ID of iMAD image\n",
" - imad_path: Asset path of iMAD image\n",
" - pmin: Minimum p-value for identifying no change\n",
" - bandNames: Band names (default: S2 visual/infrared)\n",
"\n",
Expand All @@ -461,7 +470,7 @@
" try:\n",
" # Get iMAD result and mask out water surfaces.\n",
" waterMask = ee.Image('UMD/hansen/global_forest_change_2015').select('datamask').eq(1)\n",
" res = ee.Image(EXPORT_PATH + imadID).updateMask(waterMask)\n",
" res = ee.Image(imad_path).updateMask(waterMask)\n",
" # iMAD image.\n",
" imad = res.select(0, 1, 2, 3, 4, 5)\n",
" # chi-square test statistic.\n",
Expand Down Expand Up @@ -516,7 +525,8 @@
"outputs": [],
"source": [
"# Landkreis Olpe.\n",
"aoi = ee.FeatureCollection('projects/google/imad_tutorial_aoi').geometry()\n",
"aoi = ee.FeatureCollection(\n",
" 'projects/google/imad_tutorial/landkreis_olpe_aoi').geometry()\n",
"\n",
"# Spectral band combinations\n",
"#Sentinel-2\n",
Expand Down Expand Up @@ -555,7 +565,9 @@
"source": [
"Now we will do a relative radiometric normalization of the TOA image to the SR image. This is obviously a constructed example, as the target image, corrected to surface reflectance, is already available.\n",
"\n",
"First we require the iMAD result:"
"First we require the iMAD result which is produced by the following export task. However, Earth Engine\n",
"provides the asset for use in this demo, so it is not required that you run the\n",
"export cell to complete the tutorial."
]
},
{
Expand All @@ -567,7 +579,8 @@
},
"outputs": [],
"source": [
"run_imad(aoi, im1_sr.select(visirbands), im2_toa.select(visirbands), 'MAD_Im1_sr_Im2_toa')"
"asset_path = f'{EXPORT_PATH}MAD_Im1_sr_Im2_toa'\n",
"run_imad(aoi, im1_sr.select(visirbands), im2_toa.select(visirbands), asset_path)"
]
},
{
Expand All @@ -576,7 +589,7 @@
"id": "iBZwDx1WLXYs"
},
"source": [
"After the iMAD export is finished (monitor progress at [https://code.earthengine.google.com/tasks](https://code.earthengine.google.com/tasks)), we can apply the normalization procedure, choosing a minimum p-value of 0.9 (the default) for the no change pixels:"
"After the iMAD export is finished (monitor progress at [https://code.earthengine.google.com/tasks](https://code.earthengine.google.com/tasks), if you ran the export cell), updated the `imad_asset_path` below or continue to use the Earth Engine copy and run the cell to apply the normalization procedure, choosing a minimum p-value of 0.9 (the default) for the no change pixels."
]
},
{
Expand All @@ -589,7 +602,12 @@
"source": [
"target = im2_toa.select(visirbands)\n",
"reference = im1_sr.select(visirbands)\n",
"im2_toa_norm, coeffs, im_stack, noChangeMask = normalize(reference, target, 'MAD_Im1_sr_Im2_toa')\n",
"imad_asset_path = 'projects/google/imad_tutorial/MAD_Im1_sr_Im2_toa'\n",
"\n",
"# Change the iMAD image path as needed. Here we use Earth Engine's asset.\n",
"im2_toa_norm, coeffs, im_stack, noChangeMask = normalize(\n",
" reference, target, imad_asset_path)\n",
"\n",
"# No-change image.\n",
"nc = ee.Image.constant(1).clip(aoi).updateMask(noChangeMask)"
]
Expand Down Expand Up @@ -758,7 +776,7 @@
"id": "TUHFeCKCLXYt"
},
"source": [
"Again we first run the iMAD algorithm on the two images:"
"Again we first run the iMAD algorithm on the two images. (It is optional to run the task in this demo; the asset that is generated is provided by Earth Engine and imported by defaulyt later in the tutorial)"
]
},
{
Expand All @@ -770,7 +788,8 @@
},
"outputs": [],
"source": [
"run_imad(aoi, im1_sr.select(visirbands), im2_toa.select(visirbands), 'MAD_Im1_sr_Im1_toa', maxiter=200)"
"asset_path = f'{EXPORT_PATH}MAD_Im1_sr_Im1_toa'\n",
"run_imad(aoi, im1_sr.select(visirbands), im2_toa.select(visirbands), asset_path, maxiter=200)"
]
},
{
Expand All @@ -779,7 +798,7 @@
"id": "16Pqn574LXYt"
},
"source": [
"Once the export completes (check [https://code.earthengine.google.com/tasks](https://code.earthengine.google.com/tasks)), perform the orthogonal regression on the invariant pixels:"
"Perform the orthogonal regression on the invariant pixels. (The following cell uses Earth Engine's copy of the exported asset, update the path if you completed the export task and prefer to use your copy)"
]
},
{
Expand All @@ -792,7 +811,13 @@
"source": [
"target = im2_toa.select(visirbands)\n",
"reference = im1_sr.select(visirbands)\n",
"im2_toa_norm, coeffs, im_stack, noChangeMask = normalize(reference, target, 'MAD_Im1_sr_Im1_toa')\n",
"\n",
"# Change the iMAD image path as needed. Here we use Earth Engine's asset.\n",
"imad_asset_path = 'projects/google/imad_tutorial/MAD_Im1_sr_Im1_toa'\n",
"\n",
"im2_toa_norm, coeffs, im_stack, noChangeMask = normalize(\n",
" reference, target, imad_asset_path)\n",
"\n",
"# No-change image.\n",
"nc = ee.Image.constant(1).clip(aoi).updateMask(noChangeMask)"
]
Expand Down Expand Up @@ -870,7 +895,7 @@
"id": "779R1GokLXYu"
},
"source": [
"Run the iMAD algorithm:"
"Optionally run the iMAD algorithm or skip for the demo and use Earth Engine's copy of the result."
]
},
{
Expand All @@ -882,7 +907,8 @@
},
"outputs": [],
"source": [
"run_imad(aoi, im_ls8.select(visirbands_ls), im_ls9.select(visirbands_ls), 'MAD_Im_ls8_Im_ls9', scale=30, maxiter=200)"
"asset_path = f'{EXPORT_PATH}MAD_Im_ls8_Im_ls9'\n",
"run_imad(aoi, im_ls8.select(visirbands_ls), im_ls9.select(visirbands_ls), asset_path, scale=30, maxiter=200)"
]
},
{
Expand All @@ -891,7 +917,7 @@
"id": "kdy6BSlBLXYu"
},
"source": [
"Once the export completes (check [https://code.earthengine.google.com/tasks](https://code.earthengine.google.com/tasks)), perform the normalization (or rather, the harmonization):"
"Perform the normalization (or rather, the harmonization). (The following cell uses Earth Engine's copy of the exported asset, update the path if you completed the export task and prefer to use your copy)"
]
},
{
Expand All @@ -904,7 +930,12 @@
"source": [
"target = im_ls8.select(visirbands_ls)\n",
"reference = im_ls9.select(visirbands_ls)\n",
"im_ls8_norm, coeffs, im_stack, noChangeMask = normalize(reference, target, 'MAD_Im_ls8_Im_ls9', pmin=0.9, bandNames=visirbands_ls)\n",
"\n",
"# Change the iMAD image path as needed. Here we use Earth Engine's asset.\n",
"imad_asset_path = 'projects/google/imad_tutorial/MAD_Im_ls8_Im_ls9'\n",
"\n",
"im_ls8_norm, coeffs, im_stack, noChangeMask = normalize(reference, target, imad_asset_path, pmin=0.9, bandNames=visirbands_ls)\n",
"\n",
"# No-change image.\n",
"nc = ee.Image.constant(1).clip(aoi).updateMask(noChangeMask)"
]
Expand Down Expand Up @@ -985,7 +1016,8 @@
},
"outputs": [],
"source": [
"im_imad = ee.Image(EXPORT_PATH + 'MAD_Im_ls8_Im_ls9').select(0, 1, 2, 3, 4, 5)"
"asset_path = 'projects/google/imad_tutorial/MAD_Im_ls8_Im_ls9' # Edit as needed\n",
"im_imad = ee.Image(asset_path).select(0, 1, 2, 3, 4, 5)"
]
},
{
Expand Down

0 comments on commit 08e7b8c

Please sign in to comment.