Python 3 package for finding swallow roost rings on US weather surveillance radar
Melinda Kleczynski, Katie Bird, Chad Giusti, Jeffrey Buler
If you're not familiar with roost rings and would like a fun introduction, you can read about birds on weather radar.
There are many excellent resources describing the use of weather radar data for monitoring birds and other animals in the airspace. I won't list all of them here, but I'll point out three in particular which were important in the development of this package.
The paper Unlocking the Potential of NEXRAD Data through NOAA’s Big Data Partnership discusses efforts to make the necessary data openly available.
The paper The Python ARM Radar Toolkit (Py-ART), a Library for Working with Weather Radar Data in the Python Programming Language describes the Py-ART package, which is instrumental to this work.
The paper bioRad: biological analysis and visualization of weather radar data discusses extracting biological information from this type of data.
Click on "Code" (near the top of the GitHub page, in green) and select "Download ZIP." Get the RoostRingSearch.py file. For example, if you're running a Jupyter notebook, move RoostRingSearch.py to the same folder as your notebook.
Make sure any packages listed under "Dependencies" are installed.
Run
import RoostRingSearch as rrs
and you should be all set!
If you're following along with the code in this readme, you'll also need to run
import numpy as np
import datetime
If you plan on using Google Colab, run the following line to install any packages that aren't automatically included.
pip install --upgrade arm-pyart astral boto3 haversine matplotlib timezonefinder
Then restart the runtime. Upload the RoostRingSearch.py file to "Files" on the left side of the page. Run the imports listed above.
The two main functions are morning_exodus (checks several scans) and find_roost_rings (checks a single scan). Let's find some roost rings! Run the following (which will take a minute or two):
morning_df = rrs.morning_exodus('KDOX', [2020, 8, 23])
morning_df
If everything is set up correctly, you should see something like this:
RoostRingSearch checked all the available scans from the KDOX radar during a window of time on the morning of August 23, 2020. It found four distinct roost rings and reported the latitude and longitude of the center of each ring, as well as the scan on which each ring was first detected.
The first two columns are the latitude and longitude of the center of the roost ring.
center (latitude)center (longitude)
The next two columns are the coordinates of the center of the roost ring in relation to the array indices (in case you want to plot the ring center over the array).
center (array x)center (array y)
The next four columns are the x and y limits to use to plot the part of the array containing that roost ring. This includes some extra space around the ring.
min xlimmax xlimmin ylimmax ylim
The next column is the time of the reflectivity scan when the ring was first detected on the given day.
first detection
The next four columns contain the information to run the morning_exodus function.
station nameyearmonthday
The next column contains the scan prefix, which can be used to run the find_roost_rings function.
scan prefix
The last four columns are the parameters used to run the algorithm.
cutoff_distancemin_reflectivitymax_background_noisemin_signal
Running the morning_exodus function like this is great if you just want the final results. If you want to check multiple days and/or stations, you can concatenate the DataFrames and save the results. If you're interested, RoostRingSearch can also display and/or return more information for further investigation. The functions find_roost_rings and morning_exodus both have the option to select display_output = True in order to see more information.
morning_exodus basically provides a framework for collecting the appropriate data, running find_roost_rings for each scan, and aggregating the results. Let's use the scan prefix information in morning_df to run find_roost_rings for each of the scans that morning_exodus flagged.
for scan_prefix in np.unique(morning_df['scan prefix']):
rrs.find_roost_rings(scan_prefix, display_output = True)
RoostRingSearch isn't perfect - it missed some roost rings on the middle scan! But it did a better job on the next scan. RoostRingSearch will make some mistakes, but it is a helpful tool if you would like an overview of roost ring locations without having to check all the scans manually.
Consider the scan whose prefix is 2021/08/10/KDOX/KDOX20210810_1015.
To check this scan for roost rings, run
rrs.find_roost_rings('2021/08/10/KDOX/KDOX20210810_1015', display_output = True);
RoostRingSearch looks for roost rings in the reflectivity data. There is a lot of material on the reflectivity scan besides just roost rings. Some preprocessing helps clean up the array. RoostRingSearch performs the following preprocessing steps:
- Screen out low reflectivity
- If the clutter_filter_power_removed field is available, use it to screen out clutter
- If the cross_correlation_ratio field is available, use it to screen out precipitation
Here's what the reflectivity, clutter_filter_power_removed, and cross_correlation_ratio look like for this scan:
RoostRingSearch can look for roost rings using only reflectivity data, but it doesn't perform as well. For example, precipitation can form shapes that are close enough to rings to be flagged as possible roost rings.
Once the reflectivity array is ready, RoostRingSearch uses linear filters to check for roost rings. Here's an example of what one of the filters looks like:
Here's the same filter shown over the reflectivity array for scale:
The search for roost rings involves comparing the filter to different parts of the processed reflectivity array. If there is enough reflectivity in the blue region of the filter and not too much reflectivity in the orange regions of the filter, there might be a roost ring there. There is some space between the blue and orange regions of the filter to help allow for roost rings which are irregularly shaped. This process is repeated for several filters since roost rings have varying widths and sizes.
find_roost_rings has some optional inputs. Four of these are parameters which control how the function decides whether or not it found a roost ring. These parameters and their default values are
cutoff_distance = 150
min_reflectivity = 0
max_background_noise = 0.05
min_signal = 0.3
We'll discuss how each of these work.
This parameter controls the region of the original scan which find_roost_rings checks for roost rings. The arrays are centered around the radar, and extend cutoff_distance km the the left (west), right (east), up (north), and down (south). A roost ring generally can't be identified unless the full circle which would be formed by the ring lies completely within the array. For example, with the default value cutoff_distance = 150 the roost ring in the lower left corner of the following scan does not appear as a potential roost ring. (As a side note, the following line of code demonstrates an alternate input format for find_roost_rings.)
rrs.find_roost_rings(('KDOX', datetime.datetime(2019, 8, 6, 10, 35, tzinfo = datetime.timezone.utc)), display_output = True);
To find that roost ring, increase the default value of cutoff_distance to 200:
rrs.find_roost_rings(('KDOX', datetime.datetime(2019, 8, 6, 10, 35, tzinfo = datetime.timezone.utc)), display_output = True,
cutoff_distance = 200);
Increasing the cutoff_distance creates larger arrays, which can increase the run time. Also, the data quality starts to decrease for large values of cutoff_distance. This scan demonstrates that if you increase the cutoff_distance to 250, the cross_correlation_ratio isn't available for the whole array. RoostRingSearch uses the cross_correlation_ratio to identify precipitation, so there can be more false positives due to precipitation in the corners of scans if you choose a cutoff_distance of 250 km.
A scan such as this one can have a lot of precipitation, and still have a roost ring.
rrs.find_roost_rings(('KDOX', datetime.datetime(2021, 8, 4, 9, 41, tzinfo = datetime.timezone.utc)), display_output = True);
Recall that one of the preprocessing steps is to screen out low reflectivity. The min_reflectivity parameter determines how low reflectivity needs to be in order to be removed from the array. Let's look at an example, starting with the default value of min_reflectivity = 0. Reflectivity values can be negative, so that's a nontrivial requirement. find_roost_rings only finds one of the roost rings.
rrs.find_roost_rings(('KDOX', datetime.datetime(2021, 10, 1, 11, 20, tzinfo = datetime.timezone.utc)), display_output = True);
Increasing the min_reflectivity to 10 finds two of the other rings instead, because they have higher reflectivity in the ring region and the noisy reflectivity in the background is screened out.
rrs.find_roost_rings(('KDOX', datetime.datetime(2021, 10, 1, 11, 20, tzinfo = datetime.timezone.utc)), display_output = True,
min_reflectivity = 10);
Running the function multiple times with different values of min_reflectivity would find more roost rings, but at the cost of increased processing time. This could also increase the number of false positives (identifying something as a potential roost ring when it's really not a roost ring).
Remember the linear filter? A roost ring should not have too much reflectivity in the orange regions. The max_background_noise parameter determines how much reflectivity is allowed. In this scan, the roost ring doesn't register with the default max_background_noise = 0.05 because there is too much reflectivity around it.
rrs.find_roost_rings(('KDOX', datetime.datetime(2021, 8, 7, 9, 40, tzinfo = datetime.timezone.utc)), display_output = True);
find_roost_rings is able to find this roost ring when max_background_noise = 0.15. Note that increasing the max_background_noise makes it easier for a region of a scan to count as a roost ring, so in general there will be more false positives.
rrs.find_roost_rings(('KDOX', datetime.datetime(2021, 8, 7, 9, 40, tzinfo = datetime.timezone.utc)), display_output = True,
max_background_noise = 0.15);
Finally, the min_signal parameter determines how much reflectivity is needed in the blue region of the filter. Here's an example, starting with the default value min_signal = 0.3. There is a roost ring, but it doesn't have enough positive reflectivity for find_roost_rings to find it.
rrs.find_roost_rings(('KDOX', datetime.datetime(2021, 7, 25, 9, 59, tzinfo = datetime.timezone.utc)), display_output = True);
Decreasing the min_signal parameter to 0.25 relaxes the requirements enough to find this roost ring. As before, this makes false positive results more likely.
rrs.find_roost_rings(('KDOX', datetime.datetime(2021, 7, 25, 9, 59, tzinfo = datetime.timezone.utc)), display_output = True,
min_signal = 0.25);
Run ?rrs to see some basic information about RoostRingSearch, including a list of all the functions that are available. You can also get information about any function. For example, running ?rrs.find_roost_rings summarizes the find_roost_rings function.
Kleczynski, M., Bird, K., Giusti, C., & Buler, J. (2022). RoostRingSearch (Version 1.1) [Computer software]. https://github.com/makleczy/RoostRingSearch
@software{RoostRingSearch,
author = {Kleczynski, Melinda and Bird, Katie and Giusti, Chad and Buler, Jeffrey},
month = {12},
title = {{RoostRingSearch}},
url = {https://github.com/makleczy/RoostRingSearch},
version = {v1.1},
year = {2022}
}