CottonSense: A High-Throughput Field Phenotyping System for Cotton Fruit Segmentation and Enumeration on Edge Devices

Phenotype measurements of a cotton plant (such as height and of a number of cotton bolls) are currently carried out by hand. This project's goal is to allow for automatic detection and measurement of those features. To accomplish its goals, the project implements image processing and machine learning techniques. The images are captured by cameras mounted on a robot or a tractor.

Solution: A graph-based SLAM system to 3D scan, segment, and count the cotton plants and their fruits using RGB-D mapping, point cloud processing techniques, and instance segmentation models.

This work is published at Computers and Electronics in Agriculture

Citation

If this repository contributed to your work please consider citing the associated paper

@article{BOLOURI2024108531,
  title = {CottonSense: A high-throughput field phenotyping system for cotton fruit segmentation and enumeration on edge devices},
  journal = {Computers and Electronics in Agriculture},
  volume = {216},
  pages = {108531},
  year = {2024},
  issn = {0168-1699},
  doi = {https://doi.org/10.1016/j.compag.2023.108531},
  url = {https://www.sciencedirect.com/science/article/pii/S0168169923009195},
  author = {Farshad Bolouri and Yildirim Kocoglu and Irish {Lorraine B Pabuayon} and Glen {Lorin Ritchie} and Hamed Sari-Sarraf}
}

Data Acquisition

For this phase an ROS based SLAM System was designed which was assembled on a tractor.

Here's a link to a video demonstrating an instance of the scanning sessions being operated on ROS:

Here's some images of the Data Acquistion phase:

Scanning during the day with Natural Lighting	Scanning at night with Artificial Lighting

And here's some instances of PointClouds obtained from a row during one of the scanning sessions.

PointCloud Obtained from the Top Camera (Zoomed In):

PointCloud Obtained from the Bottom Camera:

The PointClouds being aligned with a Colored ICP algorithm in Python using the Open3D library:

Fruit Segmentation and Enumeration

2D

These videos demonstrates our Cotton Fruit Segmentation and Enumeration system. An instance segmentation model was designed and trained to segment the cotton fruits in 4 different stages of growth. Also, a custom tracking algorithm was implemented to correctly enumerate the number of fruits in each growth-stage throughout the video.

The bounding-boxes are only shown for the objects that are being tracked. These tracked objects are eventually counted (for the right class) if the required conditions are met.

Video captured and processed Mid-Season:

Video captured and processed Late-Season:

3D

3D Segmentation by Projection

3D Detection of OpenBolls using Image Processing

You can see how the algorithm is detecting the Open Bolls even when it is surrounded by branches and leaves:

And here's some snapshots of the detection in 3D from different angles: