Robotic Arms in Orchards

Farmers rely heavily on the availability of seasonal laborers to pick and process their crops, however it is becoming more difficult to find people able to do this kind of work. Many laborers working in orchards or fields are undocumented immigrants who have come under greater scrutiny in the past decade. This shortage of workers creates uncertainty for farmers who run the risk of losing their crops to rot if there are not laborers. Companies such as FFRobotics and Abundant Robotics have recognized the opportunity for a robotic system to assist farmers in harvesting their crops, in this case apples [1]. These companies are leveraging the development of highly efficient graphics processing units (GPUs) which allow for the use of machine vision and image processing techniques to recognize objects and their positions in space. Once an apple is located, an end effector picks and places the apple into a bin with varying degrees of success. For this to be a financially viable investment for farmers, the robot needs to operate 24/7 with success rates of greater than 70% of total apples picked. Developers of these mechatronic systems have sought to reduce the noise in the orchard environment by tailoring not only the datasets used to train their deep neural networks (DNNs), but also by restructuring the trees themselves. By making the trees geometrically simpler and accessible to the robotic arm, they have much higher success rates because any noise or obstacles that would prevent the detection of apples in the frame of the camera are reduced. Farmers are seeing the viability in making the investment in these kinds of solutions, and as the cost of these robots are reduced over time, more farmers will be able to participate in making the same investment.

The Abundant Robotics team received large investments in its development program from Google and the Washington Tree Fruit Research Commission. They use several robotic arms all outfitted with vacuum tube end-effectors which attach to the apple and pick it from the tree and place it into a collection bucket. More details of their robotic arm are not provided in the report. The FFRobotics team used a retractable end effector which would grip the apple and rotate it to break the stem from the tree, and then drop it into a bucket. They significantly reduced the degrees of freedom of their robotic arm by mounting it to the back of a moving vehicle. The vehicle would move down the rows of trees and grab apples that the camera would detect, as depicted in figure 1. The image would get processed in the GPU and then a trained deep neural network would compare the image provided to a ground truth data set that the individual weights within the hidden layers of the network were developed from. Once an apple was found within the frame, the location of that apple would be computed and then the robotic arm would be commanded to move to that position and grip the apple. Closed loop control for this operation was considered, however at the time of writing the article, this operation was too computationally expensive, and they felt that the success of open loop control was good enough for practical use.

Figure 1. FFRobotics robotic arm mechatronics diagram.

The FFRobotics team worked hard to ensure that the datasets they used to train their deep neural networks were as accurate as possible in detecting apples in the orchard. They would prepare images of apples from the orchard which it would be deployed in and scan 3D models of the environment to be able to determine the critical points and intersections of the tree. They also employed the use of lighting during night operation and needed enough visibility in order to perform their machine vision computations. They made it a goal to be able to not only pick apples, but other fruits as well using additional image training datasets and creating new DNN weight sets to be able to deploy the robot in other environments. This flexibility of adapting the arm to new environments performing the same task on objects that are not visually the same make the product financially successful. They can move between farms during the off seasons and continuously make use of their technology. It is only a short period of time before farming and other industries look to replace repetitive labor-intensive tasks with robots capable of performing just as well with minimal errors.

References

[1] Tibbets, J. H. (February 1, 2018). “Not too Far from the Tree.” ASME. Mechanical Engineering. February 2018; 140(02): 28–33. https://doi.org/10.1115/1.2018-FEB1