OxRAM-Robot-Arm: Chess Piece Recognition (my first Machine Learning project)

Hi, this is one task within the Project OxRAM-Robot-Arm: https://github.com/OxRAMSociety/OxRAM-Robot-Arm

My background is materials science. I am focus on electrochemicamistry. So no robot background. I will make this blog easy to read for everyone. So if I can get this work, you can get it work. :)

Story

Yeah, to increase the readability, let's start from a story. I am an materials scientist working on energy storage. I am familiar with beakers and lab work. My only experiences with coding is learning C++ years before to pass the exam... Zero experiences of Machine Learning and Robot. Happy.

I started to learn 3D printing in 2019 and was very lucky to get access to various of 3D printers (Form2, MarkForged M2, UltimakerS5, Photocentric, Solidscape). I spent about half a year to learn each of them from scratch. Two FMD. One SLA. One DoD. One DLP. I got familiar with different filaments, resins and print heads. Each printer has its own advantages and limitations (avoiding the word 'disadvantages'). Then I was able to fix some problems of them and do the maintainess. Also able to match the application with printers (from materials and resolution point of view). A bit proud.

Then 2020 was a bit special. I had more time to hug my computers and explore coding. From MatLab to Python, step by step, all small tasks. In 2020, I gained more programme skill.

So in the skill set pocket, there is a '3D printing'. There is a 'programming'. Let's try to find some interesting application to merger them together and learn from practice! Yeah! ROBOT!

Then I found the university society https://github.com/OxRAMSociety and this OxRAM-Robot-Arm project. I love this team. All team members are very patient with all my questions. I started my very first Robot and Machine Learning journey.

1. Objective

This project is to build a robotic arm using open source softwares and 3D printed parts. The first application is to play chess. One of the first steps for the arm is to recognize a chess piece via machine learning techniques. Which come to the theme of this Blog (finally).

2. Lable images (LabelImg)

1. The first step is to label the images. The package that I am using is LabelImg (https://github.com/tzutalin/labelImg) on Mac.

The annotation was saved in a txt. file in the same folder. All images were labeled in the same way. 2. Save all the labeled image and annotation in a folder 'images' and compress to 'images.zip' 3. Upload this 'images.zip' file to my google drive under folder named 'yolov3

3. YOLOv3

Then I use YOLO to achieve the machine learning.Follow the instruction in https://pysource.com/2020/04/02/train-yolo-to-detect-a-custom-object-online-with-free-gpu/

4. Trials on Pawn recognition

1) First test on 30 images

Believe me, it is always good to start from a small amount of data to try and get a feeling of the workflow. I got 30 images. Lable them one by one. And then did the training for half an hour. After some debug, there is no error message anymore! Although it was only able to recgnize the pawn in a easy image as the following.

2) Increase the size of testing data set to 67 images

It seems that the previous methodology works. Now let's increase the size of testing data set to 67 images, and train the model for 6 hours. The results show both success and fail cases. Success cases:

1. All the image that with only a pawn can be recognized

2) Pawns in different focal plane can be recognized

3) Pawns in different tilting angle can be recognized

4)Pawns from different camera angles can be recognized

5)A pawn among other chess pieces can be recognized

Fail cases:

1)Can't recognize a pawn behind other object

2)In some cases, can't recognize a pawn in a different focal plain

3)Can't distinguish a pawn that the top share the same color with the bottom, when it is close with other chesspiece.

4)Can't recognize a pawn when it is decorated with a crawn to the top.

Pacially seccess: In an image with darker light, and different focal plan, only some of the pawn can be recognized.

Always success when the top part could distinguished from the bottom via color.

Always fail in some cases (the light is less real)

Conclusions:

1. It is recommended to design/3D print a chess piece that the top is distinguished from the bottom
2. From the failed cases, it is possible that the bottom of the pawn is not necessary to be labeled in the 1st step.

3) Lable to top of the chess piece only (63 training images and 4 testing imzges)

In this trial, I only label the top of the pawn.

Following the same training workflow, the learning results are much more reliable. All the previous failed cases are successed this time.

Interestingly, although the pawns with a different focal plain can be recognized, the middle pawn with a crown is still challenging.

Testing images (the machine never see these images before):

Conclusion:

1. To label the top only helps the following training
2. The training results is good enough for the robot arm to recognize the pawn in a real life image.

Limitation:

Lack of training images. All the images are online resources. Although I achieve a good training model with only 63 images, more training image will defenitly increase the stability and realiablity in following steps.

5. Trials on Knight recognition

To apply the previous methodology on Knight recognition, 167 images are used as training data. The results shown that it is important that the image has a high-enough resolution. Which relates to the camero and reaction spead of the carema that attached on the robot arm.

(to be continued...) (it is just a draft for me to remember what has been done/fun. Bear with typos and etc. please...)

So, here is my first success on Machine Learning. The purpose of this work is to suppor the robot arm project...