Cuphead CV

This project was originally created and developed in May 2022.

Currently, this project features a computer vision program that captures the Cuphead game window in real-time and detects objects in the game.

For that purpose, I fine-tuned the YOLOv5 model to detect objects from the game. I created a dataset with images of the game and annotated them using labelImg.

The program uses win32gui to record the game window. The code to record the game window was based on the following YouTube playlist: OpenCV Object Detection in Games.

Model in Action

Here are some videos showing the computer vision model in action.

• Model detecting objects in a pre-recorded video: YouTube
• Model detecting objects in real-time: YouTube

Jupyter Notebooks

These Jupyter Notebooks contain details of how the project was developed and explain how the Python code inside /src works.

• notebook.ipynb: info about how the project was developed and how to use some tools.

• yolo_training.ipynb: quick tutorial on how to train your own dataset for YOLOv5.

Dataset and Model

Link to the dataset and the model used in this project: Dropbox

Next Steps

• Create an adapter to send keystrokes to the game.

An adapter was created to automate sending keystrokes to the game. It was developed using the PyAutoGUI library. That adapter can be used to control the game using an automated algorithm.

• Train a Machine Learning algorithm to play the game.

A next step is to train a Machine Learning algorithm to play the game. Maybe a good path would be to use a Reinforcement Learning algorithm (Deep Q-Learning) to train an agent to play the game.