Using OpenCV and Tensorflow to identify a Boggle board in an image and decode what letters are on the board.
BoggleCV is deployed on my online boggle site, try it here! You can take a picture of a boggle board with your phone, or upload an image.
Here are the various stages of the computer vision and machine learning pipeline, with a sample image.
|Image with Box Drawn||Mask to Find Blue Pixels|
|Corner Finding||Warped Image|
|Row/Column Sum||Letters Cut Out|
#convolutional (2-dimensional) layers model = models.Sequential() model.add(layers.Conv2D(64, (3, 3), activation='relu', input_shape=(IMG_DIM, IMG_DIM, 1))) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(32, (3, 3), activation='relu')) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(16, (3, 3), activation='relu')) #data is flattened from 2 dimensions to 1 model.add(layers.Flatten()) #a fully-connected layer is added for more flexibility in combining features model.add(layers.Dense(100, activation='relu')) #the final layer has 1 output for each letter of the alphabet, and they are expressed as percentages that it thinks it might be of each letter. usually, one letter is close to 100% and the rest are close to 0%, which means it is very sure of its decision. But if it is less sure of a result, the number would be lower, e.g. 80%, and it might have 1 or 2 other options that were 15% and 5% for example. This information can be used, possibly to ask the user for a better quality image, or a confirmation that the letter is correct. model.add(layers.Dense(26, activation="softmax"))
Note: steps 8-10 only have to be done one time to set up the pipeline.
Right now, the model has about 88% accuracy on images it has never seen before. This is a pretty good start, but I am planning to improve it by deploying this to my site where people can play boggle, and create a system for uploading images and correcting any mistakes that the model makes. Then I will take this new data and re-train the model to achieve higher accuracy.