To determine the exact color of a wall, a person usually has to take a small paint chip off the wall and bring that chip to a store like Lowes or Home Deport where they use a specialized tool to find the exact color code of the paint chip. The goal of this project is to develop a program that can take an image of a wall and predict the exact color code, saving the user a trip to the store.
Different lighting conditions pose an immediate problem when finding the exact color of a wall from its image. For example, a color that is closest to a light purple color code could appear darker under dark lighting conditions and cause the program to predict a dark purple color code. To remedy this problem, we use a color calibration tool. This tool includes a red circle, a green triangle, and a blue pentagon. The color calibration tool is placed on top of the wall/paint sample, and the square hole in the center of the tool contains the sample whose color code needs to be found. This tool helps with adjusting for different lighting conditions because the program now takes the colors of the red, green, and blue shapes (the reference colors) along with the sample color as input. Going back to the previous example, the software can recognize the difference between a light purple wall under dark lighting conditions and a dark purple wall by checking the reference colors. In the former case, the reference colors will be dark while in the latter case, they will be brighter.
We used Valspar color cards (with associated Valspar color codes) to collect training and testing data. Each color card was placed underneath the color calibration tool and pictures were taken from different angles and under different lighting conditions. Most of these pictures were taken using a smartphone camera. Each image was given a filename that followed the format of “<Valspar_color_code>_<version_num>.jpg” (for example - “1008-5C_2.jpg”)
The model will take 12 numeric inputs - the RGB values of each reference color and the RGB of the observed sample. It will have 3 numeric outputs, which are the RGB values of the color code associated with the sample (found using the Valspar website). The first step in preprocessing the data is to extract the RGB values of the reference colors and the sample from the image. To achieve this, we used YOLOv8. An annotation tool was used to annotate the reference color shapes and the square hole containing the sample. The YOLOv8 model was trained on these annotated samples. Each image has 4 bounding boxes (3 for the reference shapes and 1 for the sample) and the bounding box information for each image file was stored in CSV format. Another script then uses the bounding box details to extract a 20x20 square of pixels from the center of each bounding box and calculates the mean RGB values for each square. This gave us the 4 RGB values (12 numbers in total) that are the features. The outputs, which are the RGB values associated with the color code, are found manually using the Valspar website. The input and output values are combined into a CSV file which acts as the training/testing dataset for the machine learning models. Before the data was split into training and testing sets, it went through some further processing. This involved removing samples with NA values, along with removing anomalous samples where the RGB values of the reference colors were unusual (for example, if the R value for the red circle is less than the G and B values for the red circle, then that sample is discarded).
Since the outputs are numeric and practically continuous, this is a Regression problem. Four Regression models, Linear Regression, Ridge Regression, Decision Tree, and Random Forest, were implemented using Scikit-Learn. Additionally, a Neural Network was implemented using PyTorch and trained on the data. The Neural Network has a structure of 12->128->64->32->3 and uses ReLU as the activation function for each layer, along with an Adam optimizer. The data, which consisted of 2844 samples after data preprocessing, was split into 80% training and 20% testing sets. Two measures are used to measure the correctness of a model on the testing set. The first measure requires that the predicted RGB values produced by the model be associated with a color code. This is done by finding the RGB values of the color code closest (as measured by Euclidean distance) to the predicted output RGB values. The prediction is “correct” if the RGB values of the color code closest to the predicted output are equal to the expected output RGB values of that sample. The first measure, named accuracy, is equal to the number of correctly classified test samples divided by the total number of test samples. The second measure consists of calculating the Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) for the test samples (the error is calculated between the predicted output RGB values and the expected output RGB values).
As a reminder, the ideal model should have a high accuracy and a low MSE. The results are displayed below -
From these results, it’s clear that the Random Forest Regressor was the most successful if we are taking both high accuracy and low MSE into consideration. Decision Tree had a slightly higher accuracy (45.9 vs 42.7) but a much higher MSE (391.9 vs 214.9) compared to Random Forest. Neural Network had a slightly lower MSE (204.4 vs 218.5) but a much lower accuracy (19.7 vs 42.7) compared to Random Forest.
We tried further optimizing the Random Forest, Decision Tree, and Neural Network using Grid Search Hyperparameter Optimization. For Random Forest, the improvement was negligible. For Decision Tree, the MSE decreased from 365 to 317.9, but the classification accuracy also dropped to 31.8%. For Neural Network, the MSE dropped by a significant amount to 175.7 (RMSE was 13.2) and accuracy increased to 36.2%. The Neural Network was optimized using the Optuna library, and the best structure was 12->80->16->3. Overall, the Random Forest Regressor still seems to have the best performance, with the optimized Neural Network coming in second.
The best classification accuracy (45.9% using Decision Tree) was quite poor even though the best RMSE (13.2 using Neural Network) was not that bad. The main reason for this is that the color cards are very close to each other in terms of RGB values, so even a slight error can cause the predicted color to be misclassified. The most straightforward solution to this problem is collecting more data for each color card under different lighting conditions and angles. Additional image processing techniques or data augmentation can also prove to be helpful.
Another area for exploration is the design of the color calibration tool. Perhaps having more reference colors on the tool or printing the tool on a better material (ours was printed on paper) could help. Creating a Valspar version of the Pantone Color Card (which has many of the Pantone colors on the card as a reference) can be useful.