- Project Description
- Database
- Workflow
- Neural Transfer Style
- My Model
- Results
- Organization
- Next steps
- Links
Art has been a popular form of expression throughout history, capable of evoking emotions, stimulating thought and awakening emotional responses in those who appreciate it. Several studies have proved that art and creativity are attributes that have played a significant role in the evolution and natural selection of human beings.However, understanding how and why paintings evoke emotions is still a challenge.
In this context, emerges the Dat(A)rt project, which aims to explore the interaction between art and the viewer using technology as an ally. The main focus has been the application of image style transfer technique, allowing to blend a classic artwork with a selfie and thus establishing a dialogue between both forms of expression.
The data handled in this project is based on the "WikiArt Emotions" dataset. This dataset was compiled from WikiArt and annotated by crowdworkers for the research paper entitled "WikiArt Emotions: An Annotated Dataset of Emotions Evoked by Art", by Dr. Saif M. Mohammad and Dr. Svetlana Kiritchenko. As a result, 4,019 images were obtained representing 22 categories of four major styles: Renaissance, Post-Renaissance, Modern and Contemporary.
Each artwork in the dataset was annotated by ten crowdworkers, who were presented with a set of 20 emotion words classified as "Positive Emotions", "Negative Emotions" and "Mix/Neutral". The crowdworkers were asked to make annotations based on:
- Scenario I: presenting only the image of the work (without title)
- Scenario II: presenting only the title of the artwork (no image)
- Scenario III: presenting both the title and the image of the artwork
In addition, each individual was asked to rate:
- The artwork on a scale of -3 (strongly dislike) to 3 (strongly like)
- Whether the image shows the face/body of a person/animal
- Whether it is a painting or a different thing (e.g., a sculpture)
Neural transfer is a technique in deep learning and computer vision that blends a content image and a style reference image to generate a new output image that combines features of both. The content image provides high-level information and structure, while the style image captures textures, colors and patterns. These features are extracted using a convolutional neural network to achieve a new image.
A convolutional neural network (CNN) is a type of machine learning algorithm used in image processing and visual recognition. It works as follows:
- Image input: reception of an image as input composed of pixels, which represent the different colors and structures in the image.
- Convolution: application of filters, small matrices on the image that calculate a weighted combination of the pixel values at each position to highlight particular features in the image.
- Feature maps: generation of feature maps highlighting the regions where they were detected.
- Grouping: pooling operation to the feature maps to reduce dimensionality while preserving key features.
- Fully connected layers: use of fully connected layers to classify extracted features and make predictions. These layers take the input features and process them to generate the final network outputs.
- Learning and optimization: adjustment of filter weights and fully connected layers using machine learning techniques. The network is trained by gradually updating the weights to minimize the difference between the network predictions and the actual image labels.
- Prediction and output: prediction and classification of new images based on their training.
My model uses style neural transfer and convolutional neural networks (CNN) to fusion a content image (selfie) and a style reference image (chosen from the dataset based on user sentiments) to generate a new image that combines features from both. In my implementation, I use the open source Tensorflow library, together with the Tensorflow Hub extension, which facilitates the use of pre-trained machine learning models. To improve performance and model loading time, I set an environment variable to load the models in compressed format to allow faster data processing. I also use Google Magenta arbitrary image styling model, which specializes in image styling transfer.
- Tensorflow: open source library used for machine learning tasks and neural network model development.
- Tensorflow Hub: Tensorflow extension that provides pre-trained machine learning models.
- Environment variable: used by TensorFlow Hub to compress the format of the models.
- Google Magenta: arbitrary image styling model.
(video folder)
Notebooks
- Image Extraction: extraction of all the images of the database into folders by artistic movement.
- Data Cleaning: transformation and cleaning of the data.
- MySQL Import: import of the final .csv into MySQL to perform some queries.
- Take Your Picture:capture of a picture with the webcam by using OpenCV.
- Neural Transfer Style: implementation of style transfer between the extracted artworks and the picture captured by the camera by using a TensorFlow.
Data
- WiKiArt Emotions (folder): original .csv files from WikiArt Emotion DataBase.
- Palettes/Movement (folder): .csv files resulting from the color analysis.
- Artwiki_queries (file): .sql file with queries to explore the data.
- WikiArt Emotions Clean (file): final .csv after applying EDA and ETL processes.
- df artist (file): final .csv filtered with the average of positive, neutral and negative emotions by artist.
- df movement (file): final .csv filtered with the average of positive, neutral and negative emotions by movement.
Images
- Color palettes by artistic movement resulting from the color analysis performed.
Pages
- Different pages generated to create the final interface using Streamlit.
src
- Modularized coding and encapsulated in functions without Streamlit.
video
- Video demostration of the final interface.
- Extend the database by collecting information from more artworks, more movements and more authors.
- Experiment with more complex neural networks to improve to generate images of higher quality and accuracy during style transfer.
- Create color palettes by artist to explore new metrics and patterns, such as the evolution of color preferences over time.
- Optimize some redundant code features.
- Consider feedback from real users to identify possible improvements to the interface and functionality.