With ART.E.C. (Augmented Reality Trough Emotional Computing) we aimed to discover how an interactive art museum experience could change by introducing an emotional factor. Through the use of emotion recognition models and bio-metric sensors, we built an art installation that would react to the spectator’s emotion, by changing the appearance of the painting thanks to the power of AR.

HOW EMOTIONS PLAY IN ART EXPERIENCES?

We were tasked to find a novelty way to put emotion recognition models in art experiences, finding ways in which Since the beginning we decided that our objective would be to enhance the artworks while maintaining the power of communication of the original artwork, making little changes in the compositions, colors, or adding sounds with the scope of not overcoming the artworks, as it was already in the scope of the artist to makes the viewers reason and feel their message.

---

DEVELOPMENT

We started by testing and collecting emotional responses to artworks by measuring the facial changes and the BPMs of our test subjects, and then we used this data, together with already existing and open-to-public facial expressions image databases, to mash together our emotion recognition model using Teachable Machine.

The emotion caught by the camera would then be compared with the BPM levels read through a ring wearable, that would connect via Bluetooth to the phone.
The AR app was built with the Unity engine using Vuforia, which we set on 4 paintings that are present in the Museo del 900 collection in Milano.