AI studies storytelling in movies to learn about emotions

0
16


Machines could soon be writing the latest Hollywood blockbuster, thanks to a breakthrough in teaching AI about narrative arcs in storytelling.

Experts trained machine learning algorithms to recognise positive and negative emotions through thousands of hours of video footage, including Pixar’s Up.

By the end of the process, the neural networks were able to predict an audience response to a film on a second by second basis.

The technique could be used to create AI systems that are capable of generating their own tear-jerkers.

Scroll down for video 

Machines could soon be writing the latest Hollywood blockbuster, thanks to a breakthrough in teaching AI about narrative arcs in storytelling. Experts trained  algorithms to recognise positive and negative emotions through thousands of hours of footage, including Pixar's Up

Machines could soon be writing the latest Hollywood blockbuster, thanks to a breakthrough in teaching AI about narrative arcs in storytelling. Experts trained algorithms to recognise positive and negative emotions through thousands of hours of footage, including Pixar’s Up

ANALYSING PIXAR’S UP USING AI

As part of its training, the AI looked at the famous opening sequence of Up, a 3D computer-animated film that was a critical and popular hit in 2009.

The movie focuses on Carl Fredricksen, a grumpy senior citizen who attaches thousands of balloons to his house in a quest to fly to South America after his wife, Ellie, dies. 

Wanting to devote most of the movie to Carl’s adventure, the screenwriters had to come up with a quick way to provide the complicated back story behind his trip. 

That’s where the opening sequence comes in. 

It’s silent, except for the movie’s score, and an emotional arc emerges as scenes of Carl’s life play on the screen.

The algorithm was able to map out this arc, rating scenes portrayed for their negative or positive emotional impact.

The software was developed by experts from MIT’s Media Lab as part of its Storytelling project.

This uses machine-based analytics to identify the qualities of engaging and marketable media. 

By developing models with the ability to ‘read’ emotional arcs and narrative content, researchers aim to map story structures across different genres and media.   

The team created machine-learning models that use deep neural networks to ‘watch’ small slices of video, including movies, TV, and short online features found on Vimeo, to rate their emotional content.

The models consider all aspects of a clip, not just the plot, characters, and dialogue but also more subtle touches, like a close-up of a person’s face or a snippet of music that plays during a car-chase scene.

When the content of each slice is considered in total, the story’s emotional arc emerges, in a process the team terms ‘visual valence.’

In a written statement, a spokesman for the MIT team said: ‘Was it possible, our team asked, that machines could identify common emotional arcs in video stories.

‘The typical swings of fortune that have characters struggling through difficult times, triumphing over hardship, falling from grace, or declaring victory over evil?

‘If so, could storytellers use this information to predict how audiences might respond?

‘[We found] machines can view an untagged video and create an emotional arc for the story based on all of its audio and visual elements. 

‘That’s something we’ve never seen before.’

As part of its training, the AI looked at the famous opening sequence of Up, a 3D computer-animated film that was a critical and popular hit in 2009.

The algorithm was able to map out this arc, rating scenes portrayed for their negative or positive emotional impact. The high and low points of the montage are marked on this graph. The higher the score, the more positive the emotion

The algorithm was able to map out this arc, rating scenes portrayed for their negative or positive emotional impact. The high and low points of the montage are marked on this graph. The higher the score, the more positive the emotion

The algorithm was able to map out this arc, rating scenes portrayed for their negative or positive emotional impact. The high and low points of the montage are marked on this graph. The higher the score, the more positive the emotion

The movie focuses on Carl Fredricksen, a grumpy senior citizen who attaches thousands of balloons to his house in a quest to fly to South America after his wife, Ellie, dies. 

Wanting to devote most of the movie to Carl’s adventure, the screenwriters had to come up with a quick way to provide the complicated back story behind his trip. 

That’s where the opening sequence comes in. 

It’s silent, except for the movie’s score, and an emotional arc emerges as scenes of Carl’s life play on the screen.

The algorithm was able to map out this arc, rating scenes portrayed for their negative or positive emotional impact. 

The technique could be used to create AI systems that are capable of generating their own tear-jerkers

The technique could be used to create AI systems that are capable of generating their own tear-jerkers

The technique could be used to create AI systems that are capable of generating their own tear-jerkers

One of the highest peaks corresponds to images of Carl as a happy child, for instance, but there’s a big drop shortly after, when young Ellie scares him in the middle of the night

To measure the system’s accuracy, the MIT team asked volunteers to annotate movie clips with various emotional labels. 

They were also asked to identify which video element, such as dialogue, music, or images, triggered their response and these insights were used to refine the AI models.

By comparing all of the emotional arcs uncovered by their AI, the researchers found a simple formula followed by most movies and short films for their emotional arc.





Source link

LEAVE A REPLY

Please enter your comment!
Please enter your name here