TEDxCentralSaintMartins - Nadia Berthouze - What our Bodies can Teach Computers

so the focus of my talk is a a body and I would like to see body as a mean for emergence and I hope this will come through the talk and so I want to focus

on the doal role of of body and on one side through the body you express your emotion to other probably you can read I'm pretty nervous I've been sitting there all day waiting for this time uh

and the other side it may be less intuitive is that through our body we are talking to us we are affecting out the way we feel we're affecting our

emotions so and I I I want to to talk to about these two aspect and these two two rle of body offer very interesting uh opportunities and challenges to people

who develop technology interactive technology in particular and so now we have technology that sense our bod so this also open into interesting possibilities so we we shake our iPhone

and he knows uh what we want to do uh we have technology probably some of your kids have technology that uh sense your body and you can interact with the

computer game through their your body so can we use all this information that the sensor that are integrating the technology we use every day can cture understand our feelings so going back to

the first talk this morning by Michael can technology go into our shoes and understand better how we feel and support us better or even as a second talk at the end said help us to know

more about ourself and leten to Rehabilitation physical rehabilitation area I'm working in how important is to understand how

the patient field so that we can uh adopt online the the the physical activity need to do the program so that it fits its need and we can also provide

him psychological support and courage uh him boost his com confidence so but why body when we we think to uh emotion and to recognize

emotion in other the first thing that come to our mind is the face is the vocal expression and for sure this a very important Channel and a lot of work showing this but also there are evidence

a growing amount of evidence showing that the body speak as much as the face and this can uh show the uh the complex emotion that uh we go through and here

just two animation that I creating from real data from people we simply ask to uh

walk and need and in what in one case is a uh yeah one person the difference that one person is a patient with is a is a

person with chronic pain or low mood in the other case is a controlled participant and hopefully from this simple representation of the body you

can recognize which one has more controll way of of moving and quite a a

a Down Low mood so can can we get technology to do the same to recognize this cue and then use this CU so that uh

it can support us much better and so this is what we're trying to do in our lab we are using a lot of sensor to continually capture information about our movement our

posture we have sensor place on different parts of the body Limbs and trunk and the head and uh we can capture information that describe the

configuration of the body how bent we are how open our body is uh the our speed the jerkiness of our mood uh

movement but we also have uh we use electromyography sensor here that allow to capture the activity of the muscle

that may form us about the motion the person is going through high contraction or some abnormally patterns so like a Vance to use a part of our body because

we are afraid to increase injury and so using all this information this large amount of information we collect continuously we can use special

Argy that to try to make sense between the Cs of uh in this numbers and F order that uh or regular pattern that uh allow

us to discriminate between different emotional expression and here's some example of the system can find just by looking at

uh the the degree of bending of the top uh of the trunk and the head uh being able to discriminate between High mood and and low mood or also other type of

information in this case is the amount of movement and looking at this over time and how and looking at how the the variability uh of this measure allow us

to to identify very different profile that reflect different state the person may be in and how does this technology perform just by using this law information so

without contextual information and pretty well the the result of our system here are in blue and uh was about recognizing four different emotional

state in the computer game context so naturally IC expression people didn't know what we were doing and we can see that if we aage this result they're

pretty similar to the red line that represent the the agreement obtained by human Observer judging the same type of posture and of course we are not reading

what the person is really feeling we are reading what from what side we can perceive from this uh um configuration and dynamic information and if we

compare this result to Uh current result obtaining in The Challenge on Rec automatic recognition of naturalistic facial expression we can see that the

best results in yellow were pretty similar to the one we obtain uh on just using the body information so quite powerful uh modality for reading how a

person uh feel but this is not all the story because um well we live in a multiculture for example we live in a

multicultural societ so we we we mix up with people from different culture and since we read so much from other people body how sh we that a person from

another culture will read correctly the way uh our body our body expression well uh this um we did some uh study on what

is called to investigate the universality of this body expression and in fact we found a very mixed picture there are um expression that all agree

uh correspond in this case to suddenness this bent body and the arm abandon along the uh along the body uh but that's not

the case for this second set of of posture where uh the Japanese culture that was invited in our study tended to completely separate from Sudden and say

well this is more frustration anger while another group of people from a different culture were seeing this as another way the people could express

sadness I'm not going in this talk to to discuss why this happened on uh is is really true and and but what what is

interesting here is is saying that there is a um a differ in way we read other people body and maybe not just due to culture but it might be other Factor

like even gender or we also behave quite different different situation so a system like I showed before that learn from a set of data and then use what you

learned to uh just to read whoever it may not work as I spent 10 years in Japan and the first time I moved there I was very surprised with certain

expression I was not expecting a response of what I was saying I was doing but little by little I learn to understand that and even I realized that

my body was starting to express the same uh to use the same web language in in a particular situation so through interaction through exposure I learn and

I adopt so can the system do that and through interaction with the person continuesly learning and and and uh get to understand his friends or his user

much better here sorry for the not very nice graph but some uh result we obtain in a study we we run with uh having a a

system interacting with a student during ill learning activity and see if the system starting from zero not knowing anything at this body language and see

if it could develop a code um to to understand what the person was feeling and then use that information to adopt so we could see

that uh in our case in the um the system as soon as we started to see a certain type of of similar expression say oh

this must be a particular type of of State on a particular type of state and then if he was getting feedback would also say oh they say well I'm I'm a little bit down and the results are not

good so let's call this said and then saying oh look another one this is pretty different from the other it must be another type of a faty state and and then if I get Fe back on a look at the

result that must be a good one and and so on so new category where emerging but what's interesting is that if another category was emerging

and course got the feedback of this is called interest but it was somehow interfering with the previous one that I learned because they share a lot of featur that share a lot of Comm so quite

quiet so on the system was started to uh unlearn a little bit what s and and having again to try to understand and find the feature that allow this to

separate and through continous exposure this curve could go up again and continue to grow so it was a continuous learning and un learning to adjust to this person and knowing this person

better so rather than something Pro defined and where even the language was Pro defined by new words could come out it could be even words that didn't make a sense in any of the language we

currently speak so body is quite an important uh interesting modality that can make uh

help us to uh have a a a communication Channel and merge between us and technology and we can communicate of um about our fact we can get technology

to better understand us and better support us and it be a very interesting modality because in some situation we may uh not have FAL and vocal expression

available we think um monitoring people on a large environment outside but there are also situation in which privacy is a concern and maybe FAL expression or not

the best way to to support a person but uh what is more interesting is then to also put all this modality uh

together um but this is only one side of the story uh there is another side of the story as I said at the beginning uh our body doesn't speak only to others

but it speak also to ourselves there is incredbly uh increasing number of evidence that the way we stand the way we move is affecting the way we perceive ourselves

and we perceive the environment uh around us object products and even people and for example staying in a position that is related to uh gener

negative State even if it's been uh imposed on us so it doesn't really represent what we feel in that moment this tend to bring us to develop negative thought about us about our

performance independently of how we are performing the opposite if we our uh body Stace you can try to straight up your body and seeing how this is trying

to B bias you a certain direction does this happen in in when we are interacting with technology and can we exploit this uh mechanism if this is the

case and was um there was one question on the wall was interesting is technology reducing our ability to feel and we saw that probably uh is not like

that but I hope the body can even uh bring us to feel uh emotion uh much more than before when we are interacting with technology not just through technology

here there are two people playing the same game uh the first person is an expert player and you can see is that he knows very

well how to use the Wii he's not using too much a body very control movement to make sure he get the point right no emotion like only at the last moment when he missed the point is this turning

of the face the same game exactly the same a very different style of playing uh is at the beginner but he got a still make a lot

made a lot of points but as you see is involved completely through his body and we can see his face a lot more emotion coming up so there is a seems to be a

different experience that may come through that may emerge through the feedback we get through our body we run different control experiment in which we

put naive uh player to play different game and we impose on them certain movements or to control the game you need to do uh this so for example you

say to fly you need to swing your arm with fluid and large movement and then we say well we change the setting now you have to do uh very jerky and small

movements so combination of different quality of body movement and what in fact we observe some interesting effect and where the quality of the mov where

changing the way um we affecting how people were feeling and their self-report of their emotion at the end of of the game so just to conclude so

the body become an interesting and interesting modality not only for talking and and creating this challenge between us and technology and getting

technology in our shoes and support us but also uh letting the experience is not predefined by who designed that technology but the experience submerge

through the involvement of our body and is also an interesting mean to steer it from a design perspective and I conclude here uh so just one more slide as there

was at the end and here is as I said at beginning we are working in a large project for a physical rehabilitation of people with MUSC skeleton chronic pain

and here the idea is to motivate them to be Physically Active we are using all the modality to read their facial expression in collaboration with

Imperium working on face and uh and voice and we're working on body and for us become interesting to see this movement that relate to hesitation or

stiffness or a synchrony in the way the muscle are user may indicate anxiety or avoidance of doing movement that uh May

according to their thought bring to uh injury and I would like to conclude by thanking uh all people working with me my collaborator my student and also the

other people on the ground [Applause]