Stanford Seminar - Generative, Malleable, and Personal User Interfaces

so I'm very excited to be here to share the research we have been doing in the foundation interface lab atsd uh I've been very fortunate to work very closely with my uh wonderful colleague Jim

Holland and a bunch of my amazing PhD students so uh in our lab you know the overarching research question we are exploring is what are the foundational

principles of a productive and conval information environment and in this environment we hope human cognition digital computation and information representations can be seamlessly

integrated and for the past few years of course a more concrete question uh not surprisingly is how will AI transform our information activity and

environment and the title of this talk has already given the answer away right so we think the generative malleable and personal user interface is a very promising Direction and we are not the

only group of people thinking about this and as I would try to uh point out this vision for such interface has been uh pursued or uh being pursued for decades

in our community so the goal of my talk is sort of try to uh share with you guys what we have been imagining uh what the what's the look and feel of such interface and what is our current

approach towards it so if you think about you know the types of using different ways of using AI in our information activity there are

three main types Right One is using these model to generate specific type of content and this way of using AI can speed up the process of information activity by supplying us the content we

need another way of using AI is through these chat Bots right uh aside from you know refine your writing or help you uh draft an email uh one thing that's really interesting is how it can help

with exploratory information activities such as understanding a new domain through a query series of Q&A now of course chatbot is very limited right because many complex information

activities require diverse sources of information and different functionality and that's why probably the agentic workflow uh seems just more compelling

right anthropic releases computer use uh uh demonstration and recently open AI uh operator and these AI agents can sort of receive a user's request and then

manipulate the apps or browser on your computer and I think this just a remarkable Tech achievement but I think this approach has a serious problem and if we are

going down this path we will not not be able to fully leverage human cognition and AI to uh achieve productive information activity now before I take a HCI

perspective on this let's take a historical view on this Pro or this phenomenon right because I think the computer use demonstration and the open AI operator is the perfect example of

what we used to say or Mark used to say history doesn't repeat itself but often rhes so the Stanford audience should know this example really well you know uh back in the first Industrial

Revolution right the steam engine was the main power source right and each Factory will have you know they had this one steam engine that drive the entire Factory so basically through the axles

or P you know on the ceiling and also the you know P you know the belts and the pulley this one steam engine is able to drive all these individual machines on the floor and uh so when you're

designing the factory you have to consider how the power is delivered you know the machine that needed the most power must be positioned very close to the power source because if it's very far you know the axle will be broken

right because of torque so basically the layout of the factory is uh designed to accommodate the power delivery mechanism and electricity drove the

Second Industrial Revolution right and so what people did uh so Eric bson and Andrew mccuff found that people simply just bought the largest electric motors and replace the steam engine they just

put it where it used to be right they didn't change the factory layout they didn't change how things are manufactured and as a result the productivity bely improved for you know

over 30 years and it was only things were only changed when new layout was uh designed or new manufacturing workflow were introduced and the productivity

skyrocketed so what's so critical about this layout well the layout or the new workflow meets the natural workflow of the

material so let's take a look at our graphical user interface now you know if just for the sake of metaphor let's consider this application-based graphical user interface to be the first

UI Revolution right and with this kind of design we are the power source that drive the each individual app to achieve

our information activity right now um and if you actually compare this with the fact it's quite similar right each individual app is kind of individual machine right and they work together to

help you achieve the uh activity and instead of just manufacturing physical uh materials we're producing you know digital materials so what are we doing now with

AI well those computer demonst you know computer use demonstration is basically trying to replace human with AI right now ai is the power source that drive

these uh machines so we're substituting human with AI now if we take a page from history we probably should think about how we should redesign this information

environment right and actually you don't need to take the page from history like there has been a lot of evidence in HCI that argues the uh application Centric information Paradigm is not very

conducive to complex information activity so bony Nardi who studied extensively how human uses computer to accomplish information activity summarized really well she said the

traditional information information uh interface paradigm does not support the complexity flexibility or sociality of human activity and argues that we should move beyond the rigidity of separate

applications or you know file and folder system now to figure out how we should really move Beyond I think we need to answer three questions one is what is the natural workflow of information

activity right and the second is why has the application Centric information interface Paradigm fail to achieve this workflow and third is what's this time

with AI right so let's dive into these questions now uh there has been a lot of work in hcii that study how human engage with information activity and hats off

to you know these seminal Works uh and if I ask you maybe one phrase you will come up to describe the workflow of information activity could maybe is it's this iterative process of collecting

processing and editing information so I think this statement is true but not very helpful uh for two reasons first I think the word iterative which we often

use in HCI is really sugarcoating the complexity of human information activity um so and also like I don't think it really reveiews the key reason why we

have to go through this messy process so I think the human information activity or you know some people have also have this thought is that the human information activity is very Dynamic

it's actually a very chaotic process right uh you you know you need to find all kinds of information sources try to determine which one is useful or not and then you know revise your goal you know form new understanding and then go find

other things and I think another aspect that we should talk more is personal and what I mean by that is you know when you engage with information activity you

have a specific goal right you may are trying to look for a specific piece of information and it's so unique that no single source of information out there will meet your goal right so you have to find you know a lot different sources

try to piece things together to uh meet your information need and same for function you know applications right you may want to process information or see information in a certain way but there has been there's no applications out

there that meet your specific need so if we step back a little bit and look at how we uh there is what the mismatch is so basically in an

application or maybe a web page even is a specific system that developed by A specific group of developer they somehow develop this understanding of how they

think user uh you know what the user would need and they're they're correct for most part and they propose this specific user interface that contains specific information and provide specific

functionality a user and their very specific context they may actually have a slightly different goal right they may want a slightly different interface and they may want different pieces of information on

functionality so this is also true just for you know just collecting information right no you know you may the author of their AR article couldn't predict right what kind of information needs the

readers will want so uh David carer phrases really well he said that no developer can predict all the ways a user will want to record annotate and manipulate

information right so here's an example for you guys so this is the aab BNB app you know I use it a lot to find a place to stay when I travel now when I look

for places to stay I care about both the price and the rating right but if you look at the interface there is no rating on the map right for me to see the rating I have to click on each

individual one and then the detail view will show the map like show the rating right and you may say well you know there's a hard design tradeoff there's not much space on the uh map we can't

display too much fine okay well another factor that I consider is I usually want to travel with my dogs I have two dogs so I want to make sure the place I'm going whether they're pad friendly so I

can consider whether I will bring my dogs right but to find that detail on this interface right now you can't right the developer decides that for this entity the information that user should

see is the you know price rating number of Reviews free canel whether it's free cancellation or not with the style of the room right but I just care about whether it's Pat friendly so I have to dive into this detail get into another

detail page right to be able to find that uh feature so this is what I'm talking about right I have a yes you have a question is it okay yes so I'm curious

how you see filters then so you so here yeah I understand your question it's like what if I can select a pet friendly as a filter right but I haven't decided whether I will because I need to consider all these factors right so what

I kind of want is whether there's a place maybe has a doc friendly icon right there could be places that do not have it right but if you use the filter all the other things are filtered out right so this is exactly what I'm

talking about you know the software yes they provide some kind of ways for you to get there but not exactly what you want right so here's another example

this maybe uh so like the developer of this website right determines what information you need right as a student you may curious about the faculty's office hour you may care about their

research area well you have to dive into each individual one to find that detail right so let's zoom out a little bit right I think uh our Pioneer doas inar

Point out this problem really really well is that the argumentation means may you know if we sort of see the information system we sort designed the system based on our

understanding of a large statistical population uh and no attempts has have been made to fit the individual needs and to develop system that really I mean he was trying to augment human intellect

the system should provide opportunity for significant adaption of the individual uh to the individual's characteristics now I think one thing that's really interesting is he point

out this uh problem before digital application were even invented right before even personal computer and I think fundamentally this really isn't an HCI problem it is kind of an economic

problem right it is just more economic to abstract the common needs of a large population and develop you know reusable solutions for them instead of trying to develop a really customize a solution

for each individual user and it's really for this reason that there's no grant writing or research proposal app there's no family travel app there's no

organizing or dinner party app because everybody does it differently right it's just not e like economic to make an app that way so then what do we have well we

have rigid apps that developer think okay here's how you should use it or we have bloated apps right one way to support all the users need just like let's dump all the things they could you

know uh be potentially uh need into one app and then becomes a super complex app that you cannot use so that's at the application Level what about the activity level right if

you think about how app is developed the developer can you know they prioritize optimizing the user experience within the app right but the operating system who's supposed to provide more structure

for us to coordinate our activity they you know what do they provide virtual desktop or you know another just like uh so but then the user actually right when they need to work on these information

activity they need to coordinate a lot of you know applications so for example research find that data science workers need to juggle 13 categories of applications in their daily work now and

also like when I'm working on my research Grant well I don't have an app right I have a lot of apps and you know I have to juggle all these things to achieve my activity same for organizing a dinner

party right and you may use a different set of app right and that's the whole point we all have our own workflow so at the end of the day you know we suffer from very fragmented

workflow right and really challenging information management and just on a daily basis we have Messi desktop and tab explosion so I think this is the problem

of the application Centric interface Paradigm we have rigid containers and Bloated containers we have fragmented workflow and challenging information management but now we have ai right

things should be better so let's take a look so last week last weekend I got my hands on to open AI operator I pay $200 to be able to use it and so the you know

I was trying to test with a you know case of or organizing a dinner party because it was the Chinese New Year um so I opened it here's uh you know sort of like my interaction with it so I

start with I'm thinking about organizing a dinner party so I reply with these chat messages like here are the things you should think about I'm like cool that's useful you know I probably will

start think you know about the menu so I you know followed up with uh operator and you know give it a more PR I mean operator asked me about my preferences

so I replied and then it will it give me this uh initial manual well so you know I was curious right so I was like okay give me the recipes of these manual then the operator you know

goes ahead and does its kind of job right it opens this browser already and so it will just you know copy the text from the chat search the recipes online

and you know this is sped up like five times okay so uh you know just doing all these browsings and then at you know after I think five minutes or maybe four

minute it returned and a s this list of uh menus and recipes with links right so I'm just I wasn't very happy with the menu to be honest but I was just trying

to keep it going to see you know where things will break right um so I asked it to generate a list of ingredients and I also want it to help me check whether these ingredients are in store for

example in Costco or you know hofus so I basically give this prompt and then um I you know went to grab lunch and by the time I come back it was still working

and I so bad I was like think about the trees that are being burned by uh the operator and the water that being used to cool it uh so I mean it was just running it's just so long so I had to

pause it so otherwise maybe just run forever so uh I paused it um and so I looked into this right so what what was the problem so I opened the uh drop down

list where you can see all its operations so here is just a tiny bit of it and I mean we're all from computer science you know the one of the reason things don't end it's just somehow got

into a weird Loop right so basically operator somehow start to Loop Itself by over this subset of of ingredients uh for unknown reasons so you know I pause

it then I just you know whatever I move on right so the next thing I ask is you know I'm want to get recommendations for a few bottle wines because I want to you know treat my guest well and replied am

I able to help with this I mean the first time I like wait wait what's going on why and here's my resp response it's response um so I was like I I think I understand why I was but then there's

nothing I can do about it right it just tell you gave me the same uh response so I accept the reality and then I move on so uh the next thing I asked is like

well let's move on to the activity after dinner you know any recommendations and then operator become cheerful again and then give me you know recommendations about board games or whatever so I also had a few other interactions with it

it's like you know what board games you know would be suitable for group of 10 you know how do I set a doodle up to you know coordinate with them right so after 4

hours uh I still wasn't able actually to get to the dinner plan that I was happy with but I was very exhausted uh because I have to do a bunch of screen recordings and all lot of stuff and waiting and then checking and I think

maybe the open AI operator was exhausted too so here's what the interface look like when I stopped right uh what did you notice

I still got a browser exploded with tabs right and there is also this very very very long chat history so then where is my dinner plan

well there's no concrete form of it right it's somewhere in this chat it's somewhere in all these tabs right and this interface is kind of the same as

before right it doesn't have much meaningful information organization uh which make it really hard for me to overview manage or iterate on this

activity so wouldn't be great if I have this environment where it can aggregate only the relevant information and functionality but also provide really

effective organization and representation of overall activity so this is what the HCI Community has been calling the activity centered interface Paradigm or activity centered

Computing now before a large language model this was actually really Tri because to achieve this really personal environment you need to be able to create environment that support any

user's information activity they may of be be of any domains any complexity and their goal may change just during the activity right they may you know the goal May Shi like

shift so how do we approach maybe with help with AI well the current approach is let's just generate the sequence actions right on top of the existing

Paradigm what we were trying to think is would it be possible to actually generate a good environment and then have the actions running on top of it so

uh we took a spec uh this specification based approach basically when a user give a prompt like I'm thinking about organizing a dinner party it will first generate what this uh what kind of

organization you may need for organizing a dinner party and then it will map those things to a UI and then the UI will be rendered as this

environment so let me give you some example right imagine I'm thinking of organizing a dinner party you know you may care about people location time right and the UI specification will tell

you or guide you gaide you guide the system like okay what's the level of detail what's the representation and what's the interface organization that you may want to use and then it will

generate you know put the right elements together now you may have new needs or new desire for your activity and if you have Fallout prompts it will sort of try to change the activity specification

like you know dishes ingredients or you know food restrictions when you are thinking about the menu right and then the activity environment will change so you have this graphical user interface

that uh based activity environment so basically this pipeline uh works this right when the user have a prompt and their data the activity specification

will guide the system like okay what are the information entities we provide what is the structure among the this activity and the UI specification will basically

guide the system to choose the right UI and then compose everything together so um on top of that we also want user to be able to directly

manipulate this activity specification so that I can further adjust the UI based on my needs because I know AI will make errors right and I also want to be able to adjust the UI right so I want to be able to adjust the overall

information environment but I also want to be able to adjust individual UI elements and then eventually hopefully I will have generative maleable and personal

activity environment so does this work um so you know we have this pipeline uh and we're like you know will it actually generate something useful so my PhD

student Rema uh build this system jelly which explores this uh Pipeline and you know the efficacy of this pipeline so I'm going to play a 4minute video of

some a user using jelly to uh achieve information activity so let's take a look let's look at how Millie used the system jelly to help her relocate for a

new job in San Francisco she opens jelly I'm moving to San Francisco however jelly thinks she's trying to figure out a moving plan so

provides her with moving services and a budget she deletes the fields she doesn't need and further prompts I'm finding a place to live tell me

about different [Music] areas I've heard some of them could be

[Music] sketchy to support mely reviewing how safe each area is jelly adds the safety level and relevant information to each

area to better compare these areas she transformed the structure into a table and ranks the areas by safety level she definitely doesn't want to

live in this area and this commute time is also an important factor for her she tells jelly my company is on Market Street how does a commute look

[Music] like jelly ask her company in the home panel and the commute time for each area in the table it also suggests the two

areas with the shortest commute time seeing this Mee Narrows her option down to Knob Hill and Mission District she then ask Jelly what are the

apartments in those areas jelly then presents available apartments on the map with the details relevant to her interests such as safety level and commute

time after some exploration Mee has her short list of Apartments a few days later the hiring manager tells Mee that she also has the option to work at the New York office so

she opens jelly and changes the location from San Francisco to New York everything updates automatically and she can reuse the structures for her decision-

making Mee decides that she prefers the San Francisco office after she moves in she wants to throw a dinner party to celebrate her new job and new

home she tells jelly I'm hosting a dinner party with my friends instead of directly generating the space jelly asks who are you going

to invite and then present her contact list M selects Alice Mike and Sarah jelly then asks when is the party and present a calendar looks like next

Friday night would be a good time with these details jelly then generates a home panel displaying the guests menu and activities she reviews the dishes

suggested for her and realizes some of them are not suitable Alice and I are both vegan to accommodate this jelly adds a dietary preference to all the guests

updates the existing dishes and adds a dietary suitability property for all the dishes with the connections established by the underlying schema jelly updates the dishes that don't meet the

restrictions Millie deletes some of them and ask for more options seeing these she also realizes Sara prefers gluten-free

[Music] food having figured out the dishes Mee asks I'm new here where to buy all the ingredients jelly generates a shopping

plan which contains a list of ingredients she needs to buy it also shows the stores on the map and Millie can review the items that she needs to buy at each [Music]

store while Mee is shopping she receives a phone call from Mike asking whether he can bring his friend divid of course she adds divid to the guest list and the dishes and ingredients are

updated accordingly looks like it will be a fun party [Music] all right so that's the jelly system like I know maybe the text is too small

the video is probably too fast this thing will be online very soon then you can take a look later but I just want to sort of step back and sort of compare these different approaches right you

know the current approach is try to generate these sequence of actions right on top of existing interface and at the end you have this really messy information environment and you don't know where your activity really is

whereas our approach is trying to generate the activity environment right using this underlying activity specification and the AI can automate things on top of this environment and

the user can engage with the environment directly to uh sort of achieve their information needs so uh let's break down the system just a little bit uh so to help you

understand maybe like how the system work and there's a bunch of details in the paper but basically uh this interface has you know three views right there's this activity specification the

user prompt region and the generated interface and this is just the design of our demo system right in the future system you can imagine the activity specification doesn't need to always be there right it can be hidden from the

user or the prompt region can just be one small text box or through voice input right so the UI will just be the generated UI and so the system work this

way right when I have a a prompt it will generate this activity specification it's just the model of the activity right the overall structure and you know based on that we'll generate the a

graphical user interface so the way we generate things is BAS based on the type of data in the uh in the activity specification so for example for

Destination I will generate a tax box for uh moving date because of time I will give a you know dat picker for the different areas because the list it's array of uh data right I'm going to

provide a list and here each area is mapped to the individual item in the list and the apartments within the area are shown on on a map right because I have the geographical location of them

and each individual apartment will be shown twice right you know one as this card and then be able to expand to show more details so this is sort of roughly how

the system connect all the parts together but basically the idea is that you know we have this generative and manable activity model right that drives

the generation of the interface now I want to note that the data that the system use right now is generated by a large language model and uh we do not

advocate for this because the data will have errors right we're what we really want to focus on is just generating the overall specification itself and in the future we can base on this specification

and use the correct apis to pull in more reliable data and another part that jelly briefly touches on is these UI specification

right and uh you know what we did is we have this simple set of UI mapping rules and of course well that rule is small enough for us to T like test the pipeline but definitely has its

limitations right if you look at the UI right this UI do not have the most effective organization of the different panels right uh also like here you know

the user can click on each item and then see the details but it has the same problem as what I was showing in the Airbnb example right uh what about the apartment type what about the moving date what about whether it's pad

friendly I want to be able to see these things on this uh each little card here so clearly uh there are some limitation of the Jolly system oh here's another one right what if I don't want the map

to be open on the side what if I want it to be open in place maybe just uh because I'm operating on different screen or maybe this is my preference

right so what we really want is a very expressive UI specification that can describe uh really high quality really

diverse UI and this specification needs to be both AI or purple so that AI can you know specify here is the right UI configuration for you to have the UI but

also enduser manipulable so that I can adjust errors that AI makes so how do we approach this thing

right like how do we describe the like unlimited like the just like numerous uis out there so uh we are quite

inspired by the work that done by uh uh Christopher Alexander's work on pattern language for uh architecture and this work this line of work has actually huge impact on software engineering and

already on user interfaces you know people have been coming up design patterns for developing softwares and good user interfaces so what is a pattern well according to Alexander's notion each

pattern describe the problem which occurs over and over again in an environment and then describe the core solution uh uh to the the problem and in such a way that you can sort of use a

solution a million times over without doing it twice the same way that's super appealing right I mean that's what we want we want have some way of generating UI but we want it to be able to adapt to

each individual user so how do we like achieve that well he said well you need to identify the invariant in this problem right you want

to identify the common properties among these problems uh and then you know based on those properties you can vary a bunch of things so in HCI as I mentioned

we have been trying to uh sort of build upon this notion of design patterns and we have a lot of work in this area you know even like practitioners like this is Apple's design guide for you know

these are the patterns that like apple promotes but traditionally these patterns need to be carried out by human designers right what we want is we want this pattern to be uh AI operable and an

user manipulable so what are all the patterns out there like you know if we can identify all these patterns and make them more uh operationalizable uh potentially we can

uh achieve this technical pipeline right so we looked into one specific pattern which we call uh which is the overview detail pattern so this is one uh project

we Al recently did so if you don't know about the overview detail pattern is this pattern that's just super simple it has two views one overview which shows a bunch of information entities and a

detail view which give you the you know uh detail information about What entity but it serve this very essential need when you're in an information activity right you want to browse a large amount

of things and then you want to dive into specific ones and for this reason the overview detail pattern just everywhere right when you shop on Amazon the search without returns overview you that into

it you get a detail view Airbnb is a overview detail uh pattern follows that pattern right the map is an overview the each card is a detail but there is more detail views behind if you click into it

booking a flight your email uh interface is also an overview right you click into it you get into detail or this is the Google Scholar or Google search page right it returns all the results and

then you know you can dive into specific detail so all these interfaces as I said right it uses a single instantiation of the pattern right you canot do sort of

adjust it well some sites give you a little bit customiz ibility but not fully but this single instantiation is not going to meet everyone's needs right

so what we did is like okay what what if we can make this pattern malleable so Brian L uh my PhD student L this project and we're trying to follow you know

Christopher Anderson's notion maybe first we should identify what are the common properties right so what we did is we uh collected 156 uh diverse

websites and then we try to identify the instances over overview detail pattern right and you see like we were able to identify more than 300 from 156 website

that means just this in this like pattern is just really ubiquitous and we distilled three key Dimensions the first one is layout uh basically tells you what are different layout in the

overview or what's the layout between overview and detail view the composition which is really the mapping of data across this different views and the content right what kind of

content will be shown in the overview and what kind of content will be shown in the detail view now based on this uh we were able to develop a system or several demonstration system to show

whether we can uh manipulate the configuration of overview detail interface so let's take a look at some example applications so this is a

shopping uh uh application that we build with this malleable uh overview detail interface so typically it works the same way as you eBay or Amazon right you click into one you will be able to see

the details and then you can go back and you find another one you see the details here so here I'm trying to look for uh the properties of couch well when I buy

couch the most important attributes I care is the dimensions right the width height and length because I want to see whether it fit into my room so here basically in each uh detail page I'm

looking for these three attributes but you know clicking into each IND individual page to see these attributes is really tedious right why not just show it in the overview so what I can do

is I can activate this U attribute mode and I can select the attributes that I care and I say surface this in the overview and in the overview now I have these attributes so this way I don't

need to get into a bunch of these like items to figure out whether the dimension fits so in addition to add details to the overview you can also remove details

you don't care even though the developer things you should care uh and you can just hide them right you can remove it from your overview page now you know I still need to sort

of like open these uh individual items to see other attributes maybe texture or whatever and it's still quite tedious what I can do is I can adjust the configuration to have things side by

side so I can easily go back and forth right and if I have details that I want to add I can just you know drag the attributes from the detail view into the overview so here I'm just dragging the

couch into the overview so this way I can just browse the overview really quickly all right so what I'm doing here

I'm just trying to find okay interesting couches I guess okay so all right what's next well yeah sometimes like the attributes you care you know the system may not have it

right for example I care about the total cost uh but then the system doesn't have that attribute but I can prompt a I say I I want this total cost attribute so what it can do is can generate this new

attribute by combining several existing attributes so here uh it also shows the attributes it uses to you know uh put together this uh attribute and all the

AI generate attributes will have this Mark indicate this AI generated error may exist right so I can also change the overview layout based on the attributes

I'm interested in so here this is the attribute of the sell's review count so if I select that now because it's a single numerical value the system

recommend hey do you want to view this actually in the histogram or if I can keep go you know going and select other attributes and now because I select two uh you know numerical value it recommend

a scatter plot now I can you know activate this scatter plot and then be able to see these uh things in a different Arrangement now I can like I also care about the color of the couch right but

then it doesn't have that particular attribute so I can prompt AI say well I'm being a nerd here I was like what's the hex value of the couch well so you know what AI can do is you know it can

look at the text maybe there is a color attribute or it can look at the image you know generate the color uh property and with the color property uh I can you know select it and then say I want to

show it in a color space now this allow me to browse all the couches you know based on color so another thing that is really

annoying uh when I'm shopping for things is you know there is this inconsistency of the of the name of the item and that's not because the platform's fault

because these names are provided by the vendors right you know vendors are not going to talk to each other say hey I'm going to use this convention you should use that so uh what we can do is we can

ask AI to reformat the attributes so that I can easily identify the attributes so as here I said uh just show me the brand resolution and refresh

rate uh I think it's frame rate yes and then the names will be reformatted so that I can easily compare these things and here's another example of

booking Hotel so finally I can add the rating onto the map so here I can click on a hotel that I'm interested in turn

on the attribute mode and I can Surface the reading attribute on the map right this way I can look at both the price and rating at the same time and only

click on the ones that I'm interested in which avoid all a lot of unnecessary clicks right so I can open uh the one that I care collected by clicking the collect button on the top left in the

overview uh in the detail view but do opening this detail view again again is tedious right so I can pull up that you know uh collection button into the map

as well and this way I can just collect these hotels that I meet my criteria and I can collect more and more and after a while I can open the detail views of the

collected items to compare them right here's another example so when I'm getting food delivered or using these food delivery apps I care about the

calories but most of the uh you know shops don't show calories in their uh overview description so what I can do have to do well is to click into each

one and see the find the detail but here I can Surface it to the overview and this would allow me to just look at the overview to find the ones that I'm

interested in so let's zoom out a bit right the overview detail interface is a ubiquitous pattern because it supports one essential need of uh iteratively

browsing many items to identify the ones that we're interested in and then dive into specific ones for more details so in this work we identify the key

dimensions of overview detail interfaces and and by supporting an user to manipulate the interface along these Dimensions we can achieve malleable

overview detail interface and what we are working on right now is to based on these dimensions and turn that into a specification module which is pretty

straightforward right so basically we can uh consider all the data um with follow the uh an array of Json basically

right and the Json describes the attributes of an uh like item right and by just connecting an array of Json with this specification module it can turn

that data into a configuration of overview detail interface you know figuring out basically what AR tributes to be shown in the overview what attributes to be shown in the detail

View and what are the composition and spatial layout between the overviews and the detail views Bally you know that's the data format right we can turn it into a suitable configuration of

overview detail and that will be part of the UI specification it's just a specific module right so uh this is just one example of a pattern that we studied

actually people have studied other patterns for example uh B at all they study the dashboard pattern in information visualization or information management systems and there are other

patterns that we can identify May some of them may be like more application uh or maybe more uh scenario related like messaging right there are so many messaging applications out there email

slack Reddit they you all use different kinds of ways like UI but what if we can identify some common uh properties and then uh be able to have it be malleable

and there may be other patterns too uh we are also interested in like actually go a level down to identify a or to develop a unified specific

language for user interfaces and this way we can make sure that all these modules are consistent but also I think in the field of HCI we don't really have a language to talk about UI right A lot

of times just like oh you should have that widget you should do this like it's very based on intuition it's we're using very imprecise vocabulary but having this un you know specification language

probably will help us better describe user interfaces so hopefully all these things would come together and you know become this UI specification module that animation was a little dramatic uh so

yeah you know this is sort of the pipeline that we curently explored but I think there's one other component that will be super interesting which is you know it is really not it's fine to have

ai generate and let the user be able to manipulate it but it will be even better if the default generation just meet our personal needs right so how do we

achieve this personalized generation and I think it's really important to uh have uh you know context of what the user have been uh trying to do what's the

problem what's the configuration uh you know configured and based on that you know future generations of for the user uh you know will you know have Rich

data to sort of like generate the personaliz interfaces so this is the approach we are currently exploring to uh achieve this generative valuable and personal

environment so let's go back to these questions what is the natural workflow of information activity well I think you know it is uh you know very Dynamic and

very chaotic and very personal right and why has the application Centric interface Paradigm failed to reach this workflow that's because app are pretty rigid they're impersonal they're bloated

but they're also Silo and the operating system provide no ways for us to coordinate them and what's different this time with AI well I I'm not arguing

well we can use AI to generate content we can use AI to generate chat messages we can use AI to generate actions on top of existing uh interface but I think what we're exploring is use AI to generate the

information organization right so that human do not need to put all these organizations together so as I mentioned earlier at the beginning of my talk this Vision

towards generative malleable and personal information environment goes all the way back to the beginning of digital Computing right uh Li lier proposed that you know the human brain

and the Computing machine will be coupled tightly together and for that to be tied you want to be able to you you have to adapt to individual users needs

right and we know dogas unle bar point out that A system that augment human intellect should uh be able to adapt to individual's characteristics alen K you know also

explored this really personal and dynamic media and my colleague Jim Holland uh you know introduced this notion of distributed cognition into the field and

Susan buer also introduced activity Theory into our analysis of interfaces basically try to really argue that we should not look for individual artifacts

like apps but really look at the overall activity and divid carer you know you know have run this decade long Endeavor on Hast stack that try to create this

personal information environment and recently uh M and Wendy Makai you know proposed this concept of information substrates and a world without app to

try to help us uh get to this uh really flexible information environment so I think we are at a really sort of historic moment where the vision of

supporting Dynamic and complex information activity with an intelligent and malleable and personal information environement is kind of achievable so I'm clearly very excited about this and

I hope I passed a little bit of excitement onto you guys but that's it thank you [Applause] I'm

oh hi I've been in industry for a long time and I really love this activity centered approach I'm way past human centered design right uh into activity

centered design and so um I like this idea of closing the loop and then knowing what is the final determination of the task success or the activity

success and I wondered if you had any thoughts around that so uh I think think really you know the ultimate success is you are trying to you are helping the user to achieve their goal right of the

activity and and in the process hopefully they don't need to feel very disoriented right they they have this you know really aggregated environment where the materials they need are in the

right place uh shown in the right way so I think you know a lot of times we're just putting it up with the messy organization and we don't want to spend effort doing that because the pay off is

unclear right so what we're doing here is we can have ai do that for you and provide lightweight interaction so that you can flexibly change the organization right so you don't need to do all the

manual work so hopefully by doing that you can always stay on top of your activity and be able to you know accomplish your activity faster or yeah and maybe this concept of when something

is done is we're just beyond that at this point you know that it's just really the continuity of of life and fitting the fabric of someone's life and

there isn't the need to Define what done is and then it becomes well how satisfying or how support was that activity yeah so like for example when I'm working on a research Grant right

you know that is one activity but then later I will pick it up right I will you know that will sort of take another Branch or maybe evolve to become another Grant right so I think uh one thing that's super interesting is just see how

this environment evolve as your daily life goes right yes thank you so much for the presentation I'm wondering if you have a

user interface that keeps changing uh the user often the first time they use an interface they have to learn it in a way right they see it and they understand it and the next time it

goes faster what are your thoughts on this uh learning activity of a new user interface when it's always different yeah so like a typical user interface is very static right so that you know you

can learn you can accumulate your learning over the over the course but I don't think it's necessarily the case that this generative interface is incurring more learning costs right as

especially when maybe personalization kick in that you know when you're planning a dinner party this time you you you somehow figure out the structure but next time you know you reuse the structure right and then you can

gradually expand it uh to uh you know it will involve right so it's something that you crafted so hopefully the learning is not that much and do you have a follow

huh yeah yes thank you for talk I'm interested in you're discussing a little bit more about like potential risk of personalized interface thinking about

like um there if there be like a challenges around like information disparity become like enlarg because I'm seeing different things from what you're seeing even though we're the exact same

app like I know we were getting out very much around like more neut example but I can think of example of like oh like Shu and I bought the same display but because our um our app are different

where or UI are different it's personalized we don't even know we bought the same one that doesn't give us that Common Ground um that we used to have when we have like more static you just curious about how you think this

type of meable or personalized interface will potentially address this RIS so uh the biggest risk is you will have a personal Silo because everything is

generated just for you and you are not doing like one thing that's really good for information foraging is you get to see a lot of different things even though they may not be the thing you need right but you get to explore a

really large area and I think that's the problem with all generated content right when you prompt for have a video like okay this video is good enough you're just going to refine it right you're not going to like maybe browse online to see

what are all the possible videos you can create right so you know when we run our user study a participant said nice this is generating the information that I need need uh well other websites they

sometime you know I will come back to the parts that they provide right so I think one way to do address this problem is still keep the you know peripheral or other interesting information uh

entities in the environment so they can say oh you know this may be interesting I want to integrate it or this is not interesting that you know I'm just going to remove it right so I think that's uh one of the bigger risk I see and we also

did a research project that really tried to encourage people to diverge when during their information activity uh so we have a paper last year Kai is really try to encourage people to do a lot more exploration like even if you don't want

to diverge we force you to diverge kind of situation uh so uh yeah that's problem I'm uh thinking about the future

of of these I find it instructive maybe to think about like the most widely used malleable user interface of all time which is the text file you know like Untitled one or whatever it would be M

and you know why do we use use that in preference to things that are like more powerful um I think that there's like you know hyperbolic discounting is a hell of a drug MH

and people are just muddling through whatever they're trying to do right now yes and in doing so like it's just good enough just good enough right and then

it becomes hard to convince them like oh if your data were just slightly more structured yeah you know you you could get all of these nice filters and stuff it's like well I'm busy I just need to finish the thing I'm curious like it

feels to me like that's been one of the major challenges of like this sort of line of work in the past has been it requires pretty substantial upfront like

setup setup and drinking of Kool-Aid um and not in a bad way just like it's it's much more like a expert user interface tool sort of more akin to something like

Unix or the turn or the command line than to something that like everyone would pick up and run with yeah and so I'm trying to think whether the directions you know to what extent you

think that what you're doing here essentially can become the next version of like the Blackboard or the text file right rather than becoming sort of an

application you know like uh rather than being sort of the generic space or new space of applications it may still be an application but I think you just want to

get it to a point where it's like easier than just do copy pasting stuff around yeah I it's hard it's hard I think you get yeah I think you know with maybe

maybe just thei what AI generate is good enough right maybe people are not going to even Mal like do malleable interfaces so I'm actually really curious about this like if we deploy the system out

there right if the AI you know at what part point you the user will actually just like I'm going to change this because you know I really like I think that's the best right or they're just going to take whatever AG well I can put

up with this right so I'm curious I think there's a lot of like cost uh evaluation there in terms like the the cost in changing it the cost of just staying in your work and then put up

with the existing one uh put up with the AI generator one but I I think one thing that's super interesting is though like if we um some user are power user right they will maybe change this a little bit

maybe some these good uh configuration can spread or you know to other you know us so I don't know like I want to actually deploy this and really see how it works work and how you know how

people balance the cost of these things yeah um not to like make it about the economic model but since you had brought it at one point that like that's often the reason why that people are building these more rigid and um bloated

interfaces I was wondering like if this is successful and we we're able to have this really broad like widespread valuable interface um I guess what do you think the role of like the companies

that currently like produce these interfaces would be like like are they more like data providers basically where they just share out the data and does the problem sort of move Upstream where they maybe they change what data they

provide to because they have like maybe the way that they design the nurs often is because of certain incentives they have on like what their company is trying to achieve like yeah I'm just

curious how how this like ecosystem might change yeah I so the the answer is I don't know it but I'm very curious about it I think if you look at how

um uh how content are being served on the web right you know the the company will serve a HTML that has the all the style has some part of data right so I think you know in the future maybe they

will still serve the data but they could also serve a template but maybe it's a just malleable template for you right maybe Expedia just like I really think the default configuration for this should be this even though but you still

have the ability to change it and I think uh maybe some company May will just give up on the UI side and say Hey you know there are really good

well-designed UI module that's there and I can directly connect my data into it and they will have a reasonable UI and I think it's that's really interesting I

think right now the designers have two jobs I think one is well um figure out what is the best representation of the

data but second is try to identify the sort of context where there's information is being used I think maybe in the future the job of some designers

who are Des designing these templates is to really design these UI uh packages in a way that is super flexible but without they don't need to assume okay in what context I want to show this right so

they just need to Def Define it in a really flexible way um but I think it's question I want to talk to some business they see things um

yeah yes um this topic reminds me of an adjacent area of Information Management which is note taking uh companies like

Rome and obsidian try to transform note taking from a linear and compartmentalized approach into like a networked thought um so they kind of

mirror how nodes in our brains connect to each other um and softwares like that have a super loyal customer base but it

hasn't expanded that much over the years and so I'm wondering are there similarities that could be drawn between that model and like

this so I think what I'm trying to achieve with malleable interface is that um the user they don't need to choose a camp right they can say that I I I'm

really happy with the Notions block style right occasionally I may use obesity and you know graph right or you know someone who's really into obesity and they can just like sometime be able

to connect the data together right so um I think I think hopefully people don't need to be

very loyal to one particular way right I think they they have the ability to choose from a variety of things and based on their context like we actually

interviewed a writer uh who uh use obesity and you know uh to manage their thoughts and he happened to be a programmer but he had to like build a lot of these customized software to

extend it to really fit it into their a workflow but that's not possible for every user right so you know there's a very high skill ceiling

there sorry skill floor all right I think we are at time so I'm going to call it thank you for again for your Bo