Should you be a Data Scientist or a quant UX Researcher?

we're live we're live are we should we refresh oh yeah let me refresh oh we're

live we're live hello um well I'm Ethel I am a quantitative user experience researcher at Google and this is my colleague

Connor and today we have a ask me anything if you want to ask Connor anything subscribe to his channel Lincoln

bio but today we're talking about Quant uxr super excited yeah I have a long list of questions for oh my God okay I hope you have five hours to B it exactly yeah but there will be a recording

posted after so we'd love to see the engagement love to see comments we're happy to respond to comments in the in the comment section as well if there are any questions post live stream um but how about we get started let's get started so eth I'd love to hear about

your path to to quantitative user experience research yeah for sure so um I've been a Quant uxr for about three years now so it's still relatively

recent um but I would say it's definitely not a straight line I kind of meandered a little bit and just kind of fell into this field which ended up being something that I really enjoy

doing um but kind of long story short um I pivoted my career so I was in finance and Consulting and then I wanted to do something more technical so I went back

to school and got my master's degree in data science and that enabled me to transition into something more technical um but I didn't become a CO XR directly

after that I actually uh changed roles a couple times being a data science consultant and then like being a Analyst at Google and then eventually within

Google I made a transition to become a Quant user experiened researcher so it's kind of a long journey I was say overall it took me six years uh in total and I

wouldn't say that I planned for this career but uh I was really lucky that I found out that this is something that aligns so well with my passion because

I'm really interested in just using data to improve people's lives and I feel like a Quant uxr does exactly that awesome yeah that's super interesting and I'd love to hear more about what

differentiates Quant uxr from data science because I often hear you know that there might be some overlap in terms of the skills data science is usually thought of as like sort of an intersectional interdisiplinary field where sort of a combination of

Statistics subject matter expertise um as well as computer science so I'd love to hear about what Quant uxr is relative to data science yeah so I would say maybe overall the biggest distinction is

about the user experience experience part because um for user experienced researchers we're focusing on understanding our users so for data

scientists maybe our methodologies are very similar but our focuses are very different so as a ux researcher you really want to kind of think like a user

like try to be the evocate for your users and know their pain points pretty well and have a good graphs of what's I guess not so great for them and what

they like about certain things um so that's the ultimate goal for a user experienced researcher uh for a data scientist I think maybe we're using the

same methodologies like surveys or live experiments or L um by they scientists might be thinking about other objectives like how do I make the model more

efficient or if this specific feature is uh performing well and by well that can Bean like monetization wise and other things but for a user experienc

researcher we're focusing on improving the user experience so ultimately that's our I say North Star metric uh it just our Guiding Light is our users and I

would say that's the key differentiator with a career in data science not so much the methodology part more so about

um the ultimate goal and uh I guess also um the affiliation like we affiliate ourselves with our users and try to be the advocates for our users

awesome yeah no I love that framing of being an advocate for the user I would love to hear about what methods you use to measure you know success in a role like quantitative user experience research how do you how do you see if

you know the experience for users is improving or or the opposite yeah so specifically about um how to understand

users in a quantitative way right um so I would say there are many different methodologies but uh I will probably break it down into three different areas and this can overlap a lot with a data

science career um so if you have any questions feel free to pop it into the chat um so I would say the first of the three areas would be surveys I would say

that's very common for Quant user experience researchers because that's a really good way for you to hear directly from your users um so survey methodology

is very important and um and you can ask survey questions in many different ways um and the difference between a Quant

and a qual researcher would be a um for a quar researcher you can sit down with a user and ask them questions oneon-one but as a Quant you want to do this at a

large scale so doing one-on-one interviews is not as possible so um so yeah so surveys would be one so you can have a large audience and you can

collect information from users um at a large scale uh and then the second one would be logs analysis meaning that you

can look into the interactions of users uh through the backend database and uh kind of using proxies to figure out how the user's interaction is uh I do want

to clarify one thing though like we don't look at specific users interactions so if Connor searched for something on Google I would not be able to see that Connor specifically

I no I already know everything you search for so I don't need to use the database um but yeah so um I everything

that we look at are basically anonymized uh and we aggregate things meaning that um ultimately as a Quan We want to look at things at Large Scale right that also

give us the statistical significance so there's no need for us to just pick out con Connor and try to understand like what he specifically does but then the logs

analysis can give us a pretty good understanding of the actual interactions of users with the products that you're looking at so that will be a second methodology and then the last one would

be live experiments so I would say this one overlaps a lot with a lot of other technical roles and is not so common

within uh qurs but uh as I said earlier because as qurs our main goal is to serve the users so if we're going to launch a feature then we do want to test

it out to see if it's going to impact users negatively or positively so Le would be a live experiments would be another another methodology that you can

use to reach our users and get their feedback and the last one you might combine that that with a survey and with logs um but then the setup of the Le itself can be something a researcher can

do as well interesting cool yeah and and going back to I like I like how you distinguish being like a traditional uxr versus a Quant uxr and how scale comes into play um and like how much value there is there I imagine that it's

difficult and you'd have to work very cross functionally so I'd love to hear more about like what cross functional work looks like as a QR like who are your common stakeholders in terms of

conducting research as a QR yeah oh my God actually this is a long list um but I would say it also depends

on your product and also the specific team that you're in so probably not a fully representative list by at least

based on my experience um your common stakeholders I would say are probably from three different areas one is the product managers um so they set the direction for the product so as an

advocate for the users you to have very close relationship with the product manager to make sure that you can uh influence the role map of the product based on the user feedback so that's

very important and the second one would be uh other researchers as you said like qu researchers because a lot of times combining the qual research with a Quant

research is very powerful so for example um with a qual research maybe you saw a couple of things that users kept bring up but you only have five users that you

talk to one1 so this might be a good collaboration opportunity that as a Quant you start asking those hypotheses in your surveys for example to see okay

at a large scale do all the users or most of the users have those problems as well so uh your core researchers is definitely another uh another St

stakeholder actually I'm going to extend that a little bit just people in ux user experience in general so that can be your designers as well uh and that can also the user experience Engineers

sometimes so depending on how big your user experience team might be but I would group all the ux related stakeholders into the second group and then the last group will be your

engineers um so these are the people who we actually building the product so knowing uh how much it takes for them to build certain things is super helpful a

lot of times you need to do kind of a tradeoff negotiation with the engineers because they don't have unlimited resources right so they they want to know okay what is actually absolutely

necessary and what is good to have and then being the advocate for the users you can help steer the engineering team on which direction they should focus more energy on because uh uh maybe

there's something that would take I don't know four months to build but your research so shows that 99% of the users had a really big problem with it so maybe it's worth it even though we're

going to invest four months of engineering work work into it yeah so I would say those three areas are the main stakeholders but of course there's so many so many other functions that can

come into play like program managers and uh also just sometimes leads in general right like maybe for smaller companies

you will be a CEO you just need to keep them posted uh and one that's not so obvious uh because they don't really show up in your day-to-day alignment

conversation is actually your users I feel like as a user experienced researcher I take a lot of responsibility and making sure that I'm representing my users in the right way

right yeah so like I guess those are kind of broad stakeholders but in your day-to-day those three groups are kind of the people that you have to talk to almost every single day yeah that makes a lot of sense and and going back to how

you're talking about you sometimes have to sort of negotiate prioritization I'm curious as to what dimensions you look at within like a research plan um and how you prioritize resarch when you get push back from certain stakeholders yeah

so prioritize what research to do do you do you yeah so like if if like if your research plan is is like the scope is too large and and and product managers or is like pressuring you to to pair

back you know your your different research questions how do you decide what to prioritize and what to De prioritize yeah that's definitely an art I wouldn't say that I have the ultimate

solution to um but how it usually happens is that uh there is definitely a tradeoff in research between more fundamental studies versus the more

feature specific studies so the fundamental studies are really helpful for you to just understand the product Market fit in general right kind of Set

uh kind of longer term visionwise directions um but a lot of times you do have the pressure from your enge or PM that uh you really want to know if this specific feature is going to work I'm

going to use an example like if this specific button should be on the right side or the left side like which which color is the best so you a lot of times

you kind of are asked to step in to be the tiebreaker between multiple stakeholders right um so I think what's really important is just to keep the

communication open I know this is a very unsatisfactory answer but I don't think there is a single solution to this um I

I think what's really good for a career in Quant research is that you get alignment from your stakeholders and make sure everybody is on the same page

and of course like maybe here's when you use your empathy as a researcher again to really listen to your stakeholders to see like why do you think this

particular question is important um and we try to frame everything as a research question so like kind of a hypothesis of like maybe and just like I hypothesize

that the button be on the right is just uh like too hard to see and PM is just like but everything else in the whole Suite of products is on the right hand side so it needs to be on the right hand

side things like that right um so understanding where they're coming from and having those options and listing out um the different options and oh actually there is a framework I'm not saying that

is the solution but there is a ice framework okay that you can use I use from time to time to kind of Stack rank the different questions people might have it doesn't solve everything but it

gives people framework to think a little bit deeper about okay what exactly is the question that I'm trying to answer and how important it is compared to all the other questions that people are

trying to get answers to um so the ice framework is importance uh confidence and effort so you ask people like okay what are your

hypotheses and you combine them all and then you kind of rank them by these three dimensions how important it is to answer this question and how confident are we that like this is to make the

product better and then lastly how much effort you will take us to answer those questions so as you can see like they're already three dimensions so it doesn't mean that you can come up with a really

clean ranking but it will be one way to push your stakeholders to think a little bit more about okay what are we really trying to do here right so maybe that's a conversation starter for people to

stack wreck things and come to the same page but that conversation is just so crucial to have and can go any direction

you know um but it just more so the process of doing it taking the due diligence of doing that is more important and a lot of times maybe there

is also a question a balance between okay like should I do what I think might be the best for the users or should should we just iteratively test things

uh and then uh maybe maybe the fundamental things will show up in in a later State um so yeah really like and satisfactory question sorry your

question is so unsatisfactory I mean my answer is so un where's the door no no I mean it's just really a non-answer answer but that's just the

reality of the work yeah yeah I feel like that's the reality for a lot of um cross functional work there's not like a a concrete framework to follow yeah um I would love to get a better sense of how

hypothesis generation Works within your role you did sort of touch on how sometimes you might have like um live interviews from the qu uxs um and then you might want to test out some kind of

user experience on a larger sample to see if it's impacting like um more users yeah I would love to get a sense of how hypothesis generation works and how you um you know plan research yeah for sure

so really good question I would say this is probably the most important part of doing any research um so definitely start from that uh and if you're preparing for a career in that I try to

frame everything into hypotheses right uh like what are some research questions that I have um coming up with the hypotheses though I would say kind of

depends on the stage of your product so if your product is really brand new it hasn't even been built yet it's just an idea then I think is mostly just like people talking to each right so people

who wanted to start the product are talking about oh I think there is a neat there um but then once you're able to spend a little bit more time where if you have a prototype then it's easier to

ask more in-depth questions right and then sometimes we uh maybe there's a market report you know like just U marketing has done something maybe you

just saw it online that let's think of a random example uh I don't know like sparkling water like do users like it and maybe there's sponsor yeah not sponsored I

need to cover this up I made this myself no free advertising but uh if you want to collab please send us a message later um yeah like do people like

Sparkling Waters like maybe I'm thinking about a different flavor for Sparkling Waters and cookies and cream like as sparkling water and maybe there is a marketing report somewhere already I

went at an initial stage of your product development then you probably need to rely on those things so you saw that um 80% of the population drinks sparkling

water so that is a good indication right and then and then you look into maybe there's some marketing report on just like the top flavors in general and you

s that cookie scent cream is like number 10 so that's like pretty good yeah um so you can start testing Things based on the things that you already know about

the general public already um but once you're further along in your product development cycle then you can start doing like core research and all those

kind of things and that can be where the hypotheses come from but even for those research to happen I guess initially you do need to do some guess work but ideally based on something that you

already know about the users like the general public um but I would say okay maybe two sources um something that you already

know about the users and the other is really just got Instinct unfortunately but from your stakeholders it's so important to involve them in the

conversation to make sure that they kind of have um some contribution to this whole process um so also getting that alignment earlier on than later really

makes it easier for you to get by in in general so um yeah kind of a combination of uh things you already know that can be from the very early stage just like marketing research and later on it can

be privious research that has already indicated something uh but you can never get rid of kind of that gut feeling and people will have gut feeling but sometimes they're right because they

have been working on the product for so long so they kind of know like this is the issue that kept popping up so it's kind of a mixture of the two awesome yeah okay cool yeah maybe zoom out a

little bit just about like the role of qu user experience research just because you know I think the role is not as known in Industry um so I'd love to get

a sense of whether how you feel Quant uxr at Google is representative of Quant uxr across industry or or what makes it unique or if you have any insights into

that yeah I have to say like this is a very Niche role I think within Google we have probably less than 200 people

globally who are Quant uxr um and then in the industry I think nowadays there are more companies that have this role but previously it was really just big

tech companies I know meta has uh this role and then Google of course and the role has only been around for maybe a decade or so so it's still relatively

new uh um overall I think the reason is probably because of what we do is at an intersection of so many things um so

I'll get into what exactly they do uh in a little bit but I would say for now this row is so with bigger companies because it's kind of a very specific

slice of user experience research right and then you kind of need to have enough resources to get somebody to work on something a little bit more specialized right um and the definition is slightly

different among different tech companies even um but I'm going to talk a little bit more about my experience so far about what I do and compare that with what other companies are doing um but I

would say in general just think of it as understanding users at a large scale right so to do that like for example we talk about the main three methodologies

so all of those is for us to be able to understand users at a large scale and the fact that it was formed is more so

because um because of the user experience Focus um so it's kind of Grew From the necessity of for example Google

having so many users yeah so it's not enough to just do core research right like oneon-one if you want to one-on-one interview all of the Google users that's

that's going to take ages like googil years I don't know Google is a number right yeah um so it stem from that like with a larger company with more

resources with more Workforce and uh with more users right then you can have this overlap to have somebody who's specifically is studying a bunch of

users um yeah and then we try to talk about the main methodologies that we use already a little bit so those are the things that I mostly do compared to other companies though I would say I

have heard that some companies care more about the surveys um some companies care more about mix method so like Quant researchers are more like mix methods so maybe you also

like do diary studies which is more of a qual research methodology yeah but for some other companies qual researchers and Quant researchers both do that so um I would say if you're interested in this

field definitely look into the job posting or maybe talk to people in the industry to understand it a little bit more because the field is pretty Niche and pretty new so everybody's still defining it right now it's kind of like

when data science was popular at the beginning like everything is data science and it's covering so many things so kind of do your due diligence a little bit awesome yeah I'm I'm a big

fan of like of like nonlinear career paths or you know when you didn't go necessarily go to school thinking that you'd end up in in the role that you you are in I'd love to get a better of sense of like of the Quant ux researchers that

you work with at Google or you know at Google do you have a sense of like what their career path has looked like and whether it look similar to yours you know coming from you know finance and then into data science and then into

Quant uxr is it similar to that or is it sort of diverse yeah I would say it's pretty diverse now previously especially when it first started with just like one

person at Google actually I think qu uxr the whole role started at Google um and I think um there is a book that she talk about how he started everything so if

you're interested I'm going to link that in the comment section later um so definitely check that out if you're interested in the history of of that um but at a beginning I would say it's

mostly from people who have been in user experience already I see and then mostly people who have really high degrees maybe even like a PhD in Psychology uh

things like that so people who are really technical um but also have experience with user experience right uh but then that evolved a little bit more

um though when I say evolve is still kind of relative right because it's still such a niche area area so for example we only have like less than 200

people so within this is impossible to be too diverse I guess is representative by about 200 people in general um but I do see that a lot of people kind of do

not have that linear path either especially because it's a newer role right so most of the people have done other things before joining it um so I do appreciate that about people who are

in the same field and recently I also see that more and more people without p phds are in this role cuz I think when I join maybe 90% of the people have phds

and I personally do not have one I really admired the people who got their phds so I guess when it first got in even I thought that was super lucky yeah

um but uh um yeah my background give me that user empathy and the technical skills but there wasn't any training even to like specifically for you to

become a Quant researcher right now we have human computer interactions which is like one like pretty common degree to get to get into this field um but back

then that wasn't even a field actually so you kind of just come in from so many different fields but you have the common passion of understanding users at a

large scale cool yeah um I sort of have a curveball question for you oh wow okay let me take a sip of water yeah you're gonna need it am I spit outo

um okay so I'm curious as to you know I know AI has been talked about a lot large language models have been talked about a lot I'm curious as to within

Quant uxr how you see Ai and large language models either impacting or or improving the work that you do in your day-to-day or if it already has yeah um yeah love to hear I love this question

uh I mean like did you do some research about my background and methodology I have to um wow I don't know if it's a curveball I think it's just like it's a

pled question now um because if you don't know I'm passionate about programming and I love just large language models in general actually when I first started working as a Quant uxr

my main goal is actually to use language models to help me understand users better awesome um so my perspective is going to be super biased but I'll try to

be more objective guess um I think it's definitely changing the role a little bit it's changing how we're working a little bit but overall not so much yet

but it's changing how we work in two ways though first it actually opened up a different field for us to do research in so oh interesting um for example like

Google has Gemini right so um you can actually do research on how people interact with AI so that's like a new opportunity a new field but you apply

the same methodologies um just with a different type of interaction that has never existed before like users have never interacted directly with large language models before so that's a new

area for you to understand oh like what is this very novel way of interacting so that's one opportunity actually and the

second way that AI has changed our field is um um it can be a collaborator and I'm saying that with a kind of a caveat

some people are saying that maybe they're going to replace researchers I personally still do not believe that but I I think it's good enough that you can really FASTT track some of your work

already um I I actually have done a couple of talks around this like how I built a text classroom model for researchers to just like um understand

user feedback a little bit faster grouping things into themes and all that like uh more programmatic way of thematic coding for researchers so I I

still use that every single day you know um and it can just be a um yeah like sometimes I do that with my my own

documents just like what did I think recently and then I just like piped that through large language models like oh okay like these are the main things that I've been working on um so in that case

I feel like it's really helping me to be more efficient uh at work um or should I say higher velocity which is the theme

yeah um yeah so I would say from a very biased perspective I think it only open up more opportunities for this field um

and I don't know what's going to happen in the future like maybe it will be more powerful maybe he can write service on his own but at the current capacity is a

collaborator is definitely not a leader right yeah um yeah no i' I'd love to get a sense for for the audience out there um anyone interested so far in in

quantitive user experience research um what what recommendations would you have for um for someone wanting to get into the field yeah okay so

um wow I well because my journey was so nonlinear so it's hard to say like what what I did right verus wrong um I would

say maybe the most important thing would be to just talk to people who are in the field uh to get a better understanding of uh if you're interested in this um

and the second thing is probably patience um because it's still a pretty Niche field so it might take some time if you're truly interested in in this

for an opportunity to show up um and we'll be the next thing I would say maybe the the last thing is that your journey as a quantitative user

experience researcher doesn't have to start with a job right I think you can even just do it on the side and I feel like that's one thing that I did no matter what position I had uh and then

and then when I realized that a Quant user experiment researcher is a role and was oh my God this is exactly what I wanted to do but what I meant by that was that uh I actually already started

doing social listening even before I started working for Google and but it's kind of a longer story but overall it it's so stupid but uh but when Game of

Thrones was so popular um I had an argument with my friend on like whether the ending of the show was satisfactory

and I was very unhappy with and and she was like no I think like uh I think she deserved it the main character um so I

of course uh being so humble and so willing to accept my failure I was like I don't believe that let me go online to see what people are talking on

social media so what I did little did I know was already a user research is I scraped Twitter Twitter yeah back then now was called x i script Twitter for

just like what people were talking about and then like clustered their responses and then I kind of uh grouped them by how many post

mention each character and then what's the sentiment of that post uh so I had a whole bar chart of like for each character so many posts were

about them uh and then the post is mostly on average like positive or negative so I think Indian and I had a indication of like people are very

positive about Jon Snow in general um but overall relatively negative about what happened to the main character well I forgot her name uh kisia I don't I

don't remember her name anyways it was it was just a fun site project I do not remember anything anymore but then that whole process itself was actually quantitative user experience research I

just didn't know that my users at the time was everybody who was on social media who have watched Game of Thrones and wanted to say something uh and then the research part is really I have a hypothesis of like for certain

characters people are feeling pretty negative about what happened right and then and then my methodology was using large language models at the time it was actually like small language

models um to do analysis um so your journey can start at any time actually as a qual user experienc researcher so you don't have to wait until an opportunity show up but if you're really

passionate about the general idea of understanding users at scale you can do it every single day unfortunately you might not be paid for that but you will have the experience of doing that yeah I know I love the the sort of

self-learning approach through you know doing your own small projects on things that you're interested in yeah know that sounds super cool um let's see if there are any questions from the from the chat yeah let's take a look looks like we

have Alex who's a fan of ice the ice method oh nice yeah wait see somebody knows about it yeah oh actually Alex and I work together he has a PM so shout

awesome shout out Alex so yeah Al's introduced the ice methodology to me so time cool anybody else no it looks like

anyone anyone who has questions feel free to to leave them in the chat happy to answer answer any questions yeah and uh we're also happy to schedule another one if you're

interested yeah there will be a recording so if you're watching a recording and you have some questions still uh I'm going to recruit Connor again

he needs some good karma so I'm just recruiting him for a followup if you're interested well yeah those are all the questions that I had this is awesome

yeah this a great chat I I learned a lot yeah yeah and I hope that you were also uh happy about what we're talking about if you're not you're you can also

leave a comment and then I'll do uh my language models on your thank you so much for joining us yeah thank you for joining yeah I'll see you next time see you next time

bye