Terminal Tech Talks: building production-grade deep learning systems and growing from zero to $100M

okay so we'll get started my name is Mohammed I lead our product teams at company called dessa we are a company based out of Toronto

just right on King at Spadina we've been around for about two and a half to three years now we are primarily focused on building machine learning tools

specifically for high performance data scientist to be able to scale up their machine learning efforts this involves tooling to maintain model team production and to be able to rapidly

scale the development of our tools so Desa elite so for product teams there we do a lot of work in the machine learning side you may have heard us from the muse and a few other things that were we

built a lot of research papers around it but yeah and oh yes so my name is Henry I'm the co-founder CTO snap travel travel chat company based here in Toronto

I started company about three and a half years ago and since then we've grown to about sixty people in Toronto another 80 90 overseas for customer service we've

done over 150 million dollars in sales all over messaging and have over 4 million users all over the world so it's been a pretty incredible journey over the past three three and a half years

and you know happy to share my learnings and experience along that journey awesome thank you and that's what today I think because we have such a technical

audience in the room what I wanted to really cover is you know what's it like running a startup from the seed stage talk about a little bit about infrastructure in terms of machine

learning wait what are the some of the aspects that we cover in the C stage and how that differs from the seed stage to a series a yeah and the seed stage you're probably you know very scrappy

and your development work you probably have just yourself and few others working with you we're gonna see is it when you're trying to scale up you you will probably have to worry about scalability and a lot of other things

like that right so just to begin with heavy I know you already mentioned a little bit of a snap travel but let's let's dig into that a little bit deeper and talk about

the technology aspects of it wait I know the stamp travel provides a broad platform for you to be able to book hotels but what about the technology aspects of in Hajj you guys get started

really sure so I'll talk about technology first a little bit and then how we got started so technology itself it's a pretty standard modern web architecture we have our it's a very

micro service based probably 40 50 different services all talking to each other and we have everything from our data pipeline service to our recommendation service or NLP service

our chat bot service as well as our web application itself and external internal api's so it's everything kind of works together across services to put them with deliver the experience and the

experience for the end user is you talk to our bottle on messenger or SMS what's up by Alexa you tell that where you want to go your preferences just like just as if you were to talk to a travel agent the Bob will use NLP to understand what you're saying understand your

preferences and then give you the best hotel recommendations and best deals all over messaging that's kind of how the product works and that's kind of the tech architecture in terms how we actually got started when we started

three and a half years ago I left my job at Google and we had no idea what we're doing so me and my co-founder we knew we wanted to work together but we didn't have an idea or vertical industry so what I had

will figure some things out we're testing ideas iterating pivoting so that time was very much a rapid iteration process like how do we build a landing page I would build MVP how do we you know get customers valid an idea

eventually we ended up in the travel space and that's when messenger and rest release really search app bar SDK where you can build BOTS so we thought okay there's something here can we bring the

traditional travel agent experience back to the sort of online travel world of Expedia and booking where everything's I know depersonalized that's how we got started but instead of building anything

or any NLP or any anything really we've wanted to validate ideas so it was me an iMessage where you can message me and I was the bot so I could look things up for you you know you would say I'm gonna go somewhere and I would kind of go around

the web find things for your sanity over messaging but the crazy part was that people wanted to people gave us the credit card so we're messaging when I was something here you know you trust this service to give your credit card and book stuff so when you do something

here we've both the prototype at first it was we're using external api's so external API is for our recommended for our NLP for the bought for the you know

state management and that works you know okay well and then over time we got better we got more customers more data and we built our own technology in-house and now all of it is in-house and you know we've been outperforming any

external provider API since like two years ago so that's kind of how we got started in the journey over time that's amazing actually I wanted to dig into that a little bit deeper you said that you know you were just like trying out

new your ideas and trying to really find product market fit right now where is where's the market situated right now and you know what can you build to be able to do and so you had this crappy ways for you know you are the body

yourself right but like in terms of like experimentation right what are some of the other strategies that you guys took to be able to really experiment and and and did you do any kind of like do you

monitor any kind of KPIs for you to be able to tell like hey there's actually an opportunity here and I want to be able to dig into this a little bit further right so so for starting a

company there's one set of processing metrics and there for growing a scaling company is a different set for starting a company in one of my favorite books for anybody who's interested in this is running lean not methylene startup but

running lean by aishwarya he goes into a very specific step-by-step process on how to you know test an idea of value in India what kind of signals metrics and you know conversion rate all these things for validating an idea so we kind

of just followed that book you know reach out to people you know we would send like 50 emails see how many people open how many people replied how many people like converted and that whole funnel we would track just to gauge

intent and gauge sort of the stickiness of a product or the of an idea I mean that's how we got started and in this space and then as we scale the company a little bit different than we took turned

it more to I first user feedback like talking to users and what the light would didn't like and once you had enough users would be a be testing so testing different products different flows different funnels looking at how different people behave to different you

know placements different features different buttons how to affect their conversion and then also once we got the product conversion metrics down we will look at the acquisition channels so now it's like we have a great product how do

we bring people to it so where do we spend our money how do we and money with some robust return that spend if I spend a dollar do I get 80 cents back I don't have 20 back a dollar 50 how did the channel scale and what are the opportunities in the China how

much do i bid how much did I pay per impression and these all the very metric driven initiatives we took throughout the stage as a company so we've been always metrics driven and data-driven in our company that's part of core of who

we are both my co-founder and I are were studied computer science and it's one of our core values within the company especially in a very competitive space like BTC travel which is probably one of the most competitive spaces out there so

we're very metrics driven that sense and what you measure changes as you scale the company but it's been metrics driven from day zero to you know now Dave or whatever a thousand something right and that's that's really interesting then

and that's you know thanks for answering that from the but that's a metric from like the you know your business to the consumer side right and the people who are using the app but you also have a

b2b side of things right from the admin side because you probably have to form relationships with these external hotels and other business models right how do you approach those and like and you know

when you are a smaller company you know now you have like over a hundred million dollars in sales but before that when you're such a small company how did you get all these hotels to trust you to be a partner and really get there yes so

luckily I mean this is probably not a super satisfying answer but ultimately you know we're selling you know we're driving sales for people and everybody when you're on the south side it's running on the buy side it's a lot

easier like I'm not selling something to a company right I'm not doing enterprise sales I'm not going out there trying to sell you know the solution to a call or Marriott I'm just helping them sell their inventory they get the sale and I

take a cut so from that perspective it was actually quite easy everybody I always say everybody needs more distribution and we're just an extra distribution channel and we had some good credibility and backing because

it's a new channel I was unique people like to hype around that that helped that definitely helped but at the end of the day it was a buying process and buying is one of the easiest things you can do when you have money to spend

whereby people want to sell things to you right ok cool let's dive into the technology part of a little bit deeper I know you mentioned at the beginning that

you at the beginning of the stage you had the api's right that you were using and it's probably like an external API for NLP that you were using and then

slowly you transitioned into building a model yourself and and really like investing more in the research part of things right so why did you make that decision and also what were some of the

api's and and processes that were using at the beginning sure so for our NLP API we use an external vendor it was called immature it was a travel specific NOP

company that had a NLP for travel agents well I was great you know it said everything we need and then we reasoned them for half a year happier to about I think a month and then one day like yeah

sorry we got bought by booking.com so we got to shut down okay that kind of sucks but actually was a blessing in disguise because it made us really you know fit spend time and energy and effort to

build our own models that eventually outperformed that there's by the way but that actually was the driving force for us to better own NOP and I think that was very much a blessing in disguise do you have some background and machine

learning as well cuz I know you did your masters in Georgia Tech is it yes so while I was doing this but just aside no I was doing a master's but it was a great program I must say I just wanted people don't know it's the OMS singham's

om SCS program from Georgia Tech it's a full degree top ten school full degree there's it's a hundred percent online but there's no online distinction full degree eight thousand dollars ten courses and top ten degree a really

great deal really great program great people and for them it's about improv it's about encouraging accessible affordable public education and but by dropping the prices from you know traditionally 50 80 K to eight thousand

dollars they're making education accessible so for me I was like you know it's a great program it's a great deal and you know snap travel we love giving good getting good deals so we did it I did it and then I did a machine learning

major in that programs and I would admission in him before so I kind of have a background I'm not an expert and I wouldn't say I'm a you know world-class leader right but I have some background which you learn that's awesome so um but what was that like so

you know they tell you like hey today we're gonna shut it down booking.com is taking us over you have X amounts of days to migrate over right how did you maintain that part of the business and also like how did you start really

like getting deep into the model building and and finally built something that is production grade yeah that's a good question I mean that these kind of things definitely take time what we we

did we had we were working with somebody from a colleague of my classmate friend of mine from Waterloo he turns out to be a person who just loved languages like NLP in languages and he was doing a

master's and I was doing his PhD and he's like hey you know I think we can do something better we're like are you sure cuz even took him like seven years to get to where they are today and even then they're not like amazing and they're gonna we'll figure something out

so okay so then we just did something quick I'm quick and simple and it was mostly rhetorics actually at a time but it worked you know it worked we put it to our data and we compared to each and there's a guy I actually perform pretty

well sometimes even better well like a there's some hope here so let's keep refining it let's keep adding to it and we're pretty scrappy as a start-up you know we're both in terms of like time in terms of resources even training data so

we thought okay let's use this you know regex mostly regex base model with maybe some you know open source models let's use that for now the bootstrap and then we'll use that to collect more data we'll have the customers help us train

our model in a way it's like if the customer says this is incorrect we'll take the data and we'll learn it and get the correct data from the customers so over time we built the framework with both those systems around that we built

the data pipeline architecture around that and just kind of you know piece by piece we click to more data improve the models and now we have a pretty sophisticated model that's obvious not

just regex but this is a model that does more it's a multi-part model that does you know any RIR and intent detection using various different models and it combines together to create that

experience it's a fun little sidetracked but you know regex is actually true and complete I don't know okay so that's one key takeaway and I think the second key

takeaway is also that you really need good friends to be able to run a business and and have them as a saving grace when when things go wrong now I know that you said that

you guys started building your model and but you also published a paper right right can you tell us a little bit about that yes so again this kind of goes back to

supporting also working with the friend who's now he still works with us but he's also doing a PhD in linguistics and you know for them it's how do we now publish some of our research because I

was quite industry-leading and we're pretty open about sharing so you can find it online we just presented at the I Triple E AI for industry conference where actually got the best conference

paper which we're pretty proud of but essentially it talks about how using NLP AI in industry at scale right and I think a lot of people talk about things in it so I talk about

machine learning and AI as like interesting projects or interesting academic papers but to take some of that stuff and apply in real life at scale you know over to millions of customers

and 100 millions of sales it's I think that's what's unique about the way we look at the ecology and yeah so we did that that's pretty cool and you can check it out online we should talk about our models and different layers the

different different types of models we use and how we use it you know it's it's actually really interesting because in in Toronto Toronto right now is like an AI hub right we have Geoffrey Hinton we

have U of T the vector Institute and lots of other really interesting things that are happening in the in the area but even you know so so many of us some

of us from Desa we actually teach a course in U of T on applied applied deep learning because what we found is that people always get stuck in the POC stage of things right they will do their

research right they will go through tons of research and development and they'll burn through a lot of research scientists efforts and then they'll have a model but they will do nothing with it you know they just have it somewhere

there right but putting it into industry putting into a use case like like snap traveling into many other companies that are doing for example there's company like route insurance there's also

companies like uber who are really using machine learning to drive their business is really something that still needs to be solved at scale yeah I think one of the things I learned is ultimately tech

she has to drive some sort of business value at least a start-up stage and you know it just comes this is more about sort of building a company where you know for the seed round for the you know pretty serious day or whatever you can

you know kind of talk about tech tech and hype and get people excited for certain point but at some point when your growth and when when you're going out for the growth round technology has to drive that business value because

ultimately that's what technology that's it you know makes people work faster work smarter more efficient whatever it is and for us that's what the approach we've always taken is use great technology use it when it's where where

it makes sense and use it to drive business value if it's ultimately if it doesn't help the customer it doesn't help the company doesn't help us it's it's just great project and great paper but it doesn't actually drive of value

that we that I think with good technology should drive yeah totally but in terms of like engineering culture how do you how do you how do you bring that out into your culture I know before just

before we sat down here you mentioned you know you really build technology just to status to satisfy the business needs you know nothing more than that nothing too extra right try to keep it

lean try to keep the technology lean as much as lean as possible but how do you like ingrained that in your culture now that your grow your scaling your company right now after a series a how do you in gain that in your culture at the moment

yeah that's a good question so I think for us you know there's a sort of notion of people want to work on interesting tech problems just physical type problems and and for us we really try to

line people around this concept of you know personal growth as well as winning right because there are still very interesting problems and they've been solving many many in different ways and

we look at and for me engineering is about solving problems whether you use tech we're not used tech if you can solve something faster without using tech and it's works and it scales great right why why do you need to you know architect this huge crazy tech thing

that's hard maintain how to reason about so for me it's about how do I solve problems and how do I grow in that process because solving problems effectively I think is that is a true measure of a skill right to solve

something in effectively but and overly complex is actually not in my opinion a measure of scale it's maybe a measure of I want to do something cool or fine so for me we try to find people who are aligned with that we try to agree in our

culture very metrics driven in that sense and realign our company towards I know it's or star metrics around your revenue growth retention stuff like that and that's not say we don't use technology

and leading-edge technology right if you look at our paper some of the models when we use are really really fresh and really brand new and using at scale in production you know we're doing some

really bleeding edge stuff with with learning or excuse me with optimizing pricing and bidding on our hotel side and how do we price things and that that's very hard very interesting but it

also drives a lot of business value so that's kind of what we try to align ourselves on and I try to make sure that people always learning and growing and many many many ways and that could be infrastructure architecture it could be

you know process as well but so we still have alignment and we make sure people are growing and I think that's ultimately what's what's most important right and even even from my experience what I've seen is that in in start-up

worlds many more companies have actually failed due to over engineering then do you do under engineering because what'll happen is you know they'll get they'll get a seed of round they'll get funding

right and then they'll spend a lot of time trying to perfect a particular technology instead of experimenting as a start-up the best the best thing that you have as a start-up is agility wait

you're like a small ship that can change course really really fast right but when you over index on a particular engineering problem or try to perfect something over optimized for speed and things like that I mean there are

situations where you might need to do that but and that might be or you know yeah you're cutting edge development but a lot of startups have actually failed

due to the fact that they over engineer way too fast in their lifecycle right okay cool so now what I'll do is I'll

divide the next few sections in into to two different parts the first part is let's put ourselves in your shoes before the series a right before when you were

just starting out precede seed era right and then we'll talk about and then we'll put ourselves in the shoes of you know after you right now what are you doing right now but let's talk about

engineering before doing the first few parts right how many people were in the team how did you folks like you know what were your engineering practices how

did you put processes in place to be able to really develop and move as fast as possible yeah so when we first started it was I just made my co-founder we would just you know I would build things and here

we go kind of pitch around and then we will move back to Toronto in 2016 and we were in DMZ about four or five people I'm again super scrappy just you know work on stuff there's no road map

there's no sort of proper sprint process it's just hey let's go ship something like you know we have a initial idea of the vision and we wanted to build let's go ship it and then we talked to customers and we figure that out so it

was very very lean and process like if at all because everybody you know we sat in this pot and we just turn around talk to people and then as we scaled over time we had a little bit more process so we started doing like a light kind of

Kanban process you call it we use new Trello a little bit maybe in the scrum master for another kind of scrum like tool and then as we grew more and more we kind of matured as an organization we

had different teams we had a p.m. we had

a p.m. who can help us drive processes and then we also moved to more proper sprint process and we had different teams so each team had their own swim process that's been planning and review backlog remiel all that stuff so yeah I

think it's sort of the organization evolves over time and it all revolves based on the needs and continuous improvement and feedback and these are the things that we learn over time we kind of see what works what doesn't work

what we're where do we see processes breaking and do we hire by people to help us improve that and we try to change our process to its or accommodate for that right but indeed in the

beginning stages when you were doing you know you've been the CEO you're probably doing a lot of architecture design you're so right how did you make those decisions like how did you know like you know now is

the time to use a pops-up mechanism for example or now is the time for us to scale up and and optimize our front-end to make it faster or any of those aspects yeah those are good questions I

mean the the front-end definitely a piece were I think I probably under invested a little bit me being not a traditional content person I think one of the mistakes I made was not investing

as much in our front-end because that ultimately we are still a b2c company and front-end matters we've been we were we took two very dog calm approach which is like just throw a bunch of tests see what

converts and then whatever converts you just keep it and that kind of became you know a little bit of a cluster Mack net actually of different colors and symbol lowers and stuff like that so we had to

revamp our front-end took a little way too long to do but now we finally finished our revamp our front end that's an example of a bad actors decision and we probably should you know hire too strongly to get from the start to really

make sure we use the right design system the right patterns there there's the back end I guess I better start up before this I was like Google I were to start up the I was there like when there were 10 people the CTO was exitos inga

so I got a lot of perspective just from other mentors of the people in industry also himself experienced a lot of just reading as well so for the back and stuff I feel like we've made mostly good

decisions we decided to micro services pretty early on and that's mostly helped because we can deploy things faster each service can be owned by a team or or each team can will multiple services and

they can be independent independent monitor tracked and all that kind of stuff that really helped with our deployment process and then in terms of the machine learning architecture we had

other people in company who were really strong at that so I delegated that to them I didn't design entire workflow pipeline from like training validation to deployment to inference a back to

training and all in a cycle and that you know they've been really they've handled that really well same with our data architecture with a data guy on a team who drove the process end-to-end we don't wait some mistakes

on tracing about some platforms we use certain tools that weren't scalable when we do but overall like delegating the pieces what strong leaders helped a lot and yeah I fortunately had some experience and

overall architecture and back kind of things right that's awesome can we dig into the machine learning pod a little bit further you mentioned that you have a machine learning pipeline

right what is the pipeline look like right now and and what was the rudimentary pipeline and how has it evolved so far because I know a lot of companies don't actually have a pipeline

originally you know they build a bunch of models and then one of them goes into production at some point right there's no like continued deployment cycles there's no way for them for someone to convert

a pickle filed into a into a micro service in an easy way right so what does the pipeline look like right now yeah as a question I mean I think it's still pretty scrappy right now when we first started it's just summers laptop

run children to put attributed notebooks and stuff like that right and then over time we had a more than normal phone process so we we we added a data

labeling tool called Pagosa where we have labor labor data and that I first know just gave a CSV we upload cs3 and then we use that and then we eventually move that to the data warehouse so now we ingest a little into snowflake so

basically everything is locked so we had a locking infrastructure to log every event and every sort of click and label data we log that to a genomic event

stream through through Kinesis now flow Indy to snowflake and then we have our labelers labeled data in snowflake and that we use there in snowflake to Train

that data it's still actually you know in a local desktop because it was pretty expensive doing the cloud with the GPUs so we just bought like a giant giant GPU machine and we just trained it locally

and then we deploy them train model to s3 and then when we deploy our services it downloads it the model that the pickle file essentially from Esther into our server and it runs that file so it's

still pretty scrappy but it works it's it's at least it's you know version pipeline and stuff but we try to keep pretty lean on that part that's awesome but it's actually really good that you

guys are doing a lot of the training and development within your on premise right because I'm sure there's these issues about data data sets that you don't want

to put a public write at the same time GPUs are so expensive these days right I think a GPU cost like 2 dollars and 73 cents per app per hour which is which is very very expensive but in terms of okay

so now you have put you've trained a model you put it into production right do you do any sort of more entering on the model itself or how do you know when is a good time to retrain the moral or

recalibrate it or how do you Ben figure out like if there's any like your standard deviation is shifting or there's any concept chips that are happening at the moment yeah that's something that's a good question is something we're not super actively doing

right now so right now it's mostly just a batch process we you know like we collected data so we collect feedback data from customers where well sometimes they pick up if the models not confident it'll ask is this

correct yes no it's just what you meant and based on that data will collect feedback and also we have agents in my customer service agents that can also train the models so a lot of times well surface suggested replies to our agents

and who can say this is relevant yes/no and we'll use that data to try not to also inform them of our model but the actual online training it's not fully online yet we had some debates and Charlie about this whether we should

make it fully online or not and we decided that it's okay to just keep a batch so what we just do now is we every couple weeks or a month we would get the new fresh data passed into our labeler

joining with our customer feedback data training the model calculate our f1 score and then due to the poor turn the motor Claddagh etc I see okay that's cool okay one last question on this on

the precede side of things right also I'd the stuff is like after a series it's not really appreciate it was just like true parental books and like that's it okay slight push to production okay cool

but but from the PC side you know you're a CTO which is in the PC side is about to you know raise a raise some capital right what was one it could be a podcast

could be a book it could be any kind of advice that you had at that time that was really helpful for you and in what context like technical fundraising in

terms of technical concepts technical concepts that's a good question for me to Mars I've been mostly focused on the process side of things you know I I

think I used to think about tech and DevOps as a technology thing but I've learned that a lot of it actually is processing and it's about addressing systemic process improvements as opposed to like

just a technology improvement and that's something I've been really focusing my time line across organization as I learn about or experience building a team building agriculture building the right

tools and systems a lot of this comes back to to process and it's not and it's the people behind the process that really matter not just technology because technology comes and goes and tools come and go and what's the hottest

what's hot today could be cold tomorrow but having the right people the right process the right sort of workflow really helps the team do more again when that pastor helps the culture be more efficient so that's some

of the stuff I've been focusing on and so they're not unfortunately not a direct answer to your second question but it is related because your tech output ultimately depends on your process and the people yeah I think that

makes a lot of sense I think even as as DevOps engineers we we tend to over index on what technology to use what CICE pipelines to use but instead if you don't have any kind of like behavioral

change within your organization itself all of those are really useless right okay so now let's fast-forward a little bit after where you are right now wait you've raised a series and now you're

trying to scale up the company right you probably had to hire a few engineers outside of yourselves right so what's what's the headcount like at snap travel at the moment yes there's about 60 people in Toronto mostly product

engineering and then another 80 overseas for customer service awesome so after you raised this capital around you know why did you decide to stay in Toronto and and not go back to say

California or somewhere else and what did you decide to hire some of the core engineering talent and your torrent of office itself yeah ass good question so I actually started the company in SF I was working

at Google in California I quit to start the company and we were there for behalf of year until we found you know our initial idea and we moved back to Toronto in mid-2016 and that was kind of

due to many reasons one being my co-founder his wife is here and family's here most so I grew up mostly in Toronto and I went to want to loo as well so part of its like family and family but

also it's just that the access to talent um I think Toronto has a lot of great Canada in general has a lot of great talent and it's not as competitive as it isn't it a big area where you know one

day somebody's raising twice you're raising and offering to pay your engineers twice I mean it's still competitive don't get me wrong you know people try to poach all the time but it's not like that crazy so I think it's

just access to talent and also on top of that there's the level of focus where I think there's always the next you know unicorn the next hot thing elsewhere in Toronto you can really

execute and grow the company so that this combination reason yeah I think that's what I found is well Desa is also based in in toronto region and many of our the reasons we've hired

so many people in the toronto region is because iran has just been so great for the AI community or the machine learning committee you know there's like thousands of meetups that are going on every single day there's people writing

research papers from Toronto like on a daily basis right and we've just had such incredible access to talent just it like you know a walkable distance from Toronto you know to be able to find this

talent yeah and also we actually want one of the things is we've hired a lot of people who wanted to come back from the valley so one of our managers was a single well I think over half our team

has some sort of experience in the US and in and that's actually I think is good I think you know as a young person it's good to get that experience in the u.s. to you know be kind of in a place

u.s. to you know be kind of in a place which is hyper competitive and also has a lot of smart people but also people wanted to come back for a family to settle down whatever and we get a lot of talent that way and you know that's that

we sort of for us we think we try to build a fast growth Silicon Valley style company within Canada so we capture a lot of the talent here I think it's very similar to you know terminal what

they're doing did you do a lot of the hiring yourself at the beginning yeah so a lot of hiring myself and then we hired a HR person to help with that and that's helped tremendously

but yeah hiring is a sort of a company-wide initiative I see what were some of the things that you do you have a particular style in your interview process like how did you reach out to

people to be able to get them to join snap travel at an early stage yeah I think it's anywhere and everywhere you know it's like LinkedIn and AngelList email caldrich or whatever like whatever

channel works and people reply you you go and you know events in person whatever and and yeah hiring is tough it's always tricky and that's a constant iteration process I wouldn't say I've

you know nailed my hiring funnel I still think we can improve on that and we're kind of continue trying to optimize with like different messaging different reach out doesn't follow up with periods

different you know articles we said it's it's a it's it's a competitive process awesome how do you think your role has changed now when he have scaled the company a little

bit as a CTO versus what it was before the precede stage yeah that's leffen one thing is I quote a lot less now so we know early days I was coding almost everything now I'm

sort of coding very little if at all any day and I do maybe 10% at night when I'm I hope but a big part of my role now it's about how do I focus on the process

and the people within the company how do I make sure you know the leads and team needs are successful do they have the right support the resources the training do they have do I have the right people in place should I hire somebody should I

bring somebody on board should I swap teams for some things a lot of that is around the people process the process of things I also do for me it's just I like a lot of strategy stuff so I began an

awesome strategic side of the company from a purely tech product side my day-to-day now is a lot around process like what do we do why do we do it how do we do it and do we have the right

people to do it so that's been sort of a shift for me cool awesome I think I'll ask one last question before we jump into a little bit of Q&A I know we've been talking for some time now let the

audience have some questions but one last question this is more of a future facing and also reflecting on the past right we've seen chat box become a really Fame

you know big thing about four three four years ago right so there was this you know peak hype cycle for chat BOTS at some point and then after that it slowly died down some business models have come

out of it right some have gone successful some not but the hype has really been stable right but we're seeing a little bit of a pickup again right you know companies like yours snap travel there's a lot of other companies

that are also coming up right now who are doing different kinds of NLP work for other aspects of different kinds of industries but why do you think that that that cycle happened and also what

does this mean for staff travel in the future and what do you see that conversational a I go at a little time yeah it's a question I think you know I don't know people remember when messenger just launched chat but SDK was

a crazy hype wave and I kind of crashed but I think it comes back to what I was saying originally is that technology ultimately has realvalue if you look at the history of

the humanity technology is always created value and when you're building on hype when you play you know if you play with the trap lots and early 2016 you'll be like 80% it don't create any

value right it's like a chatbot that you know tells you like what's the weather well why don't I just open what I have and so things like that where it was it was around creating hype and novelty for the sake of analogy and I wasn't around

kidding me whereas we've always been about creating value for the customer and part of its convenience part of it as recommendations part of its price part of the service but ultimately for

us as a company we were very focused on creating value for the customer and that's ultimately why we're able to grow and sustain you know and that in the high wave came and came and came and went but we've been growing consistently

you know doubling tripling every year for the past three years because the focus on the customer and ultimately I think that's what matters and that's what's gonna win and so in terms of looking forward I think a lot of

companies that survived the the crash was one of them that crashed but the depth what the companies are focused on creating value I think you know my friend Mike had a de right there about

creating using technology to empower better customer service and they're creating real value to automate you know cosmodromes agents save companies money and create maybe even better chat

experience chat service experience things like that I think will emerge and yes I believe there's world where chat is better but it's not better for everything and that's where companies

have to really pick their battles to make sure they're ultimately creating real value right yeah totally and and I totally agree with that it's the same thing with any kind of like machine learning work weight machine learning has really existed for a very long time now

but it's just now with the advent of like cloud computing and advent of like being able to use GPUs remotely it's just much more accessible that a lot of people can create value out of it right

and that's where we're seeing a huge rise in the in the peak right now what you say so yeah sure I think you know it's like yeah there's a cause that's of creating value and I think it's well now we're seeing companies take that

technology and really automated scale and how many things to do things better faster cheaper