Getting to know Temporal

[Music] hello and welcome to this session uh we're really excited to have you here today what we want to provide is really

a basic Foundation understanding of of temporal the technology and how it's used in in with bi-developers across lots of you know leading organizations all over the whole planet um

I'm joined today my name is Jim Walker I'm the VP of product marketing here at temporal and I'm joined today by two my esteemed colleagues that I'm going to let them introduce themselves so max let's start with you I'm Maxim I'm a

co-founder of temporal project and now I'm with CEO in of the company that's right my ultimate boss that's right in Ryland uh I'm Rylan Goldstein I'm the head of

product here at temporal and I'm an early employee of the company yeah I mean how long have you been at temporal Ryland uh like three and a half years a lifetime a lifetime and portal terms for sure so between the two of you I think

we have more knowledge than almost on the planet around temporal uh well there's a there's a couple other people that have a lot of really great knowledge um what we'd like to do is just run through a couple just a baseline understanding of kind of where

it fits we'll talk a little bit about the history of the project and how how Max started and and doing all this work and then a little bit about some customers um so just to get started

um you know as as application Architects and as developers you know we we draw pictures uh and often these pictures are very very generic on this slide you'll see uh you know a very generic

understanding of a system this is just a simple order flow diagram um and what happens is is we design the system and then ultimately you know developers and and operators and Architects have to go out and Implement

these things uh we'll take this we'll cut it down into you know microservices we'll use cues we'll use timers and databases uh what ultimately getting gets delivered in production is fairly

complex and it may seem simple uh you know at its at its surface layer But ultimately inside what we're building the complexities really start to compound

and honestly as you add more features they get exponentially more complex right and so while these these new architectures these modern architectures kind of Promise efficiency and ease and

kind of you know a simple way forward to to implement these applications they often present a lot of different problems because what can go wrong will a database can fail a service can go

down I mean I've spent years in the kubernetes space you know pods coming up and down all the time cues can back up and and ultimately like you know apis

time out all the time uh and and how do you deal with these things as a developer I think in a in a pure state where none of these things go wrong things are pretty simple to deal with

but when things go wrong uh you know we're set dealing with rollback or or retry policies and these sort of things and I think you know people struggle with this sort of stuff and ultimately it detracts away from doing what we

really want to do which is basically deliver the business logic that we've been we've been asked to deliver and and really ultimately even when things go right we still have problems in these in these complex architectures how do you

share State across various different things and I maybe not everybody thinks about it estate but uh where where am I and do I have insight into this overall process for each person that's being

executed across this you know this this complex flow you know can I do uh can I can I perform transactions to a single database across multiple different multiple different Services maybe you

want to do that um you know how do you implement long-running timers how do you wait a month for something to happen let alone a month maybe even three hours for that matter or even 10 minutes even a minute

um are we just putting pauses in code you can't really you know pause pause code so these things get get get really complex even when everything is is working well

and you know I think ultimately and we kind of said this earlier you know as you add more features as you add complexity to the application as the architecture expands it gets even more

difficult and even more complex um you know if you think about retries and rollbacks across three services well what happens when it's 15 or 20 or 30.

the complexity just really compounds and then as you scale and you get more customers you know reliability can suffer as well because there's just more components that can go down and so ultimately you know I think developers

are left dealing with this like they get caught holding the bag right because they're the ones who have to develop all this stuff and they they lose productivity um it gets more difficult to deliver

features faster uh you know the the reliability of our systems as we mentioned kind of goes down and again I'll come back to this this really important point it's really really difficult to get this kind of end-to-end

insight across all the process flows that are going on in in whatever the system is and the application that you're that you're going sure we we may have some observability and some tracing across the services but what's really

going inside that service and how do we know what's actually happening end to end and so ultimately you know I think over the past couple years you know I've seen this happen anybody could go to microservices.io and look at some

development patterns on how these people are how people are actually doing these things development patterns are interesting and I think it's uh the natural inclination of developers to share and figure out how to how to do

this over and over again and I think some of these patterns like a dead letter box uh you know like a saga pattern these sort of things uh they're fantastic uh but but often they'll serve

only one part of the problem uh and I think even further complicating the issue is is we leave it to humans and each developer to implement them in their own way

so they're also prone to to errors or complexity and I think as we move on and I and I think you know ultimately there is a better way to do this Beyond kind of development patterns Beyond just kind

of manually coding all these things and you know one of the reasons I joined temporal was exactly this I think this is a problem that I think I was aware of when I was a developer but I never knew there was going to be a way to actually

figure this in fact I think I I stopped being a developer because I hated doing all this kind of chaos the stuff that I didn't want to do I wanted to focus on business logic and when I think about temporal I I think about it as you know

a developer is is now able to kind of you know build for kind of the the single positive state that they want what is the what is the the way I want people to actually flow through my

application and not worry about the retries the rollbacks uh you know the the the cues the timers uh and and temporal really abstracts away all of that

um we allow developers to code in whatever language well not whatever language but we have sdks across you know go PHP typescript Java python guys correct me if I miss one in the end

there but um uh dotnet.net that's I knew it Max coming soon that's right that's right that's right um you know we allow you to code in your language uh you know Define this workflow which is this end-to-end kind

of positive state that you want in your language and then instrument your code using an SDK to to interact with with temporal so it is really about abstracting in a way this complexity so you can focus really on what matters and

it has a fundamental impact um really on the way that we think about applications I I like to think of it as almost a paradigm shift because I think uh you know the developers I know that have used temporal it's like kind of

once they see it they can't unsee it it is really a a different way of thinking about your application I think it's a incredibly interesting way to to approach software development so

um but I'm joined by you know that's my own my understanding of it I've been at the company and introduced to temporal about eight months ago but I'm sitting with the person who actually created the

project and Max this is not the fist like your first incarnation of this approach is it right I mean there's a history here of you doing this for quite some time right

uh yes uh it it took us probably 15 years to get where we are it's a long time so where did it start like I mean how long ago and how long ago was 15 years as you

told me but where did it start for you I started probably okay I joined Amazon in 2002. yeah yeah and back then Amazon

in 2002. yeah yeah and back then Amazon was I think one of the first companies which decided to kind of break the big monolith into smaller pieces which they

called services and now we call them micro Services back back then it was all services and the reason they did it not because of some kind of

abstract desires or architectural principles it just was taken 45 minutes just to re-link the binary of Amazon website back then so imagine the

developer experience um and uh and I think clean build of that uh binary was 18 hours wow so uh the reasonable solution was to break it into the pieces and then these pieces

had to talk to each other and I was part of the infra team which was responsible for pops up kind of all the messaging between those micro services on the back end and at the end I was checked it for

the service kind of the storage for queues and later it was adopted actually as a simple queue service AWS sqs uses that as a back end uh is somebody who

was responsible for practically talking to every team which was adopting our technology and that ended up kind of getting exposure to a lot of use cases and became pretty clear that Q is

actually not good way to Link services together if you have complex transactions and Amazon Fulfillment flow is non-trivial it actually even back then it was non-trivial I cannot imagine

what it is right now and so we actually uh kind of started to think how to solve that and orchestration became pretty

clear answer to that and Amazon had two kind of multiple iterations on that and uh so practically which resulted as a Amazon simple Buffalo service which is

still publicly the best service but we had two internal versions of that before and then yeah later co-founder of temporal Samar who worked with me on a

simple workflow into Microsoft and uh long story short where Microsoft has a Azure durable functions which are based on the same idea and later we came to Uber and

started like one of the projects we started was Cadence which uh Grew From like zero to 100 use cases within three years and it was open

source from the beginning and I started to get popular like adoption outside of uber and we started a company three and a half years ago and we started we

created a team portal which was kind of continuation of a fork of cadence and we added a lot of new features and now we have to borrow so I I'm sorry but I lost track there how many different it was

what there was one five different incarnations there were some others in between and also it's not only about back-end service it was about client-side experience yeah and we uh I

built at least probably four different Frameworks uh for client-side and first ones were kind of more traditional ones like uh which was kind of oh you create

this DSL and you kind of have objects instantiating them this like there go like uh syntax tree of things you have a sequence you have like split join you have kind of all it's just anything to

think there are like million incarnations of those yeah and it worked uh but I quickly realized it was not a very good developer experience because I just uh much much harder to understand

and troubleshoot and so on so we kind of ended up we implemented this what we call AWS flow framework which was the first idea kind of similar to what temporal does so you just specify your

logic and code directly without any intermediate representation and it didn't we didn't nail the developer experience probably not many people know about simple workflow but later

Microsoft Samara did I think a awesome job with dotnet I think framework like a weight I think and a tuber we did go SDK which was actually synchronous and I think we kind of nailed the developer

experience while doing that well it's interesting I think it's the kind of the core principles that when it is Step functions as well and I think there's this you know it's by my experience as

far as I talked to you know people users of this and I think there's been more than just the five or six that you and Samara have done I think everybody's trying to solve this problem in some form or fashion I think you know once

you see what the problem is it's like oh gosh how do I how do I deal with this you know I think what you all did at Uber was was amazing and you know creating the Cadence project and open

source m T license and then starting a company what makes it right this time um you know like this this incarnation of temporal by the way I'm a absolute believer as well right so what

why is it right this time around can I try to answer it for you first that'd be great it's right this time because it wasn't right all the other times interesting that's good I like the one most constant message that I've heard

from him is that like when people are asking you know like why should we just assume you know all this stuff and you made all these right decisions he's like well I didn't initially that's right first time I made the wrong decision and now I have this long list of decisions

that I should make correctly the next time I have to make them yeah so that's at least what it seems like for me yeah I think the idea is that it's not like we dreamed this overnight it was

just a lot of iterations and a lot of different mistakes and certain things we took us for up to five or six years just to come up with the right abstraction and I think just as a project why I

think we've got to try it is that first we've got developer experience I think correctly and it was a big deal at the same time we built a lot of projects like that they focus on developer experience and forget about the back end

right but because we built simple workflow service before as AWS service so we built this as open source project as a highly scalable practically cloud-based architecture from the

beginning and it was practically so far we couldn't find the point when it breaks yeah it can saturate almost any store practically any storage we we oriented with 300 Cassandra clusters we

have our custom storage which is even more performant and we always able to create a new database to 100 and then and the third thing was because

we builted a tuber as an open source project and uh initially we didn't get like huge adoption but we were running hundreds of use cases high frequency is a tuber uh so practically we kind of

nailed the operational uh excellent yeah because we had to run it for Mission critical use cases and so it means that we kind of got developer experience we got it as open source and we run it at

scale uh at the large company and then uh we've got adoption from external companies which was amazing because like one of the first users were companies

like coinbase Airbnb box uh like doordash and uh this cap these companies uh certain hashicorp and these are not like companies which would take these

things lightly but they trusted us uh because again because all of these reasons yeah I think it's really interesting and I you know I I concur on the developer experience for this is just fantastic you know being able to do

this across multiple different languages you know polygot applications that we have it it's right and I think if you're going to change the model you kind of have to do it across all you couldn't do this in a single language you know what

I mean and I think the the operational kind of getting it to it to a point where it's great um this is all amazing uh do you want to show us the technology Max I mean you

want to give a quick demo okay yeah let's do it I I've been in very few companies where I just asked the CEO to do a demo uh but that's our best demo person right there

because he built it uh so first we run the temporal Service uh which kind of beckoned and usually it's a large-scale distributed system but you can run it as

a single binary with a in-memory database for developing purposes how can someone get that binary if they wanted it yeah you just to do Brew install temporal and if you're on Mac

and in automatic there's a release on the GitHub and you can curl it and install it like a normal Linux package so I want to demo a very simple uh money

transfer use case so practically we have two operations we want to withdraw and we want to deposit money back and if you want to write that like the business logic of that is just two lines

of code right so we call this draw and we call deposit uh obviously nobody writes code like this uh in normal life like at least in

production because obviously if something fails in the middle here uh your process crashes and you lose money which is actually not a good idea

probably if you're a bank so the interesting thing that you actually can write like production code like this with Steam portal so the whole idea is

that temporal guarantee is completion of this code we call it durable execution uh of workflow kind of interchangeably just for kind of Legacy reasons but any

workflow code is guaranteed to complete and keep running in presence of any failure so it means that for example if a failure happens a process crashes

after line 35 here that uh temporal will automatically move this state of this particular manual transfer to a different machine and recover that and

it will keep running so as engineer you don't even notice uh in like as a programmer that there was a failure so it's very powerful because you practically do need a huge class of

issues that disappears you don't even need to think about process crashes other thing if you call this draw service and this service is not reliable uh temporal will automatically retry that as long as necessary so there is no

limitation of duration of retries unless you specify it explicitly in the configuration options and all this retries are fully configurable it's exponential retries and so on and there are a lot of features like rate limiting

flow control and so on which I don't have time to spend on so uh if you want to do this money transfer how would you initiate that so this is actually workflow code and it implements an

interface in this case transfer interface you see I'm in Java uh as we said we have uh sdks and a bunch of other languages but for Java you would

not take a workflow is this workflow interface and Main Muffler method with workflow method otherwise it's just normal Java code so do we implement this in a way that was meant to be very adhering to

temporal or did we try to marry this to the actual idiomatic way that like a Java developer would Implement something in a framework themselves no this is just normal Java right but it's nothing magical but you can use all the Java

constructs so for example if you want to say okay what happens if withdrawal succeeds and deposit fails and in them I didn't do that but then it will just use normal uh

uh uh exception handling from java right and you would run compensation here like let's say put in money back on the original account or you can Implement

Saga full Saga pattern here if you want to so uh but this is like just normal Java code if you want to have a loop if you want to have if like it just you can use all object and design you can move with

another method to even class so just there is no any limitation what you can do here there are some some rules but like you have the full power of java when implementing this code

so let's just try to run it and so uh the idea is that there is this backend service which I just started before uh kind of we have it running in this

window but and also uh when you start the workflow you actually call into that service uh so the only requirement is that temporal Services run when you're

starting that the way you would start it you'll connect to this service all this kind of code is just to connect to the service and then you will uh

just call uh start on that transfer method which we just described here and when it will start is accepted uh

the uh this process will exit and transfer will continue so let's request the transfer it will take some time mostly because it will try to recompile also grade always

not the fastest thing in the world okay we request a transfer we can see that it printed uh okay we don't even want to see what printed imagine it's a distributed

system with thousands of processes running uh temporal provides UI so we can uh this is again uh this local binary which I was running or includes

UI it just hit it on the local host and we can see that there is instance of this workflow running by the way this workflow ID is assigned by you so it's usually a business level uh identifier

and you can use NV granting uniqueness it's fully consistent system so you'll be guarantee uniqueness of workflows by ID so things like idem potency and uniqueness I it's easily to guarantee

because you don't want to transfers with the same ID going at the same time right and do users have any control over that behavior in terms of the ID usage and reusage policy so you cannot

have two workflows open at the same time to save ID ever it's guaranteed but after it's closes you can choose they want to run workflow again with the same ID or they want to run only if it failed

or never run it so you can say never allow running again if it completed or failed for example so here we will see that workflow we have some workflow with

this ID which we started and we can see input arguments of that so these are arguments I passed as a parameters uh there is no result because it's not

completed yet and it will say there is no walk around why because in temporal temporal doesn't run your code even workflow code it's a citizen external process and you deploy this process

anyway you want to just part of your service it's like the way you would for example have your process talks to a database and this is your process talked with temporal backend and workflow code is not running this way this workflow is

not making progress more like qued up in the system so let's run the Walker so I have kind of uh and also in temporal we have a separation between

workflows and activities so workflow would be the code which is uh as I said durable execution is Fault tolerant survives failures and activities just code which talks to external services

and it can fail anytime and we just usually retry it the work focus is doing the orchestrating whereas the activities are the tasks yes it's a workflow it's orchestrator right it's orchestrates

orchestrates those so we we started to run a walker so we can refresh and also just wanted to clarify it makes it sound like what you're saying is that temporal server itself is never running your

actual code this is always taking place on the worker and just events are transparently communicated back and forth on your back yes it's a it's a temporal Services more like middleware right it's kind of it's kind of

encompasses queues and databases and all of that so from your point of your run code your application runs both workflow and activity code and connects to the backend server for State Management and queuing and also durable times and so on

so now we see what happen is that we have a deposit pending activity and it means like how this draw activity probably completed and we see that it's

already it was executed five times and there is a failure going on so we can go and check on that failure and help an account

implementation line 36 and let's find account implementation line 36 and yes and there is exception here actually put this exception just

for the demo purposes but we can see that this activity has been retried and deposit so let's fix this in real life you would just probably fix bug in your code and always fix this service because

maybe there is some other issue is that and redeploy in this case we just restart the uh uh where is it

account activity worker yeah here it is let's just restart this process I use a new code and let's go back to our UI and see what happens so at this at this point

temporal server has statefully and durably tracked the position where my code is executing in the workflow and essentially it's just trying to execute this activity right now and activity is

obviously failing and so what you're trying to do right now is just fix that activity implementation so the next time it retries it'll completely yeah and also obviously I can also kill the

workflow worker which holds the workflow code so we can restart that and we will see that it will recover to exactly the same stated it was so it will adjust it

will like both local doesn't even need to know it was restarted right so we were at the workflow with the line uh uh 38 and right now probably it will

just come already completed because uh it retried the activity yeah so yeah workflow is completed and we can go back and check okay we executed this uh activity

uh first activity withdrawal it was scheduled uh at this time these arguments of this activity it was

started by this process so you see every Everything which happens with what happened then we've got activity

uh deposit scheduled and this activity was executed eight times it failed seven times this is stack trace an exception why it was failing and then it completed

it some time right and then workflow completed so think about it I do need to restart workflow I didn't to do anything I just fixed my bug and the system just

recovered itself and he just won but if you have like million of those running at the same time it's pretty cool that you don't need to do anything it just code recovers automatically in presence of a process crashes and so on you don't

need to do anything as we saw that quote doesn't uh contain any it doesn't need to contain any air logic related to intermittent failures or outages this

area located here is only happens if you have business level failure right for example deposit is not possible because account doesn't exist but at the same time you don't need to handle a situation when process crashed because

it just handled automatically I don't want to spend too much time on that but for example we also have unit testing framework and it means that you can test these workflows even if you have long

grinding uh for example you can absolutely write production code which is workflow.sleep and say something like

is workflow.sleep and say something like duration of this right and sleep like for three days here and it's a blocking call so

it's okay for the process to for example sleep for three days and then continue and then you can Union test it in milliseconds because we automatically skip time for example okay uh so just to

review that you can write a normal code and you have full visibility into this code processing and everything is recorded and you see all I put inputs and arguments and this code recovers

automatically after any intermittent and deployment or failure very very short introduction it's a short introduction but it's very powerful as well Max and I think you

know some of the things it's like yeah there's retries rollbacks we've saw a bunch of stuff I mean even timers were in there at the end right I mean I think schedules is interesting why are people

dealing with KRON today like you use temporal to run things like that and this was just money movement right this was like a bank transfer which you know I think is one of those ways in which people use temporal today you know

high-end transactions where like a bank like you said like that can't fail man it's money right like you it's it's got to work right it's got to be reliable it's got to be durable I think is the word that we like to use around temporal

a fair amount um you know but like this kind of durable transactions is kind of one thing I think people use it for like say you know business processes I don't know I'll ask you I know rylan's one of his favorite use cases is more of a

business process you know people use it for like Inventory management or uh you know you just just understanding things and and having insight into things I I've even seen you know the I think one of the common use cases is like

infrastructure deployment um you know I've seen people end to end you know bring software or you know bring Hardware up run software bring the hardware down like it's incredible like

some of the things that people are doing with temporal um Rylan you've been on the front edge of a lot of uh people using temporal and you've collected a fair amount of like user stories what's

your favorite user story I I think it's impossible to probably just choose one I will say that one thing I want to just make sure I don't uh forget for you know what Max is kind of showing there is that like temporal itself almost does a

disservice to its own technology by making things look so easy and so much more simple than they actually are and so even just something basic like like implementing you know retry statefully

that's an immense amount of code like any developers had to do that no that's daunting and like that's just something that's implicit in the demo that you just saw and so I just wanted to make sure that that was highlighted there in terms of the users I'll choose like a

couple um I think you know some of them are very exciting because they're products that I use and the fact that they're a user of ours is just cool in itself and So like um one example in

that uh areas Yum brands they're the company behind you know a lot of major fast food establishments that probably almost everyone has eaten at once or twice in their life and I'm someone who

eats like hey man tacos pizza and chicken I'm in exactly right and so it's uh it's hard not to say that they're you know an amazing customer just on that basis as someone who spends a lot of

money at their establishments um so I think you know the other part I love about that is kind of going back to this idea that things can look very simple or sound very simple but in reality not be that way you might think that a process

like ordering food or having it delivered isn't isn't that much stuff there isn't that much work there but it's clear from you know the presentations that people have given like Matt from brands that there's an

immense amount of complexity there you could spend days just talking about the complexity of delivering someone's food that orders that you know a Taco Bell and so I think that you know the fact that we took something that's so meaningful to a lot of people's lives

and we were able to you know make it deliverable at such a high scale and with such a great developer experience that's super great exciting for me to see I think another type of customer that's so so exciting is one where I'm like a very strong user of their

products or you know I really like the the brand they've created like datadog um who for like a lot of developers is like a really representative company of modern Cloud development modern Cloud

Trends uh and so you know datadog is unique because they're not just going in on temporal for one use case at this point they are probably one of the most aggressive users of temporal um even beating us out in a lot of ways in terms

of how they adopt and use the technology and so you know it's amazing when you have users that actually are teaching you about your own product on like a regular basis and you know not just teaching us but contributing and you

know helping us deliver things like the temporal CLI experience and all of that and so I think datadog has been like a wonderful partner but also like one of the most Exemplar users and contributors to everything we're doing here at Temple

yeah and the the taco I'm sorry but I got to go back to Young Brands the taco one is the one that actually sold me on temporal in the very beginning it's very simple to understand because I think everybody's gone through that experience

you walk into a store you go to a kiosk you hit the thing you order something the funds get held for a second your credit card doesn't get charged until you get that product right so that happens it goes to the cloud somewhere

Central server it goes back to the store a chef picks it up pushes a button makes the taco pushes a button when they're done puts it on a window the guy up front grabs it puts it to the front they push a button and then your number comes

up and Ryan goes and grabs his taco and then you get charged the amount of complexity in that small little what was that 10 seconds is intense and and all of the things that can go wrong in that

in that equation is is incredible you know that's just one time do it a million times do it a million times and and like if you think about like the Taco Bells like they're in all in like cities with amazing Wi-Fi like a store

can be completely offline and when we talk about durable execution right this this ability to kind of withstram pressure or damage and whatever that is and your system continues no matter what

it's incredible an incredible like use for them because I mean okay maybe they're making 50 cents a taco I don't know what it is but like that's money in the end right and so that's the one use Case by the

way that sold me on temporal I think that that that talk from our conference last year our replay conference was was really really just amazing so um when I really think about temporal I think about kind of like what we can do

that that that word durable uh becomes really really important and you know at temporal we Define this as durable execution because of exactly that it can whisper him pressure um you know this this is this is a

framework that allows you to survive damage of these complex systems that you're building but ultimately when we look at that you know it really helps you kind of deliver more features faster um I I always like to use words it

allows you kind of more elegantly fail you know you might still fail but it's going to be very elegant and you don't have the code for all those kind of situations and then you know maintaining the Integrity of your data and having

Insight end to end to all your processes I think is just you know truly incredible and and I I think don't any fails when you want it to fail that's right because it fails on business problem failures but not on the infrastructure Affairs intermittent

failures deployments and all these other things that's right it's truly amazing and Max Kudos like I'm I'm honored a worker because I really think it's changing the way that developers think and and really changing the way that

people kind of approach problems I've seen it happen time and time again I don't know Rylan what is the best way to get started with temporal so I think some of some of them you saw a demoed you know even with what Max did I think

you know starting with our temporal CLI and just having a very good local development experience um I think like you know philosophically I would say go and watch the presentations from you know amazing

people like Matt mcdole from Young Brands uh and you know uh Drew Hoskins from stripe and Jacob up from datadoga these are all people who are really really amazing at representing the

problems that they solved with the technology and how it transformed not just their lives but all the developers lives that they work with and so that's at least for me what kind of has the biggest impact and kind of and and quite honestly all it's open source so you can

go and start to use temporal today we talked a little bit about date dog their temporal CLI um you know and and if you want to deploy on the cloud we make it real easy for you to run it as a managed Service

as well uh but you know join the conversation our team is uh ever present including Max in slack and support and all the various different places so you

can interact with us we're more than happy to work with you to get you started if you have any questions but Max Ryland thank you for doing this today I hope you enjoyed it did you enjoy this Max absolutely awesome blast

awesome buddy I appreciate it I like talking to you guys anyway so yeah um ah that's thanks buddy thank you everybody