Christopher “Vjeux” Chedeau: Excalidraw: Under the hood of the virtual whiteboard

foreign [Music] I'm so excited to be here part of the next JS conf my name is Christopher Shido I'm also known as visual on the

internet and today I'm going to talk about escalator so actually draw is a tool called the virtual whiteboard but in a sense it is a way to get all of your thoughts onto paper and communicate

with each other so the way it started was January 1st 2020. so at Facebook we have this

2020. so at Facebook we have this process called peer feedback where you have to give feedback on the people that you're working with and this is very important

because this is a way for everybody to start learning and growing and being better now I'm a big procrastinator

and so when it happens I'm basically cannot focus on doing the things I'm supposed to be doing and I've started using a technique recently called active procrastination

and so what this means is I'm basically know that I'm not going to do the thing I want to do like pure feedback and so I'm actually going to like focus on something else and so in this case what

I wanted was to write a blog post and for the past 10 years I've been using a tool called zwibler that makes handwritten like diagrams the

unfortunate events is that that day the zwi blood soul wasn't working anymore and so I was like hmm maybe I could write it it doesn't seem super

complicated like seems like fun and so I started using this Library called rough Jess you know how to make this handwritten lines and I use code sandbox to prototype it so I put it on Twitter

and mentioned my surprise there was a lot of excitement about it so the next day I decided to create a GitHub repo and to add the Verso integration so it gave me two things so the first

one is people could actually submit per request and I could see live what it was doing and so I didn't know to I didn't have to import the pull requests that locally developing it I could just like

play around with what we are proposing it was much easier to get uh the live collaboration aspect going the second part is with one click I was able to get

a website up and going so I didn't do to do like devops or anything it was just working and so with this integration like my goal was to build a community around

this because I'm only one person and I still had this feedback to write I wrote it don't worry about it so uh what I did was my playbook with

all of the previous Open Source Products I've been running is to create a list of like the things I wish the project would have and so I think I created like 20 of them and a lot of people started

contributing and this was amazing so like a lot of energy and like the project moved really fast now let's talk about all of the

nitty-gritty details of how rendering Works within escalator so we are here today at next gscon which is Oliver reacts and the most fundamental idea of react is

that you get a piece of State and you're implementing a render function that takes this piece of state and generation outputs and so in the context of a scary draw

every single time you do a mouse move events within the app we're going to generate a new piece of states which is the list of all of the things you're seeing on the screen so you're going to

have like rectangles Stacks arrows circles and then you're going to do some magic that is going to generate how do you render it using this handwritten like

feeling first let's talk about text the font right now looks handwritten and this is not a coincidence alien or rap actually

took a handwriting and converted it into a font and so she did 20 years ago the good news is the font is amazing and it really looks handwritten and I really

like the like the feeling and the look and feel the challenge is when we started skating it didn't support any action created characters

so I'm French and I like to do my Oxon circumflex but I couldn't do it in these phones and so I actually reached out to her 20 years later and she agreed to

update the fonts and add like all of the regular accents and also a Cyrillic and Greek character sets so now this phone can be used by a lot more people

and not only this she also open source the font so you can go on GitHub and actually submit a proposal for improvements of the phones

so this is one of the great thing about the internet and open sources you can like reach to like people have like specific skills uh to make the project better

the second piece about the scary draw is all of the shapes that you can write for example you can do arrows rectangles

circles like all of those good stuff and in practice you can reduce it down to how do you display a line that looks handwritten and for this we use the

library called rough.js and in a nutshell the library takes a point a and a point B and gives you a list of operations in

order to make it look handwritten you can write you can look at the website it has a very uh in-depth documentation of how the algorithm Works which is super exciting

but this is out of scope for this presentation so we know the output of this is a list of bezier curves that we need to draw

and for this there's two ways to do it in the browser one is you can send it as an SVG node and let the browser do its things and the second thing is to use a

canvas element which can draw them and actually give you the pixels and so this is all our architecture basically draw

we start with a higher level Json list of all of the shapes for the text we send it to the fonts and for the shapes

we deconstruct it into a series of lines that we send to rough.js to get the list of operation and we render it and so you may think okay we're doing all of this

every single Mouse move it must be slow now in practice with the recent Improvement about with all of the browsers and all

of the machines you can render like hundreds and low thousands of elements at 60 FPS and so this is plenty fast for all of the general use cases uh you're

using for escalator now one of the challenges we started implementing zooming out and not only this but people started using a schedule like vehemently with

the infinite canvas and have like really really large documents and so with those two things in mind we were able to see in practice like tens of thousands of

elements and this started not scaling anymore and so for this we implemented one performance Improvement and this is implementing caching

and so caching is a way to trade off memory for CPU so in this case what we have done is to store two pieces of

information the first one is from an element high level description what are the list of the bezier paths we need to do and the second one is from the list

of busy pass what is the canvas that's going to be generated and so we are storing those two things in memory

and now the rendering operation is going to be much faster for the things we already computed because now we can just for every single

element in the canvas if they haven't changed now we can just get the corresponding canvas element and just do a copy to the final canvas and doing a

copy of like row bits of memory is the fastest things computer can do so this is like almost instant and so this is the thing that we've done

uh with the optimization button the rendering now let's go back to the timeline so we are in January 2020 and as you all know

three months later in March uh kovid happened and everybody was sent home and needed to interact with each other only virtually and so one of the big aspects

around this transition was this kind of like physical whiteboard experience turned out to be actually very relevant into how people were working and a lot

of people started thinking about how do I have the equivalent online and at that time a lot of people started looking at the Excalibur in order to be

able to do it and they actually started using it but there was one big issue facing with this system is that at that point a scary Road did not have any

multiplayer future so you could work on it on your own computer but you couldn't collaborate with other people and so one thing that we've done is uh

during this weekend right after everybody was sent home a few of us hacked together a multiplayer feature and this is again like one of the reason

uh why react is so good is that if you remember it's all about taking a piece of state and rendering the UI from it and so now we what we need was just to

take this piece of states and synchronize it with all of the other Trends and if we were able to do this nothing in the app would change we would

be able to get the live player feature walking so we went into a bit of thinking about like how do you synchronize uh these

states effectively and one of the system that we've done is very simple but very effective so it relies on the fact that in the multiplayer version of a

whiteboard if you are able to see the cursor and like what are other people doing at the same time you're very unlikely to have two people working on the same uh piece

of uh diagram together what's going to happen is you're going to wait to set somebody else is going to work with it and then when they're done now you can start working on it and if

both of you are really working together then you can expect as a user that things are not going to work well and so for this with this observation in

mind we use a very simple synchronization algorithm which is to add one piece of information on every single element which is a version

counter so anytime anybody changes this element the version counter is increased by one and so now when there's two people working on the same elements for example

changing the size of the element or the background color and the two versions are being updated now the one

with the highest version is going to win now in practice if two people are really truly synchronizing at the same time one thing we can do is to have every single

person also generate a random number and whoever has the highest random number is going to override uh the other's action and so with this small trick now we're

able to get the synchronization working and there's only one other aspect regarding deleting an element so using tombstones so we are marking the element

as being deleted so that if somebody else synchronizes that the element exists it doesn't reappear out of nowhere so with this two setup we've been able

to get the synchronization working and while it was very simple to implement in practice it's working extremely extremely well for this particular kind of application

so now that we've seen how multiplayer is working I want to talk about something else and this something is how well do I sleep at night so now you're probably wondering why am I talking about this

and so in practice one of my biggest worry with this is that in this setup the server actually sees in plain text all of the scenes that people are

working on and not only this if we Implement a persistent storage now the high chaos can see all of the scenes

being shared like since day one and one of the tricky challenge with this is like this is a side project for me and I didn't have like the resources

of earlier who was willing to spend the time to actually secure uh the server and make sure like no mistakes were made and so in practice what that means is I wasn't sleeping well

but the good thing is I work at Facebook at Mira and we have one app which is WhatsApp that is fully end-to-end encrypted and I felt like oh this may be

interesting for this use case so what does end-to-end encryption means it means that the clients is going to like see all of the data and before sending it to the server it's going to encrypt

this data so like the server only sees like an encrypted blob of things and it's going to send the blob of things to the other

clients and because the different clients are able to exchange like the private keys to read the data they are both able to read it but not the server

and so this is the same concept that we implemented with excavator now in a different context because we're on the web and so the

very much like the very thing that unlocked this for the web is the URL and one other thing you may not know about for the URL is that there's really

two parts of this the first part is the thing that you uh think about for a URL like HTTP and like the domain and like after like the slash like all those things and there's also like the hash

sign and one of the very interesting thing is that anything that is written after the hash sign is never sent to the server

so what this means is the server like when you load a URL like gets everything before the hash sign the browser gets it and then sends to a server a please give

Windows HTML for this part of the URL then the server sends back the HTML and then some JavaScript is being run clear inside and this JavaScript has access to

uh whatever is after the hash sign and so now we have a system where like the hash sign is can be used to store the private key that the server never sees

but all of the clients are able to like see it and share it and so with this system we're able to get end-to-end encryption working on the client

the other palette was needed for this is how do you actually encrypt and for this there's a new web crypto API that is implemented by all of the browsers now that lets you do all of the

cryptographic operations to be able to implement the system and so thankfully like the web has all of the Primitive needed for the

end-to-end encryption and so now I'm sleeping much better at night knowing that like even if something happens to the server like none of your data is

going to be seen by other people so far I've described the architecture of the open source version of SQL which is hosted on xcaregrow.com now

main design decision around it is with have the notion of an authenticated user in practice every user is has the data

in the local storage of their browser and this is how like we make it secure now this adds constraints and challenges so

for example if you want to share a bunch of different drawings with many different people and find it soon like who has access to what you can do it but you're going to

do it outside of the system so you're going to send links to a bunch of people and make sure like they don't see uh the right links and everything but this is pretty tedious

and so this is like one of the challenge around this the Second Challenge is around uh the funding model so in practice opens uh excalatory is open

source and we have a way for people to donate to the projects but what we've seen so far is that even though a lot of you actually contributed and please we

want more contribution right now the amount that we're receiving through that channel is not even enough to fund like one person working on it full time

and so for those two areas one of the thing that we've decided to do is to actually create a companion around the schedule and specifically with a product

for SAS and so if you think about Google Docs or figma you're able to get like all of the files and you're able to get like all of the fine grain control of like who sees what you're also going to

be able to get authentication and use your company's Authentication and then one advantage of actually dropping your company is now uh the

company can actually uh like sends uh like the form request for Excel party uh authorization so we can give you like the privacy policy uh the like the

security compliance like all of the Audits and all those kind of things which as an open source project is really hard to do and finally uh this is

able to fund uh the open source version so right now uh thanks to this we have like eight people working full-time uh on Excel draw both the open source

version and the SAS version so if you're interested in contributing you can use Etc plus and pay for the subscription

this is now the set time where we need to pathways I really hope that this talk was useful to you and I want you to leave with three

takeaways the first one is the react model is very effective for this kind of application having the data being separated from the

rendering and from the synchronization meant we could build like all of those three pieces independently and none of them if there were bugs or design

decision that affected one would affect the other the second one is where data lives and who has access to it is very important

and only going to be more so and I hope to show you that using cryptography and end-to-end encryption we can change the Paradigm and we can actually make the

data more secure for the people and finally the open source funding model is something that a lot of people talk about and I hope to show you that with

this excali draw plus SAS products there is a way to fund the open source version in a way that's effective and useful for everybody

so thank you and see you next time