Transforming Code into Beautiful, Idiomatic Python

you can be sure you're getting the straight story or they will correct me right in the middle so this was labeled a novice talk an intermediate talk and an advanced talk Al together I've uh I

put some of each in uh it's got a lot of code in it you don't have to memorize it as it uh goes by I'll actually give you all my slides they're already uploaded they are actually meant to be something that you take back to work and use

immediately go through all your codebase and find every where I said don't do this do this instead and swamp it and your code will be beautiful faster more idom and more pythonic and less like C

Java C++ and other languages not that those languages are bad they just have different idioms uh uh than we have uh I'm curious how many people in the audience have uh taken a class from me

at one point wow I'm I've uh taught over a thousand engineers in the uh uh last year and I've gained an appreciation for what things are really beautiful in

Python works great for people so it looks like I've got a number of former students here how many perspective students none oh okay lots of them good enough and as all of my students know

practically every example starts out with Raymond equal red Rachel equal blue Matthew equal yellow in case you're wondering who they are they're right at the back door right there uh the lovely

lady in little boy that is Matthew and Rachel all right I was supposed to introduce myself Raven hinger a python core developer uh if you would like to have fun of doing

this we can play a game whenever I put up a construct and say this is awesome clap if you uh know whether or not I wrote it okay so I get to talk about some of the things I created there are

some awesome things in there I didn't create your game is to decide which one so clap if you think you saw one that is neat and is uh something I wrote uh the other interesting thing on this slide is

the uh at Raymond H uh I use my account Twitter account differently than other people I try and teach python through Twitter I don't tweet when I arrived at an airport or when I left or anything like that it is technical tweets and so

I don't waste your time I teach python in little 140 characters at of time which is a very interesting challenge because you can give one little example and sure enough someone will treat back

but in Python 3 it does are inversion 2.5 it's 140 characters you don't get to put all the footnotes just saying so without further Ado let's start at the beginning the uh uh novice part and work

our way up pretty much uh in every other language I've used you use uh indices uh quite a bit to do array uh lookups pretty much when you use an index in Python unless it's a fairly exotic

circumstance you're almost always doing it wrong we've worked really hard to see if there uh to make better ways to do it so you don't have to manipulate uh uh indices I'm also going to show you something more advanced very few of you

probably know about the else clause on four Loops I'll show you what it's for very few of you probably know that the uh iter built-in function can take two arguments I'll show you what uh what

it's for but overall our goal is to uh aim for fast clean idiomatic python how do you Loop over a range of numbers simple enough you make a list and loop over the list what is important about

that example Python's four is not the same as it is in other languages if someone comes to my glass and says I know what a four Loop does because I know C that is not what this does we probably should have named hours for

each what it does is loopover collections it uses the iterator protocol it is in no way like the floor that you grew up with as is there a better way well we can use the range

function the range function the output of it is that list up above in other words these two things uh do the same thing in exactly the same way many people come to python see the second one

and say to themselves it's the same it's the four Loop that I learned in C or basic or some other language it's not what happens here is range produces that

list then the for Loop Loops over the list I know what you're thinking if I do range 1 million that list is going to be kind of big and in fact on a 64-bit

build it'll consume 32 megab of memory is that awesome no not awesome so there must be a better way it's called X range X range uh creates an iterator over uh

the range producing the values one at a time which is better range or x-range X range which is ugly x-range X range is a horrifically

bad name we didn't know how bad until Python 3 came along in Python 3 we got rid of of the old range and renamed X range to our range so I started going through my programs where I used xrange

everywhere and I took off the X and they were profoundly more beautiful I didn't know how ugly the X was until I got rid of it you'll like Python 3 better it looks better remember beautiful is

better than uh uh ugly looping over a a collection how would a c programmer do it well they would say for I equal 0 I less than N I ++ look up the uh I color then they get to Python and say how do

you do that in py they do it this way some of you already know this how are you supposed to use this uh information you're supposed to after this talk go back and look through

your entire code base and do a GP for that whenever you see that don't do that do this instead even in professional code bases with really good programmers as I go from company to company looking at their

code I see the first one all over the place just fix it it's simpler easier to read better and in Python it's faster how to loop back

backwards oh by the way somebody clap for xrange that predates me I go back uh almost 13 years as a 12 years as a core developer somewhere in there X range was

uh just before me I'm not responsible for the X okay how to Loop backwards a c programmer knows how to do this for I equal n minus1 while I greater than uh

minus1 count down by minus 1es that translates directly into python the idiom that you learned on the first day of uh class NC Works ex uh perfectly well in Python and it's grotesque it's

horrific and it's what we had to do until we introduced a better way the better way is to use reverse which is faster the first one or the second one second one which one's more beautiful

second one which do I see in a lot of code bases first why why would people do that the answer is they're gravitating toward back toward the mother ship and

the mother ship that is where they came from prior to python that was the way to Loop backwards and almost every other uh language you know and because you've learned it you gravitate toward it you

use indices all over the place and you try and write for Loops as if Python's four meant the same thing I bet if we renamed it four each it wouldn't be as pretty but everybody would use it

correctly that's how you Loop backwards I heard no claps what's special about reversed I did that okay looping over a collection and the indices at the same time C

programmer would have no problem because they are already looping over uh I they can print the I position and the I color uh so the uh output of this would be

zero red one green two blue three yellow how do you do it in Python without indices the answer is Ed enumerate good call Larry he wins I did a numerate so

en numerate is a simple clean way to do it it's fast it's beautiful and it saves you from tracking the individual indices and incrementing them what's the cue here whenever you're manipulating indices directly you're probably doing

it wrong just saying go scan your code base you'll find this somewhere take it out replace it with a numerate it's fast and beautiful and readable how to Loop over two collections at uh uh once every

C programmer knows what to do take the shorter of the two lists the minimum of the lengths Loop over the indices and look up the I name and the I color why would they do such a thing because it works in every other language they've

ever learned what's the python Way ZIP is it really the python way actually it goes back 50 years it was in the very first version of uh Lis if you the uh original paper that came out on Lis zip

was in there zip has a deep history it is a proven winning performer it's what you want to use do you love it yeah I think it's clean and beautiful is there

anything wrong with it what what's wrong with it a to Loop over this it manifest a we took started with two list it manifests a third list in memory that

third list consists of tuples each of which is its own separate object with pointers back to the original in other words it takes far more memory than the original two uh uh lists combined this

is no fun it doesn't scale what's this whole scaling and speed thing it used to be if you asked me how to make python uh how to make any program go fast I'd teach you about Loop

unrolling and remembering previous calculations and whatnot but on Modern processors only one thing matters is the code running in L1 cache because if it's you have a cash miss the

Intel optimization guide has this horrifying line in it that says the cost of a cash Miss is that simple move becomes as expensive as a floating Point divide it can go from a half clock cycle

to 400 to 600 clock Cycles you can lose two and a half orders of magnitude by not being in cash if these lists are really big do you think uh that zip is going to fit into cash I don't think so

there must be a better way and it isip izip so izip is better yep that was me I did that one I got in at just the right time iterators have just been created I like

make an iterator out of everything they're like wow R's really smart like no quo made the iterators I just put them everywhere so uh it was his uh a brilliant idea and it's gone very far

it's one of the things that makes python beautiful and fast looping uh in sorted order so we uh can Loop over a collection by doing sorted colors it's pretty easy to take any for Loop and

just drop sorted in it and now you Loop over it in sorted uh order Merz okay yes I did sort it too okay so uh how to Loop backwards reverse

equal true simple enough how do you do a custom sord order this was the traditional way you made a custom comparison function that compared two keys and returned either minus one one

or zero depending on whether it's less than uh equal or greater than Harold it's just grimacing but there's others who are not grimacing there's others say that's the way I learned it in see

that's the way Q sort works and those of you who are older who you learned with Q sort and compared comparison functions you're going to have a hard time letting go of this in fact you'll fight with me you'll come with me over and you will

try and invent examples of where you have to have a custom comparison function and you might not even listen to me when I tell you that's horrifyingly slow it's no fun to write a a function like that you can write one

the shorter this gets the job done and how many uh times will this function be called well if you have a million items in a list uh and you're doing a sort the number of comparisons is in log in a log

of a million Bas two is uh 20 so that's 20 million comparisons is there a better way so that's 20 M million calls to that compare function which is long and slow here's a better way sorted colors key

equal uh length the key function gets called exactly once per key which is better 20 million or 1 million 1 million is the function shorter for key it's

almost always shorter uh so those of you who grew up on comparison functions will probably argue with me and say I can invent a comparison function where you can't make a key function and if you get

really creative and work really really hard at it after a hundred tries you're going to find one that I can't do or can't do easily although I do have a function that will convert back uh if uh

necessary how do we know that key functions are sufficient who likes to sort all the time SQL people they sort all the time they're ordered by this that and the other do they pass in custom compare functions you they have

key functions order by sum of relative frequency order by uh this field plus that field if they can get by with key functions you can too this code is

shorter more beautiful faster and should you abandon your key uhy uh comparison functions absolutely in fact I've abandoned them for you and I ripped them out and they're no longer in Python 3 goodbye comparison

functions all right how many of you knew all of that stuff already okay let's see if I can uh take you to some place you haven't been before the traditional way to uh do a

loop over a uh a function call that has a sentinel as you do a while true Loop when we do an F read reading a block of uh 32 bytes at a time eventually we run

out of data and when we do the what F read returns is a sentinel value an empty string whenever it's an empty string you can break out of the loop so we pin the uh build the block so one at

a time by the way how should you once you've the output of that is a big list of uh strings how should you connect them together join how should you not

connect them together plus oh hey you guys are all on top of it so this is a traditional way you will see this code all over the place did you know that the iter function can take two arguments where the first argument is a function

that you call over and over again and the second argument is a sentinel value The Sentinel value this says it will call read uh 32 over and over again looping one block at a time we get to

use four Loops which should have been called for each which are fast and beautiful instead of uh the while now some people would argue because I had to use a partial in here that is slightly less readable than the original but

remember what I made here I didn't just have to hand it to a for Loop you know the first one these two are equivalent in terms of what they uh do but I didn't have to use this with a for Loop the

moment you've made something iterable you've done something magic with your code what have you done uh with your code as soon as something is itable you can feed it to set you can feed it to

sorted min max heapq sum many of the tools python consume uh iterators as soon as you've uh made something iterable it works with all the rest of the Python toolkit so the part to

concentrate on is not the for Loop part it is the two argument form of iter add that to your toolkit if you've not seen it before it's a good time to learn uh in order to make it work you the first

uh function has to be a function of no arguments how many arguments does uh F re take one how do you go from one to zero partial partial takes a function of many

arguments to a function of fewer arguments if you haven't tried this before go home and play with it and learn something new welcome to the world of uh functional programming and the magic of this is there are lots and lots

of functions uh in Old especially in older apis that are intended to be called over and over again until they give you a sentinel value it's called a control uh break style of uh programming

it used to be very widely uh practiced until there was a certain little hicup there was an insurance company that was processing big decks of uh uh Punch Cards for uh insurance claims and uh

they Brun to the deck and then they stick in another deck more claims another decks and at some point they needed to tell it to stop so they put in a control break field a sentinel value that told it when to stop one day they ran through a deck of cards and it

stopped right in the middle they reran the deck stopped right in the middle it stopped at the same card every time this is a true story I got it from uh programming uh pearls the uh cause was

uh the claim came from uh Ecuador the capital is keto and quit was the control Blake symbol and when it hit the city and said quit it did there's a reason we don't do this anymore it's the same

reason that we don't terminate our strings with nulls anymore because sometimes we going to stick nulls inside the string fair enough but if you encounter an API like that the two argument form of iter takes it out of

the old world and into the new world of uh iterators who learned something new all right distinguishing multiple exit points in Loops I didn't come up with

this quido didn't come up with this this goes back to the go-to uh Wars lots of people hung on to go goto uh and wouldn't let it go until every un known

use case could be uh replaced so Donald n sat down and itemized the most common use cases of the go-to and he came up with some structured equivalent that would do the same and so one problem was

uh when you do a something like a for Loop you need a a flag variable to say is something been found or not found now keep in mind this example one we already have a built-in find so you don't need

this code to begin with number two I could have exited out early um uh with a return so I know it's a little simplistic although it is new's example uh his point was typically code like

this occurs intermeshed with other more complex code in other operations so that there is not a shortcut out the usual solution to the problem is to put in flag variables which slows down your code and makes it less readable we'll

start with found equal false if we find the uh Target value is uh true uh we will uh change the flag and then act on the flag at the end there's a better way a way that is shocking and jarring to

most people coming from other programming languages because they've never seen it before and it was n's ideas not quidos and the uh we actually have else Clauses on four lips remember the for

Loop has essentially got an if inside and it's saying if I haven't finished the loop keep doing the body if I haven't finished the loop keep doing the body what construct is normally

associated with if else so what the else means is I finished the body is there any more of the body of the loop to do no else so you can think of it that way

inside every four internally there's a uh an if and go to and this is the else associated with that if some people have a hard time remembering that way if I could go back in time and talk to quido

if he give me the keys to the kind machine I would say back when you first made this language else was exactly the right term because it's what newth use and people knew at that time all fours

had embedded an if and go to underneath and they expected the else but in the future no one will know that because we're all using structured programming already so why don't you call it no

break if it was called no break everybody would know what it did there are two ways to exit this Loop you can finish it normally or you can break out search uh search your house for the keys

there are two outcomes you find the keys and come out or you search all the rooms and there are no more two possible outcomes they're distinguished with the else if that was caus called no break you would know what it did if we finish

the loop and didn't encounter a break return minus one if we finish the loop normally return I who learned something new now you know where it came from Donald nth guess what uh quido was

reading when he came up with this Donald n guess what whether he was thinking about the future no he would have called it no break at which point everyone would know what it did just like if we called Lambda make function no one would

say what does Lambda do it would be called make function all right dictionary skills uh those of you who've been in my classes before you know I I start out with dictionaries at the beginning I cover

them the second day the third day and the last day because there's two kinds of people in the world people who've mastered dictionaries and total goobers all right they are the fundamental tools for expressing relationships linking counting and

grouping here's your core dictionary skills how to Loop over the keys for K and D nobody clapped I didn't do that one quido did

that one but I got to he was sitting on The Wire what should uh the for Loop would do with the dictionary half of the people wanted uh the for Loop to return the key and the value at the same time the other

half just wanted uh the uh key I went and researched what other programming languages did went back into small talk uh a grip through um a lot of existing uh code doing counts and see what people

most needed most of the time when they looped over I look to see what was consistent with uh if you wanted to treat a list as a dictionary the indices

uh of a list are parallel to the keys in a dictionary and I kind of Led Out a argument and it believe it tip the scales and that's why it's for K and D is there another way to Loop over the

keys yes you could just ask for the keys and loop over when should you do the second not the first it's when you're mutating the dictionary and the first way if you want to mutate the dictionary you can't do that while you're iterating

over it in fact in any programming language for the most part if you mutate something while you're iterating over it you're living in a state of sin and you deserve whatever Happ happens to

you in this case though d. Keys calls

the keys argument and makes a copy of all the keys and stores them in a list at which point it is you are free to go mutate the dictionary and delete all the uh keys that uh uh start with r leaving

just Matthew that's kind of the way it is around the house uh all the keys that started with Aragon and now it's just Matthew I brought him to Pon people just come up to me

and oh and they look it right at the baby we deleted all the keys starting with r okay uh next

one more ways to I'll Stand quite forward if I pitch forward you'll know what happened okay looping over to the Keys and values at the uh same time one way is to Loop over

the key and then look up the uh value is it very fast no because it would has to H uh rehash uh every key and go uh do a look up on it if you actually need the

values there's a better way the items and so we we're using tupple unpacking uh here if you need both Loop over them directly no lookups are are involved is

there a better way yeah because items makes a big huge list the better way is iter items so iter items will return an iterator you might clap on that one I didn't do

that I think that one was Barry Barry Warsaw okay y for Barry okay another Berry looping over the dictionary uh in

does that say from pairs in pairs whatever to lip over them together oh constructing a dictionary yes so zip is uh fantastic because as it assembles the

pairs the dictionary Constructor you might not have known will accept a list of pairs or any iterable of pairs so the easiest way to assemble these two into a dictionary is to izip them together if

you Marvel at this one and I think you should the thing to the thought to go away with is the parts in Python fit beautifully together how do you take two list and join them them together seamlessly and construct a dictionary it

is 1 2 3 four words of python it doesn't get much more uh beautiful than that look back through how you're building up your dictionaries if you already have the inputs available izip is a fantastic

way to do it was izip a better way to do this than a zip yes now it still has to make a tle on each iteration

right no I went in put inside I checked the reference counter after the dictionary has consumed the tupple we Loop back around to make the next toule we reuse the previous one so it can

actually build this uh without in any intermed uh intervening uh calls to the uh alligator it just takes one topple and Rees it over and over again in other

words is this fast yeah absolutely wow I've still got 15 minutes left that's great how to count with a dictionary you guys probably know a number of ways to count when you teach uh people python dictionary show them

get first uh and show them the most basic methods first so these are the most basic ways to Loop over a dictionary Loop over the colors check to see if the color is not there if it's

not there add it since now a square bracketed look up is uh conditional it can fail if the key is not there raising a key error but in this case we've just put the key in uh so we know it's there

this will this last line will always succeed this is a simple basic way to count everyone should know how to do it don't immediately start them with the most advanced thing because if a person can't uh do this they will be helpless

on any more comp complex problem with dictionaries should you start them right away with default dicks and whatnot no start them this way what's the uh next level of improvement over this if I

don't want to use anything exotic and I want to use a core dictionary API those of you were my classes I threaten you all the time with Matthew will remain fatherless unless you know this particular method I threaten to

keep people an extra day if they don't know this the method is get yeah set default uh sets in the case of accounting all we need is uh get and so the code up above simplifies to this we

get the color if the color's missing return zero and add one to it we don't need a set default all we need is the uh zero we need to look up to not fail is

there a more modern way yeah what is it default dick when I answer a question on stack Overflow and I put one of these first two please don't immediately go change

it to default dick all you're doing is taking somebody who couldn't count in the first place and then taking to where they have to import the collections learn the distinction between a regular dictionary and a default dictionary they

have to know the in about Factory functions they have to know the int can be called with no arguments producing the value uh uh zero uh and then when they get something back it's not actually a dictionary it's a default

and, needs, to, be, converted, back for some use cases in other words if you hand this to a beginner you've usually made them uh worse off make sure they know the first idioms uh uh before they

drive on but that said what I use is this or I use collections counter okay how to group with the dictionaries uh this is an example I've used over and over again how do I uh uh

group these uh names together I forgot what I've grouped them by uh their length of their first letter the traditional way use uh that a person should learn first is start with an empty dictionary the key of the

dictionary is what you're grouping by so uh Raymond is of length seven that will be the key and the value will be a list of all of the names of Link seven if you'd like to accumulate a lot of points

on stack overflow know this because this question get asked gets asked about once a week what should you immediately take them to when someone wants to group the

collections module how about show them Set uh set default so this one uh the output of it by the way is Roger has five letters uh

Raymond Matthew Melissa and Charlie all have seven uh Rachel and Judith uh have H six by the way if you're grouping by anything else you only need to change the key line maybe the key is name zero

that will group people by the first letter name minus one uh key could be uh the number of e's in the name you can Group by just almost anything using this idiom there's a better way though it's

set default we actually need set default because we want to return the list so we can append to it but we also need it to be inserted in set default is just like get but it has a side effect of inserting a missing key for a long time

this was the idiom for grouping in Python I think it is not particularly beautiful uh uh python though uh the the word set default is really bad and everybody thinks it's awful but no one

can think of a better name every other name we've ever experimented with had like about 50 letters into it well this goes into a dictionary it looks to see if a key is there if it's not it uh takes the default value inserts it into it and returns it so you can group with

it that would actually be the best name but it's a little long and now the uh modern way transfold your uh transform your code into this uh default dict uh list that will create a

new list and it is far more beautiful than this original six lines and slow four lines and fast that is the new idiom for how to group things in Python you must know how to uh do it but not

only must you know how to do it my presentation is intended as a checklist for you when you go back check out your code base and find out look where everywhere you're doing this or this

replace it with this if you do all the replacements in my slide deck it'll narrow uh speed up your code uh quite a bit it'll make it more maintainable and more beautiful uh an interesting one is

a pop item uh you might clap on this one or might not I can't remember I made either pop or I made pop item but I'm not sure which one it was a long time ago it was my first or second

contribution to python was putting in Python it must have been the second because the first contribution was I volunteered to put Doc strings in a whole bunch of modules do you guys use any dock strings yeah I put I put about

half of them in there originally they just were mostly empty is that a good way to join an open source project yes if you go through uh putting doc strings everywhere one

people love you for it two you make the code more usable but as a side effect you actually learn what every module does or you can start another way how about you take the most popular important data structure a dictionary

and mangle it and transform it into some radical way and change its performance characteristics is this a good way to start no someone recently did I growled at them earlier it was the only person I

growled at at uh Pon okay so a pop item I might or might not have put in it removes an arbitrary um uh item the interesting thing about it is it's Atomic so you don't have to put locks

around it so it can be used between threads to automically pull out a task who learned something new all right linking dictionaries together this kind of code is reasonably

common we have one dictionary which has uh some default values for some parameters in addition uh we call arars and prar some command line arguments

that are optional so a use can specify the command line the uh user or color or they cannot specify it and lastly I have a uh a third dictionary which is uh not

showing here the third dictionary is OS Environ which is not really a dictionary but it looks like one which gets environment variables it is common to want to uh chain these together and the

traditional way to do it one that I actually found in the standard library was you copy the uh dictionary full of defaults then you do an update from the environment that way you have some some standard defaults and then if someone

specifies an environment variable it overwrites environment variables take precedence over the internal defaults but a commandline argument should take precedence over the environment variables this kind of code is uh very

common how many of you have written some code like this ever is it the right way to do it well no it copies data like crazy if you want your code to be fast

don't copy like crazy so chain map has been introduced into uh Python 3 and it links them all together it leaves the three independent dictionaries and just looks up in the first one command line

if it doesn't fall do it there it looks in the environment if it doesn't find it there it falls back to defaults this way is H fast and beautiful and it's my config parser is no longer slow thanks for the Applause on that one that was me

improving car Clarity I have so few minutes left but I was going to leave five minutes for Q&A do you guys want to blow off your Q&A and get more of these slides cool no questions you don't even have time for a yes all right wherever

uh you have positional arguments and indices that's nice you can do that in any language keyword and names are better the first way where you're using indices that's convenient for the computer and fast in a language like C

but the naming things is how humans think so a way to improve your uh oh did we start with the answer on that one there we go oh they were out of

order so the top one is the kind of code that I see all over the place in client saw customer bases it call S Twitter search Obama uh false 20 and true raising the question what is the 20 the

false and the true to uh you would have to have memorized the argument signature in order to check that a simple way to improve the readability of your code is to go find everywhere where you're making an obscure call like that and

just replace it with keyword arguments it's an easy thing to do it slows down your code uh just a little bit but really what are you trying to save microseconds or hours of uh programmer

time hours of programmer time those are the ones that uh uh cost you so uh this is a simple transformation in fact if you're a junior programmer just starting out and you would like to improve your company's entire code base go through

and do this everywhere make sure you don't do it in the middle of a tight Loop but mostly uh everywhere and it will make the code better profoundly better and who will be the first consumer of this you because

you're the new person to the company and don't know the code base you'll know it really well after you've uh done this it's an easy way to improve uh quote quality name tuples it used to be if you

call Doc test test mod it returns 04 at the Tuple is that a good thing or a bad thing are you happy or sad when you get 04 you don't know that is what it

returned for most of its existence now it returns test results failed equal zero attempted equal four are you happy or sad which is a better output is the second output substitutable for the

first sure name tuples are a subass of tupple so they uh they still work like a regular tupple they just tell you what they say and how the way you make the name tle is simple you just say we're defining test results as having two

Fields failed and attempted easy enough this is a very easy way to improve your uh code base basically all over the place go put name tles and now all of your output messages and error messages

will be uh much more readable the person who benefits from this will be you there we go all right was the name tle guy unpacking uh

sequences Raymond hinger who's a young man in HEX um I can pull out the fields uh this way why would you do it this first way the

uh the answer is is that's what you do in almost every other programming language uh that you know give myself another 5 minutes for Q&A time there okay so people do this mainly because it

works in all other programming languages and they do it out of habit the better way is uh the one that's listed here SE uh tle uh unpacking and it pulls out the uh four fields for you the second one's

more readable and it's faster uh this is an easy change to make it's an easy thing to GP for everywhere you see brackets zero brackets one brackets two brackets three you know what's going on replace it with unpacking your code is

better and faster easy change so how do you do uh simultaneous State uh updates the traditional way to uh write the uh uh

Fibonacci is to take a temporary variable for y add up your new uh Y and then uh use your temporary variable I hate this code I've written code like it

a lot of times because I started with a 1967 version of Dartmouth basic and it was all I had but there's a better way you can use the the topple packing and

unpacking don't overlook how important it is it's profoundly important the problem with that uh okay well first I'll show you the correct solution the correct solution the way you'll often

see it written is with uh simultaneous variable updates the Y and the X+ y use the old values of X and Y uh to build the Tuple then they get unpacked and

stored in the variables the X and Y are State the state should be updated all at once if you don't update the state all at once and put it on multiple lines dur in between those lines the state is

currently mismatched at one point x uh Y is the new Y and X is the old X this is a very common source of uh problems and plus the ordering matters here if I mck

up the order of these three uh it breaks the code and it's a hard error to uh see the last thing I uh don't like about it besides the risk of order is it's too low level of uh is uh on the next slide

I'll talk about taking an atom and breaking it into subatomic particles this is been broken into Som atomic particles what does this say this says take Y and store at t x + y to Y and T

to X the second one says update these variables according to those equations and so you transform one to the other the second way is a higher level way of

thinking it doesn't uh risk getting uh uh the order wrong uh and it's uh fast in Python please transfer form code like that into this lest I let that go it's got a whole

additional slide and a half of okay okay two slides for this this is such an important part don't underestimate the advantages for this if you break this out into a p

pieces you risk ordering problems also you are making it too Atomic you are losing the ability to chunk the uh uh your thoughts into think higher level thoughts for example a problem I give

when I uh teach a scientist is I give them the function uh influence the influence of one planet over another all they have to do is plot the orbit of the uh uh planet and you have some people

get an isol iCal orbits and other people where it just kind of zigzags away the ones who get it wrong are the ones who wrote exactly this code except they didn't use the temporary variables if

the first thing they wrote is X is equal to X Plus DX plus T they're toast at that point why they've updated the X and now this one gets computed with a new x uh rather than the old X it's a very

common problem the other half of the people who write these temporary variables how did they know to do that the answer is they've all been burned by this problem before the correct answer

is this do the calculations on the right with the uh old values of the variable the old X the old y the old DX old DX Dy take their partials and then only then

update all of the variables this is of profound importance not just for scientific Computing I can give other examples where people are doing a simple mortgage calculation with uh principle and interest and whatnot and the very

first thing they do is update the principle by principal minus equal payment they're toast because their interest payment is going to be wrong the interesting thing to me is not that they get it wrong uh when they program

in any language the interesting thing is I can give them the same problem in Excel and they always get it right why is it that people get it right in Excel and wrong in programming languages the

answer is in Excel you take all the state on each row on month uh one here's the current principal interest and uh uh whatnot on month two it's this and people naturally refer to the uh uh they

take their formulas and they refer up to the previous row all Excel people do this essentially they are doing exactly this operation you could view this as what's on the right is referring to the

previous row and what's on the left is the new row and that gets uh uh iterated in other words it's a very natural style of uh thinking please don't write code like that write code like that this is a big deal that we've got it in this

language it will save you from a a lot of trouble got one minute efficiency I'll do efficiency fast basically just don't move data around unnecessarily concatenating strings in

my classes I tell a Naggy joke around this in order to hammer home this is quadratic Behavior don't add your strings uh this way instead join them it sounded like most of you knew that

already go check your codebase and see if your codebase knows that just saying updating sequences if you see a dill uh

zero a pop zero or insert zero you're doing it wrong I go into uh customer s they say here's a million lines of code and it runs really slow can you make it go fast as if I'm going to be able to

read a million lines of of code but 15 minutes later I come back and said I've sped up your code what did I do I Greed for these three things pretty much everywhere they did a Dell name zero a pop zero or insert zero they were using

the wrong data structure what's the correct data structure de deck yeah I did that anyway a deck La let you do a uh uh you can delete names

zero efficiently a pop left efficiently or a uh a pin left uh efficiently uh decorators and context manager I have no seconds left but we're going into break

time minutes oh 5 minutes woo we got we have time for decorators and context managers which completely Rock we're out of uh novice territory and into really good stuff these are fantastic tools for

refactoring uh your uh code but good naming is essential because it provides macro like capability meaning you can hide all kinds of awful actions behind the macro or you can uh be very clear so

remember the Spider-Man rule with great power comes great responsibility all right so I want to uh factor out some administrative uh Logic the

business logic here is opening a URL and returning a web page the administrative logic is I'm caching it in a dictionary that way if I go look at the same web page over and over again I just simply remember it you'll see code like this

all over the place in Python which TR was trying to cash their lookups what I don't like about it is it mixes the admin logic with the business logic and

it's not reusable simple fix at Cash it's actually the lru cache in Python 3 I have backported it for people who want to scan for uh the backport uh you can

start using it today that said these things are pretty easy to uh write on your own uh so I really want to demonstrate the decorators here less than what I've uh written with this is reusable I can put at cache in front of

any pure function a pure function being one that Returns the same value every time you call it random. random is that a pure function no because it gives a different value every time you it pal is

that a pure function y same uh answers every time the business logic has been separated from the admin logic and I've gotten reusability the way I write it is with a simple caching uh decorator it

only takes a few lines I would like for your utilities directories to be full of little tools like this so that elsewhere in your code you put at cache and the problem is solved uh factoring out

temporary uh uh context for decimal so we get uh we copy the context change the uh decimal Precision to 50 do a calculation and restore the old context

this is saving the old restoring the new that happens over and over again there's a better way with the local context the context manager here makes a copy of the context puts it in place does a

calculation restore it which is easier to get right the first or the second second one it has reusable Logic the uh pretty much anytime you're setup logic and tear down logic get repeated in your

code you want a context manager to improve it it can profoundly clean up your code base especially if you're doing this sort of thing all over the place okay the traditional way to open and close uh files everybody knew they

were supposed to do it they wouldn't do it you had to do the try and the finally and see python closes for you anyway but the simple way the uh new way is beautiful what did the wi statement

factor out for us it factored out the setup logic and tear down logic it set up the try finally for us most of you knew that one already does your codebase

know go fix it how to use lock simple way to make a lock the old way is acquire the lock do a try finally do you have to use a try finally absolutely if

you don't you don't release the lock under some situation where an error uh happens in here what happens if you don't read the uh lock uh release the

lock do puppies die every time uh dead puppies so that's what you're supposed to do but you had to indent twice spell out finally put colons people knew they were supposed to do it

and probably wouldn't the new way is with lock I've actually separated the administration logic of getting the lock separate from uh uh the print statement uh I don't know who came up with context

managers I need to go find them and thank them because they are wonderful uh factoring out temporary context by the way most of you guys knew with lock

already right and you already knew this uh the uh way to close uh open and close files here's one you didn't know uh I would do an OS remove to file and then I catch an uh an error there's

an another way to do this of course you can check and see if the file exists before you do it is that the right way no because it has a race condition in it and so this is the uh correct way it's

also irritating here's a better way with ignored oh you've never seen ignored it says do this code how come you haven't seen ignored aren't you watching my check-ins I made this check-in a few days

ago you guys aren't working off the head on python 34 yeah anyway if you want your own I put it on the next slide uh you put those uh handful lines in your code uh hey it's like 10 words of python

stick that in your utils directory and you too can ignore exceptions it gets rid of the idiom for tri accept pass who learned something new cool all right

more new things factor out uh temporary context did you know that help since it's help to standard output so you're going to have to cut and paste it to store it in a file that's irritating

can't we just redirect it sure what you can do is uh open a a file redirect standard output temporarily assign it go to a TR finally on the help and then after you've uh done the help capturing

the output to the file restore it is this any fun no better way with redirect standard out to the file and it's done pretty much you can get back in the business of writing your functions with

print statements and then wrap them with widths to send them to files send them to standard eror send them somewhere else this restores the beauty of using uh uh print uh everywhere and this is

the uh context manager for if you don't have this and of course you don't cuz I uh haven't checked that one in yet that'll probably go in on Monday so uh um Nick and I have to have a couple

words about he's uh very happy with uh this part what we are unsure of is whether we should say in fact you guys can just vote The Proposal is if file object is

equal to none how about we automatically create for you a uh string iio object capture the string so that uh you can do with uh redirect standard out no

arguments s s and then do a S.G value

and capture the string yeah do that you want that all right Nick has decided okay Nick is the uh Nick is the

man for um uh context uh uh lib so we negotiate everything there just like somebody wants something in collections they have to come to me they want something in tools they come to me all

right Jackie came to me said no he tried to alter one of my combinatorics I'm now wishing I had said yes though all right

so uh concise expressive on liners this is the uh uh very uh uh last one but it's an interesting thought when people first come to python we teach don't put too much stuff on one line there's an infinite amount of vertical space

available to you uh in your code take advantage of that on the other hand don't take um uh single units of thought and break them into subatomic uh particles it actually makes makes your

code harder to I understand every single line but yes do you understand the Galt of that my rule for uh what goes on one line and one logical line did you remember earlier when we did the um uh

tle unpacking with the planet positions that was one logical line even though I I actually typed it on four so uh one logical line means one statement so my rule is what goes in uh one line is what

you can express in a single English uh sentence give me the sum of the squares of the numbers up to 10 this is one way to do it you start with an empty list this is the way we

used to do it in the olden times when I first came to uh uh python I'd have you clap on some but uh Alex put some in there I'm the one who made some go fast though so uh I put all the optimizations

in so it doesn't create a new object at every uh iteration it actually just keeps a running total uh in C there is a better way which is to use uh uh the

square brackets and put this in one line why is that better well the first one tells you exactly what to do step by step the second one says what you want it's more declarative it just says I want the sum of this I

read it left or right I want the sum of the squares of I taken from 1 to 10 the same way you would write it in mathematics I can the second way is better because it's a single unit of thought and the first one is too busy

telling you how to do it and not what it's doing fair enough oh is there a better way yeah take out the brackets

generator Expressions I did those yeah so yeah my contribution to python I came along and saw these square brackets and I took out an eraser and I erased

them and it made everything go a lot faster that creates a generator version of this instead of filling up memory making it go fast did you guys have a good time cool

[Applause] thank you all uh for coming to the presentation and do me an honor get uh take these slides go back to work and

have somebody if not yourself look at every one of these find them in your code base and put them in it'll make your code faster better and more beautiful thank you very much