Give Me 30 min, I'll Make CUDA Click Forever

You're going to learn CUDA in 30 minutes and make your code 100 times faster. We're going

from zero to massively parallel, fully optimized code right now. This is the code that makes it happen. It looks a little intense. I know you're seeing special keywords, pointers,

happen. It looks a little intense. I know you're seeing special keywords, pointers, arrays with strange names like a tile and b tile, thread idx, block idx, and these mysterious synchronization calls. By the end of this video, you will understand every single line

of this code. You'll know what kernels, threads, and blocks are. You'll understand the difference between slow global memory and fast shared memory. and you will know exactly why this code is so fast. That's my promise. And listen, this isn't just theory. We measure performance in Gflops.

fast. That's my promise. And listen, this isn't just theory. We measure performance in Gflops.

That stands for giga floatingpoint operations per second. We're talking billions of calculations every single second. It's the standard for high performance computing. And the rule is simple.

Higher is better. Now look at this table. This is the performance gap. An optimized multi-core CPU gets us about 50 g flops. That's our baseline. We'll call that 1x speed. Now, a naive CUDA

implementation on a GPU. That gets us to around 3,67 G flops. That's a 60 times speed up. But

that's not where we stop. The optimized CUDA code, the code I just showed you, gets us to almost 4,600 g flops. That's nearly a 100 times speed up. How is a 100x speed up even possible? The answer

isn't just in the code, it's the hardware itself. And this is the most important mindset shift you need to make. To program a GPU, you have to stop thinking like a CPU programmer. Let's start with an analogy. We'll call it the kitchen analogy. First, let's talk about the CPU. A CPU is like a

an analogy. We'll call it the kitchen analogy. First, let's talk about the CPU. A CPU is like a single worldclass master chef, a genius at complex sequential tasks. Incredibly fast at any single thing. Now, let's talk about the GPU. The GPU is an army of thousands of junior chefs. Each one can

thing. Now, let's talk about the GPU. The GPU is an army of thousands of junior chefs. Each one can only do one simple, repetitive task, like chopping a carrot. Okay, quick quiz. Who do you hire for one fancy meal? The master chef, of course. the CPU. But if you need to chop 10,000 carrots,

who wins? It's not even a contest. The army of junior chefs, all chopping in parallel,

who wins? It's not even a contest. The army of junior chefs, all chopping in parallel, will finish the job orders of magnitude faster. And that is the core idea of CUDA. CUDA is the art of breaking your problem into thousands of simple, identical carrot chopping tasks. This isn't just a

story. Look at the numbers for an Nvidia A100 GPU. You have 6,912 CUDA corores. Those are your junior

story. Look at the numbers for an Nvidia A100 GPU. You have 6,912 CUDA corores. Those are your junior chefs. And it can manage over 200,000 threads. That's the number of chefs executing in parallel.

chefs. And it can manage over 200,000 threads. That's the number of chefs executing in parallel.

So your job as a CUDA programmer is to organize this army. And CUDA gives you a simple structure for this. It's called the CUDA hierarchy. Number one, a thread. This is a single junior

for this. It's called the CUDA hierarchy. Number one, a thread. This is a single junior chef running your code. Number two, a block. This is a team of chefs at a cooking station.

And what's really important is they can cooperate and share info quickly. And finally, number three, the grid. This is the entire collection of blocks, the whole kitchen. Now, let's see what this looks

the grid. This is the entire collection of blocks, the whole kitchen. Now, let's see what this looks like on the actual hardware. This diagram shows the compute architecture in the GPU. The entire

chip is the grid or the entire kitchen. Inside you have all these SMS, which are like individual cooking stations. Each station runs a block of threads and inside the stations are the cudaores

cooking stations. Each station runs a block of threads and inside the stations are the cudaores which are the individual threads. Your chef A, chef B, and so on. Now notice how each station has its own little workbench called shared memory. And each chef has their own personal storage called registers. We'll come back to that because CUDA optimization is all about memory access

registers. We'll come back to that because CUDA optimization is all about memory access patterns. Let's break this down using our kitchen analogy. First, you have global memory. Think of

patterns. Let's break this down using our kitchen analogy. First, you have global memory. Think of

this as the main pantry. It's huge, but it's far away. Every trip to get an ingredient is a long, slow walk. This is our performance villain. Then you have shared memory. This is the workbench.

slow walk. This is our performance villain. Then you have shared memory. This is the workbench.

It's tiny, but it's right next to the chefs. Grabbing an ingredient is instant. This is our performance hero. If you remember one thing from this chapter, make it this. Look at this table.

performance hero. If you remember one thing from this chapter, make it this. Look at this table.

You've got global memory, the main pantry. Who can access it? All threads in the entire grid.

the C question mark. It's huge, 80 gigabytes. But look at the latency, the walk, very slow. We're

talking 400 to 800 clock cycles. Now compare that to shared memory, the shared workbench. Who can

access it? Only the threads within one block. The C question mark, it's tiny. 164 kilob per SM. But

the latency, ultra fast, around 20 to 30 cycles. And finally, you have registers, the chef's hands.

Only a single thread can use its own registers, but access is instant. Basically one cycle. The

difference is just staggering. So let's look at that GPU architecture diagram again, but with our new understanding. The entire GPU chip is the kitchen. Way off to the side is that big global memory, our main pantry, which is slow. But inside each SM, each cooking station is that fast shared memory. And for each individual CUDA core, you have instant registers. You can see the hierarchy.

memory. And for each individual CUDA core, you have instant registers. You can see the hierarchy.

The fastest memory is the closest, but also the smallest. And that gives us our optimization strategy. It's a simple three-step plan. Step one, minimize global memory access. Avoid those long,

strategy. It's a simple three-step plan. Step one, minimize global memory access. Avoid those long, slow walks at all costs. Step two, load data into shared memory. Make one coordinated trip to the pantry and bring everything your team needs back to the workbench. Step three, reuse that cached

data for as much computation as possible. That is the secret to unlocking incredible performance.

It's time to write our first recipe. We're moving from theory to code and writing our very first CUDA kernel. But what does hello world even mean for a GPU? A GPU doesn't exactly print text to

CUDA kernel. But what does hello world even mean for a GPU? A GPU doesn't exactly print text to a screen. The parallel computing equivalent is vector addition. Why start here? Because it's what

a screen. The parallel computing equivalent is vector addition. Why start here? Because it's what we call embarrassingly parallel. What does that mean? It means the calculation for each element is completely independent of all the others. It's the perfect crystalclear way to see thousands of threads working at once without any complex logic getting in the way. It's beautiful. So here's the

problem. Vector addition. Given two arrays A and B, we need to compute a third array C. The formula

problem. Vector addition. Given two arrays A and B, we need to compute a third array C. The formula

is simply C of I equals A of I plus B of I. Super straightforward. And our plan is just as simple.

We will assign one GPU thread to calculate one element of C. So if we have a million elements, guess what? We'll launch a million threads. It's that direct. Okay, here is the complete

guess what? We'll launch a million threads. It's that direct. Okay, here is the complete working code. Now, I know what you're thinking. Whoa, that's a lot of code. Don't worry. Don't

working code. Now, I know what you're thinking. Whoa, that's a lot of code. Don't worry. Don't

try to understand every line just yet. We are going to break this down piece by piece. So,

how do you compile and run this thing? Well, if you have the CUDA toolkit installed, it's pretty easy. First, you save the code as, let's say, vector_add.cu. [Music] Then,

you compile it with nvcc, which is the NVIDIA CUDA compiler. The command is nvcc vector_add.cu-

vector_add. And finally, you just run the executable dot /vector_add. And here's the output.

you'll get vector addition of 1 million elements and then a little verification line. That's it. We

just added 1 million numbers in parallel. Now, let's break down exactly how this code pulled it off. Every CUDA program is split in two. You have the device code, which is the kernel. This

it off. Every CUDA program is split in two. You have the device code, which is the kernel. This

is the code that runs on the GPU. It's executed by thousands of threads. This is the recipe for our junior chefs. Then you have the host code which is your main function. This is the code that runs

junior chefs. Then you have the host code which is your main function. This is the code that runs on the CPU. It manages the whole operation. It's the project manager. First up, let's look at the device code, the kernel. Here it is. It starts with this special keyword double global. This

tells the compiler, hey, this function is for the GPU. Now, this line, this single line is where the magic happens. It distributes all the work across all the threads. int i equals block idx.x* block

magic happens. It distributes all the work across all the threads. int i equals block idx.x* block

dim.x plus thread idx.x. This is how each of our thousands of threads figures out its unique ID.

It's how one thread knows its job is to calculate C of 12 and another knows its job is C of 5,000.

Let's break it down. It's simpler than it looks. Block idx.x X is basically the thread asking which team, which block am I in. Block dim.x is asking how many threads are in each team. And thread

idx.x is the thread's personal ID within its team. Let's use a concrete example to make this crystal clear. Imagine you are thread number 12 and you're in team number five. And let's say there are 256

clear. Imagine you are thread number 12 and you're in team number five. And let's say there are 256 threads per team. The unique global index I is calculated like this. I equ= 5 * 256 + 12 which

equals 1,292. And just like that, this thread knows its one and only job is to compute C of

equals 1,292. And just like that, this thread knows its one and only job is to compute C of 1,292. Simple, right? Okay. Part two, the host code, the master plan. This is so important.

1,292. Simple, right? Okay. Part two, the host code, the master plan. This is so important.

Every CUDA kernel follows this exact five-step pattern. Burn this into your brain. Step one,

allocate memory on the GPU. Your CPU's RAM and your GPU's VRAMm are totally separate places. You have to explicitly allocate GPU memory. So, we declare our pointers dev_a,

places. You have to explicitly allocate GPU memory. So, we declare our pointers dev_a, dev_b, dev c, and then we call cuda mallec to create space for them on the device. Step two,

copy data from host to device. We have to copy our input arrays A and B from the CPU's RAM over to the GPU memory we just allocated. We do that with CUDAM copy. Now, a quick pro tip. This is so important. The PCIe bottleneck. This CUDA memopy step can absolutely kill your performance. Look at

important. The PCIe bottleneck. This CUDA memopy step can absolutely kill your performance. Look at

this table. Think of the PCIe bus as the highway to the kitchen. It's got a speed of about 32 GB per second. Not bad, right? But now look at the GPU's global memory. Walking to the pantry, that's

per second. Not bad, right? But now look at the GPU's global memory. Walking to the pantry, that's around 2,000 GB per second. And shared memory, the workbench, 19,000 GB per second. The difference is insane. So, here's the key insight. A truly fast CUDA program minimizes data transfers. You want

insane. So, here's the key insight. A truly fast CUDA program minimizes data transfers. You want

to send data to the GPU once, run as many kernels as you can, and only copy the final result back at the very end. Okay, back to the plan. Step three, launch the kernel. This is the moment of truth.

We tell the GPU to execute our kernel using this special triple angle bracket syntax. Add kernel

blocks per grid, threads per block, and then the arguments. This is us telling the project manager to unleash the chefs. Step four, copy the results from device to host. Once the GPU is finished, the result is still sitting in its memory. To see it on the CPU, we have to copy it back. So,

we call CUDA memcopy again, but this time with the CUDA memopy device to host flag. And finally,

step five, free the GPU memory. Good housekeeping is important. We have to clean up after ourselves and free the memory we allocated on the device using CUDA free. And that is the full CUDA cycle.

Looking at this diagram, you can see the whole flow. On the host CPU, you allocate GPU memory.

You copy data in. You launch the kernel which runs on the GPU device. Then you copy the data out. And finally, you free the GPU memory. It's a beautiful logical process. You've now seen a

out. And finally, you free the GPU memory. It's a beautiful logical process. You've now seen a complete CUDA program in action. You know how to write a kernel. You know how each thread finds its unique job. And you know the five essential steps the host uses to manage the entire operation.

unique job. And you know the five essential steps the host uses to manage the entire operation.

We're ready for the main event, matrix multiplication. This is where we start building the code that delivers that 100x speed up we promised, but we're going to start by building our villain. That's right. This first version is the obvious brute force solution. It's functionally

villain. That's right. This first version is the obvious brute force solution. It's functionally

correct, but secretly it's horribly inefficient. And here's the key. By understanding exactly why our villain is so slow, you will understand the key to making CUDA code fast. Our strategy is simple. We're going to launch a 2D grid of threads and assign each thread to calculate exactly one

simple. We're going to launch a 2D grid of threads and assign each thread to calculate exactly one element of the output matrix C. Straightforward, right? All right, here it is. The complete villain code. Now, don't worry. I'm not going to read this whole thing to you. Let's just hit the

code. Now, don't worry. I'm not going to read this whole thing to you. Let's just hit the highlights. Inside our kernel, Mattal naive, the first two lines are exactly what we saw before.

highlights. Inside our kernel, Mattal naive, the first two lines are exactly what we saw before.

Each thread calculates its unique row and column index. Simple. Then we have a boundary check, of course. And the heart of it is this for loop. This is the dot product. Sum plus equals

of course. And the heart of it is this for loop. This is the dot product. Sum plus equals a at row * k + k * b at k * n plus call. It's the mathematical definition of matrix multiplication translated directly into code. And the main function, it does exactly the five steps we just learned. It sets up our 1024x 1024 matrices. It calls cuda cuda fme to send the data up. Then it

learned. It sets up our 1024x 1024 matrices. It calls cuda cuda fme to send the data up. Then it

launches our kernel, copies the result back down, and cleans up with cudafree textbook. So how do we run it? Easy. We compile it with nvcc mmattmall naive.cu-matol cu-mol naive. Then we run it dot

run it? Easy. We compile it with nvcc mmattmall naive.cu-matol cu-mol naive. Then we run it dot /matol naive. And what's the output? naive matrix molt 1024x 1024x 10,024. And the verification C at

/matol naive. And what's the output? naive matrix molt 1024x 1024x 10,024. And the verification C at index 0 is 1,024.0. It works. Get this. We just performed over 2.1 billion calculations in under

a millisecond. That is impressive. Impressive. But this is where our villain reveals its fatal flaw.

a millisecond. That is impressive. Impressive. But this is where our villain reveals its fatal flaw.

Let's expose it with a simple 4x4 example. We launch a 4x4 grid of threads. Each thread computes one element of C. So what does thread 0 do? Its job is to compute C at 0 comma 0. To do that, first it reads A at 0 comma 0 and B at 0 comma 0 from global memory. Second, it reads A at 0a 1 and

B at 1 comma 0 from global memory. Third, it reads A at 0.2 and B at 2.0 from global memory. And

fourth, it reads A at 0.3 and B at 3.0 from, you guessed it, global memory. Okay, makes sense. Now,

what does its neighbor thread 01 do? Its job is to compute C at 01. So, do you see the problem? It's

a disaster. Here's a diagram of what's happening. Every single one of those threads in a block is making a separate individual trip all the way out to slow global memory. It takes 400 to 800 cycles for that data to arrive. And look, the super fast shared memory, the workbench that's right there inside the SM, it's not being used at all. Every access is the slow one. Let's look at

the villain's crime directly. Here's a table of the reads. Thread 0, 0 reads the first row of A, the values 1 2 3 4. Thread 0.1 reads the first row of A, the values 1 2 3 4. Thread 0.2 also reads 1 2 3 4. And thread 0.3 reads the exact same data. The entire first row of A was read four times

from slow global memory. Each thread made its own independent slow walk to the main pantry to get the exact same ingredients as its neighbors. This is a catastrophic waste of memory bandwidth. Our

GPU isn't crunching numbers. It's just waiting. We are completely memory bound. Now, let's scale this up to our real 1,024x 1024 problem. Each element of matrix A is needed by 1,024 different threads.

Each element of matrix B is needed by 1,024 different threads. This means every single value is reread from slow global memory. 1,024 times for our 1,024x 1024 matrices. That adds

up to over 2.1 billion redundant global memory reads. This is the villain we need to defeat. Our

kernel is drowning in slow, unnecessary memory traffic. The solution isn't some obscure GPU trick. It's a fundamental algorithm. It's called tiling. And before we write a single line of CUDA,

trick. It's a fundamental algorithm. It's called tiling. And before we write a single line of CUDA, we're going to understand the pure logic behind it. The core idea is simple. Work smarter. Let's

go back to our kitchen analogy. The naive approach was like individual trips. Every chef would say, "To make my one dish, I need to fetch all 1,024 ingredients from the pantry myself."

It was a mess. But the tiled approach is about teamwork. The head chef says, "Team, let's make one coordinated trip. Bring ingredients to our shared workbench and reuse them." This

simple change from individual trips to cooperative fetching is the secret to high performance. To

prove this isn't some GPUon concept, let's look at the naive matrix multiplication in Python. First,

we have our function matt_naive. It takes in matrices A and B. You can see the code here.

It's exactly what we expect. For each element in the output matrix C, we loop through row, we loop through column, and then we compute the dotproduct with a third loop over K. Standard

stuff. Now, here is tiling in Python. This is where we isolate the logic. The function is matt tiled. It looks almost the same at first. We still have our loops for row and column, but look

matt tiled. It looks almost the same at first. We still have our loops for row and column, but look closer. The key difference is a new inner loop for phases. For each phase, we compute a partial sum

closer. The key difference is a new inner loop for phases. For each phase, we compute a partial sum using a small tile of the data. We have another little loop inside that does the work for that small tile. And then we accumulate the result into C at row column. It's the same math just broken

small tile. And then we accumulate the result into C at row column. It's the same math just broken into smaller phased chunks. Let's walk through a problem to make this crystal clear. We're going to process our 4x4 matrices using 2x2 tiles building the result in phases. On the screen, you can see

matrix A and matrix B, both 4x4. First, let's remember what we need to compute C at 0 0. We need

row 0 of A * column 0 of B. That's the vector 1 2 3 4 * the vector 10 50. If you do the full calculation, 1 * 10 + 2 * 50 and so on, you get 900. Okay, that's our target. Now, let's do it

the tiled way. Phase 0. We process the first two elements. We use the first tile, which is just the first two elements from each vector. So, we take the first two from row 0 of A, that's 1 and two, and multiply by the first two from column 0 0 of B, that's 10 and 50. Our partial sum is

1 * 10 + 2 * 50 which equals 110. We'll hold on to that number. Next phase 1, we process the last two elements. We use the second tile. That's the last two from row 0 of A, which are 3 and 4, times the

elements. We use the second tile. That's the last two from row 0 of A, which are 3 and 4, times the last two from column 0 of B, which are 90 and 130. That partial sum is 3 * 90 + 4 * 130 which comes

out to 790. And now we combine both phases. We just add the partial sums from phase 0 and phase 1. Our phase 0 result was 110. Our phase 1 result was 790 and 110 + 790 equals 900. It works.

phase 1. Our phase 0 result was 110. Our phase 1 result was 790 and 110 + 790 equals 900. It works.

This is the key idea. Break the dot product into tiles. Process each tile, then accumulate. Okay,

this might seem more complex, so why bother? The answer is simple and brutal. Memory traffic. Let's

quantify the benefit with our 4x4 example. Look at this table. In the naive kernel, each of our 16 threads reads four values from A and four from B. The total global memory reads 128. But

with the tiled kernel, we load data for each of the two phases once for the whole team. The total

just 32. That is a four times reduction in memory traffic. Same math. A quarter of the slow walks to the pantry. Now scale that up. For our 1,024x 10,024 matrices, the naive approach resulted in

about 2.1 billion global memory reads. The tiled approach about 134 million global memory reads.

That's a 16 times reduction in memory traffic. We have found our hero. Tiling is a pure algorithmic optimization that drastically reduces our reliance on slow memory. We'll combine the tiling algorithm with the GPU's key feature, that tiny ultraast shared memory workbench. This is the grand payoff.

This is where we defeat the memory bottleneck and unlock the 100 times speed up. Here it is, the final optimized kernel. The code you saw at the very beginning of this video. I know this looks like a lot, but don't worry, we're going to break down the most important part, which is right inside that global function maple tiled. You can see the shared keyword creating our workbenches, a

tile and b tile. And the real magic happens inside that main for loop. The rest of the code is very similar to what we had before, just setting up our matrices and launching the kernel. And how do you run it? Simple compile with NVCC and run the executable. So what's the result? The payoff for

our 1024x 1024x 1024 matrix multiplication. We hit 4589.96 gigaflops. There it is. We went from about 3,000 g flops in our naive kernel to almost 4,600 glops. That's a 1.5x speed up over our already

fast GPU code and a staggering approximately 100x speed up over an optimized multi-core CPU. We did

it. We crushed the bottleneck. So, how does this code actually work? The key is the main for loop that processes the tiles. This is the fundamental pattern of tiled computation. I want

you to burn this into your mind. Every high performance CUDA kernel follows this pattern, the three-step pattern. For each tile, every block of threads executes this sequence. One,

load from global to shared memory. Two, hit a sync barrier. And three, compute, reading directly from that fast shared memory. It's a simple, powerful loop. Load, synchronize, compute. Let's break it down. Step one, load the coordinated fetch. We declare our shared memory workbenches for a tile

down. Step one, load the coordinated fetch. We declare our shared memory workbenches for a tile and b tile. Then look at these two lines. Each of the 256 threads in our block grabs exactly one element from slow global memory and places it onto the fast shared workbench. It's a

perfectly coordinated team effort. Next, step two, synchronize the barrier. This one line of code sync threads is critical. This function creates a barrier. No thread can pass this point until every thread in its block has arrived. This ensures the workbench is fully stocked before anyone starts

cooking. You can't have one fast thread running ahead and trying to use ingredients that a slower

cooking. You can't have one fast thread running ahead and trying to use ingredients that a slower thread hasn't even fetched yet. And finally, step three, compute the payoff. Look at this inner loop. All reads in this loop for a tile and b tile come from ultraast shared memory. This is

inner loop. All reads in this loop for a tile and b tile come from ultraast shared memory. This is

the data reuse payoff we've been talking about. Each thread can access any of the 256 elements on the workbench almost instantly. No more slow walks to the pantry. Let's look at the final scorecard.

When we used a multi-core CPU, we got about 50 g flops. The Master Chef is fast, but he can only do one thing at a time. Then we move to the GPU with our naive Cuda kernel. We jumped to over 3,000 G-flops. The army of chefs is working, but they're all making slow, redundant trips to the pantry.

G-flops. The army of chefs is working, but they're all making slow, redundant trips to the pantry.

The code was memory bound. And now, with our tiled and shared memory approach, we hit nearly 4,600 g flops. The army works in teams. They make one coordinated trip, use their shared workbench,

flops. The army works in teams. They make one coordinated trip, use their shared workbench, and crush the job. We are now computebound. Visually, the journey looks like this. A tiny bar for the CPU, a huge leap for the naive GPU, and another significant jump for our final optimized

GPU code. Here's the bottom line. By changing how we access memory, we reduced 2.1 billion redundant

GPU code. Here's the bottom line. By changing how we access memory, we reduced 2.1 billion redundant global reads down to just 134 million. I need you to understand what you just implemented is not a toy problem. This is the beating heart of modern artificial intelligence. When you run TensorFlow

toy problem. This is the beating heart of modern artificial intelligence. When you run TensorFlow or PyTorch, they call a library called Kublos. And Kublos is Nvidia's hyperoptimized version of the exact tiled matrix multiplication you just wrote. Every multiplication inside GPT,

inside stable diffusion, and inside Alph Go is powered by this exact principle. You now

understand how modern AI actually runs so fast. It's not magic. It's a clever algorithm combined with massive parallelism and a deep understanding of the memory hierarchy. And with that, you've done it. You have officially crushed the memory bottleneck and unlocked the massive parallelism

done it. You have officially crushed the memory bottleneck and unlocked the massive parallelism of the modern GPU. Let's just stop for a second and recap what we've covered. In just the last 30 minutes or so, you went from zero to understanding GPU architecture. You wrote your first CUDA kernel

commanding thousands of threads. You identified the problem, redundant global memory access.

And you defeated it using the tiled algorithm with double underscore shared double_memory. You didn't

just learn some syntax. You learned to think in parallel. That cryptic code from the beginning of this video, well, it's no longer a mystery. You now know this line. You know this declares the ultraast shared workbench. And you know that this line doubled sync threads with parenthesis

is the critical synchronization barrier that makes teamwork possible. You now know why this code is so fast. This is a massive accomplishment. I said, "So where do you go from here? What's the next

so fast. This is a massive accomplishment. I said, "So where do you go from here? What's the next level?" Well, if you want to take your kernel from the 4,600 g flops we hit to over 20,000 g flops,

level?" Well, if you want to take your kernel from the 4,600 g flops we hit to over 20,000 g flops, here are the next steps. First, there's memory coalescing. This is all about optimizing the patterns you use to access global memory. Then you have warp level optimizations. This is about avoiding divergence in those little groups of 32 threads to keep the hardware 100% busy. Of course,

there are the CUDA libraries, things like Kublos and CDN. These are handtuned by Nvidia engineers and now you actually understand how they work. And finally, there's a pro-level technique called asynchronous streams which lets you overlap your memory transfers and your computations.