ControlNet Tutorial for ComfyUI.

today I'm going to show you control net

and how to take an image like this and

turn it into this and this and this and

this this is going to be a control net

tutorial in comfy UI let's get

started I tried doing 100 sit-ups but I

failed because my stomach couldn't take

that AB

use so this is what you see when you

startup comy right and this is no

control net here so I'm going to show

you how to create one from scratch and

I'm also going to show you how to get

more advanced workload like this and

this I have just downloaded this is

available for my Patron subscribers if

you want this you don't need this but

I'll show you how to make it so if you

want it easier if you just want to

download and get started available on

patreon now once you download a control

n workflow either if it's this one or

another one you're probably going to see

an error like this so you have following

node types that were not found that

means you need to go into your manager

here and if you don't have the manager

you can find my how to install comfy

guide in the comments or up here

somewhere that's going to guide you how

to get comfy running press that then

you're going to go into install missing

custom nodes and all the missing nodes

are going to be here you can see I only

have this one so I'm just going to press

install here and then it's whenever

that's finished you're going to restart

button down here and this is what

happened for me now you're probably

going to have more here so make sure

that you select all of these and then

press install we are restarting and once

that is finished you can see also in the

terminal

the status of that so here's stuff going

on right whenever that's finished this

little thing is going to be removed and

now it's done but but but but Seb what

about these they're still red yes you

actually need to refresh the browser

window as well and now we can see that

all of our nodes look fine if they don't

make sure to again check your manager

make sure everything is you can press

update here update all update comy

recheck install missing custom Noe

and restart your comfy again there is

some other things we're going to install

as well so we're going to go into here

install models we're going to search for

control net and we're going to check

here both the type and the base we want

the type to say control net and the base

to say 1.5 so we have a lot of these now

they are about 700 megabytes each I'm

going to download uh the canny I'm going

to download the line art the open pose

the scrip Ribble soft Edge probably

anime line art and the tile as well now

those are fairly big like I said so um

get the ones that you feel that that you

need but uh if you have the space you

know just get uh the same ones that I do

just make sure that whenever you've

install them you scroll down again and

uh recheck that there is control at an

SD 1.5 now you might be asking well

aren't there control nets for sdxl and

other models yes there are there are

some sdx control Nets you can download

them and use them if you want to uh I do

prefer using control net for stabil

Fusion 1.5 models because those control

Nets are uh in general a little better

but that might change over time now I

got a lot of these installed now you can

see in the green one here I got the cany

the depth the line art open post

scribble soft Edge the anime line art

and the tile to apply this we're just

going to press close here and then if we

press refresh here we're going to be

able to see them in our load control net

nodes here so can here for example this

is the one I downloaded I have multiple

so I have a pth or a safe tensor if

you're downloading from the manager

you're probably going to have the safe

tensors but if you have a pth that you

downloaded from hugging face it doesn't

matter that's fine too but for this

tutorial we're going to use the safe

tensors that we downloaded from the

manager we're also going to be

downloading this QR code monster model

and it's available in the description

just get this one curac code monster V2

if you can't find the V2 you might be

over here you got to find the QR code

monster for sd15 here you go into files

and versions and then you're going to go

into the this little folder here this

ISS V2 there you have it go into your uh

comy folder models control nut and just

put that right in there now if you're

using my workflow you should have it

running by now and U What's Happening

Here is that you have an input so this

is going to be your input image now

right now we have this man here and we

also have the prompt old man closeup

portrait linen shirt back background and

this is what we want to see not what

we're actually seeing this image we're

then setting a size here 768 x

768 we're then loading a checkpoint

though so this is a model this is not a

controller model but a regular stable

Fusion model you can get these from Civ

TI or or any other model site now if you

don't have this vae here you actually

need to go into the manager again

install models search for 840 just

install that like we did with the

control models we have two nodes that

load Aura here now they are disabled by

default you can either press uh contrl b

or right click and click bypass to

enable or disable them bear in mind you

need to download Laura models to use for

that those go into the models Laura

folder we're not going to dig into luras

for this tutorial so these are only

available here if you know what they are

and want to use them so what's happening

here now we have loaded an image we have

this hold man here we have the size set

which was set by default we have the

prompt so that's what our image is going

to look like and then we have have a lot

of nodes here the blue ones here which

are uh different control Nets and they

tie into the output here so this is uh

the end result or this is where the

generation gets done so if I would press

Q prompt here now I would get an image

of an old man and it might or it might

not resemble our input image as you can

see we have some lines here from the

glasses but they're not really looking

uh like like the input that we had and

that's because that the control n that

we are using now and as you can see all

these are in low opacity and purple so

they are disabled it's only this control

net that's enabled and this is using a

canic control net and it's applying that

with a strength of 0.5 the strength is

uh usually set between 0o and one so if

we change this to one and then we have a

start percentage and an end percentage

this is where the control net starts and

ends so think of this as a slider

between 0 and one so we're going to

start at zero that's the first frame and

then going to end at one which is the

last frame so now we're going to run the

control net high strength L for the full

duration now if you press Q prompt again

you will see that our image more closely

resembles our input now the class

glasses are full now we haven't really

delt into what these controllers do yet

so let's start looking at what's

happening here first off all these nodes

can easily be enabled and disabled right

by pressing the little bypasser here and

that's possible for all of these control

Nets so now they're all disabled so if

we would press Q prompt here now we are

not getting an image that

resembles this one here so we could say

mid

shot or actually remove closeup old man

midot and you can see it's now a mid

shot so it doesn't take into account

this image anymore because we have

disabled all of our control Nets so if

we enable a control net here let's say

this one line art soft Edge cany now

they're enabled again and even though we

are still having mid shot in the prompt

it's forcing our control net image into

our result and that's really what we're

looking for with control Nets it's all

about finding how much of these control

Nets you want forced into your image

let's look at what all these control

Nets actually do let's take a different

image let's take this ballerina here

with our hands behind her head we're

going to ignore the prompt right now

we're just going to leave it at the old

man midot and then show now if we double

click we can type here and we can add a

node and we want the preview image so

the preview image is well it's going to

preview an image and we want to see what

all these control Nets are actually

doing to your image so what you can do

is you can take a pre-processor so in

control Nets stay with me there are

three parts so you're loading a control

net model you're loading a pre-processor

and you're applying your control net so

we're we're loading a control net model

line art we're applying a line art

pre-processor so the pre-processor takes

our input image and turn

it into an image that control net can

understand and then it applies that

control net onto our image now let me

show you what that is we're just going

to press Q prompt here and as you can

see here our line art pre-processor has

turned our image over here into this and

this is different for each control net

right so if we add some more previews

here let's just drop them in and we can

check okay what does the soft Edge look

like we can take the pre-processor here

of softedge connect that there we can

take the canny the pre-processor connect

that there let's check the dep we can

connect that there we need to enable

this let's also check the open post

because that that's pretty cool so the

pre-processor is um always under the

load control net here in the in this

workflow so if you generate this again

now or Q prompt we can see a lot of

other control net pre-processors so we

have the soft edge here which kind of

gives you uh soft edges of the image we

have the cany which gives us a detailed

lines of the input we have the open post

which is one of my favorites is actually

seeing the post of the image without

forcing any anything else on the image

so this is kind of cool because if you

have a skinny character wants to turn it

into a fat character or vice versa uh

it's not going to be constrained by the

body proportions it can just see the

limbs of the character which is is very

cool and we have a depth map here so

this is what happens inside of the

control Nets right so it takes your

input image applies a control net

pre-processor and then from that image

it uh applies the control Nets to your

image now we we enabled a lot of control

Nets here and we have have um this woman

we have a prompt that says old man mid

shot but if we look at our result here

we can see this is not an old man it

just looks like our input image that's

because our control Nets are too strong

here right so let's start by disabling

here our control Nets so we have no

control Nets enabled now let's enable

the open pose we have an open POS it is

applying 0.9 strength so that's fairly

high and an n% of 0.95 that's fairly

high as well we can just run this for

100% now it's going to retain the pose

of our input image but nothing else

which means that it's not constrained by

the lines or anything so our old ma man

here getting us the same pose as the

woman now if we feel the open pose is

not enough and we want to retain parts

of the input we can enable something

else we can enable the depth here for

example can generate again we can see

that our character now is starting to

resemble our input

so you need to play with the values here

and raise or lower the different control

Nets that was an added bonus I have a

put in an upscaler here so if you want

an image upscale you can just enable

this here it's using the tile model so

you need to download that if you didn't

do that it's also using an upscale model

if you want to download one of the UPS

scalers just go into manager install

models search for upscale and here you

have a bunch to choose from now if

you're downloading workflow sometimes

and your load control net model is

turning red or purple around the edges

but you know that you have downloaded

that specific model you just need to

reclick this and reselect them mod it

could be that the workflow creator has a

different name for their model or that

it's a safe tensor or it's a pth or a

ckpt which is checkpoint but uh just go

in and reselect the model now one more

thing when it comes to this these

controllers right we talked about

loading it we talked about a little bit

about the pre-processor and we talked

about applying it now for each of the

pre-processors there are settings and

we're not going to dig deep into all the

settings but one of them that's

available in most pre-processors is the

resolution so if we for example let's

drag and drop a couple of pre-processors

here canny I'm just copy pasting these

from this one and then we're going to

load a preview image again I'm copy

pasting these like connecting them and

I'm changing let's lower the resolution

here let's have one low one mid and one

high I'm using the anything everywhere

nodes which means I don't have to

connect an image here as you can see

this is glowing so it has an input but

if you are not using this or if you're

using another workflow you need to

connect a FAL image here into your

images down here but as you can see it's

not required for me now if I would run

this you can see here now that even

though that all of these pre-process are

canny they are of different resolution

and that's going to affect your image

right so if you use this one with a

resolution of 128 you're going to be

able to see these uh kind of pixels in

in your image if you are using a more

high-risk image in this 512 one we can

see it's a little better and in the 1024

our lines are even smoother just bear in

mind it's also going to change what's

going on in your image so here you can

see in the lower resolution one we're

actually getting more face detail but

this can also be changed with uh the low

and high threshold so let's change the

low threshold on the high resolution one

you can see that we're getting some more

detail into back into the image but that

can be a good way to check what's going

on if you feel that you aren't getting

the results that you are looking for

let's remove these again now if you just

want to create a good-looking image

enable the line art enable the soft Edge

start asking for what you want man in

tropical Jungle Safari clothing I'm

going to lower the strength here a

little bit there you have it it's kind

of too strongly resembling our input so

let's disable the line art I'm going to

lower the softage here a little bit I'm

going to enable the open post instead

let's try again oh maybe too strong

still lower here the open pose and lower

the control net uh we're getting closer

but it's still kind of giving us these

um non-clothing Vibes so if we disable

uh the edges here now we're just using

an open Poe for example and why we're

getting those results previously is um

as you can see in this image the lines

here on the close or the the body is so

tight that it has no room for any

clothing basically that's when you don't

why you don't want to have too much on

your control net there oh and just as a

reminder if you are using absat it's

going to take a very long time um um

since it's an a tile up scal let's load

a default workflow here I'm just going

to show you if you want to create a

controlled workflow of your own we're

going to just move these here from the K

sampler and forward and we're going to

load three notes we're going to have a

load control net you can also rightclick

add node and find the ones that you want

we're going to apply a control net

pre-processor and this depends on what

kind of model you want to load so let's

say for example that we want to use the

depth then we need to search for depth

here let's pick here minus depth map and

then we're going to apply control net

Advanced we're connect to the control

net here the image here we're going to

take from our prompt the positive into

the positive and the negative here into

the negative how do you know what's

positive and negative well most of the

time it's the positive on the top and

the negative on the bottom but it kind

of depends on how you connect them so if

you connect this one to the positive

that's going to become the positive and

then we're going to go from the control

net positive to positive negative to

negative and then we need to load an

image into this so we're going to load

image we're going to connect that there

we're going to drop an image into this

going to drop this uh robot character in

here and let's say um woman

Android closeup portrait biom mechanical

and if we generate this now we will get

an image that resembles our inputs now

depending on the control net you load

it's um well it's going to be different

because there are so many control Nets

there are so many pre-processor you can

use and there are so many values of

strength that you can use for your image

that should cover the basics of how to

use control Nets thanks for watching I

hope you learned something and I'll let

see you in the next one see you