Web Scraping for Beginners – Extract Data with an API

In this full web scraping tutorial, Anna Kubo will teach you how web scraping can be utilized for tasks like market research, SEO tracking, and data

collection. She will demonstrate how to

collection. She will demonstrate how to bypass common obstacles like captures, rate limits, and bot detection by leveraging an efficient API. By the end,

you'll learn how to build a simple, effective scraper that extracts structured data for your own projects.

And thanks to SER API for providing a grant that made this course possible.

People want to collect product data, monitor competitors, track SEO rankings, or even build AI tools that pull information from the internet. But the

problem is most web scraping tutorials are either super complicated or they break the moment a website adds things such as captures, rate limits, or bot

protection. So, in this video, I'm going

protection. So, in this video, I'm going to show you a much easier way to scrape the web. Instead of building scrapers

the web. Instead of building scrapers from scratch, dealing with proxies, or constantly fixing broken scripts, we're going to use an API that handles all of

that for us. It runs searches in a real browser, solves captures automatically, and returns clean, structured data in JSON that we can use directly in our

apps. My name is Anna Kubo and I'm a

apps. My name is Anna Kubo and I'm a software developer, course creator, and today your guide into learning how web scrapers actually work, how to scrape

data from search engines like Google, and how to turn that data into something useful for your own projects, and finally, how to build a simple scraper

in just a few lines of code. So whether

you're building an SEO tool, doing market research, or just want to automate collecting data from the internet, this is a complete

beginnerfriendly web scraping tutorial.

So what are we waiting for? Let's do it.

Okay, so in order to use an out ofthe-box scraper that will allow us to scrape all these things and allow us to bypass things such as captures, rate limits, and bot detection, we're going

to use an npm package called SER API.

But in order to use it, we need an API key. If you don't know what npm packages

key. If you don't know what npm packages are, don't worry. We will cover that in this tutorial. But in order to use this

this tutorial. But in order to use this npm package, we need an API key. And for

that we need to sign up to serap aapi.com. So let's do it. Okay. So here

aapi.com. So let's do it. Okay. So here

we are on SER API. And I'm just going to sign up. And ta straight on my

sign up. And ta straight on my dashboard. We get our private API key.

dashboard. We get our private API key.

Now this is mine. Of course I'm exposing it to you. Please don't use this one because I will be disabling it. And

please do get your own and keep it safe.

The reason you want to keep it safe is that if other people get hold of your API key and use it in their projects, they could potentially use up all your credits or even rack up a hefty credit

card bill if you decide to attach credit cards to your platforms. Okay, that goes for every API key out there. So, please

be aware. Okay, so please go ahead and copy that right here. Now, the SA API is also super super useful. It also shows you how to search all of these different

things. Okay, so there is a lot and even

things. Okay, so there is a lot and even provides you with some helpers too. So

we've covered Google, there's some Amazon ones here, Amazon product API, Apple app store API, Bing, even Duck

Duck Go search API, YouTube search API.

I mean the list is pretty extensive and all the helpers will be there. I'll be

referring to this further in the tutorial, too. But for now, all you need

tutorial, too. But for now, all you need is your API key. And that's it. We're

set up to use SE API. Just to check this is working, I'm going to do a simple car command in my terminal. Putting in the only required parameters to do this

search. To be clear, what we're going to

search. To be clear, what we're going to be doing is communicating with SER's Google search API. It is identified by the URL, and we're going to be searching

for coffee. So here it is. Here is my

for coffee. So here it is. Here is my command and as you will see I put in two parameters. Q for the search. In this

parameters. Q for the search. In this

case it is coffee and then my unique API key. So here is just an example of one.

key. So here is just an example of one.

Okay. Once again these are the only required parameters. Others are optional

required parameters. Others are optional and we'll cover them in the next sections. So let's do it. Let's check it

sections. So let's do it. Let's check it out. So here I am. I'm going to just get

out. So here I am. I'm going to just get up my terminal and whack in that command. Make sure to write yours

command. Make sure to write yours exactly the same only replacing your unique API key. And let's run it. And

tada, we do get back the results of the search. Okay, so this has worked. You

search. Okay, so this has worked. You

have officially use the Google search API to scrape results from the Google search page. And here are the results.

search page. And here are the results.

Let's now explore the different optional parameters you may have. How to scrape data from search engines like Google.

Okay. So, in this section, we're going to do exactly like it says on the tin, and we're going to scrape Google, Google search, but also other things such as

Google flights, for example. So, I've

got up my code editor, and all I've done is simply created a directory or a folder called scraping tutorial and put in an index.js. JS file. So, please go

ahead and do the same. This is

essentially our starting point for scraping anything really, getting up our code editor so we can write some code.

Okay, so I'm just going to minimize that for now so we can focus on this file.

Now, the first thing we're going to do is actually initiate a new Node.js project by creating a package JSON file.

So npm init the command that we will need. Okay. And here we go. There are

need. Okay. And here we go. There are

some default questions that is asking me. So package name scraping tutorial

me. So package name scraping tutorial version one. Yes, this is version one of

version one. Yes, this is version one of my app. Description let's keep blank.

my app. Description let's keep blank.

The entry point is going to be index.js and test command let's leave blank. Git

repository blank. Keywords blank. Author

blank. License blank. Okay. And here we go. But we've just created a package

go. But we've just created a package JSON file. Wonderful. Now, if this

JSON file. Wonderful. Now, if this didn't work for you, it could be because you don't have Node.js installed on your system. So, please go ahead and do that.

system. So, please go ahead and do that.

By installing NodeJS, it will essentially allow you to run basic npm commands as we just did. Okay. So, just

go ahead and click get Node.js and download it based on whatever operating system you are using.

Great. Okay. So, we should all be here.

And now, let's just go back to our index.js file. So, we are going to be

index.js file. So, we are going to be using the package SER API to scrape essentially, which means we need this code. Okay, we're getting that package

code. Okay, we're getting that package and then we're getting the method of get JSON from it. So, of course, we need to install this package. So, once again,

let's get up our terminals. Just going

to clear this and do npmi to install the ser api package. Okay. And that is now done. So once again if you look in your

done. So once again if you look in your package json it should show up here along with the version you are using.

Now this version is important. If you

are watching this in the future and perhaps this says 421 just go ahead and put it to two so it

looks exactly like ours. save the file and run npmi again to install this version of SER API so that is compatible with this tutorial because you never

know what kind of changes could have been done down the line in version 4. So

yeah, just make sure your version of SER API is exactly the same as is written here. Great.

here. Great.

Next, I'm going to get the method of get JSON and I'm going to pass through an object. Okay. So there's my object and

object. Okay. So there's my object and my object is going to have the engine that we are going to scrape. So on this

occasion it will be Google. The query

that we are searching on Google which is going to be cafe.

The location I'm going to just go ahead and put London, England, United Kingdom,

Google domain.

Of course, you can change all of this.

It is going to be completely up to you.

Let's put google.co.uk.

Just going to change these to be double quot so it matches the rest. H L is going to be the language for the Google

search. So I'm going to put English. GL

search. So I'm going to put English. GL

is going to be UK. So GL just defines the country to use for the Google search. So it's twolet country code. In

search. So it's twolet country code. In

this case is UK for United Kingdom. And

our API key for me it is this. Please go ahead and use your own because I will be disabling mine so it will not work in

the future. And that's it. That's my

the future. And that's it. That's my

whole object. Of course, you can add more parameters to this. And all of these, okay, you can't just make them up. They have to come from a specific

up. They have to come from a specific list that is listed on SER API. Okay,

these API parameters can be found here.

Okay, so serapi.com/arch

API you get the query search. So Q for us we used coffee. There's other ways to use this too. We just use this in a very very basic way. We have the location.

Again, this has to look a specific way.

Just head over to the locations JSON API where you can essentially see the supported locations. Okay, so just go

supported locations. Okay, so just go ahead and click that to see them. You

also have all of these other things which are optional. So the ule parameter is Google's encoded location. You can

also define by latitude and longitude.

Again optional. A radius as well lets you define the distance in meters within which to bias the search results. So

that is pretty cool too. You also have some advanced Google parameters which we will not go over today as this is for beginners. But if you want to check them

beginners. But if you want to check them out, here they are. And then of course localization. So we've used Google

localization. So we've used Google domain which defines the Google domain you want to use again optional. We've

used the GL parameter which is to define the country to use the Google search in.

We've defined the language again the Google languages page is here. So don't

just make this up. There are certain twolet codes that will represent different languages. We also have CR. So

different languages. We also have CR. So

this parameter finds one or multiple countries to limit the search to. So

again, check out the Google CR countries page. And finally, you can define one or

page. And finally, you can define one or multiple languages using LR as well. So

check those out via the link here.

Okay. And of course, we have some advanced filters. We also have the

advanced filters. We also have the search type, pageionation. And then the only thing that is really required as

all of these are optional is the API key which we also already put in. Great. So

that's where you find all your API parameters. So all of these how to fill

parameters. So all of these how to fill them out as well as add the other optional ones too. Wonderful.

Let's move on. So once we send that over to SER API, we of course want to get some results. And our results come as

some results. And our results come as JSON.

So let's get them. And then let's also console log them out. So we're just going to console log them out in the terminal. Console log JSON. And that's

terminal. Console log JSON. And that's

really it. Let's save this. Now I'm just going to get out my terminal. Let's

clear it once more. And making sure we are in the scraping tutorial. I'm just

going to use Node to run the index.js.

JS file just like that and ta we get back the JSON. Okay, so here you get essentially the search results for cafe

in London using the google.co.uk

domain the language the device also has been added as desktop as I am using my desktop so it automatically picked that

up. Okay. And here it is. Here are all

up. Okay. And here it is. Here are all the results that we got. Okay. Cool. So,

we get an Instagram reel back. We also

get London's cheapest cafe has never been more than one customer. And that's

a YouTube video. We also have in position three a Facebook post. It looks

like top 10 caf try in London. We also

get related questions. Okay. The whole

page is script. What is actually also quite cool is if you want to see this visually, you can go back to serapi.com/theser

serapi.com/theser API. So the Google search API and you

API. So the Google search API and you can simply type it here. So everything

we just did. So I'm going to put cafe that was my search query remember Q cafe and location I would just put London. So

London like this and search. So once

again this has automatically picked up some other search parameters. such as my language device desktop and so on. But

essentially here we get a visual representation of what would be search if we genuinely did type cafe into Google's search engine. So that is

pretty useful too.

Okay, great. So amazing. We have that JSON. Now we can use it in whatever

JSON. Now we can use it in whatever projects we want and that is the goal and we will get to that. However, I just want to cover a few more things. So we

did write the JavaScript code for this.

So this might look familiar to you. But

if you are more familiar with Python, here is the code for that. It already

has your API key here if you do sign up to SER API. So that is super useful. If

you want to use Java, if you want to use Rust, if you want to even get this in Google Sheets, you can do so. Okay,

great. Let's move on.

How to scrape Instagram reels, YouTube shorts, and other short videos. In this

section, we're going to look at scraping short videos via the Google Short video API. This will allow us to essentially

API. This will allow us to essentially extract video titles, links, thumbnails, even the video clips themselves as well as source icons, channels, and

durations. Let's go ahead and try it

durations. Let's go ahead and try it out. Okay, so in order to get short

out. Okay, so in order to get short videos, and I mean literally the videos themselves as well as all other information, once again, this is my basic code setup and once again this is

Node.js and all I'm going to do is add the engine. So, I'm going to write

the engine. So, I'm going to write Google short videos.

And now we're going to put a query in.

So, let's go ahead and query lab boou.

And that's it.

Like with the previous scrape that we did, you can of course add other API parameters in here. We went over them in the previous section, so I will not go

over them again. And now all we have to do is run this script. So once again, I'm just going to do node and then index.js,

which is the name of our file.

And ta, we get a bunch of Lubu videos coming back to us. Okay, so there is a lot. The viral Lubu toy will soon start

lot. The viral Lubu toy will soon start its own featurelength film. Let's move

on. How to use the Google Lens API. So,

this is the last one I'm going to show you that I think is quite interesting.

The Google Lens API allows you to scrape results from the Google Lens page when performing an image search. The results

related to the image could contain visual matches and other data. And we're

going to get that. So, once again, here is my setup. as the engine. We're just

going to pass through Google Lens and then we're going to have to search for an image. So on this occasion, I'm just

an image. So on this occasion, I'm just going to search for this image of Danny Devuto. Here is the URL that I'm going

Devuto. Here is the URL that I'm going to pass through. And of course, we need our API key. Now, I just want to see the visual matches, but we'll get on to that

in a bit. For now, let's just console log everything out. So I'm just going to run node index.js again and tada. So

this is basically everything that comes back. Okay. So stuff from Facebook

back. Okay. So stuff from Facebook groups. There's a lot of stuff in here.

groups. There's a lot of stuff in here.

If you just want visual matches, I'm just going to search for visual matches.

You can go into the array and then just search for visual matches and run this again. So node

index js and now only visual matches will get returned back. So as you could see it's

returned back. So as you could see it's just images or thumbnails the whole way through. Great. Let's just open one up

through. Great. Let's just open one up to see what it looks like. And great, a visual match that was used on monmouth

timeline.org was returned. Okay, so now let's use what we have learned to build our first project, which is going to be, you guessed it, a short video scraper.

So this is going to utilize what we have learned about using the Google short videos from SER API. and we're going to build a web app that scrapes Google for

short videos via SE API and downloads them locally onto your computer using another package too. So we're going to build just a super simple back end.

We're then going to call SER API to do the search on the videos that we saw previously. The Google short videos

previously. The Google short videos engine will return the results and then we're going to download them via YT- DLP to a downloads folder. Okay, we're going

to build out a super basic front end using just JavaScript and CSS that has a basic search input and count selector.

So, we can select either 5, 10 or 15 videos to download. Of course, this project is super basic. I just want to give you the skeleton to do this. So,

you can go forth, make it your own, add features, expand on this idea, you name it. Great. So let's do it. Okay. So

it. Great. So let's do it. Okay. So

let's do it. Just as before, I'm just starting off with an index.js file. And

just as before, I'm going to initiate it. So npm in it. Okay. Just press enter

it. So npm in it. Okay. Just press enter on all of these to get the default values.

And great, that will spin up a package JSON file for us, which looks like this.

Next, we're going to have to, you guessed it, install some packages. So,

yes, we're going to install SE API. So,

let's go ahead and do it. npmi SER API, but we're also going to install Express, which is going to help us with rooting.

So, there you go. That's going to install.

And if you go down here, you will see the packages we are using along with our versions. So, just as before, make sure

versions. So, just as before, make sure your versions match. If they don't, just go ahead and change this and run npmi

again. Okay, great. Let's shut this

again. Okay, great. Let's shut this down. So now I'm just going to get up my

down. So now I'm just going to get up my index.js file again and let's write some code. So first off, we're going to get

code. So first off, we're going to get express and this requires the package we just installed called express. And now

we're going to get the path. This is

going to require path as well as fs. This is going to help us with files. So require

fs. I will talk you through these when we use them. const xc

file require child process

and const get json which as we know is from ser api.

So there we go. So the path module essentially is imported to handle file directory paths. This right here is the

directory paths. This right here is the file system module and it helps us read and write files and folders on our computer. The exec file method is

computer. The exec file method is imported to run external commands such as yt-dp that we will be using and it comes from

the child process package. Great. Now

we're going to create an express application instance. So essentially

application instance. So essentially we're getting express which we've imported here. We're calling it and

imported here. We're calling it and we're saving all of that wonderfulness under the const app so we can use it.

Let's also define our port. So const

port, I'm going to say 3000. This is

where the server will listen to. And

then const API key. Well, you guessed it. We're just going to put in our old

it. We're just going to put in our old one. So this will help us use the SER

one. So this will help us use the SER API.

Great. Next, I'm going to also define our down loads directory and we're going to use path

join directory name download. So, we're essentially saying

download. So, we're essentially saying that this is the absolute path to the downloads folder that lives on your computer. So anything we download will

computer. So anything we download will be saved there. If you don't want it saved there, you can of course change it to whatever folder you want. Now let's

ensure the downloads directory exists before the server starts. So exist

sync and then let's pass through the downloads directory. So if it does not

downloads directory. So if it does not exist then we create it. FS mukdur

sync. So we're creating a folder called downloads.

Okay, mukda is just a command to create a directory. Now we're going to use app

a directory. Now we're going to use app use and essentially enable JSON body

passing. So express JSON just like that.

passing. So express JSON just like that.

And we're also going to serve the front files from the public folder. So,

Express static path join

and then pass through directory name public. And we're also going to serve

public. And we're also going to serve downloaded videos as static files at download. So in order to do this, I'm

download. So in order to do this, I'm going to go to the downloads folder express

static down loads directory. Great. Okay, now it's time to

directory. Great. Okay, now it's time to write our search endpoint. So app get this is where the rooting comes in because if we essentially go to a

certain end point. So in this occasion I'm literally making this up. It could

be whatever you want. I've gone with app search.

We're going to pass through a request and a response. And then we're going to extract the search query and result

count. So const q count equals 5 for

count. So const q count equals 5 for example. Okay, so five is the default,

example. Okay, so five is the default, but it can be changed. I will show you what this means when it comes to running it. Now, if there is no search query, so

it. Now, if there is no search query, so we didn't search for libu for example, then we just want to send an error. So,

we're going to return rest status 400 JSON and then just send back an object with

the error. And we're just going to put

the error. And we're just going to put the string of query parameter Q is required

just so we know why we got an error.

Now, we're going to use get JSON, which you should be familiar with. And we're

going to pass through, you guessed it, an object. An object that takes optional

an object. An object that takes optional parameters such as the engine, which in this case is going to be one you used before, which is Google Short Videos

as the query. Well, we're going to pass that through from our front end. So,

we'll just leave it empty for now. API

key is going to be API key. Okay, so this actually is going to

key. Okay, so this actually is going to be from our variable stored up at the top of the file which we should keep secret. Okay,

don't upload this project as it is onto the internet. That is something that you

the internet. That is something that you will have to do. Okay, so great. So

there we have it. And then we get the JSON and

const videos equals JSON short video results. Okay, so we're getting the

results. Okay, so we're getting the short video results from the JSON that gets returned back to us and storing it as const videos. Alternatively, if that doesn't come up with any results, we're

just going to return back an empty array. Now this we're going to just

array. Now this we're going to just slice through either 5, 10 or 15. And

for that we're going to use the slice method and start counting from zero and then pass int. So turn it into an integer the count which will get passed

from the front end as well. Great. So

let's get the res JSON and just send back videos. Okay. So the video results

back videos. Okay. So the video results will be sent back to the front end as JSON. Great. I think let's start our app

JSON. Great. I think let's start our app for now and see how this looks. So app

listen listen out to the port and then we're just going to console log server running and at our local host and whatever the port was

which was 3000.

Cool.

Having a look at this we just need to remove that and put that outside here as well. Okay. So it lives outside of the

well. Okay. So it lives outside of the previous function.

So we've written our back end essentially. Now in here I'm going to

essentially. Now in here I'm going to make a another directory. So let's

create a new folder. I'm going to call it public. And in here I'm just going to

it public. And in here I'm just going to create a new file, the index html file.

I'm also going to create a styles CSS file and a JavaScript file called app.js.

app.js.

Wonderful. And now let's link them all up.

So, we're just going to put some boiler plate in here. So,

doc type HTML and put HTML language English. And then

in here, we'll have the head, which will have some metadata for us as well as the title. I'm just going to

call this short video scraper like so. Meta Char set UTF8.

So you should be familiar with this.

This is just super standard boiler plate name viewport content

width equals device width and initial scale equals 1. Great. Okay. Then we're

going to get the style. So, I'm just going to link this up.

Link href styles CSS real style

sheet. Now, in the body,

sheet. Now, in the body, I'm going to keep this super simple. I'm

going to have an H1 element that says short video scraper.

Let's also put the emoji here and then a div.

This is going to have an input a select dropdown for the

count. So 5, 10 or 15. I'm going to put

count. So 5, 10 or 15. I'm going to put option five. Also put

option five. Also put 10 and 15. Of course you can have as many as you wish. I also need to put the

value. So value the string of five

value. So value the string of five value keep this as 10 and keep this as 50. Let's give this the ID of count as

50. Let's give this the ID of count as well. So id count.

well. So id count.

Great. This input is going to be for type text which is actually the default.

I'm going to give an ID of query so we can pick it out. And let's also have a placeholder that says search for short

videos dot dot dot. Okay, so that's the placeholder text. And I'm just going to

placeholder text. And I'm just going to put a value and I'm going to hardcode it, but you can also override it of cool.

So if you want, I can also put this on multiple lines so you can see it. There

we go. That looks good. So, now that we have that, I'm just going to maybe give this the class name of control so we can

style up everything in this div. Uh, I'm

also going to put a button. Okay, so

this button essentially will allow us to search for our search term and the amount of videos we want to return.

Great. I'm also going to give this the ID of search button so we can pick it out in our JavaScript. After this div has finished, I'm just going to put a P

element that has the ID of status just so we can see any updates and put them in here in between these two P tags. So

just some text. And next I'm going to create a div that has the class of

down load bar. Okay. And the ID

load bar. Okay. And the ID of down load as well. Okay. And this is simply going

as well. Okay. And this is simply going to be for our button. So I'm just going to add a button here that says down load

or Okay, great. And maybe let's give it the idea

great. And maybe let's give it the idea of download button as well. Cool. And finally, I'm just

as well. Cool. And finally, I'm just going to add a div. And here is going to be all our results. So, let's give this

the ID of results. And I'm also going to give the class name of grid. Just like

that. And finally, after all of that has done its thing, I'm just going to put in the script source app.js. So we read our JavaScript file, which we are yet to write. But let's add some styling first

write. But let's add some styling first so we see what this looks like. So first

off, I'm just going to remove any default padding or margin. The body I'm just going to put the font familiar serif background color black color of

text whiteish and min height 100 viewport height with a padding of two rim. All H1 elements are going to be

rim. All H1 elements are going to be text align center with margin bottom 2 font size 1.8 rim and the color white.

Next we're going to have the controls.

So I'm going to pick out the element of the class of controls and I'm just going to use display flex so I can space everything quite nicely with a maximum

width as well as add some margin and align all items in the center. Now the

input with a type text that lives in control. So I'm being super strict is

control. So I'm being super strict is going to have the following padding as well as border radius. I'm going to add a border, a background, and a color for the text as well, as well as define the

font size. And finally, the select

font size. And finally, the select element that lives inside the controls is going to have padding, a border radius, border, background color, color of font's going to be white, and I'm

also defining the font size. Now, the

button, the button, okay, is going to have some padding, a border radius, border none. The background color is

border none. The background color is going to be this purple. Font color is also going to be white. The font size is going to be one rim. And the cursor is going to be a pointer when you go over it. It's also going to have a nice

it. It's also going to have a nice transition. So we will define that in a

transition. So we will define that in a bit. So this is going to be an

bit. So this is going to be an animation.

When we hover over the button, it's just going to change a darker shade of purple. And when the button is disabled,

purple. And when the button is disabled, we're going to make it gray. And the

cursor is not going to be allowed. In

fact, maybe let's just get rid of that transition. It's not really necessary

transition. It's not really necessary for this demo. The status. So the

element with the ID of status, I'm just going to change the text as well. So

it's going to be kind of an off white as well. And now the download all bar

well. And now the download all bar itself is going to also just have a little bit of margin. And I'm also going to assign a max width. And finally the

grid results. So the element with the

grid results. So the element with the class name of grid is essentially going to have a grid template. So when it gets populated, it will look nice and clean

in a grid. Great. And it's going to be populated with cards which we didn't add to the HTML cuz this will be created in JavaScript. So any element with the

JavaScript. So any element with the class name of card is going to have this color background, a border radius overflow hidden, and a nice border. any

image in the card is also going to fit the width 100%. I have an aspect ratio and also just a little bit of CSS to make sure that the image looks nicely

fitted the card body now. So the card body is going to have a padding and let's also style the HV element that lives inside the card body as well as an

element called source. So, the source of the video and we're also going to have a button that lives inside the card body.

We're also going to look for an element that has a class name of done inside the card body and change it to green. And if

there's an error, it will be this red color. Okay, so I zoomed past that

color. Okay, so I zoomed past that because I don't want this to focus on styling. Let's move on. We are done with

styling. Let's move on. We are done with this file. Let's move on to the app.js

this file. Let's move on to the app.js file next. Okay, so we're going to

file next. Okay, so we're going to collect current videos in an array. At

the moment, it's empty, but we haven't run our search yet. So, I'm going to write a function that's called a sync function. Search. And here, what we're

function. Search. And here, what we're going to do is look in our document. So,

document and we're going to get an element by its ID. And the ID is going to be query. and

ID. And the ID is going to be query. and

we're going to get the value and trim it. Okay. And

we're just going to save it under the const Q because that is our query. We'll

do the same for the count because those are two things we need to pass through into our query. So let's get the document get element by ID and the ID is

going to be count. So we're looking in our document our HTML document of anything with the ID of count and we're just getting its value. So we're saving those two things. However, if there's no

query, we'll just return out of this and do nothing, right? Because we need a query.

nothing, right? Because we need a query.

However, if there is a query, we will continue. So now let's get the search

continue. So now let's get the search button. So once again, let's look in our

button. So once again, let's look in our document and get element by ID and that ID is search

button. And let's save it as button just

button. And let's save it as button just like that. Now, if there is a search, we

like that. Now, if there is a search, we want to disable the button. So, we're

just going to make it true. We're also

going to get the text content of the button and change it to be searching.

So, it's really visible to the user that something is going on. Next, we're going to set the status as searching. Okay. And this is a function

searching. Okay. And this is a function that we are yet to write. So function

set status and then I'm going to pass through a message which at the moment above is searching. I'm going to get the document get element

by id status. So then we're going to populate that with whatever message we pass through so that we will see it visually in our browsers. Okay. So at

the moment it would say searching but we can use this function many times to change the message to other things too.

Great.

So let's continue in this function right here. So the search function we are

here. So the search function we are setting the status as searching. Then

I'm going to use document get element by ID. I'm just going to copy that. Uh and then we're going to search

that. Uh and then we're going to search for the element with results because we want to populate it. Right? So, first

off, I'm just going to clear it in case there's anything in there from previous.

And then I'm going to get the download bar.

So, we gave it the ID of download style display none. So, we are clearing it before we

none. So, we are clearing it before we do anything, right? So, we're going to start off with a blank slate and then we're going to try Well, I'm going to

await the results. So await fetch and we're going to go to API search. So this

is a route that we defined and then I'm going to search for a query. Now the query is

query. Now the query is encoded URI component

Q and counts. And then we're just going to pass through the counts.

So either five, 10, or 15. Okay. Make

sure that says encode URI component.

And then we're going to save our data.

So await res JSON and save the data of this fetch and save it to current videos. So at the moment current videos

videos. So at the moment current videos is an empty array. We're going to override it with data videos. Or if that doesn't come back with anything or

there's errors, we'll just return an empty array. Great.

empty array. Great.

Now if current videos length deeply equals zero,

we just want to set status. Let's pass

through a message. No videos

found.

Okay. And we're just going to return out of this whole function.

Awesome. However, if everything's fine in the R videos, we're going to put set status and back ticks found. And then

we're going to return back a number of how many videos are in array. So,

current videos length. We're getting the length of the

length. We're getting the length of the array video just like that. And we're going to get the document element. We're going to

search for the download bar again.

and we're going to do style display block.

Okay, cool. And then we're going to write a

cool. And then we're going to write a function to render the videos.

Then we're going to catch any errors and set status as error if there are any errors and

literally just print the error. So get

the error message just like that. And

then finally we will disable the button disable false. So it should be clickable again

false. So it should be clickable again and get the button text content and change it back to the word search.

Great. This is looking good so far. So,

at the moment, let's go back here. Node

index JS.

How cool is that? It's all looking so good. However, at the moment, it won't

good. However, at the moment, it won't let us download. We haven't written that function yet. Let's write our function

function yet. Let's write our function to render videos. So, I'm just going to move all of this down like so.

Function render videos.

So this time we're going to look in our project and look for the element with ID of results which is the grid of course. And now

we're going to make each card that will essentially populate the grid. So we're

going to use grid inner html to do this.

And I'm going to get the current video.

I'm going to map everything that lives inside the array onto cards. Okay, so

let's go ahead and do it. I'm just going to use back ticks. And here inside this back ticks, we're essentially going to create a whole element in HTML. So I've

just done it here so it's easier for you to copy.

There you go. So that is the whole thing right here. I will be providing this

right here. I will be providing this code for you at the end of the tutorial, but essentially it's a div. We're

mapping out the ID number onto the div.

We're then adding an image which will show the thumbnail of the video. As an

alternative, also we're going to pass through the title and then another div with a card body which contains the H3 element that will show the title visually as well as the duration of the

video and the source. You also have an option to download that video as well.

Great.

So, let's move on. That is the whole element. And then I'm also going to

element. And then I'm also going to join. Great. So, once we have rendered

join. Great. So, once we have rendered all our videos, let's actually see what this looks like. So, let's go back here

and let's hook up the search button. So,

once again, let's search for the element of search button. So const search button

and then let's grab it and add the event listener to it and if we click on it then we want to run the search function

which we wrote above. So now if we put labu and 10 and search amazing we get those videos rendered for us. However,

the button doesn't work as you will see.

We are yet to do this. So, let's write that function. So, here async function

that function. So, here async function down load one video. We'll pass through

the index of which one to download.

Going to define video. uh we'll look in the current videos array and pass through the index to return the one that we are interacting

with.

We will search for the button by the button index.

So this belongs to the cards each individual one.

And once we have that button let's disable it. So disable

disable it. So disable true and button text conquals down loading. So we know that we're

down loading. So we know that we're downloading the video. Okay, so that's the visual stuff. Now let's actually

download it. So for this we're going to

download it. So for this we're going to await fetch and then we're going to go to API

download which we are yet to write.

The method of this is going to be a post method.

Headers are going to be content type application

JSON.

Make sure that's a comma and as the body use JSON stringify

and pass through an object URL video link and then the title video

title.

Okay. So we are sending information to our back end that will contain the video link and the video title and then we're going to save that data.

So data equals await res JSON just like that. Now visually I just want to show

that. Now visually I just want to show on the download buttons that it was successful or not. So if data success

and we get some data back then I want to get the button text content and just put done like that.

Okay. And then I'm going to get the button class list and add done. So that should turn the

button green. However, if it is not

button green. However, if it is not successful, then I'm going to get all this and just put failed.

Okay, so the download failed. And let's

change this to error, which will make the button red. You can also add a tick and a cross if

you want. And then if there is any

you want. And then if there is any errors, I'm going to catch errors.

And I'm just going to put the button as failed as well because that makes sense.

Great.

So now let's go back in here and handle our downloading.

One prerequisite you're going to have is that you're going to have to install YTDLP, which is essentially what you're going to use in order to download videos onto

your machine. So, if you haven't done so

your machine. So, if you haven't done so already, you can do so by reading the installation. If you are on a Mac, all

installation. If you are on a Mac, all you have to do is do brew install YTLP.

Okay, I already have it installed on my machine, but please go ahead and do that if you don't. and check out the instructions if you're using other operating systems. So, this is going to

be quite a big one. So, this is our download endpoint.

Uh, it's going to be apposti slashd downloads as that's what we chose.

And then I'm going to pass through the request and the response as well.

download singular singular download and we're going to extract the video URL and title okay from the front end because that's what we sent over. If there is no

URL we just want to return an error. So

return res status 400 and then as the JSON going to return an object that has the error and it's

just going to say URL is required because obviously one was not given.

Okay, then we're going to sanitize the video title so it's safe to use as a file name. So const safe

file name. So const safe name equals title or

video. Okay, so that's the back

video. Okay, so that's the back essentially. Then we're going to replace

essentially. Then we're going to replace any special character with just an underline. Okay, so this is some reax to

underline. Okay, so this is some reax to do that. Then also what we're going to

do that. Then also what we're going to do is limit the file name to 60 characters. And then we're going to

characters. And then we're going to write a yt-dp output path file name. So const

put template equals path join down

load directory safe name dot

let's put this in back tick actually extec file YT DLP and this is going to run YTDLP as a

child process with the following arguments. So output file path template.

arguments. So output file path template.

We're also then going to put no playlist. So only download the single

playlist. So only download the single video, not a playlist, and merge audio and video streams into one MP4

file. Okay. And then also we're going to

file. Okay. And then also we're going to pass through the video URL to download.

Next, we're also going to put time out to one, two, three, four error

SD tout sdr. Okay, so this is a 2 minute time out.

And next if there is an error. So if

error occurs with YTDLP then we're just going to log the errors into the server console like so.

And let's return an error. So res

status 500 this time JSON error

failed to download video and then you can also show the details if you want.

Um so details s we're passing that through. Let's

continue. Now we're going to look for the downloaded file in the downloads directory. So const files

directory. So const files equals fs read directory sync downloads

dur. So this is going to list all files

dur. So this is going to list all files in the downloads folder and we're going to find the file

matching our video name. So const down loaded equals files find

f start with and then pass through the safe name

and if down loaded. So if the video is downloaded we found the title. I'm going

to guess JSON success. We found the title. So, success

success. We found the title. So, success

true file name down load dead. And then let's do path

down loads and then down loaded. So this just

returns this info. So we can see else if the file wasn't found we just return back JSON

success true message download

completed but file not found.

Great.

So, let's check it out. So, now if I search for five videos on Lubu, I should be able to download. Great. So, now if we look in here and look under

downloads, tada. That video has been

downloads, tada. That video has been downloaded onto my computer. Amazing.

How cool is that? I love it. Let's

continue. We have one final thing to do and that is write a function for downloading all the videos. So let's go.

So once again at the very bottom here I'm just going to write a new function async down

load all.

Okay. So this function right here and set status is going to be downloading

all videos dot dot dot and let's write a loop to loop over all

of them. Let I equ= zero as long as I is

of them. Let I equ= zero as long as I is smaller than the current videos array

length and we increase I by one. This time

we're going to get element by ID and pass through. So use back text button

pass through. So use back text button and the ID of the button that's been rendered.

So just like that and let's save it as a button. each time we loop.

button. each time we loop.

And if button class list contains done, then we continue

and we await and we download. So we use the function we already wrote, download one I. So we're essentially just looping

one I. So we're essentially just looping over each video and downloading each one. So set status then we can also

one. So set status then we can also update in the browser what is going on.

So I'm just saying downloaded I + one of current videos

length. So it should be downloaded one

length. So it should be downloaded one of 10, downloaded two of 10 and so on.

And once that is done we can set status again. Set status to

again. Set status to all down downloads

complete. Easy. So it really is that

complete. Easy. So it really is that easy. I'm just going to hook this up. So

easy. I'm just going to hook this up. So

document get element by id download down load const

download all button equals and then I'm going to get this button use the event listener to add event

listener click and we're just going to call this function. down over four.

function. down over four.

Just make sure that is a dash.

So now let's try out. Okay. So let's

search and let's download all. And

that's downloaded videos down 105. That

should be 205 3 4 5. All downloads

complete. And if we look in here, tada.

Amazing. How cool is that? I love it.

Okay, now that we've seen that work, I'm just going to do a few things before I upload this onto the internet. The first

being making sure that API key is not just here in this file exposed. Okay,

I'm going to put it in a file that won't be uploaded onto the internet essentially. So for this

essentially. So for this I'm going to install npmi

env. So the package is called env. Okay.

env. So the package is called env. Okay.

And it just means I can now create av file. And in here I'm going to put my

file. And in here I'm going to put my API key. So let's just take it from here

key. So let's just take it from here and paste it in like so. Okay. So that's

what your file should look like. Save

this. And now I can use process env.

And then whatever we called it in the env file. So it was API key. And then I'm

file. So it was API key. And then I'm just going to require. So all of that works. Great. Now we're ready to upload

works. Great. Now we're ready to upload this onto my GitHub. Okay. So I'll be sharing the code for this with you now that we have finished. Please go forth,

improve on this, make this your own.

This is just a super basic demo for you.