AI Product Management - Complete Course - 3.5 hours - Masterclass | AI Agents, RAG, Evals, LLMs.

If you are someone who's looking to get into AI product management, then this is a mustwatch video for you. Few days ago, I conducted a free master class for

above 500 people who joined this session and then they were able to understand how AI product management is done. Now

I've decided to make this video available for everyone so that everyone can take the benefit. My only request is that before you go ahead and start watching this video, keep a notebook and pen with you so that you are able to

take notes and have your full attention in the session. I'm sure this is going to help you out and if it helps, go ahead and let me know in the comments.

Thank you and happy learning.

So, we have divided this two-day master class into three components. The first

component is geni fundamentals where we'll talk about the geni value tech types of AIPM and nature of large language models. Then we will talk about

language models. Then we will talk about inside EIPM. We'll talk about context

inside EIPM. We'll talk about context engineering, rag, prompt engineering, finetuning, AI agents and AI events.

Even if you're not able to understand anything right now, it is my promise that by the end of tomorrow, you should be able to understand these topics. And

then we are going to talk about giving you actual examples how products such as chat PR granular gamma AI notebook LM are actually built. Right? So this is going to be the session agenda for these

two days.

And the motivation or the purpose of this session is not to make you an expert in AI product management because that cannot happen in 4 hours. But the

motivation of this session is to make sure that you feel motivated after completing these sessions that yes I have a clear road map. If I can learn

this much I can also go ahead and learn and explore the entirety of AI product management. So this is to give you the

management. So this is to give you the confidence so that you are able to get started. Right? And tomorrow and today

started. Right? And tomorrow and today and in today's session I'm going to ask you in the end that whether you are feeling feeling motivated after the session or not right

now let's talk about the Gen AI revolution.

Right? The companies that you are seeing on the screen, they are now among the world's most valuable companies and many

of these companies have able to get this kind of amazing transition just after 2022.

So from 2022 to 2023, this company Open AI has got a valuation of 500 billion. This anthropic company which was not existing few years ago now

has a valuation of $183 billion and then we have something called as XAI Elon MRI which has the valuation of $200 billion.

Not only the new companies are doing well but the older companies which are betting on AI as a revolution which is Microsoft has gone from so Microsoft was

founded in Yes. Microsoft

Yes. Microsoft was founded in about 1980s 1983 and from then to 2021 they were able to get a

market cap of 1.8 trillion but in the last 3 years they have grown from 18.8 to 3.8 almost twice increase. Similarly

Google after betting on AI they have grown their valuation almost 2.5x similarly meta has grown too much from 0.3 billion or 300 billion to 1.8 8

trillion and then Nvidia which is the most which is the company which has most benefited from this AI revolution it has gone almost 10x of their valuation right

and then we have also seen some very new age companies which are actually created in this period only so ker the company did not existed few years ago is now worth $9 billion glean is worth $7

billion 11 labs is worth $6 billion and perplexity is is worth $15 billion So now there is a sharp change that has

happened in the market recently because of which people are able to understand that there is this new technology called Genai which can go ahead and do so many things together. Okay, but understand as

things together. Okay, but understand as many people have predicted this can well be a bubble. So how do you check if something is a bubble or not? It might

be possible that all of these companies are overvalued in some of the other way.

But let me ask you a very honest question. Okay, everyone go ahead and

question. Okay, everyone go ahead and answer me in the chat.

When was the last time that you have used chat GP and Gemini?

Yeah, I'm sure for many of you your one of the tab is chat GP right now.

Yes. Cool.

Right. So, understand there have been many technologies that have came and gone. For example, there was a

gone. For example, there was a uh there was a revolution on blockchain, there was a revolution on cryptocurrency, there was a revolution on NFTs. Although you were listening

on NFTs. Although you were listening about all of these things on news on multiple websites and Twitter but they could not become a part of your daily

life right so the verdict is that it might be possible that these things that we are talking about the valuation and all these could be larger than life valuation that is completely possible I'm not disagreeing with that but we

also have to understand that genai is a technology that has been able to prove itself that it is actually very helpful Because technology if it is only

increasing the valuation that might mean that yes it could be a bubble but if it is generally helping you in your daily life then it just means that it is just a matter of time that we are able to find its relevant use cases and utilize

it for work and more of our personal life right but how has it happened like what is there in the genai revolution so understand the genai revolution has

happened because of a innovation in technology which is called as large language model. Okay. So what is a large

language model. Okay. So what is a large language model? A large language model?

language model? A large language model?

A large language model is a neural network trained on massive amounts of text to learn patterns in language enabling it to understand, generate and reason with natural language by

predicting the next more likely most likely word or token in the context.

Right? This is the Wikipedia definition of large language models. But what

exactly is a large language model? Let

me go ahead and tell you. Please

everyone focus on the session super important part.

Yes. So we are talking about large language model.

So a large language model is it is kind of a code or an algorithm.

It is kind of a code or algorithm or a piece of technology that is very good at predicting the next word in the sequence. That is get very

good in predicting the next word of the sequence.

Right? For example, I just take this sentence.

I will take this sentence. I will

increase the size.

Right? So let's say if I enter this particular thing to a large language model, if I tell a large language model that a large language

model is a then it is going to return me neural. It is going to return to me the

neural. It is going to return to me the next word. If I give it a complete

next word. If I give it a complete sentence that a large language model is a neural then it is going to return me network and then I will keep on iterating so that I'm able to get the

complete sentence. So what is a large

complete sentence. So what is a large language model? Large language model is

language model? Large language model is the next word or token predictor.

And when you do this repeated number of times, you get the complete sentence, the complete book, the complete essay, the complete summary. Right? So this is the basic fundamental how a large model

works. You give it some sentence and

works. You give it some sentence and complete sentences. It will try to

complete sentences. It will try to complete it. And how this how does it

complete it. And how this how does it completes it? Let me go ahead and tell

completes it? Let me go ahead and tell you. Okay. So understand first of all if

you. Okay. So understand first of all if you guys are able to observe I have written token in the bracket. So what

happens is whenever you give a word to a large language model. Okay. So let's say I

language model. Okay. So let's say I give a word to language model that he lives near the river bank.

He lives near the river bank. Okay. So

large language model is a machine. It's

an algorithm. It's a code.

it will not be able to understand what are these exact words. In order to make sure that the machine is able to understand everything, what we do is

we convert this into numbers. So in

order to make sure that the machine is able to understand things efficiently, we convert these into numbers. These

numbers are called as tokens.

Let me show you how it works. So I'll go to a website called as tick tokenizer where if I write this sentence

the sentence is he lives near the river bank right now the large language

model has converted this the large language model has converted this into these tokens. So these are the tokens

right? Every model can generate

right? Every model can generate different kind of tokens. Every model

has different kind of combination of tokens. Okay. The tokens are generated

tokens. Okay. The tokens are generated to make sure that the large language model is able to efficiently utilize its memory and context because numbers are

easier to interpret for machines rather than the words.

Right? And in total in English language we have around 26 alphabets and we would have so many maybe millions of

applications, millions of spellings, right? Vocabulary. In GPT4 we have 100k

right? Vocabulary. In GPT4 we have 100k tokens, right? So what is a token? Token is a

right? So what is a token? Token is a numerical representation of your text.

Token is a numerical representation of your text. Right. And at a high level,

your text. Right. And at a high level, three words in English is equal to sorry

at a high level in English two words are equal to three tokens at a high level like not this is not

completely true in all the cases but for most of the times on an average two words is equal to three tokens. Why

tokens are important? to understand if you go to any of these application let's say let's understand open AI pricing so if you look at pricing of any of

these models you'll be able to understand that they work on tokens how many tokens are input how many tokens are generated how many tokens are cashed

right so for every 1 million tokens that you give to charge GP it is going to ask you for the input it is going to take $1.25 $25 from you. Right? So what is a

token? Token is a

token? Token is a conversion of your words into mathematical numbers so that the model is able to understand them very

very efficiently.

Right? How many tokens are possible in a model is called as the vocabulary size of the model.

How many tokens are possible in a model is called as the vocabulary size of the model.

Right? So this is this is called as the tokenization. Right? Now what

the tokenization. Right? Now what

happens the purpose of an LLM is that if you give it a set of tokens let's say 1 2 3 4 5 it will try to predict what is

the next token then if you run it again then this becomes your full sentence up to six then it will generate seven and then you have new sentence 1 to 7 and

then it will generate eight. So it is going to go ahead and complete the sequence. Right? That is at the most

sequence. Right? That is at the most fundamental level how large language models work.

Right?

Everyone a quick yes or no if you're able to understand everything so far.

A quick yes or no before I could go ahead.

Wonderful. Great. Now let me go ahead.

But this is a simple explanation. But

how does it happen? How does it predicts? Okay. So what happens is

predicts? Okay. So what happens is this is how a large language model works. Okay. So

there are three stages. Whenever you we create whenever we create a large language model,

we use these three stages. Every large

language model is created by going through these three stages. The first

stage is called as a pre-training stage.

The second stage is called as the training stage. And the third training

training stage. And the third training is called as a post training stage. Right? So in the

training stage. Right? So in the pre-training state, understand what happens is the models are able to predict the next token very accurately

because they are very good with pattern recognition. So what we do is we need to

recognition. So what we do is we need to train this model on a lot of data. We

need to train these models on a lot of data. Okay. But before that let me tell

data. Okay. But before that let me tell you let me help you understand why this data is necessary. Okay. So how many of you

can tell me what is this?

Tell me in the chat what is this guys? Go ahead and tell me in the chat

guys? Go ahead and tell me in the chat what is this?

Wonderful. That means you guys have studies in the uh in the elementary mathematical classes, right? So y= mx is

nothing. It is a very simple equation.

nothing. It is a very simple equation.

It is the equation of a line. any

straight line. This is y-axis. This is

x- axis. This is the slope and maybe this is the intercept.

Right? So this is a simple equation that is y = mx + c. Right? Now if I want to know

what is the value of y at this value of x. If I want to know what is the value

x. If I want to know what is the value of y at this value of x. So x is the input and y is the output.

So now if I want to know what is the value of x at this what is the point of y at this value of x because there are two things right now I might not be able

to understand because there are two parameters which are unknown to me I will not be able to understand but what if I already know two values of input and two values of output so let's say

somehow I was able to find that if x were our one if x is equal to

1 then y = 2 and when x =0 then y = 1.

Right? So let's say somehow it is given to me that this is true. This is called as training data. Training data is

training data. Training data is something where the output and the input is known to me. Right? Right now what

I'll do is I will try to find out the value of m and c. So y = mx + c. I'll

put the value of x and I'll put the value of y. So 2 is equal to 1 into m +

c. 2 = m + c.

c. 2 = m + c.

So c = 2 - m.

Right? I also have the second value. So

what I'll do is y = mx + c. That means 1 is equal to

m into 0 + c. c = 1. Right? Now c = 1

and c = 2 - m which means m = 2 - c. 2 -

1 is equal to 1. So c= 1 and m= 1. Now

the equation becomes I have found all the parameters. I was

able to find the value of M. I was able to find the value of C. So C is 1 and M is also one. So the equation is Y = X + 1. Now you give me any kind of input in

1. Now you give me any kind of input in the world. Give me any kind of output in

the world. Give me any kind of output in the world. Give me any kind of input in

the world. Give me any kind of input in the world and I'll be able to tell you what is going to be the output.

Right? You give me any kind of input in the world and I'll be able to give you the output. Right? Now if you go ahead

the output. Right? Now if you go ahead and amplify the same thing, if you go ahead and amplify the same concept to billion number of times. So now what

happens is a large language model is a equation like this

but it contains almost greater than a billion parameters.

Right? So in order to find two parame in order to find two value of parameters m and c I needed to

have two values of input and output also called as training data. Training data

that I already know right in the case of a large language model a large language model is nothing. It is a very complicated mathematical equation

plus the equation contains a lot of parameters. How many parameters? Maybe

parameters. How many parameters? Maybe

billions of parameters.

Right? So in order to determine these parameters, what I need to do? I need to have billions of training data, which is I need to have

large training data, right? And then this is repeated

right? And then this is repeated multiple number of times to get the values of these parameters. So how do we get this training data? We get this training data from the pre-training. So

before we train any model, what we do is we crawl the internet.

We look at the forums. We take the books that are written already, literature, well-written books. And then

we convert this data, we clean this data, and then we train on this data.

How do we train? How do we train on this data? Understand?

data? Understand?

Any piece of text that you have that piece of text can be converted into two parts the input and the output. For example,

let's say in the training data I was able to get this.

So I have written a blog around a IML product management. Okay. So let's say I

product management. Okay. So let's say I have this sentence. So I will collect

sentence. So I will collect I will collect a lot of data. I will

break down this data into small parts so that I'm able to feed it to the large language model and that then what I'll do is let's say one of the training data is this

one of the training data is this. So

what I'll do is I know that this sentence is correct. So

what I'll do is I will just feed this part I will feed this part as the input and this part as the output

because what does a large level model does if you go ahead and give it if you go ahead and give it any sentence it will try to complete it. So I'll give

multiple billions of rows and billions of uh let's say components of data in the form of input and output and when the large language model will learn it will learn from these patterns and then

it will be able to give you almost real kind of reasoned answers right a quick yes or no everyone if you are able to understand this

I have tried to help you understand large language models with the help of a very simplistic A linear equation.

Yes. Cool. Now let's go ahead.

Yes.

Cool. Now we have understood this. Now

let me go ahead and take you one step ahead. Okay. So now everyone please look

ahead. Okay. So now everyone please look at this very carefully.

So how does a model trains itself. What

happens is this is what the structure is. You have the training data. In the

is. You have the training data. In the

training data, let's say you have this sentence. You are going to break this

sentence. You are going to break this sentence into two parts.

One part is this the initial part of the sentence. Other part is this the later

sentence. Other part is this the later part of the sentence. The initial part of the sentence is the input and the later part of the sentence is the output.

So ideally what you want is you want the model you want the model to behave in a way that whenever you write how LLMs

it should complete with work. Okay, this

is what you want. So what you'll do is you will break the sentence. This is one sentence. This other sentence you are

sentence. This other sentence you are going to tokenize it. Tokenize as in you are going to convert it into maybe some kind of numbers. Okay. And

every model that you are running they would have their own method. So there

are some free and available APIs available through which you can go ahead and tokenize your content into any kind of numbers. Okay. So we have done the

of numbers. Okay. So we have done the tokenization.

Now whatever are the input tokens tokens for how LLMs are going to be feed as an input

right and the output token is going to be here I'll take I'll tell you how it is going to work right so now it will be feed into a large language model okay a

large language model is a specific kind of uh I would say uh right now we are talking about a large language model which is a neural network and the Type of the neural network is transformer.

I'll talk about what a transformer is in brief in tomorrow's session. Okay. So a

transformer what is a transformer? It is

nothing but a mathematical expression a non- mathematical expression understand it thousand times more complicated than the line equation that I have given you. And it contains the

parameters the constant that I have told you right. And the purpose of this large

you right. And the purpose of this large language model training is to identify the value of these parameters to identify the value of M, C and other

parameters that it has. It can have billions of parameters. Right now what will happen? It will initially when it

will happen? It will initially when it is not trained, it will try to generate anything random. Okay? And also

anything random. Okay? And also

understand one thing that the large language models are not deterministic.

Listen to this very carefully. This is

super important to understand the language of large language models. I

told you that they are next token predictors. Predictors means they will

predictors. Predictors means they will not tell you that this is a token after this. They will give you some

this. They will give you some probabilities. So

probabilities. So he is a this is the input and the large language model is going to give you

probabilistic outputs. So it is going to

probabilistic outputs. So it is going to give you many options. It is it will tell you he is a man, he is a boy,

he's a father, he's a singer. So there are multiple probabilities, right? And then he's it

probabilities, right? And then he's it is also going to tell you the L language model that what is the probability. So

let's say man the probability is 0.5 boy the probability 0.4 father probability is 0.05 single probability is 0.05.

Right? So it will not tell you the one answer. It will not just give you one

answer. It will not just give you one answer. It will give you the

answer. It will give you the probabilities of what other words could be there and then you can go ahead and choose the top word or second word whatever your large language model can

go ahead and pick up. Okay. So always

remember the large language models are not deterministic. They are stochastic

deterministic. They are stochastic which means they can go ahead and pick random words which are going to be correct in some

context but if you go ahead and give same prompt to same model at the same time it can go ahead and give you different kind of outputs

right so this is important nature of an LLM to know and this is also going to be of foundation of when we go ahead and learn about AI evaluations Right?

And how does it generates this? It

generates on the basis of training. So

it would have seen a lot of data and based on that data it generates that yes I think this should be the probability based on what I have learned.

Right?

Yes.

Cool. Now

going back to a diagram. So what will happen here is what we have done so far is we have done the tokenization we have taken the input part we have feed it into the transformer the transformer has

generated some random results and then what will happen we have the real actual token the actual token is going to be

the meaning of work like the the the numerical token for work. Okay. And here

it is going to generate something. Now

it will be compared that work should have a probability of one or work should be at the top of the predicted numbers or the predictive tokens and then we are

going to calculate the difference that it should be word it should be work but it is not work. So it is going to share this feedback with the transformer and then what

transformer will do is transformer will correct the parameters inside and then this model will run on multiple training sets for billions and trillions

of time and then you are going to go ahead and get a model that is going to accurately predict what should be the next probable word.

Right? So what we do is we give it the training model. We have the output. We

training model. We have the output. We

have the input. Then we check we check the difference between the real output and the expected output. Right? Then we

calculate the difference which is called as a loss function which is called as a mathematical loss function. This is passed as a feedback

function. This is passed as a feedback feedback to the large language model.

And then when it keeps on doing this billion number of times it is able to set a very accurate value of these parameters.

Right? If you run it a number of times, you will have a pretty accurate model or a large language model. Right?

Understand? If the number of parameters are in a race of billions, this is called as a large language model. If it

is less than billions, maybe in the ratio of millions, it will be called as a small language model. Example of a large language model. Example of a large language model is Gemini chat GPT spell

uh example of a small language model is Google's Gemma right yes quickly everyone a quick yes or no

in the chat if you're able to understand this at least at a high level wonderful great you guys are intelligent great amazing So loss function understands the

difference between what is the predicted token and what is the actual token and then it pass on this feedback to the transformer. The

transformer adjusts everything accordingly.

Right now a bigger question that is that should aride arise in your mind is that why all of these things are happening after 2021

like uh why all of these things are happening very recently why not before okay so understanding language

understanding language is not an easy task for machines why because of this reasons so let's say my sentence is he lives

near the river bank.

Right?

Initially, traditionally what people used to do was if they want to create any kind of AI, if they want to create any kind of predictive model or intelligent model, what they will do is

they will work on word by word.

You might have seen autocorrect in your mobile phones, right? Whenever you try to write something, it will only catch the last word and then based on it, it is going to predict. It will not look at the entire sentence. Okay. But people

who are able to understand, people who understand NLP or natural language processing or programming, they were

able to understand that it's not enough.

They were able to understand that if if we want to really understand language, we need to understand the meaning of each word with respect to that sentence.

For example, in this particular sentence I have written he lives near the river bank.

So in this bank means the side of the river.

Right? But if I did not know this sentence, I could have as well concluded that this bank is a financial institution.

So in 2017, a group of computer researchers created a research paper, published a research paper. They all were working in

research paper. They all were working in the Google brain and deep mind team. And

the name of the paper was attention is all you need.

And they told they came up with the hypothesis that if we really want to truly understand the language models, if we truly want to understand the language

rather than looking at things token by token, we should try to find out what are the relationships of these word with entirety of the sentence. Initially

these these predictive models can only look at one word at a time but now with this approach we were able to go ahead and consume all the sentences and essays and whatever is

the context window at once. So every

word is now able to map it relationship with other words. So these words are created into tokens and vectors and then the relationship between all the vectors

is calculated real time and then as a whole we are able to understand what does this paragraph what does this sentence what does this context says

that was a very big shift of how genai was born okay but now because of that one problem occurred the problem was the sequential

processing of data was easy It can be done on a CPU but if you need to process multiple sentences multiple parts of the sentences at once understand when I'm

talking about the sentence he lives near the river bank

at once the algorithm or the computer has to calculate this relationship of he with lives with

near with the with river with bank. Then

parallelly it needs to understand the relationship of lives with he near the river bank and then it also needs to understand the relationship of this with

this. So now understand this is

this. So now understand this is complicated maths.

This is resource intensive maths. Resource intensive

intensive maths. Resource intensive maths. It will take time. It will take

maths. It will take time. It will take effort. It will take a lot of processing

effort. It will take a lot of processing power, right? So in order to create a

power, right? So in order to create a good solid large language model, you need two things. First, you need a lot of data on which you can train so that

AI is able to find patterns. Plus, you

need super huge processing power.

And now people started thinking where do we get this processing power from?

because quantum computers are not yet invented at least at the scale for practical purposes and CPUs are very much limited.

Then people thought that these kind of metric multiplication like one to many multiplication also happen at gains

and games are running on GPUs.

So people understood that yes transformer is an architecture. So the

neural networks or the algorithms that operate in this way by calculating the uh the relationship of one word with

every other word. These kind of algorithms are called as transformers.

They work on the mechanism of attention which is one word is paying attention to every other word in the sentence.

Attention means one word is paying attention to every other word in the sentence to derive its meaning. Right?

And then the neural networks or the algorithms that use attention as a mechanism to predict are called as transformers.

Right? And transformers are best run on the same technology that runs these complicated games with is GPUs.

Right? So this is a history about how GPUs were best suited for running large language models. And that is also the

language models. And that is also the reason why we were able to get a huge jump here. We were able to get a huge

jump here. We were able to get a huge jump here.

Right everyone, a quick yes or no if you're able to understand this.

Yes. If I see one more LinkedIn link in the chat, I'm going to block you. Don't

distract other guys, please.

That's so large language models run on GPUs. They are trained on GPUs.

GPUs. They are trained on GPUs.

Right? Now,

so we have understood this. Now let's go ahead. Okay. Now the third part, I have

ahead. Okay. Now the third part, I have told you about the two parts already.

The third interesting part of a large language model.

So we have talked about pre-training, we have talked about pre-training, we have talked about training. Now the

third part is post-training.

Now after you generate, after you train a large language model on all the internet data that is available, you get a personality like a baby. A baby has

the ability to think. It has the ability to perceive its surroundings and they will also be able to go ahead and maybe blabber a few words, right? But they

don't have a personality yet. So

depending upon a baby is born in a Hindu family or a Muslim family or a particular culture, they are going to get their nature. So if you are reading Quran, you are going to think this is the God. If you are going to read maybe

the God. If you are going to read maybe Gita, you are going to understand that maybe this is the God. Okay? Reality

could be something else altogether.

But based on how you are conditioned, based on how you are trained after you are in the world, that is called as postraining. That gives you a

postraining. That gives you a personality. So postraining is very

personality. So postraining is very important. It converts

important. It converts the next token predictor machine next token predictor machine to a

helpful assistant.

It converts the next token predictor m machine to a helpful assistant. Right?

So what we do is we hire real human laborers like what what do we call it? People who go ahead and label the data. So these

people give accurate information to the large language models and then this data is feed again into the training and then the large language model gets a

personality. For example, there is one

personality. For example, there is one thing that large language models have struggled for a long period of time.

Okay. So large language models generally tend to hallucinate. Although the

hallucination has reduced very uh very recently but still there is let's say a lot of hallucination still.

Okay. So Meta came up with this very interesting perspective. What they did

interesting perspective. What they did was they just give some sample input and output to the large language models.

In the input they told some random question like question for which the answer is not possible right for example they created some

random names and ask GPT or ask let's say llama that who is this person initially before this training llama or chat GPT they are going to give you random answers even if the person does

not exist they will try to frame something that is called a hallucination they are going to give you wrong answer but they'll give you with complete confidence right so what meta did was it gave them some questions which did not

have any answer and in the output they mentioned they asked the llama model to give the answer as I do not know about this I do not know about this and then

once the large language model started understanding that yes I can give the answer that I do not know about this that is okay right to show you some other sample data

okay so this is how the data looks like for example we want to understand about summarization we want to make sure that our model is able to summarize. So we go give this kind of input. Summarize the

following passage about neural network in one paragraph. A neural network is composed of layers of neurons that transforms inputs. Okay. So we'll give

transforms inputs. Okay. So we'll give this prompt and then this is the desired output. This will also be feeded into

output. This will also be feeded into the large language model and then we will be able to retrain it again and again in this data to make sure that we

are able to get the right answer.

Right? we are able to go ahead and get the right answer. Okay. And then these are the methods in which we are going to test. It has to be clear factual summary

test. It has to be clear factual summary and neutral to right. So we give all of these feeds to large language model after it is created to the base model and then we are able to get a more

intelligent model.

Right? Now guys, all of this data, all of these files, all of these presentations, the notes and everything would be given to you if you remain till the end of the session today and then we are going to go ahead and uh and you go

ahead and give the submit the feedback form. Okay.

form. Okay.

Yes.

Cool. So I hope everyone was able to understand how large language models works. Everyone please tell me on a

works. Everyone please tell me on a scale of 1 to five if you're able to understand whatever I have told you so far. On a scale of 1 to five, please.

far. On a scale of 1 to five, please.

One you did not understand, five you understood it well.

Cool. We have 821 people. Amazing.

Great.

Super. Super guys. Amazing. Amazing.

Thank you.

Yes.

People who are not able to understand, just look at the recording. I'm sure

you'll be able to go ahead and understand this well. Yes. Cool. People

are not able to understand, please focus better. Please focus better in the

better. Please focus better in the session.

Yes.

Cool. Let me just go ahead.

Yes. So now based on whatever I have told you now let us go back to economics. Okay.

So we have talked about mathematics, we have talked about computer science, we have talked about psychology of a model.

Now let's back go back to economics and the business. Okay. So

the business. Okay. So

as a product manager you need to understand at what levels is the value created in the large language models by the large language

models. Okay. So the first layer is the

models. Okay. So the first layer is the infrastructure layer. This is the layer

infrastructure layer. This is the layer on top of which large language models operate.

Okay. What they do is here we have companies such as Nvidia and Google's Vert.x AI which provides the tensors or

Vert.x AI which provides the tensors or the GPUs where these models can run. So

they provide computation data platform GPUs and all right. So the core technology that you need which is infrastructure which is uh how where do you deploy

these models is provided by these infrastructure layer companies.

The second layer is the model layer companies who are creating and owning these models. Companies such as OpenAI,

these models. Companies such as OpenAI, Anthropic Meta Grock uh and Deep Seek these are the companies which are owning these models and they will give you large language models,

small language models. They will also give you fine-tuning. What is

finetuning? We are going to discuss in the coming sessions. Right now, this is the core. Now, on top of this, there is

the core. Now, on top of this, there is a bigger opportunity that you as a product manager can utilize all of these

technologies like these models in order to build some amazing use cases. So,

lovable gamma chat GPT Gemini they are all based on this model and the infra layer. So you as a product manager

infra layer. So you as a product manager can play a very important role in the application. Application layer is how

application. Application layer is how are you able to utilize the power of a large language model in order to solve the problems for your customers.

Right? So internet is provided by someone else. The routers are provided

someone else. The routers are provided by someone else. But Amazon has created an application on top of internet which is able to give you the e-commerce at your doorstep.

Tinder has created an application on the top of internet so that you are able to find your next state. Similarly in AI you should be able to create a lot of applications on these on the top of

these large language models so that you are able to create a lot of value right and the fourth layer is you can

use these tools you can use these large language model based tools such as chat GPT such as GMA such as Kurser in order to make your clients and make yourself

more productive. So service agency

more productive. So service agency companies such as TCS, Realast and all they will use these tools and then they will help the company achieve their outcomes.

Right? So this is the whole value chain.

People create infra then some people create model some people create application based on model and some people create services. Okay. But this

is where you can drive a lot of value without investing a lot of money.

And this is value at scale.

Right? All of the companies here, all of these companies are same companies. They don't own models. What they do is they work on top

models. What they do is they work on top of models created by other companies, right? And they are able to generate a

right? And they are able to generate a lot of value.

Right? Now, based on this gen AI value stack, we can divide multiple kind of AIPM jobs. Okay? So, first of all,

AIPM jobs. Okay? So, first of all, understand that every PM is an AIPM. If

you are using any kind of AI technology in your life, GPT, Gemini or anything, you are an AIPM. So I divide AIPM into two parts. The first part is the AI

two parts. The first part is the AI enabled PMS. These are people who are using products like Chat GPT, Lovable, Jira or any

other AI product in order to make themsel more productive in order to solve problems for their customers. in order to save time in

customers. in order to save time in order to provide more impact. So anyone

any PM who is a user of any of these AI tools they are an AI enabled team. So

100% of all of us are AI enabled PMs. Right. The second is AI product PM. Now

Right. The second is AI product PM. Now

this is where the value is. This is what we are going to talk about which we have talked in the last sess in the uh today's session as well initially and then we are going to talk about today more and in the next session as well.

Okay. So AI product PMs can be divided into two parts. One is the core AIPM.

These core AIPMs need to understand how the technology behind AI works. So

training pre-training post-raining how do you create a large language model? How do you store? How do you make

model? How do you store? How do you make it efficient? How do you consider memory

it efficient? How do you consider memory of a large language model? All of these core things are the responsibility of a core AIPM. So people who are creating

core AIPM. So people who are creating the infra or the model are the core AIPMs. These people need to be technical and they should have a solid understanding of machine learning in

order to become a core IP.

Whatever I have told you so far about how GP works and all this is a work of a core IPM.

The second kind of PMS where most of you should be going is applied AIPM. Now

applied AIPM are the people who are creating useful applications by using these core technologies. So people who are building notion AI, Grammarly, lovable, chat, GPT, people who are

building these products are applied AI PMs. Applied AI means the people who are actually using AI tools like AI

technologies such as models and infra in order to build something helpful for their customers.

Okay. So these are the only the users.

These are the core technology creators and these are the product creators.

Right? So can you guys tell me where do you fall right now?

What is your best description right now?

Where do you fall right now?

Yes, wonderful. So all of you fall under the

wonderful. So all of you fall under the AI enabled part right now but eventually you should go ahead and Yeah guys, everyone is trying to focus and

learn. Please do not put any kind of

learn. Please do not put any kind of links in the chat. I've already blogged a couple of people. Do not let me to do it more. Let us go ahead and focus on

it more. Let us go ahead and focus on the session.

Cool. Now

this is the differentiation. Everyone is

here.

If you are coming from already a very strong technical background, you should try to move here. But everyone, even if you are coming from a technical or nontechnical background, you can always move here. This is where I want you to

move here. This is where I want you to move.

Okay. Now let's go ahead. Okay. So tell

me everyone.

Tell me everyone if you are able to understand this meme. Look at this very carefully and tell me if you are able to understand this meme.

Yes, cool. So understand there are like so much valution that has been created in all of these things. Okay, this is sitting on the top of LLMs. Okay, so I'm

going to give you some examples of how various kind of technology the various kind of technologies are working on top of these large language models. Okay,

but we are right on time. We will take a quick break here. We will take a quick break of five minutes and then we will come back. Okay.

come back. Okay.

So now let me go ahead and tell you how do So now you have understood that yes a large language model. Okay. So

there is something called as a large language model.

which is provided by companies such as Open AI, Anthropic, Meta, Google and many more. Okay, these are

the large language models that are provided by so many companies. Now,

right now, this large language models can do three things for you.

Okay, the first thing is they can understand content.

They can understand what you are trying to say because they have seen so many patterns of data. They're able to understand what you are trying to say.

Okay. So if you ask it to generate a summary, it will be able to understand the whole paragraph and then it will generate the summary. Okay. The second

is it is able to transform content.

Transform means if you ask it to create a summary of some content, it will transform that into summary. If you ask it to create a quiz on some content, it will be able to transform that into

code. Okay. And the third part is it is

code. Okay. And the third part is it is able to generate content. So if you want it to generate images, code anything it because it is data it should be able to generate because it has seen enough code

in its life right. So it will be able to go ahead and generate. So as a large language model the capabilities of a so as a product manager you should understand that there are three core capabilities of a large language model.

The first is it can understand second is it can transform and third is it can generate data.

Right? So you can use it for multiple ways. You can use it

ways. You can use it to generate summary. You can use it to generate answers for your questions. You

can use it to generate code. And you can generate for almost everything that you want, right? And because they have been

right? And because they have been trained on tons of data, tons of terabytes of data, generally the accurations are very accurate. So now

what happens? So far whatever the software that we used to create so far whatever the software that we were used to create these softwares were generally of four

types. Okay. So any software that we

types. Okay. So any software that we have used so far what they will do is they will generally create a data they will read data they will update the data or they will delete the data. So all the software that we have seen let's say

Facebook on Facebook what happens you go ahead and create a profile and then on your profile some database element is created and then your people can see your profile some people can go ahead and upload some videos and then you are

able to see these videos and entertain yourself so generally most of the application before the AI world were actually create read update and delete

this is called as application but now AI has been able to give a brain to your softwares. So as a product

manager, you need to understand what this large language model can do and how can I utilize it to create better products.

Right? So

I'm going to give you this understanding with the help of few examples. Okay? So

there is a product called as granola.

Okay. And this product is similar to many other products. So it's a very simple product that if you go ahead and use granula, it will go ahead and get added to your calendar

and whenever you are joining the meeting, it will take the transcription and recording of that meeting and it is going to summarize the meeting for you so that you do not have to pick up a notebook every time. What happens is

whenever we are joining the meetings we have to write down that what are the things that are discussed so that we do not forget and then we create a minutes meeting like the minutes of the meeting right so there are multiple tools like

granola there is fireflies otter fathom most of you most of them which you can see also on the uh the attendee list right because many people have added

their note takers so now how these noteers are created what is the value behind them so I'm going to show you a simple tear down Okay.

So, everyone a quick yes or no in the chat if you understand what a AI noteaker is.

Amazing. Great.

Yes.

Yes. So now let us try to understand how this product works and this is like a very important case. It will help you understand how AI products are built.

Okay. So first of all understand everyone whenever you create any product as a product manager you have to look at five angles.

Don't just go ahead and just think about technology. Technology is one important

technology. Technology is one important part but it is not the most important part. But AI is just a piece of

part. But AI is just a piece of technology. So do not focus on just

technology. So do not focus on just using AI. Focus on solving the problem

using AI. Focus on solving the problem of the user. So whenever you look at any product, please ask yourself five questions. Right? The five questions are

questions. Right? The five questions are what problem does it solves in people's life?

Right? What problem does it solves in people lives? Okay. So

people lives? Okay. So

can you guys tell me? to think about it for a moment and then tell me in the chat what do you think what are the problems that granola solves in people's life

think about this as a product manager this is the most important value that you need to add that how am I being useful to the users what problem does it solve so tell me what is a problem that

these note takers solve Yes. Very good. So

Yes. Very good. So

at a task level, it helps you record and summarize your meetings.

But that is just an output. The outcome

is that it is able to save you time and it will help you be more accurate because when you are writing something it might be possible that yes you are missing something and you also have to

invest a lot of time and attention when you are writing things right so granular saves you time it makes you more efficient and accurate always remember

it is okay if you're not using AI in your product AI is not a silver bullet that will improve your But if the product does not have a value, if you are not able to clearly define why this

product should exist, then please do not create the product. Okay, this is the most effective value that you add as a product manager. Okay, now the second

product manager. Okay, now the second part which is a business value.

Why should the business per pursue this opportunities?

Okay, think about this for a moment everyone and then tell me in the chat why should the business pursue this opportunities?

How do you find this out?

Let us keep it collaborative so that everyone is able to learn accordingly.

Yes.

So now what we'll do is now what we will do is we will try to understand that if we are creating a meeting assistant

how many people in the world are first looking for this kind of solution.

So you will have a prediction that people who are a part of small businesses or large businesses they are going to love this kind of solution because they are going to save time and then you are going to understand how

many people are there who are working in these small businesses. how many small businesses are there and then you are going to size a market opportunity.

Okay, that is one way that it is going to help the business because there is a large market opportunity and in order to do this research you can go ahead and use chat GPT notebook LM Gemini or anything right that is the business

value now the other kind of business value is let's say we are not talking about granola we are talking about zoom right so because zoom already has a lot

of users who are using its platform to conduct meetings it is an strategic opport opportunity for Zoom. It is a strategic business opportunity for Zoom

to create an inbuilt meeting assistant.

Right? So it aligns with the strategy and it also aligns with the market opportunity.

Right? So understand everyone as a product before being an AI product manager, you are a product manager and the purpose of a product manager is to bring the user value and to bring the

business value.

Right? Now the third thing is how do we make sure that the product is intuitive for the user to use. How do we make sure that the

use. How do we make sure that the product is very intuitive.

Right? So in all of these tools you would have found that you do not have to do anything. You just install the tool

do anything. You just install the tool you add it to your calendar.

Automatically whenever a meeting occurs the person the the meeting note taker would be automatically added. Granola

goes to the next level that right now this meeting is recorded with granular but you do not know why because it is installed on my computer. It is taking over the microphone and the video and it is doing it by itself. So now you have

to do optimizations on the design part so that you are able to reduce the friction for the user. If I have to add a granular every time, if I have to add uh uh this fireflies every time when I'm

doing a meeting that is going to be resistance for me right? So this is called as being a proactive software.

So they are able to get meeting into meetings automatically. So you don't

meetings automatically. So you don't have to do the work automatically it is happening. Right? So this is the

happening. Right? So this is the intuitiveness and then they have created an interface so that I'm able to see what are the meetings that I have conducted. I can go ahead and ask

conducted. I can go ahead and ask questions based on my meetings. Right?

So this is the value part, business part and the design part. The next part that we are going to understand today is how will this be engineered.

Engineered part is feasibility part. But

please remember this line. User value

and business value as a product manager should be your first focus and feasibility should be the next focus.

Feasibility should not be the number one focus of a product manager right now.

Please start with the user problem and the business problems. That is what you add the most to the company. That is where you are being most impactful rather than only talking about engineering and

feasibility because there are people who are more eligible than you who can do the task. Okay. So how will this AI

the task. Okay. So how will this AI product be orchestrated? But before that everyone a quick yes or no in the chat if you are able to understand the three

parts user value, business value and design value.

Amazing.

Wonderful.

Yes. So there is one person who has said no four number of times.

Great. So

yes. So now we have understood the three parts. Now let me give you a bit about

parts. Now let me give you a bit about engineering. Okay. So this is how this

engineering. Okay. So this is how this product looks at the back end. Now look

at this very carefully. Super important.

If you just under understand this one product, I can assure you that you can reverse engineer almost any AI product.

So please focus on this slide. Okay. So

behind granula whenever you do any meeting on zoom on meet or anyone

granola or fireflies or otter or any of the meeting recording tools even the zoom tool they will be connecting to the meeting either they can join the meeting

as a bot or they will capture the content from your screen whatever you are speaking they will take over your mic or they are going to ask the API. API as

in they will ask Zoom or Google Meet to share the recording or the transcript if you have given the permission. Okay?

So they will get the audio of the meeting, right? Once they get the audio of the

right? Once they get the audio of the meeting, they are going to convert that audio into text transcription.

There are multiple APIs available for the same. So, OpenAI has an API called

the same. So, OpenAI has an API called as whisper where you give it audio. It

will be able to give you the transcript of that audio. There is also Google text to speech like uh speech to text where you can just go ahead and give it a

audio. It will be able to convert into

audio. It will be able to convert into accurate transcript.

Right? There are some smaller players as well which are assembly.ai and other players who can also do this job. Okay?

So now what we have is we have a transcript.

Now this transcript is available with us. What these people will do is what

us. What these people will do is what granula or fireflies will do is they will take this transcript and then they

will add a system prop.

Okay. So this is a system prop.

So now everyone please look at this very carefully. Look at this very very

carefully. Look at this very very carefully. So it is going to add this

carefully. So it is going to add this prompt. The prompt is you are an expert

prompt. The prompt is you are an expert chief of staff and noteaker.

Given the following raw meeting transcript, generate a structured and actionable summary that includes the following sections. meeting title,

following sections. meeting title, data and participants, date and participants, key decision made, action items, open questions, summary, notable quotes and then you have written some

more instructions.

Right? So this prompt might be created by a product manager at granula which will convert a raw transcript that

is generated into very helpful meaning notes, meeting notes, right? And this

prompt plus this transcript would then be sent to a large language model.

Right? This will be sent to a large language model such as Gemini or Jad GPD, right? And then it will do its

GPD, right? And then it will do its magic. It will create summary

magic. It will create summary actionables and tasks. Okay? So this is how these meeting tools work. And then

what happens? You can take it to the next step. You can make it more agentic.

next step. You can make it more agentic.

Agentic means they will take action. So

what can happen is they can also go ahead and push these tasks to your slack to your calendar or to your Jira. So now

you are just adding a meeting notetaker like a meeting assistant to your calendar and then it will get the recording. It will do the transcript. It

recording. It will do the transcript. It

will go ahead and create the summary with the help of system prompts and the transcript. then it can go ahead and convert into multiple task and then it can also if you give it the permission it can also go ahead and add

it to your Slack calendar or Jira right so this simplistic architecture which is dependent on multiple LLMs and things is now

a possibility because large language models are possible right everyone a quick yes or no if you are able to understand This

right? So I want you guys to think in

right? So I want you guys to think in this manner. Okay. I want you guys to

this manner. Okay. I want you guys to think in this manner.

Everyone a quick yes or no in the chat please.

Yes. So now large language models have unlocked intelligence for you. Okay. Now

you can use them like an expert in order to go ahead and do a lot of things.

Right? So now this is like a simple orchestration. Now you can think about

orchestration. Now you can think about how you can innovate on top of the same right. So the history of granular is

right. So the history of granular is that that before granular there were players like fireflies, autoi and fedum.

What happened because of these was they will keep themselves added on. They will

keep themselves added on.

Yes. They will keep themselves added on multiple meetings and people generally get irritated if so many bots are joining the meeting. Okay. And it was also not very privacy intense in

incentive for intensive for other people. Okay. That people do not want

people. Okay. That people do not want your bots to join their meetings. Okay.

So Granla came up with a AI with a UI innovation. With Granla you don't have

innovation. With Granla you don't have to add anything. It will operate in your computer. It will listen to the audio

computer. It will listen to the audio and then it will it is going to automatically transcribe the meeting for you and give you the context right so you can do these optimizations in order to make sure that you are able to go

ahead and yes you are able to go ahead and create a lot of things right now the magic that you can add as a product manager is understand AI is

not difficult you just have to understand the concept that I'm telling you after that the next real skill that you have to develop is customer empathy Understanding

what customers want and giving them as soon as possible in the best design possible. That is the value that you add

possible. That is the value that you add as a product manager.

Okay. Now we have talked about granular.

Now let's go to the next level. Okay. So

granular is a very simple example. Okay.

The next example that we need to understand is okay. Okay. So one part is left.

okay. Okay. So one part is left.

That part is understand just building a product which is useful and business wants to build it and it is good design it is good feasibility doesn't make sure that product is

actually very very helpful right doesn't make sure that the product is very very helpful

so there is one more thing okay there is one more thing that is needed to make a product achieve its potential and generate revenue out

of the for the company which is how do we grow this product in this market which is creating a go to market strategy building

a product that is actually loved by the users creating distribution for the product making sure that people are actually using the product that is called as the growth part so how do you

grow in this market so all of these meeting assistants they are viral by default So let's say I and you did a meeting

together and because I believe in productivity I do not believe in taking notes manually. I have added a noteaker

notes manually. I have added a noteaker called as fireflies. Now when you see this that in the meeting there is another thing which is fireflies. You

are going to get curious that what is this? I should also go ahead and check

this? I should also go ahead and check what is this. Okay. So by using virality they are going ahead and creating a distribution right. Similarly whenever

distribution right. Similarly whenever the meeting has ended most of these people are going to send you emails. Let

me show you one of the emails. Just give

me a moment.

Let me check if I have those kind of emails. What is the name? Fireflies.

emails. What is the name? Fireflies.

Yes. Cool. So, let's say cool. So,

cool. So, this is the link. Okay. So generally

when we do the meeting this is the link that we received after that. Okay. So

someone added me to their meeting and they also added fireflies because they were using fireflies and after the meeting has ended it always sends me this thing that this is what you have done. This is a summary this is the

done. This is a summary this is the action item and it also tells me to go ahead and if I click on view the meeting and all it will ask me to sign up.

Right? So these are some growth tactics that these people are implementing in order to make sure that it is able to grow. The product is actually able to

grow. The product is actually able to succeed. So marketing and growth is also

succeed. So marketing and growth is also a good responsibility or a must have responsibility for a product manager, right? And understand guys, product

right? And understand guys, product management is multi-dimensional.

Not that you just create a great product and put all of your efforts into it and it is going to work. You need marketing, you need stakeholder management, you need a bit of understanding of technology, you need a bit of understanding of analytics so that you

know if you are moving in the right direction. It's a multi-dimensional job.

direction. It's a multi-dimensional job.

Right? So this is the story behind granola. Now let me go ahead and give

granola. Now let me go ahead and give you some more complicated examples.

Okay? So

tell me everyone if you ask chat GPT if you ask chat GPT that what is the most important item in my Google

calendar tomorrow would it be able to answer?

If you ask GP that what is the most important item in my calendar tomorrow would it be able to answer yes if you have not connected a Google

calendar it would not be able to answer right and if I take example of let's say this particular product called as calendar is connected to Google but if I

ask about notion then Gemini will also be not be able to answer Right.

So now we have so much data in our calendar, in our to-do list, in our Jiraa dashboard, in our stripe document like stripe dashboard. But how do we make sure that all of this content is

available to large language models so that they can go ahead and give us the answers. So rather than asking like

answers. So rather than asking like rather than thinking that where is the PD what is the task that I need to do today from my Jira or notion I should just ask a query that what are the three

things that I need to get done today and then it should give me the answer. Okay

if I want to go to this level what I can do is a very simple approach that I can do is I can just go ahead and give all

the data from my notion or stripe or atlation to the large language model.

Okay. But when I give a lot of data to the large language model, what happens?

The model runs out of context.

Okay? Because these models cannot carry infinite context. Context

means the kind of input that you can provide to these models.

Right? So now we have to work we have to work hard to make sure that these models work. Okay? And also although companies

work. Okay? And also although companies are trying to increase the context window as much as possible, there has been research that is done by great people at Chroma which is a

vector-based database company. They were

able to find that as you increase the context, as you increase the context, the performance of the model goes down.

Even if these people claim that our model can work in so much context, when these people actually tested multiple models, they were able to find in almost all the models that as the context or

the input size increases, if you increase the size of your prompts, then the quality of performance actually

reduces after a certain level. Right? So

now understand as a product manager, so even if someone is not able to understand this or this, at least you'll be able to understand this. Okay. So

when you give concise input context to the large language models, they appear to be very intelligent and very insightful and very careful. But when

you go ahead and give them too large input, then they don't know what they are doing and they get confused.

Okay. So now how to solve this? What we

do is we have another methodology called as context engineering. Context

engineering means how do you carefully curate what should go as an input to the large language model so that it is accurate so that you have given the

right kind of information so that you are able to get the right kind of insight or output from the large language model right so this is called as context

engineering so context engineering I am going to quote the person who has acquainted this term which is and Karpathi. So he says that context

Karpathi. So he says that context engineering is the delicate art and science of filling the context window with just the right information for the next step.

Art and science means you have to apply your mind. You have to think creatively

your mind. You have to think creatively to give enough context to the large language model so that it is able to answer you in the most accurate format.

Right? So this is context engineering and in context engineering we do multiple things. Okay. One of the point

multiple things. Okay. One of the point of context engineering is prompt engineering. How well you write the

engineering. How well you write the prompt is a part of context engineering.

Okay. But apart from prompt what happens is you need to give some data about your software. For example,

software. For example, if you want to build a software called as notion AI. Okay. So

let's say you will to you want to build a software called as not not motion AI but let's say you want to build a Bloomberg chatbot.

What is a Bloomberg chatbot? Bloomberg

is an organization which has access to a lot of financial data available in the world.

Right? So now what you want to do is that whatever the data is available with Bloomberg you should be able to send a question to a large language model in a natural language and then it should be

able to give you answer based on the content that is available with Bloomberg.

Right? So here the problem is you cannot store all the content that Bloomberg has in the context window. You cannot have all this content as the input. Why?

because it is going to run out of context. So what you will do is you will

context. So what you will do is you will carefully build something called as a R a pipeline. Now this is a very important

pipeline. Now this is a very important concept.

Even if you remotely want to become an AI product manager or AI practitioner, you need to understand this. This is

like a very basic thing that every person who calls themsel AI enabled has to understand. Okay. So rag means

to understand. Okay. So rag means retrieval, augment and generate.

retrieval, augment and generate.

Okay. So what happens when someone comes to your house and they

tell you that I need some sugar. Okay.

When they tell you, "I need some sugar."

You don't go to your study room and you do not look for sugar in you do not start by looking for sugar in your cupboard or in your

wardrobe or in let's say your library or maybe under your sofa. What you do is you straight away understand that sugar is an item that might be available

in my kitchen in a particular shelf. It

might be possible you don't know that which part of shelf that it is put on but you know a particular shelf or maybe you at least know that it is going to be found in the kitchen. Similarly, if your kid asks for a book or if they ask for

money, you do not go ahead and go anywhere else in the house. You know

where the money is put. You are going to just pick the room where the money is.

Then you are going to pick a particular wallet and then you are going to see in the wallet that yes in all of these four to five pockets there could be a place where I have put the money. Right? So at

a high level you know that where this information is the same analogy can be drawn to rag. So in rag what we do is

the first part is we get all the data. We get all the data. For example, we have PDFs or a

data. For example, we have PDFs or a better example is we have Bloomberg data.

So Bloomberg data is they have a lot of files. So we are going to break down

files. So we are going to break down files into small small chunks maybe in the form of paragraphs. Okay. So we have one chunk, we have another chunk, we have another chunk. We are going to

create all of these chunks which are small enough and meaningful enough.

Okay. We are going to break down this data.

Now in this data what we'll do is what we'll do is now we have the chunking of data. Now what we'll do is we will convert these data chunks into something called as vector embeddings.

Now listen to this very very carefully.

Super important. If you miss this you will miss rag. Okay I'll repeat this again.

You will take the data whatever data is available you will break it down into small chunks and then when you have these small chunks you will create you will convert these chunks into vector

embedding. So what is a vector

embedding. So what is a vector embedding? So quick oneonone on vector

embedding? So quick oneonone on vector embedding that like large language models we have also created algorithms

which are able to create any form of text into a numerical

representation numerical representation as in let's say I have a word called as cat right so I have created a mathematical

representation of cat okay what is a mathematical representation I have assigned some numbers okay so let's say the number is 1a 2a 3 this is let's say

x-axis y ais z ais okay right now to show you because we can only look at 3D I'm showing you three dimensional but in reality it can be n dimensional which is n is a very large number okay so we take

words we take sentences Then we generate vectors from these sentences. Vector is this like this

sentences. Vector is this like this point right? And then what we do is

right? And then what we do is we keep on generating these vectors.

Just a moment.

Yes. So we keep on generating these vectors and the beauty of these vectors is that somehow when you search for cat and when you

search for lion lion okay because these belong to a same family they are going to be very closer

to each other right? When you plot something like a

right? When you plot something like a pat it is going to be much closer but when you go ahead and plot something like a table then it is going to be far

from cat.

So mathematically computer scientist has been able to create this vector thing. So what this vector thing does is it will take a sentence or a piece of information. It

will convert it into a numerical identity which is a vector and then these vectors are created in such a way that if you go ahead and compare these vectors with other words based on the

distance you will understand how much this word is similar to that word or not in the meaning right. So this is vector embedding.

right. So this is vector embedding.

Okay. So what is happening here? We have

had the content and the text. We have

converted them into long sentences of vector embedding which is numerical numbers. Okay. Now what happens?

numbers. Okay. Now what happens?

And then we are going to store these vector embeddings into a vector database.

Right? So now we have the data ready. We

have data ready to be searched. So now

what will happen whenever just a moment where is that?

Yes. So this is rag.

Look at this very carefully everyone. So

there are two parts of rag. First we are creating the knowledge base. What we

have done is we have taken all the content whether it is text, PDF, transcript code or any Bloomberg data.

We have converted that into small small chunks, small information of data and after that we have created vector embeddings and then once we have created vector embeddings understand there are

standard programming code or functions to create these vector embeddings.

Okay, once the vectors embeddings are created they are stored in vector storage. There are multiple databases

storage. There are multiple databases that give you by default vector storing which is pine cone, superbase, AWS S3, vector storage. It gives you all of

vector storage. It gives you all of that. Okay. Now what happens? A user

that. Okay. Now what happens? A user

gives a query. The query is how has Tesla performed in the last quarter?

Okay. Now what will happen?

This query would be converted into an embedding. It

will again be converted into vector. Now

what will happen? We have two vectors.

One is the vectors that are stored.

Other is this vector. So if we want to see what are the vectors which are similar to this vector which is where this information is stored. Right? Where

is that let's say what where is that I would say wardrobe or where is that place where where in my kitchen the sugar might be stored. Okay. So it is

going to go ahead and understand this embedding and then you are going to search for things which are similar to this embedding in this database.

Okay. And you are going to pick maybe 1 2 3 5 10 things based on your algorithm based on your needs which are most similar to this particular vector. So

you will have things which are most similar to this particular vector. Then

what you will do is you will have some information. The information is you will

information. The information is you will go ahead and get this information and then this will be your final prompt. So

now you have taken the vector you have found what are other similar vectors.

Now you have created a list of the vectors which you have found. Now

finally you will get this.

Now finally you will have this. So you

will give this prompt to the large language model. Just a moment.

language model. Just a moment.

Yes, you will give this prompt to the large language model.

Please observe this very carefully.

Answer the question using the context below. This is the context. So, chunk

below. This is the context. So, chunk

one, chunk two, whatever has been retrieved from there. Whatever

information, okay, that information in the text form, it will be written there.

Plus this is the question from the user which is the query which is this how has Tesla performed in the last quarter right and then eventually you are going

to go ahead and get give this prompt to the large language model cloud Germany or anything and then you are going to go ahead and get your answer.

This is rag in a nutshell. So we first create a vector knowledge base then we get a query. We match that query for all the similar terms in the vector

database. Then we get all the

database. Then we get all the information. We create a prompt. Then we

information. We create a prompt. Then we

send it to the LLM and then we get the answer. So now with rag what happens is

answer. So now with rag what happens is you do not have to store all this information into your model or in the context. You can just cherrypick the

context. You can just cherrypick the information that is very relevant to that particular information the prompt and then you can get all the information at once.

Okay, everyone a quick yes or no if you are able to understand this.

Amazing. Great. Super.

Yes, people are not able to understand.

uh if you just go ahead and uh submit the feedback form in the last like in the next 10 minutes I will give you the recording so that you can go ahead and do a revision. Cool.

Yes.

Cool. Uh guys, tokens and embeddings are different. Embedding is actually a

different. Embedding is actually a meaningful representation of that particular sentence. Okay. But token is

particular sentence. Okay. But token is just let's say a vocabulary. Okay. It is

just a numerical representation of particular word. Okay. Now this is rag.

particular word. Okay. Now this is rag.

Now in order to help you understand better I have created a small app that I have white coded very recently. Let me

show it to you.

Yes. So this is a simple rag process visualizer which I created in just 10 minutes using cursor and chat GPD. Okay. And in the program that we offer at LOPM, we are

going to teach you how to do all of this. Okay. So let me show you what

this. Okay. So let me show you what happens in that. So I've told you that in DAG, what happens? You have a piece of information. You first convert it

of information. You first convert it into chunks, right? After that you create embeddings.

After that you convert it into and save it to vector.

And once the vector is created, you are going to write a query, you are going to write a query and then from that query, you are going to match what are the matching chunks, how many

chunks you want to match and then you will go ahead and compose the query as in the prompt is going to be composed and then you will generate the answer.

Okay. So let us see how it works. So I

will take this thing. So I have written like a this log a very long time ago. So

let me go ahead and do this.

So I have taken this piece of information. Let's say till this I can

information. Let's say till this I can take any kind of information.

I will go to this. I will paste this content.

Now the chunk size is 350 which is it is going to take 350 characters in a particular chunk and the overlap of 80 which is the there would be overlap of

80 characters between the first and the second and the third chunk. Okay. So I

will just go ahead and create chunk document.

Now look at this chunks are created.

This is the first chunk. This is the second chunk. This is the third chunk.

second chunk. This is the third chunk.

This is the four chunks. Understand this

chunk was removed here. So it is in this detail 4,000 words guide setting the context how to build then rather than how. Okay. So now there is overlap in

how. Okay. So now there is overlap in this chunk. So 50 characters are over

this chunk. So 50 characters are over overlapped so that all the chunks have a meaningful information. So now we have

meaningful information. So now we have created the chunks.

Okay. So what is chunking? Breaking down

information into smaller parts so that they are meaningful and smaller enough.

Okay. Understand? You cannot use the same chunking method methodology for every application. If you're using rag

every application. If you're using rag for code in order to retrieve code and in order to generate code the chunking has to be done mostly at the functional level. If you are creating for a book

level. If you are creating for a book then maybe it can be done at a chapter level. If you're doing it for a meeting

level. If you're doing it for a meeting it can be done at a maybe uh at a paragraph level. So chunking depends

paragraph level. So chunking depends upon the product that you are creating and tomorrow I'm going to tell you an evaluation that is how to decide your chunking strategy. Okay. So this is

chunking strategy. Okay. So this is chunk. Now I will create embeddings.

chunk. Now I will create embeddings.

I'll create on embeddings and then the embeddings are created.

Okay. So if I look at this these are the embeddings guys.

These are the embeddings. Okay. Okay. So

for every chunk I have created vector embeddings. Okay. And this graph that

embeddings. Okay. And this graph that you are able to see let me zoom it up a bit.

So this graph that you are able to see is a representation of the vector. Now I

will press the next button to save to vector DB and then it will be saved locally. Okay. This is not

saved locally. Okay. This is not connected to any server. So this is going to be saved locally. And after

this, yes. And after this

yes. And after this I will write a query. The query is what is AI product

management.

Okay. And I want to retrieve three chunks. Retrieve relevant chunks. And

chunks. Retrieve relevant chunks. And

then these are the chunks.

So these chunks because there is product management and all it is is being accessed here. So these are the most

accessed here. So these are the most relevant piece of information that I need to store here. So this is chunk number eight, chunk number two, chunk number three. If I click on this, I'll

number three. If I click on this, I'll be able to see the complete chunk. Okay,

this is the cosine which is how distant these things are from each other and from my vector.

Okay. And then eventually what I'll do is I'll just create on compost query.

Now the query is composed.

And this is the query. Answer the

following query based on only the provided data. User query. What is data

provided data. User query. What is data product management? What is I product

product management? What is I product management? And these are

management? And these are these are the chunks three chunks.

Okay. And then when I click on this, this will be sent to a large language model and then I'll get the answer. But

I have not connected the LM because I don't want to pay. And then we have this answer. Okay. But I can always go ahead

answer. Okay. But I can always go ahead and put my open AI key here API key in order to do this. Then the answer would be generated.

Okay. Everyone tell me on a scale of 1 to five if you're able to understand rag now.

Cool.

Wonderful.

People who are four, I would recommend you to watch the recording once more and then you'll be able to understand.

Yes.

Cool. So, this was about rag. Super

important concept everyone needs to understand. It will also improve your

understand. It will also improve your problem solving capabilities. Now, the

next part is prompt engineering. Okay. So guys, I'll

prompt engineering. Okay. So guys, I'll only take 10 more minutes and then I will share the feedback form. So if you have been here in this session for so long, please remain for another 10

minutes so that we can share the feedback form and you can learn the prompt engineering as well. Okay.

Yes.

Cool. So now we'll talk about prompting.

So understand guys, you need to understand how to give instructions to your model.

In order to give the right data, you have understood you have to use rag.

But in order to give instructions, you need to understand that you need to learn prompt engineering. Prompt

engineering, no matter what the world says, it is a super important skill in today's world. Okay? because you are

today's world. Okay? because you are using let's say AI and LLMs for so much of your time in a day. It is worth investing a couple of hours in order to

learn prompt engineering and also if you are working as a product manager as an AI product manager prompt engineering is your way to interact with the LLMs.

Okay. So it is worth investing few hours

Okay. So it is worth investing few hours of your time in order to build great products. Okay. So this is also a guide

products. Okay. So this is also a guide on prompt engineering. So understand

this products such as Fireflaws, Firefly, Gong, Granola which are almost a billion dollar companies or at least multi-million dollar companies. They are

based on these kind of prompts. So you

should know how to learn these prompts and I'm going to go ahead and give you some I'll also share this guide with you. But right now let me share with you

you. But right now let me share with you few tips to improve the prompt.

Okay. So right now everyone tell me out of 1 to5 how would you rate yourself on your prompt engineering skills? How would you rate yourself on a

skills? How would you rate yourself on a scale of 1 to five on your prompt engineering skills?

Three. Don't give yourself three. Three

is a very diplomatic answer.

Yes.

Many modest answers. Yes. any

reflective answers. Okay. So, first of all is first is please take a step back.

So, I'll not give you a very uh exhaustive guide that you're not able to even read.

I'll give you a very simple guide and this guide follows the parto principle.

If you invest what is like if you learn these 20% of the skills as a prompt engineer 80% of your efforts like problems are going to get solved. The

first piece of advice is please take a step back. Okay, understand the value of

step back. Okay, understand the value of prompts.

What happens is the unfortunate reality is that when you go to chat GPT or Gemini when you give it an very awful

prompt, it will still be able to give you an answer most of the time because it has given you an answer. Okay, answer

you go ahead and move forward from there and then you start working on whatever you are working on. Okay. But understand

always remember in many phrases in many places in your life good

can be the enemy of best.

Okay. In many places in your life good can be the enemy of best. which means you have to

of best. which means you have to understand and recognize what is the value of this task and is it worth investing a couple of more minutes or five seven more minutes in order to

write this prompt well. Okay. So the

first advice that I have for you is please take a step back and understand if it is worth investing time in that particular prompt if it is giving you that many results. Okay. The second part

is right role. Whenever you go ahead and start using an NLM please try to give it a role. role means try to say it, try to

a role. role means try to say it, try to tell it that act as a person with 160 plus IQ or act as a seasoned product

manager or act as someone who is a seasoned psychological therapist. Right? So give

psychological therapist. Right? So give

these personas to the LLM so that it is able to perform well. If you want to experiment, you can share two prompts with chat GPT or china or anything. In

one give a very like give it a very solid persona in one don't give it a persona. You'll be able to observe the

persona. You'll be able to observe the answers.

Right? So the first piece of advice is please give it a right. The second piece of advice is please write the output expectation. Output expectation as in

expectation. Output expectation as in you want JSON as an output. You want a table. You want to make sure that these

table. You want to make sure that these mistakes are not repeated. Please write

it explicitly. You have to be very very specific. If you're not specific, it is

specific. If you're not specific, it is going to do anything and then you cannot go ahead and control it. Right? The

third part is multistep. Multistep means

rather than telling your large language model to give you answer at once, ask it to think in a step and give you a step-by-st answer. If you know the step,

step-by-st answer. If you know the step, you can tell it the step. If you do not know the step, go ahead and ask it to tell you the step. For example, rather than asking chat GPD that how would I

improve this product LinkedIn, I would ask it that I want to improve LinkedIn.

Please follow these steps in order to suggest me the improvements. First

understand who are the different users.

Then understand their important problems. Then give me what are the uh what are the top priorized three problems. Then give me the solution of these problems. Then prioritize these problems and prioritize the solution and

then give me a particular solution. Now

when you are focus like forcing chat GPD to work in this manner you are more probably like to get a better answer.

Okay. So please do the thinking. Let it

do the thinking in multiple steps. Okay.

After this give examples. This is also called as few short prompting.

Few short prompting means you are giving it few examples.

You are giving it few examples that this is the format that you should follow or these are the examples that make sure that you are able to go ahead and get the right content. Right? And then we

have constraints. So in example what you

have constraints. So in example what you can do is whenever you are writing any piece of blog or any piece of article that you want to learn from

go ahead copy the best author that you know copy their blog and then give it to Chad Gigi. Okay. For example, what I do

Chad Gigi. Okay. For example, what I do is whenever I write some piece of content because I have put a lot of efforts in writing this particular guide, I copy this guide and I go to

chat GPT and I ask that go ahead. So let

me show it to you.

So this is Chad GPT. I am asking Chad GPT to write a detailed article on AI

on mistakes that people make while learning

AI product management.

Right now we will see the output.

Yes. So you can see that it is writing.

It is doing like a pretty interesting job right?

In the meanwhile, I'll also open a new chat.

Right? So it has written this. You can

see that it is uh well created. This is

point number one. Point number two, point number three, four, this, this, this, this, this. Right? I'm still

writing. Right? This is good one. Right?

Now, what I'll do is I will copy this article.

I will copy this complete article.

Maybe a Ctrl C.

Right. So, okay.

First I'll go to this then I will write. So I've written the sim similar prompt. Now I'm writing

make sure to follow the similar structure voice and helpfulness.

from this example.

Okay. And I can go ahead and either I can go ahead and uh give it the link. It

will be able to crawl the link or I can just go ahead to be assured I can go ahead and give it this content.

Okay.

So now I have given an example. Let us

see how it works.

Yes. Now you look at this. Okay. Rather

than straight away jumping to what you should do, what you should not do, rather than straight away jumping to actionable, it is first setting the context. Why do you need this? This is a

context. Why do you need this? This is a quick context and then it is able to give you multiple mistakes and then how you can

go ahead and avoid these mistakes.

Right? This is a slightly better structure than you could have got in the first way. So this is called as few

first way. So this is called as few short prompting where you give examples to your large language model as in how they should operate.

Right? And

the other part is the similar example which is constraints.

Please give constraints. Constraint as

in be helpful like this. Okay. Or

constraint as in if you want constraint in terms of what do we call it? If you want large language model to not do something then

you should always include them as explicit constraints. Okay. For example,

explicit constraints. Okay. For example,

sometimes large language models assume a lot. Large language models actually

lot. Large language models actually assume a lot.

Large language models assume a lot. So

you can go ahead and tell them that please do not assume only take check the information that is available in this context that I have shared with you.

Similarly, you'll understand more about constant when we talk about iteration.

So understand prompt engineering is called engineering. Prompt engineering

called engineering. Prompt engineering is called engineering because it is iterations which is you tweak your way around. You

give it a prompt. You understand that I got the output but the output does not follow these rules or the output is not good in this particular way. So you are going to understand the output and then

you are going to give it the constraint that boss make sure that you are not taking unnecessary assumptions. Make

sure that you are not using examples which are let's say very much vague or generic. Use specific examples. Okay. So

generic. Use specific examples. Okay. So

when you go ahead and apply all of these steps and when you keep on iterating on top of all of these six steps you are going to learn how to design better

products.

Okay. Now even after I give you all of this I know that most of the people are lazy and they are not going to follow this. So now I'm going to give you a

this. So now I'm going to give you a prompt guide for busy people. Okay. So

four things that everyone should learn and then we are going to wrap up the session. First is please build a prompt

session. First is please build a prompt directory. Whenever you come across good

directory. Whenever you come across good prompts have a notion document or a Google doc where you are storing all of these things. Okay? So that you don't

these things. Okay? So that you don't have to find your way through good prompts every time. Right? Second is

quick shortcuts. Okay? Keep a bookmark where you are able to access this library as soon as possible. So what I do is whenever I want to write something I do not want to find a writing pad.

What I do is I just click on this new tab and then I have all of my notes from notion available here.

Okay, Chrome new tab is the name of the extension and then it is going to load here. Right now I think notion is down

here. Right now I think notion is down so it is not loading but now it has loaded. Okay, so all of my notes and

loaded. Okay, so all of my notes and everything is here. So what I've done is I have set up a Chrome extension where whenever I click on this new tab button it will automatically load my dashboard.

so that I don't have to run here and there. I can just go ahead and click on

there. I can just go ahead and click on this and then I can start writing my notes and my prompt directory is also there. Right?

there. Right?

The other piece of advice is use a prompt manager. Okay? So there are

prompt manager. Okay? So there are multiple prompt managers. You can use any one of them. And you can also use this prompt wild card. Okay. If you are too lazy to write your own prompt, you

can also get this wild card which is analyze the following prompt idea.

Analyze the following prompt idea.

Insert prompt here. Rewrite the prompt for clarity and effectiveness. Identify

potential improvements or additions.

Refine the prompt based on identified improvements. Present the final

improvements. Present the final optimizing prompt. Okay. And also there

optimizing prompt. Okay. And also there is a prompt creator from anthropic. Okay. So there

is a prompt creator called as anropic prompt creator. Okay, where you can go ahead

creator. Okay, where you can go ahead and create prompts. So it would be somewhere or have to login in order to see this. But you should be able to go

see this. But you should be able to go ahead and generate this from cloud console.

Okay. Once you log in, it will give you a prompt uh creator so that you can go ahead and create your own prompts which are much better than what you would have gone ahead and written by yourself.

Okay. So this is about the guide to prompt engineering for busy people. Now in tomorrow's session what

people. Now in tomorrow's session what we'll do is we will talk about notebook LM how does it works. We will talk take the similar kind of case studies. We'll

detail it enough. Then we'll talk about fine-tuning. Then we'll talk about how

fine-tuning. Then we'll talk about how to choose between fine-tuning, prompt engineering and rag. And then we'll talk about AI agents. Then I'll give you how products such as lovable, bolt,

emergent, and visero works. And then we are going to go ahead and talk about MCPS. And then eventually I'll give you

MCPS. And then eventually I'll give you some examples of how to write AI evaluations. Okay, this is going to be

evaluations. Okay, this is going to be tomorrow. But guys understand whatever

tomorrow. But guys understand whatever I'm telling you in these two sessions this is just 10% of what we offer in our bigger program at hello. Okay. So at

hello we have created the world's most detailed AI product management program.

Okay. If you want to understand anything about the program just go to this website called hellopm.co click on this video and then I'll explain you what that content is. Okay.

This is the most detailed, the most practical and the most long-term support AI product management program that you can find. The content that I've told you

can find. The content that I've told you today is just 10% of what we teach in the program just in the first two classes. We will make sure that tomorrow

classes. We will make sure that tomorrow you'll get a much more power pack session. So thank you everyone. Bye and

session. So thank you everyone. Bye and

take care. If you have any queries, you can use the same form to publish these queries. Tomorrow we are going to meet

queries. Tomorrow we are going to meet at the same time. Make sure that you also go ahead and bring your friends.

Thank you everyone. Bye and take care.

You have been a great audience. I hope

you got some value from this session and the two hours that you have invested.

We'll meet everyone tomorrow. Thank you

everyone. Take care. Today is the day two of AI product management masterass and we are going to give you almost everything that you need to get started

in your journey as an AI product manager. This is the foundation. Now you

manager. This is the foundation. Now you

can go ahead and explore the whole canvas in order to go into detail. But

these are the major building blocks that you need to absolutely know if you really want to call yourself as an AI product management enthusiast. Right?

And because AI product management is so new, even if you are coming from a very senior background or you are just aspiring to be a product manager just straight out straight out of college, this content is going to be equally

useful for you. So yesterday what we have done, we have talked about the agenda. I talked to you about what is

agenda. I talked to you about what is the GI revolution. lot of companies are building value on the top of this geni revolution then even in our normal lives

AI has found its way right yes cool I will stop the chat for some time so

that you guys are not getting distracted yes cool so AI has also found the way onto our daily lives and almost every

day we are using chat GPT or Gemini or some of the other AI tools tools in order to make ourselves more productive and grow the impact of our work. Right?

And then I talked about the most fundamental unit why this AI boom has started which is because of a particular technology called large language models.

These large language models, these large language models are algorithms that are trained on massive amount of text and they have learned the patterns of

language really well and they are now acting as the brain of the computers.

And now what is happening? They are

enabling us to understand, generate and reason with natural language by predicting the next token most likely or the token in the context. And I have told you yesterday how all of these things actually work. I have given you

the whole architecture of what an LLM is. The content is like you will go

is. The content is like you will go ahead and take all the content as a training data from the internet. You

will break it down into input and output. Then you are going to tokenize

output. Then you are going to tokenize it. You are going to send the input from

it. You are going to send the input from here. The neural network or the

here. The neural network or the transformer is going to predict the next token. And then whatever is the right

token. And then whatever is the right token from the uh training data, you are going to compare it. Then whatever is the difference, the feedback is going to be sent to the transformer again. And

then it will keep on iterating. When

this cycle happens billions and trillions of time, you will have a very mature model, right? But this model is only a base model. It is trained on some

random data from the internet. So it it has certain kind of biases. It

hallucinates a lot. So what we do is we go ahead and do something called as post training. Questioning means we are going

training. Questioning means we are going to have some human labelers or we are going to have some data analytors who are going to give some specific input

and output maybe in this particular format.

Wait.

Yes. Maybe in this particular format where we are going to insert this kind of data like well articulated inputs and well articulated outputs so that the human

can the uh uh algorithm can go ahead and learn from the same. Okay. This is

called a supervised learning. And one

part of that learning is also called reinforcement learning through human feedback which is humans are telling how what is correct and what is incorrect.

Right? Humans are reinforcing the correct things by giving the rewards to the LLM. Now after that

the LLM. Now after that we talk about the genai value stack. So

there is multiple kind of values. For

example there are companies who are creating infrastructure such as these big Nvidia and Google. Then there are company who are creating the models. You

know about anthropic and open AI. Then

there are companies who are building on top of these large language models. This

is where most of the value unlock is.

And then we have companies and people who are actually using these AI tools in order to make their clients happy, solve their problems and also solve their own

problems. Right? And then I discussed

problems. Right? And then I discussed about type of AI PMs. Everyone who is using AI is AI enabled PM AI tools such as chat GPD and all. Anyone who is using AI tools is a AI enabled PM. And then we

have other type of PM which are more serious PMs. These are PMs who are either working on core technologies such as databases, models, infrastructure and then there are people who are working on the application layer and these are

people who are actually using these LLMs and models and everything in order to build some truly AI products. Agents are

one of the examples. Chat GP is one of the examples. Grammarly is one of the

the examples. Grammarly is one of the example. Lovable is one of the examples.

example. Lovable is one of the examples.

So if you are building these products not building with these product you are building these products then you are an applied AIPM and the purpose of this session is to help you understand the core AIPM part and the applied AIPM

part. Okay, for this part we have

part. Okay, for this part we have already uploaded a lot of content on our YouTube channel that you can see right then

I talked about how this application these meeting assistant such as granola works okay and not only granola but if you take any of the meeting assistant such as fireflies

fed them or anything everyone works in a similar format the format is whenever you want to create a product as a product manager manager understand please do not look

only from the engineering or the AI angle. You are the product manager who

angle. You are the product manager who is responsible to bring the user outcomes and the business outcomes. So

please look at these five things. The

first thing is what is the value that is being generated for the users? Why would

the users use it? Second is what is the business value that is generated. Then

there is how do you make design so intuitive that people are able to use it easily and then how it is feasible with the help of engineering. So all the system architecture is going to come

here and then growth and adoption. In

granular we knew that this product helps us save time by recording our meetings and sending us the summary. How does it helps a business? The business should

invest in granular in creating granular because there is a large opportunity in meeting apps. So it is going to make

meeting apps. So it is going to make money. Similarly, how do we make this

money. Similarly, how do we make this intuitive to the users? By making the design very intuitive. How do we make it intuitive? We make the design intuitive

intuitive? We make the design intuitive by making sure that just after logging and connecting connecting granular fireflies. You don't have to do

fireflies. You don't have to do anything. Whenever a meeting is

anything. Whenever a meeting is scheduled, it is automatically going to join that meeting. It is automatically going to generate the summary. It is

automatically going to send the summary to everyone who was invited to that meeting. Right? With just few clicks,

meeting. Right? With just few clicks, you should be able to do so without your involvement. So that is the

involvement. So that is the intuitiveness. Similarly, how will this

intuitiveness. Similarly, how will this be engineered? So I have told you about

be engineered? So I have told you about this.

So let's say Zoom and Google meet or any kind of platform is the place where you are conducting this meeting. Then Fireflies or granola

this meeting. Then Fireflies or granola is going to either use the API for Zoom or Google or they are going to do the

audio capture through a bot. Okay, like

you can see that there are a lot of bots in this meeting who are actually recording the meeting on the behalf of their administrators. Right? After that

their administrators. Right? After that

that audio is captured and then there are tools called as audio to text transcription machine learning models.

Right? These models such as whisper by open AAI, speech to text by Google, they are going to convert the audio into text

transcription right and then you are going to attach a system prompt. So you

as a product manager of granular is going to add a system prompt and then you are going to add a transcript. Let

me show you that system prompt that I have shown you yesterday.

So this is the prompt that you will paste. The prompt is very simple. You

paste. The prompt is very simple. You

are an expert chief of staff and note taker given the following things. Please

go ahead and summarize the summary of the please go ahead and summarize the given transcript in this particular format.

Right? So now what will happen? We have

this sub we have the system prompt and then the system prompt is going to be combined with the transcript and then it will be sent to large language models

such as Gemini or Google Gemini or open uh open AI's APIs right understand this is happening at the back end you do not know what is happening you are just using granular at the back end granular is doing all of this and then it will

generate the summary actionables and the task and you can take a next step you can make it into an agent by actually sending the tasks on a

calendar on your Jira or on your slack.

So this is how any application which is a note takingaking assistant can work right even if you go ahead and understand a bit of by coding you can also go ahead and create your own

personal AI meeting assistant. This is

the simplest architecture that teams use, right? And after that we talk about

use, right? And after that we talk about a very important part that yes in this example we were able to work with prompt engineering but how do we go ahead and

build applications which are able to work on the large set of data such as data which is on your Jira dashboard, data which is on notion

AI, data which is on stripe.

So what we do is we cannot let's say I have a notion database I want to ask some questions on the top of the database. So I cannot just go ahead and

database. So I cannot just go ahead and put all of these content in the text in the context. Why? Because the LLM is

the context. Why? Because the LLM is going to go out of context and then there are researches that are conducted which have proven that with increasing

context the accuracy of the LLM incre decreases. With increasing context, the

decreases. With increasing context, the accuracy of the LLM decreases. And after

this, we talked about the simple thing that if you have if you are working with LLMs, try to make your inputs concise otherwise they can go ahead and hallucinate too much and they are going

to run out of context and give you irrelevant answers. And then we

irrelevant answers. And then we presented you to con context engineering. In context engineering, we

engineering. In context engineering, we first talk about rag. So what is rag?

Super simple.

What we do is rather than putting all the documents let's say I want to build a functionality where on my notion document or on my Google drive I ask a

question that what is the main what are the tasks that I need to do tomorrow okay and somewhere in one of the PRDs one of the documents it is

written that I have to do these tasks tomorrow okay so in a non ideal scenario what will happen I will try to get all the content of the Google Drive I will put it in the LLM context of the LLM and

then I'll ask the question among all of this content what is something that I need to do tomorrow okay and then what will happen because it is lot of content the LM the LLM is going to run out of

context so what we'll do is rather than going ahead and taking all the data at once to the LLM we are going to only find the content that is relevant right

so what we do in the what we do here We simply go ahead.

We simply go ahead first whatever data that we have we break that data. We

break that data into chunks.

And after chunking the data into small enough and meaningful chunks, we are going to convert it into vector embeddings. And after we have converted

embeddings. And after we have converted into vector embeddings, we are going to store in a vector storage. Once the

vectors are stored, we will go ahead and get the user prompt, find all the relevant content from the vector storage, then convert it into text and then ask this question to the LLM that

based on these information please try to answer this question and then it will give you answer answer the question from the limited context that you have given.

Right? And I have also shown you a visual example with a rag model that we have created like a rag vibe app that we have created.

Right? So this is about rank and then we talked about prompt engineering. Okay, I have shared

prompt engineering. Okay, I have shared with you some very simple but very effective tips which is try to understand the value of prompt engineering. Make sure that you are

engineering. Make sure that you are giving it the right role so that it is more authoritative and you are getting the right answers. Then mention the output expectation very clearly. give it

do something called as chain prompting where you are going to mention ask the LLM to think in steps so that it is able to go ahead and give you what is the reason behind this thinking and then give examples this is called as few

short prompting when you don't give example it is zero short prompting when you give example it is few short prompting right and then mention the constraints understand all of this is called as iterations because once you

give it a prompt you will understand that maybe there is something missing I should have mentioned this clearly and then you are going to iterate on the prompt and then you'll understand what are the constraints that you need to put okay and then I have shared with you

some advices and some wild card prompts so that you can go ahead and get better at prompt engineering and writing better prompts right understand prompts are your doorway to seize this AI

opportunity right it is not just plain English you need to go ahead and think deeper about the prompts so as to get the better answers right you have a very

o uh a very obedient and a very powerful individual at your disposal. If you ask it like uh awful questions, you are going to get awful answers. Now it's

your responsibility to make sure that you're asking it better kind of questions. Right? Yes. So this is what

questions. Right? Yes. So this is what we have done so far. I think we have spent 15 minutes just in the revision.

Now everyone please tell me go ahead in the chat if you have understood everything so far. How many of you are aware about this product called as notebook?

So in case someone has not used I'll just give you a glimpse. Okay. So let's

say I want to learn about AI product management. Okay. This is the topic for

management. Okay. This is the topic for today.

But sadly I do not have time. I do not want to invest time. So what I'll do is I'll go to this platform notebook.google.com.

notebook.google.com.

Okay. I will click on new notebook.

Right? And now what I can do is I can give it lot of documents, lot of resources and then I can ask questions and summarize these documents from this

interface right on chat GBT when you give so many document documents it runs out of context but it is this particular platform notebook is working on rag that I have just told you so it will be able

to answer let me show you how it works okay so let's say I want to understand about a IML product management so I'm going to insert some resources okay so the resources At hello PM we have gone ahead and done

something called as the AI sprint which has everything that you need to understand as an AI product manager. So

I will click on this.

Yeah. So I'll click on my channel and then I'll click on playlist and then I'll click on the AI sprint >> and then >> okay so this is one video. So what I'll

do is I'll copy the URL.

I will go to notebook. I am now collecting the resources. Okay. So my

resource are these URLs. So I'll click on YouTube. I will paste the URL here.

on YouTube. I will paste the URL here.

Insert. So this would be inserted in some time. Second. So now what is

some time. Second. So now what is happening the Google uh uh Google notebook alm is actually taking the transcript of the video it is breaking

the transcript into small small parts and then it is vectorizing all of these transcripts okay all of the contents out there. Second,

there. Second, I will take maybe uh uh the second video.

I will put it again in the YouTube part.

Insert. Now I want a blog. So a IML this. So I've written a blog as well. So

this. So I've written a blog as well. So

I will go ahead and just add it here.

And then I would put a link of website.

I can put multiple URLs as well. Okay.

So now I have all of this context ready.

These are long 1 and a half hour 2 hours videos and this is a very long I would say 3 4,000 words of vlogs. Okay. So now

you understand that yes I have all the context available. I can go ahead and

context available. I can go ahead and ask it any kind of question and then these questions would be answered from the data that I have given. Right? So I

will ask a question how do I what skill does a

AI product manager needs.

Now this answer would be grounded in what is written in these sessions.

Right? So if you want to do any research of the world you can just catch up some YouTube videos, some interesting podcasts from the founders and industry leaders. put up some blogs, put up some

leaders. put up some blogs, put up some reports in PDFs from Mackenzie and other companies and then you should be able to do this. Okay, now it has given me the

do this. Okay, now it has given me the answer. But this is not the beauty. This

answer. But this is not the beauty. This

is okay. This is cool. But it also has some wonderful features which is it can help me create mind map. Right? So if I click on this, it will create a beautiful mind map from all the

concepts. Okay.

concepts. Okay.

Another important part is reports. So if

I click on the reports, these are all the report that it will create. It will create a complete blog

create. It will create a complete blog post which I can read. It will create a study guide so that it will be having a short answer, quizzes, some uh revision material, everything. Then a briefing

material, everything. Then a briefing doc so that I can go ahead and just understand what is important in these particular three four resources. And

then there are multiple things. Okay,

concept explainer, process overview and everything. So it is now able to

everything. So it is now able to understand the content and based on content it is suggesting me that this kind of reports are possible from this

and I can also give my own report right.

So let's say I want to get a briefing.

So now it will take some time in order to generate this particular report.

Okay. And then I can also go ahead and create mindm flashcards quiz and everything. And also eventually I can

everything. And also eventually I can also create an audio overview. Okay. So

this is what this product does. This is

almost a number one research tool based on AI which almost every product manager and every individual should use. Okay.

So this is notebook LM for you. There

are so many use cases. You just think about it and notebook LM should be able to do these things. Right. So my

recommendation to everyone would be that after this session go ahead and try to play around with notebook. Super helpful

tool. Right? Yes. Everyone a quick yes or no if you're able to understand this tool.

Amazing. Thank you. Yes. So now

let us try to understand and also what I'm telling you is not just how notebook LM works but also any of the tools any of the AI tools which are

based on documents. So in notebook we are giving it a lot of documents and transcripts and it is working on the same. On notion AI we are actually

same. On notion AI we are actually giving it all of our databases the notion document that we have created it is working on the same in Salesforce Einstein we have all the document that we have created in our salesforce all

the sales record all the CRM all the emails all the customer conversations and then there is an LLM on top of it.

So the mental model is that if you have a lot of data and if you want to put a LLM on top of it in order to make it more queryable with the help of a natural language then this is what you should

do. This is the same architecture for

do. This is the same architecture for all of these products. Okay. Yeah. So

what we do is we follow the same format. Question

number one is what problem does it solve in people lives? This is something that you need

lives? This is something that you need to answer as a product manager before even building the product. So why does notebook help why is notebook helping helpful? Because it helps people conduct

helpful? Because it helps people conduct research by helping them read so many documents and summarize these documents at once.

It helps people read and learn about exhaustive full topics within a lot less period of time.

It also helps people create some effective nodes for their learning, for their research, for building any product or for doing anything. Okay. So, there

is definitely a value in the product, right? So, it will help people become

right? So, it will help people become more productive and take better decisions in shorter period of time, right? What is the business value? Now

right? What is the business value? Now

Google has understood that people are using chat GPD a lot and Google has understood that now if people keep on using chat GP and perplexity the research that people used to do on

Google is now going to get reduced. So

they have to create a product that will make sure that they are getting their lost foothold. And now notebook alum has

lost foothold. And now notebook alum has become popular that people are have started using it. So it is a strategic bet for Google to create this kind of product. Right. Third is how do we make

product. Right. Third is how do we make it intuitive to use? So Google has made it very simple. So you can see the UI.

It is very simple. It tells you. So

Google might have thought that why not we go ahead and allow people to just enter content. Okay. But content could

enter content. Okay. But content could also be in the form of YouTube. So they

explicitly mentioned that yes people want to learn from YouTube so you can enter YouTube. People want to upload

enter YouTube. People want to upload their PDFs. So they are allowing you to

their PDFs. So they are allowing you to upload the PDFs. So they understood that from what kind of resources people would want to ideally get the information and they have enlisted all these resources.

Similarly they were able to understand that most of the people are very lazy in terms of prompt engineering. So they are giving you some automatic templates of reports. So rather than you thinking

reports. So rather than you thinking that what is the question that I asked from this document they will give you automated question in the form of report that I have just shown you.

Right? So this is how you make the product more intuitive. Right? And if

you guys want to learn how to make the product more intuitive, a simple exercise is go ahead look at 10 top AI products and then reverse engineer what are the things that they are doing in

order to make it more intuitive. You can

follow the similar kind of principles.

You will be ahead of 80% people who do not know anything about UI, right? And

then we have how will this be engineered, right? How will this be

engineered, right? How will this be engineered? This is the structure.

engineered? This is the structure.

Please look at this very very carefully.

Okay.

Yes, we have 510 people in the chat.

Everyone please write focus if you are ready to go through this. Please write

focus if you are ready to go through this.

Amazing. Great. Cool. So focus here everyone. So here what is happening is

everyone. So here what is happening is first is the resources.

Resources are in the format of articles, videos, PDFs and text which is videos it will not be able to understand. So what

it does is it takes a YouTube videos and YouTube videos comes with a transcript.

So it is understanding the transcript and not the video. Okay. Second is PDFs to convert PDF into text.

We can either if the PDF is created digitally, we can go ahead and convert it into text like through some PDF extractors. However, if it is an image

extractors. However, if it is an image that we have taken photos of and then we have converted the PDF, then we have something called as OCR or optical image recognition, character recognition,

right? So these things would be creative

right? So these things would be creative to understand it is not just that it building an AI product is only about implementing the LLM. You have to do this dirty work as well which is making

sure that all the content is in the right format which is text right after this after this

and even if let's say we have audio right now I don't think it allows audio but if it allows audio what it will do is it will get the audio from you it will give it to the back end like from

the back end it will give it to Google speechtoext API or whisper API and then it is going to get the transcript and then the transcript is going to be added here. Okay. So as soon you add the

here. Okay. So as soon you add the sources the sources are extracted they are chuned and then they are chuned at two levels document

level and the paragraph level. Right? So

it tells you that from which document and which part of document is this content coming from. So whenever you are storing these chunks there would be a link to that document whole document like the source and there will also be a

link to that particular part of the document which is the chunk right I have shown you this example yesterday with that thing right after this

we have embedding so now this will be stored in Google's own database for embedding and understand whenever you change the content whenever you update the content this is going to get refreshed

right till Here we have not used the genai so far. We are only using engineering computer science engineering so far and some machine learning

algorithms. Right now what happens your platform notebook is going to understand what this content

is all about. So it will try to go ahead and create highle summaries of all the documents that you have created like highle summary for every document automatically and then it will try to understand what are the different kind

of reports that I can generate and for generating these reports it would have some kind of templates already. So let

me show you so this is what Google might have created at the back end. Okay. So let's

say if we think that there there could be a summary that we can create. So it will go ahead and give this kind of prompt which is summarize the key ideas in the

plain English focus on main findings and implications. if they need to create a

implications. if they need to create a study guide. So here what is happening

study guide. So here what is happening is we have these report prompts right it will show you that these are the things that you can do with this report and

then the user input whatever the user has inputed the prompt and the retrieved context. Retrieve context means whatever

context. Retrieve context means whatever is matching with that prompt. Whatever

the words that you have mentioned in that prompt, whatever is matching in the vector database that is going to be fetched and then that is be that will be sent to the LLM. If it is Google, the

LLM is most probably going to be Gemini.

So this is the prompt library. Okay. So

if the summary type the report type is summary, then this is the prompt. If it

is a study guide, then generate a structured outline with key component, definitions, and questions for self- testing.

Right? Mind map. Organize the main ideas hierarchically with branches for non for subconcepts and examples.

Right?

And then reports would be generated and then reports could be summary, mind web, guides, FAQs and then what will happen?

You as a user will give you the feed will give the feedback that it is good or it is bad and then it is going to go ahead and autocorrect these prompts and you can also go ahead and put your own

prompts. So what is happening this is

prompts. So what is happening this is augmenting this part is retrieval and this part is generation.

Right? So we are first creating the documents then we are generating then we are uh going ahead and giving it to the LLM and then we are generating and going giving the output to the user right so

this is how notebook LM works in a nutshell it is a pure play rag application right everyone a quick yes or no if you are able to understand

H now one question to everyone question to everyone okay let's say if I ask the question to notebook alm

that summarize all the contents summarize everything that you are like from all the resources summarize Now what do you think will happen? Think

about it for a moment and tell me in the chat what will happen then? Let's say if I give it a query, give me a summary of this content. What will happen

this content. What will happen now? So summarization. Yes.

now? So summarization. Yes.

Cool. So understand summarization is a non-typical task. Why? Because

non-typical task. Why? Because

summarization is the most I would say popular task but it is a non-trivial task. Because in the prompt what would be mentioned?

Summarize this document.

When this is mentioned in the prompt when it will search in the rag model when it will search in the embedding what will it search for? Summarize

article. And now how will it find that where is summarize and all because it is doing a vector like search. So it might find some chunks where summarize brief executive summary these kind of things

are written but summary would not be accurate.

Right? So what is the solution? The

solution is we do not implement this normal rag pipeline in these kind of applications. What we do is

applications. What we do is we have something called as hierarchal hierarchal rag.

I'm not sure what the spelling is. So

hierarchal rag what it is is that I have source one, I have source two, I have source three.

Okay. So now first of all understand the problem clearly. Okay, I have created

problem clearly. Okay, I have created chunks. Let's say there are 10 chunks

chunks. Let's say there are 10 chunks that are from this document. There are

10 chunks from this document and four chunks from this document. If I ask LLM the question that please summarize this, then what will happen?

It will make the vector of summarize, right? And summarize vector might be

right? And summarize vector might be equal to brief.

Okay.

or maybe some other things like synonyms of summarize right and it will go ahead and try to find summarize brief and some things in these chunks and whatever whatever are the chunks that are relevant it will go ahead and give these

chunks let's say one chunk from here one chunk from here and one chunk from here so we have got three chunks right but understand these chunks are incomplete they have mentioned summary

or brief or in short something but they do not contain the true summary they just contain these words right. So what we will do here is we

right. So what we will do here is we maintain a summary we maintain a summary at a high level.

So for all of these documents as soon as you upload this document in notebook LM it is first going to generate the summary of the whole document. So it

will send it to the LLM. It will

generate the complete summary so that it understands what the report is. Then for the other it is going to

is. Then for the other it is going to generate the summary. For other it is going to generate the summary without you saying anything. And then whenever you send a query of summary it will understand what the equation is. And

then it is going to answer by concatenating or copying or taking all of these summaries into the context.

That way context is also not increased but there would be some more LLM calls.

Right? So how we have solved this problem? We have taken all the sources

problem? We have taken all the sources automatically without even the user saying so we have taken the summaries of every individual source.

Right? And once we got the summary from the LLM now whenever a user is asking question about summary we are going to combine all of these summaries in order to get a better answer.

Right?

Yes, everyone a quick yes or no if you're able to understand this and like this guys although rag is simple okay rag is not a

rocket science but you have to understand the nuances every product is unique so you have to understand these kind of nuances before going ahead and

calling yourself an AI product manager okay so this is about how not works and all the rag applications works in the similar format you have the knowledge base, you convert the knowledge base

into the vector database. After that,

you go ahead and get the user query, you match the pattern, you get the uh uh top similar chunks and then you put the LLM on top of it along with the system

block. Right? Yes.

block. Right? Yes.

Now in the context engineering we have so far talked about rag we have talked about uh prompt engineering and the third part of context engineering is actually

fine-tuning. So sometimes what happens

fine-tuning. So sometimes what happens is sometimes what happens is you have to retrain the model you have to change the parameters in the

model for your use case. For example,

Google has this amazing model called as alpha. Okay, Google alpha and

alpha. Okay, Google alpha and Google alpha fold or something is the name alpha fold I think. Okay, so what it does is it is able to identify the

patterns in protein and then it is able to suggest that what is going to be let's say the consequences which is the diseases which are identified from protein patterns and so many things.

Okay, but understand when Chad GPT or Gemini was trained, it does not have any information about proteins and all. It

has the information about how normal natural processing works, but it does not have any understanding about how to understand the protein models. Okay. So

now what happens is we are going to take some new data which only contains information about these protein folds and then we are going to

go ahead and retrain our large language model again. So that now we have

model again. So that now we have contextual information about this. So if

there is some information that is completely lacking that cannot be compensated with rag then we use fine-tuning plus sometimes when you have

to change the personality of the model for example this is a real use case that one of the bigger banks they used chat GPT in order to answer the customer

queries right and then they were able to find that chat GP is able to answer well but it does not have that personality of how their people were actually going ahead and answering. So what they did was they actually taken all of their

past records of the chat which humans have done they have fine-tuned GPT and other models on the top of the same and then they were able to get a new model

which is now able to act nicely right similarly Bloomberg has created a product called as Bloomberg GPD where they have trained complete Bloomberg on

top of uh the a GPT on top of Bloomberg data so that if it has that kind of context Right? It kind that kind of information

Right? It kind that kind of information and knowledge. So finetuning is adopting

and knowledge. So finetuning is adopting a pre-trained large language model which is a chat GP or any base model with a domain specific data to make it more useful for a particular task. Right? So

what we do is we take a base model llama GT sonnet. Then we do the fine tuning

GT sonnet. Then we do the fine tuning with curated data and examples and then we have a fine-tuned error. Okay. It is

one line of code which you have to use in the openi terminal. OpenAI API

fine-tune create t data.json JSON

whatever the data that you have then this is the model that you want to train one line of code can go ahead and do the finetuning but understand finetuning is the costliest of all the three method

that we have talked about finetuning takes a lot of money a lot of resources so unless you are not sure you should not start with finetuning first do prompt engineering then try to

understand if the problem can be solved with rag if it is lack of information problem if it is not solved then you should go ahead and go to finetuning Right. And

Right. And generally finetuning is of two types.

One is full finetuning. Second is

parameter efficient finetuning or PFP.

Okay. So in full finetuning what happens is whatever are the billion parameters that we have in the large language models we are going to try to update

them all. So it is very intensive right.

them all. So it is very intensive right.

So this is going to give you great results but it is also going to be very very intensive on the CPU and GPU and it is going to cost you a lot of money.

Right? Similarly we have parameter efficient finetuning. Okay. Where what

efficient finetuning. Okay. Where what

we do is rather than so I have talked about parameters in the last class. In

parameter efficient finetuning rather than going ahead and uh uh optimizing and changing every parameter we only change few parameters to make

sure that middle model is able to get that behavior without spending a lot of money on training right so it updates the subset of parameters it has lost cost and store

right and the f the training can be done faster the examples are adopters and lora Right. So this is called as fine tuning

Right. So this is called as fine tuning in LMS. Now a major question is when should I choose what to do whether I should do prompt engineering or rag or

fine-tuning. How should I choose? So

fine-tuning. How should I choose? So

this is a simple decision tree for you.

Okay. If you want to decide the right contextualization method, this thing is called contextualization or context engineering. If the base LLM is good

engineering. If the base LLM is good enough with careful prompting, then go with prompting. Experiment. Try with

with prompting. Experiment. Try with

prompt first. Before going to advanced method, try with prompt first. If it is giving you the answer, go ahead and do this. There are multiple products that

this. There are multiple products that are based on the same. For example,

granola and chat PD, they are only working on prompt engineering.

Right? Second is if it is not working, prompt is not enough. Then if the problem is only about missing or dynamic knowledge then go ahead use rag is try

notion AI notebook lm they are using rag right and the third is if the issue is about style tone or a domain specific behavior that you'll understand by

asking questions to the LLM then you should go with this and examples are dual indo max and bloomberg GP that I have already given you examples of right so guys this was context engineering.

Everyone tell me yes or no in the chat if you are able to understand everything so far.

We talked about prompt engineering, we talked about rag, we also talked about fine-tuning, right? And the question is the question

right? And the question is the question is that what you should use what you should use is

usually decided by your context in your problem. Okay? Don't think that if some

problem. Okay? Don't think that if some company was able to use something and get the results, you will also get it.

Please try to understand the user problems, the context, the limitations, the trade-offs and then choose whether you should go with prompt engineering, rag or fine-tuning. You should always

start with prompt engineering. If

there's a issue with data, go with rack and then if the issue is with personality or the base data, then go with fine-tuning. Right? There is also

with fine-tuning. Right? There is also an interesting thing that is coming across these days. I'll just give you a hint. We talk about this in detail in

hint. We talk about this in detail in our bigger program which is an amazing thing which is called as transfer learning. Okay, this is an

transfer learning. Okay, this is an upcoming very I would say very promising field and many companies are going to adopt this. Just a moment.

adopt this. Just a moment.

Yes. So there is something called as transfer learning.

So what we do is the issue with large language models is that more intelligent the model okay more it is going to cost

and more it is going to have latency which is it is going to take more time to respond. So if you go to the OpenAI

to respond. So if you go to the OpenAI API, okay, OpenAI API cost, okay, so if I go to pricing,

understand that this GPT 5 is their best model and the cost of GPT5 is for output $10 per 1 million tokens, right? And

then they have a small model which is GP nano which is $0.41 $4 for 1 million tokens

almost 25x difference. Right? So models

which are small models have less number of parameters are more efficient and cost effective but they do not have the right kind of they are not trained on more data and they do not have more parameter that means their output could

be a bit here and there. Okay. So we

have something called a transfer learning. What we do is we pick a small

learning. What we do is we pick a small language model such as Gemma or a combination of or something between the small and the large language model

such as nano which is GP nano 5 nano or 4 nano whatever you want to choose. Okay, these

things have less cost. What you will do is you will go ahead generate some synthetic data from the top models which

is GPT5 or Gemini 2.5 flash okay you are going to use Gemini in order to generate some synthetic data and then you are going to generate that data and then you are going to fine-tune these models on

that data so that these these models although they will not get as intelligent as GPT5 or or or Gemini 2.5 but still if you give on specific use cases. For example, if you are using for

cases. For example, if you are using for customer support, if you are using for coding, if you are using for coding in a particular language, if you are using for a particular use case, then you can generate enough data and then you can

retrof these models with fine-tuning and then there is going to be one-time cost of finetuning. But then eventually your

of finetuning. But then eventually your overall cost of API and inference will reduce. This is called as transfer

reduce. This is called as transfer learning. So you have a small language

learning. So you have a small language model which is efficient which has less cost but it does not know about certain knowledge. So you will generate that

knowledge. So you will generate that knowledge artificially from bigger models put that knowledge into fine-tuning data for these small language models then the small language models will become more powerful for

your context.

Right? So imagine these large language models as swords.

But sometimes for some purposes you don't need a sword. You just need a needle and then transfer learning gives the power of sword to this particular needle.

Right? A quick yes on everyone if you're able to understand the power of transfer learning.

You can explore more about this. We talk

about this in more detail in our bigger program. super helpful skill for any

program. super helpful skill for any product manager because implementing LM and understanding is is is is easy but getting the most out of from this is

super important.

Yes. Cool. Now with that being said, let us go ahead and talk about the interesting piece in the room. Okay. Which is AI agents.

the room. Okay. Which is AI agents.

So far the AI agent was like your most interesting, your most uh

intelligent manager. They are telling

intelligent manager. They are telling you that this is what you should do, this is what you should do, this is what you should do. But they are not taking action. And nothing is more frustrating

action. And nothing is more frustrating than a handsoff manager which has they don't know how to do the work but they always try to give you knowledge and gam okay and LM are the same. You give them

the input, they give you the output and they appear the most intelligent part.

But they cannot act. Act means

you do not want the LLM just to write an email. You want them to send the email

email. You want them to send the email and take the follow-ups. You want them to do your work rather than just being your uh uh uh your brainstorming buddy.

So in order to give agency to the agent in to the AI, we have something called as AI agents. So AI agent is nothing but

a very simple concept. Okay, which is we give access to some actions to our

agents. We get our AI to act. So what we

agents. We get our AI to act. So what we do is a simple example of agent is let's say I post very frequently on LinkedIn.

Okay. And if someone wants to go ahead and be updated with what is happening in the product management world, what they can just do is go to my profile, look at

what I have recently posted and then you should be able to understand what is happening in the AI product management world. Okay, but this thing you have to

world. Okay, but this thing you have to do manually. What if there would be a

do manually. What if there would be a large language model? What if there would be a software or an agent that can go to my profile every day in the

morning look at the five recent post that I have done let's say not every day but let's say in a way in a week in that week whatever I have posted they should collect all that post and then they

should give all of this post to a large language model which can summarize all the important content and then it can send me an email. This is an example of a large language model. This is an

example of an AI agent. Okay, which is I am choosing a time where I am going to crawl a profile, LinkedIn profile. I'm

going to get all of their post. Then I'm

going to summarize this content and then I'm going to get them to send an email.

The summarize part is the intelligent part that is done by AI. That is why it is called as an AI agent.

Right now I can go ahead and tweak this AI agent more. What I can do is I can allow tell the AI agent that rather than getting the post from someone, you only go ahead and send me the

message rather rather than crawling all the post, you should only go ahead and crawl the post where something is written about AI product management. So

now the AI agent will go ahead and take some calls based on or take some decisions based on the content that it finds. So when the AI agent is taking

finds. So when the AI agent is taking some decisions based on its own context then it is called an autonomous AI agent when it is just going ahead and following the instruction it is a

workflow agent right a quick yes everyone if you're able to understand the example of AI agent cool and we'll actually create that

agent today okay so intelligence autonomy and uh action Right? And there

are multiple platform guys understand building AI agents these days is not rocket science. It is commoditized

rocket science. It is commoditized information. How to build great agents

information. How to build great agents is something that is something that you need to learn. Okay. So if you go to platforms such as agents.ai, ZPR, NAN and OpenAI has very recently created

their agent kit. You can go ahead and create multiple agents. Okay. However,

there has been a report recently. There

has been a report recently and this report got way too popular and it has mentioned that 95% of most of the AI initiative most of them are AI agents

have failed and the reason being there was a learning gap. Learning gap means these project did not fail because these people were failing to engineer agents.

They were able to create agents but the problem was these agents were actually not solving any good enough problem and it was not learning from the

feedback right so anytime you want to go ahead and build any kind of product please remember this if you are creating AI agent if you are creating a normal

traditional product or even if you are creating an offline product please make sure that you start from these two If you read this report, I'll I'll attach this report in the resources

section when we are sending this to you, you will understand why not having the right product sense was a major problem why these things fail rather than just not knowing about the AI. So

understanding AI is easy but using AI at a scale is a different beast altogether.

Right? Yes. So this is simple about AI agents. I'm going to help you understand

agents. I'm going to help you understand how AI agents are built. So,

a quick yes or no everyone if you have ever built an agent.

A quick yes or no if you have ever built an agent.

Cool. A lot of nos.

Cool. So, I I'll Okay. So, understand is also a fantastic flow, but I'll show you something even simpler. Okay. So you can go to this app called relay.app.

This is free with enough credit. So I'm

using this. Okay. And it is very simple.

Okay. So I'll click on login.

I might be logged on already. Okay. So

I'm already logged in. Okay. So what

I'll do is I'll create a simple flow.

Okay. So I'll click on new workflow.

Now every agent starts from a understand guys. uh this agent I am telling you

guys. uh this agent I am telling you these agents that you can build the production grade agents are different the products that I'm going to give you examples in just a moment they are also

agent but right now we are using like a agent that you can also build on your own okay this is for your own productivity okay so the agent is

yes so the agent is will do the simple thing at 8:00 a.m. In the morning it is going to crawl my profile or any other profile that you want to. It will take

all the posts and then it will summarize these post and then it will send me an email. That's it. Okay. Summarization

email. That's it. Okay. Summarization

will happen with an AI. So add a trigger. Trigger means when should this

trigger. Trigger means when should this event happen? When should this event

event happen? When should this event happen? Okay. So I will just go ahead

happen? Okay. So I will just go ahead and click on schedule trigger. It should happen

schedule trigger. It should happen tomorrow or at today's 26th today at 8 a.m. So this time has already passed. I

a.m. So this time has already passed. I

can make it tomorrow.

Okay, it can run at this time and it should run daily. I can also make it any other time. Okay, this is how the event

other time. Okay, this is how the event is triggered. Okay, and then I'll click

is triggered. Okay, and then I'll click on done.

To make it simple, I have chosen this frequency. But you can also execute your

frequency. But you can also execute your agent on any other event. For example,

when you send a message to Slack, when you send and get an email from someone, when you go ahead and maybe send a message on Telegram, when you go ahead and click a button, when you go ahead

and get a new lead from your CRM, you can go ahead and add any kind of triggers. Okay, this is just to make it

triggers. Okay, this is just to make it simple. Keep it simple, right? Then

simple. Keep it simple, right? Then

we'll add a step. The other step is I want to add LinkedIn.

So on LinkedIn I will just click on add this tab. I will search for LinkedIn and

this tab. I will search for LinkedIn and then get LinkedIn comments person post. I want some people

some person's post. I'll click on this and then I'll select a URL. So what is my LinkedIn URL? I'll go to this LinkedIn slashin

/ ankit.

So this is my LinkedIn profile.

I'll just click on this.

I will enter manually. I will enter this

enter manually. I will enter this and then I want to get the top 10 posts and then if posts are not returned then

continue without a result and then done.

Second step is done. The third step is I have to summarize.

So I will summarize and then what will I summarize? I will summarize posts. Next

summarize? I will summarize posts. Next

it will automatically tell me what you want to summarize based on what I have sent from the last step. Okay. And then

this is the prompt.

Okay. I can do it better. I can go to chat GPT and I can ask chat GPT that write me an

effective prompt to summarize my LinkedIn posts.

Right. So I will go ahead and copy this.

Right. I'll go ahead and copy this.

Okay. But to keep it simple, I'm not using this. But you can just go ahead

using this. But you can just go ahead and copy this and paste it there. Okay.

I'll go ahead with the simple one which is summarize the attached post. Use

fewer than 100 words. Okay. Then access

to this this this this is this this.

Okay. Plain text. Okay.

Done.

Right. And then we have plus button which is I want to send myself an email whenever this happens.

Email send email to yourself. What is

the subject?

Summary from Ankit's post add body. If I write X I should if I rect the rate I should be able to get the sum AI output

summary is above okay and then I'll go ahead and click on done right and now okay so everyone so far

write yes or no in the chat if you're able to understand the flow so far it is super simple Right.

Correct. And now what we'll do is we will just go ahead and try to test this start now. I can

test this, right? I don't have to wait for morning tomorrow because you guys would not be there. So I will test this summarize with AI and then an email to yours. So I'll

click on this. This is the email. The

post primarily promotes a free multi session AIPM master class which drew over 800 live attendees. The master

class focuses on mastering LLM. This

this this and this. I hope you guys remember this right now guys tell me confident in the chat if you are confident in creating your own agents now

tell me confident in the chat. Yes. So

understand this tooling was never difficult.

This tooling was never difficult. What

one thing that AI has done is it has gone ahead and made things simpler for everyone. It has gone ahead and made

everyone. It has gone ahead and made things super simpler for everyone. Okay.

As a person, you need to understand that now you have so much power and you have the power to exercise this power. You

have the ability to exercise this power.

Okay. Now you should focus on the right use cases. Okay. So what I'll also do is

use cases. Okay. So what I'll also do is I will share with you a list. So Google

has created an amazing guide.

Google has created an amazing guide which talks about what are the different kind of use cases in different kind of industries for AI agents. I will share this guide

also with you in the slides so that you are able to understand and then create agents there. First whenever you create

agents there. First whenever you create any product understand the users and the business. An AI product manager is a

business. An AI product manager is a product manager before they are an AI product manager. Okay tools are easy. I

product manager. Okay tools are easy. I

have already taught you the tools. they

are not so difficult. Important thing is understanding to solve the right problems. Okay. And then when you create an AI

Okay. And then when you create an AI agent, these are the things that you need. So this is a general stack that

need. So this is a general stack that you guys can go ahead and take a screenshot of. In an agent, you need

screenshot of. In an agent, you need reasoning so that you can summarize, you can take decisions, you can call certain kind of tools, right? And generally LLMs are used. LM's large language models are

are used. LM's large language models are GPT, Claude, Llama and others. And small

language models are F3 and JMA.

Right? Similarly,

you need memory. For example, if you in chat GPT, it remembers everything that you have told. But when APIs are called in in agents, they might forget about your older conversation. So, you need to

save your memory. Okay. So, understand

here what has happened in this model.

What is memory? Let me tell you. So, in

this model, in this flow, it found 10 post then it gathered the data. It sent the data to the other step

data. It sent the data to the other step where the AI summarized it and then it was able to get that memory that data that summarization to the next step.

This is called as memory. Making sure

that LLM and or the whole uh model the whole agent is able to remember what you just said. Right? This is called as

just said. Right? This is called as memory. And for memory we have multiple

memory. And for memory we have multiple tools.

For memory we have vector databases such as pine cone, chroma, rag and feedback loops. Right. And then there are tools.

loops. Right. And then there are tools.

Tools can be accessed with the help of databases, visual agents and APIs. APIs

are available. There is something also called as visual agents. Visual agents

is it will if it does not have an API, it will act like a computer. It will

browse, it will click buttons in order to get the data for you. Okay. And then

so open world operator is example of a visual agent.

Right. Then we have orchestrators platform on which you can create these kind of tools these kind of agents. So

langraph crew AI NA10 is these platforms. I have given you a new example today which is array.app and

then we have guard rails. So now

understand AI agents are like fire. You

can use them to cook your own food. You

can use them to show away danger animals. And you can also use it to burn

animals. And you can also use it to burn your own hands and burn your own home.

So guardrails are super important.

Guardrails make sure that your content is relevant, your agents are safe, you have you are not abusing the tools and then you are also validating the output

before presenting it to users. Right? In

our major big program we tell you about in detail of all of these things so that you are able to build some secure agents, production grade agents,

right? Yes. And uh these are some

right? Yes. And uh these are some examples of agents which are useful.

Customer support agents which will read tickets, drafts, replies and then update the CRM meeting productivity such as fireflies, fireflies and granular that we have discussed yesterday and bizops

business ops which is sepia or enterain agents that monitor incoming data and execute automations. Right? So this is

execute automations. Right? So this is in a nutshell what agents are. Okay. Now

in the next part of the class we are going to talk about how platforms such as V coding platforms such as Lovable Berchant on Vero works. Okay. But right

now let's take a quick break. Okay. So

before that everyone tell me on a scale of 1 to five how good are you able to understand everything so far?

Great. Cool. Amazing. Cool. So now

everyone a quick break of 4 minutes and then we are going to come back.

Quick break of 4 minutes and then come back.

I think we are right on time. How much

is the time?

Oh 911. Oh 911 is the time. So

great behind lovable.

Cool.

I'm sure many of you might have gone ahead and tried your hands or at least heard about these apps like these very high growth apps called as lovable,

bolt, emergent, fezero. What they do is you give them a prompt then they work in the back end in order to create a full-fledged application for you. Right?

What these app do at the back end is something that I'm going to tell you today. Okay, I'll tell you at a high

today. Okay, I'll tell you at a high level how these things are supposed to work.

But before that we should follow the follow the rules. Okay, the first thing is what is the user value? The user

value here is that as an entrepreneur, as a product manager, as an engineer, I either do not know how to code or I do

not have enough time to spend on a unproven idea. So what I do is I give it

unproven idea. So what I do is I give it a prompt. I give I want a tool that can

a prompt. I give I want a tool that can help me build a full-fledged app that can help me test my idea. Before these

kind of apps, before AI, what used to happen? I will hire a developer. I'll

happen? I will hire a developer. I'll

build something and then it will take me a lot of time. Then it will take me a lot of money and then maybe I'll know that this thing is not working. Right?

Why? Because you only know if a product is working when you go ahead and launch it. So it was very difficult to build

it. So it was very difficult to build full-fledged apps without the involvement of a developer. These low

code tools that used to happen before that like web flow and all they were not very effective for any non-engineer.

They still used to take a lot of time.

So now with the possibility of AI we have some tools with which we can just go ahead and give it a prompt. It will

be able to go ahead and generate a full website or an app. Right? So value

unlock is definitely there. Okay. But

you should define what is the market that you are playing in. Is it engineers or hobbyist or entrepreneurs or non-coders that will define the UI and the UX right and the strategy. Why

should the business pursue this opportunity? There are so many people

opportunity? There are so many people who want to build their own products, right? And traditionally these people

right? And traditionally these people have not do not have the power to do this. Why? Because technology could not

this. Why? Because technology could not support it. Now what has happened right

support it. Now what has happened right now is that there is a technology called as Gen AI through which this is possible now. Okay. So now there is a huge

now. Okay. So now there is a huge business opportunity out there. You can

do the sizing of the market in terms of tam but there is a big market opportunity right and many product managers many entrepreneurs they want to build quick prototypes before they can

go ahead and full-fledged take the business the engineering resources there is a business value the third part is design how do we make it intuitive to use so if you look at the platform such

as lovable you'll be able to find multiple things okay first is

these These platforms give you a very good interface where you can just write a prompt like a chat plus they will let you to add

any screenshots. So most of the people

any screenshots. So most of the people they are inspired. So they will go to a website look at the website it is looking good they will take a screenshot and they want to make something like

that right and then they are also going to give you examples of how other people have done this. So you are getting some inspiration. Okay. Plus they also make

inspiration. Okay. Plus they also make sure that there is a community which can go ahead and help you. So you

can go ahead and get added to this community and then you can go ahead and learn about how other people are using Lovable. So it makes sure that you are

Lovable. So it makes sure that you are able to get on boarded on the platform very very easily.

Right? And then they also have some learning tutorials and all that makes sure that it is easier for you to use.

And they have also made sure that they have a free pricing plan.

Yes, I already have a paid plan. So they

do have a premium plan through which they can you can go ahead and try the platform before you go ahead and start spending your money. Right? So this

makes sure that the platform is usable.

The ultimate usability is coming from the simple fact that an LLM can allow you to generate like convert your simple prompt into a full-fledged website or a

web app. Okay. So this is the usability

web app. Okay. So this is the usability and then how does it work?

Okay, how does it work? So this is the simple architecture of these kind of apps. Okay, the user will give the

apps. Okay, the user will give the input. Okay, the user will give a input

input. Okay, the user will give a input that please create a clone for Airbnb.

Simple but they are going to enhance it with a system prompt. So they will write a prompt like you are a senior level

system architect and a backend developer and a front end or a fullstack developer and now you have to create the app the product that the user is requesting.

Make sure that you are going ahead and following up with uh you are going ahead and following all the engineering principles and using these kind of files and they would go ahead and give more context. Okay. So your prompt whatever

context. Okay. So your prompt whatever you are giving will be enhanced with the system prompt and then as a result what you will have first it will create a prompt to plan the whole product. Step

number one is create this file. Step

number two is create this file. Step

number three is create this function then create an API. So it is going to lay down the complete plan. After that

it will create files.

It has access to a virtual environment.

So now it is becoming agentic. Why? It

has an access to a tool. What is the tool? The tool is a virtual environment.

tool? The tool is a virtual environment.

So what will happen a virtual environment is nothing a piece of computer that is given to lovable where it can go ahead and deploy your apps.

Okay. So it has access to a file system it has access to an automatic deployment. Okay. And then it is going

deployment. Okay. And then it is going to keep on running the file on itself so that it is able to understand if there are any kind of mistakes and then because it's an agent it is going to automatically correct these mistakes and

eventually you are going to get the output.

Okay. So lovable, emergent, vzero are nothing. They are just agents which work

nothing. They are just agents which work in a loop to correct the mistakes and they also make sure that they have access to some kind of file system where they can host these files so that you as

a user can go ahead and access these files.

Okay. So let me give you a moment. Okay.

This is chat GPT.

I am giving a simple prompt to charge GBD. Okay.

GBD. Okay.

I want to create a clone of Okay. help me create a system prompt

Okay. help me create a system prompt that I can use in my vibe coding app

where user can enter their idea of app and the system

prompt will make it detail. scale

detail. scale effective to be able to create a

dependable modern app with let's say JavaScript as the primary

language. Most of the apps you will see

language. Most of the apps you will see that they are written in JavaScript.

Okay, I'll just click on this and then I'll get a very detailed system prompt.

Yes. So here what will happen? While the

prompt is being generated, what will happen here?

I will create a system.

If I want to create a clone of lovable, I will create a system. This is the box.

User will input something. This request

will be added with my system prompt and the user input will be added and then it will be sent to an LLM and then I'll get the complete set of instructions.

Right? So this prompt I have taken. Now

this is a prompt system prompt. This you

are a Yoder. This this this is this this this. And now you understand everything

this. And now you understand everything is mentioned here.

Right. This now will be sent to the LM.

Now based on this files would be generated. So it will create a plan. In

generated. So it will create a plan. In

the plan there would be some files, there would be some testing, there would be some functionality and then this will be again sent to the prompt.

Right? You can actually create your own lovable, your own emergent with the help of relay app because there also you are executing instructions step by step.

Right?

So this is how lovable and every platform works on the back end. So agent

is nothing. It is thinking and accessing tools in a loop. So they have orchestrated them in such a way that they are able to go ahead and execute

things again and again in a methodical or a step-wise manner. Right? Now agents

are built on multiple patterns. One

pattern is that there is one agent who is doing multiple task. Then

let's say if I give you people an assignment that tomorrow you have to create a case study on lovable five people in the same team. Okay. Now what

will happen? I believe that if five people are working on the same team the quality should improve. That is why I'm putting five people and I'm giving you less time. For an individual I would

less time. For an individual I would have given one week but for five people I'm giving two days. So now because I think that efficiency will increase. So

now what you guys will do is you will divide the work among yourself that I will look do the research you do the direct someone is going to do the design someone is going to do something else.

Okay. And when you go ahead and come back you are going to go ahead and combine all the out outputs and then you are going to create something which is a one case study. So this is how agents

also do the orchestration. So there is also design patterns in agents which define how different agents are going to work with each other.

Okay. So this is in a nutshell what agents are. Okay. Everyone tell me on a

agents are. Okay. Everyone tell me on a scale of 1 to five you're able to understand agents.

Yes. So please everyone please understand things from first principles which is agent is nothing. Agent did not existed. AI agent did not existing 4

existed. AI agent did not existing 4 years ago. Okay. So it's very simple. If

years ago. Okay. So it's very simple. If

you are creating a workflow which can have an AI LLM in between which can execute actions for you and do some intelligent task that is an AI agent.

Right now after this there is one problem with the agents.

The problem is that the agents in order to act they need to connect with tools.

LinkedIn is one platform, Gmail is one platform, Google Drive is one platform, maybe your SMS is one platform, your Telegram is one platform and there are so many platform, your Slack is one platform. So they need to

connect with many tools. And if you are someone who is building agents where LLM needs to connect with the agents with the tools, then it's a nightmare to read

every API and connect your product with the API. Okay. So if so many let's say

the API. Okay. So if so many let's say like one LM has to connect with multiple products then everyone has their unique API. So that is a nightmare. So now what

API. So that is a nightmare. So now what these people have done is enthropic the people behind claude they have created a simple thing which is they told everyone

that if you want to they have told Gmail slack Jira everyone and the whole world that if anyone wants the LLM to use their APIs please go ahead and create something

called as model context protocol or MCP.

So what is an MCP? MCPS is these people have created one more versions of their APIs or a layer over their APIs which can be understood by large language

models.

Right? So maybe yesterday only or a couple of days before Zumato has launched their own MCP. So Zumato is an Indian website where you can is an Indian app or website where you can go

ahead and order food. So this is the detailed article on the same where is that? Yes. So now you can go ahead and

that? Yes. So now you can go ahead and do so many things from your cloud. So on

your cloud desktop or your chat GPT desktop you can go ahead and install this uh uh this cla this summato MCP and then you should be able to go ahead and

write these kind of queries Indian food in your Indraagar pizza under 200 rupees and then you should be able to make an order from there only. Right. So now

Zumato has an API which already existed for a long period of time but you cannot have conversation with an API through the LLM through natural language. Now

they have created an MCP. What does an MCP do? MCP converts the natural

MCP do? MCP converts the natural language helps the LLM to talk with the API in natural language.

Right? That is an NCP. Okay. And there

are multiple I'll share this link with you so that you are able to go and understand how it is created. Understand

it's not difficult to create MCP. What

you need to do is if you are a product like Zumat or any other product in the world, you would have an API documentation. Take that API

documentation. Take that API documentation, go to chat GPT and ask it to convert into an MCP. It will give you a file that you have to host on your server and then you are done. You have

to do a couple of testing as well but that is done right. So that is model context

right. So that is model context protocol. So in model context protocol

protocol. So in model context protocol we have three components. The first part is the AI host where you are going to send query. So it could be claude,

send query. So it could be claude, cursor, chat, GPT. This is where your MCP client is going to be there. This is

your MCP client. Okay. Then there's an MCP server that is hosted at the person who has given you the MCP which is Zumato or Razer Pay or something. And

then there is a data source. This is

stored at Zumato or somewhere where you can go ahead and have the data or you can take actions. Okay. So any MCP server will give you a small file that is called as client configuration file.

You have to enter it into your cloud or any of the LLM that you are using and then your agent will be able to talk to these tools without understanding the API.

Right? So this is MCP. You can go ahead and read more about it. Right? Yes. So

we have talked about so many things so far.

Cool.

Everyone tell me how would you describe the session so far in one word so that I know if we are going in the right direction.

Tell me how would you explain this session so far in one word so that I can go ahead and tell you something more.

Yes.

Great. Now let me just go ahead. Thank

you. Thank you everyone. Thank you for the nice words. Now I think we are on time. So we have some more time. So I'll

time. So we have some more time. So I'll

give you one more thing. Okay, which is evaluations.

Look at this very carefully. Okay, so

there are some challenges that happen in building AI products. The

challenges are they hallucinate. They

have certain kind of biases and they are indeterministic.

Okay. But understand nowadays you people would be understanding that language models are becoming more and more intelligent. Okay. If you find that

intelligent. Okay. If you find that there is an error, you can ask it to correct that error and it corrects that error. Okay. But still large language

error. Okay. But still large language models are these powerful beast. If you

leave them without a leash then they can do a lot of disaster to your customers.

Okay. So this is what AI believes like.

Okay. It believes that it has all the content in the world and it can give answers like wrong answers with complete confidence.

Okay. So that is why we have something to tame these beast. The leash that works on your LLMs is called as evaluations.

Okay. And what happens is whenever your LLM is not working and if you're not getting the right kind of output, there are certain issues and you can resolve these issues with three things. Okay. So

when you are building any kind of GI product, you have to take three important decisions. The one important

important decisions. The one important decision is what is the model that you are going to choose. If you choose a very high quality model then you have to pay more cost. If you choose a very low

cost model then you have an issue of accuracy.

Right? Similarly rag instrumentation in rag there are multiple components.

If your chunk size is not correct then the information that it is giving you would be incorrect. If you understand I have told you that there is a vector embedding but for vector embedding also there are multiple algorithms which

algorithm should you use okay it depends on the use case. So what is going to be the instrumentation of your rag? What is

the vector algorithm? What is the size of chunking? What are the different kind

of chunking? What are the different kind of model that you will use? That is also very important.

Okay. These are the decision that you need to take. And then context engineering. What should be the prompt?

engineering. What should be the prompt?

How long should be the context? how you

should give the instructions. These

things are going to determine whether your geni app is working rightly or not.

And this is actually engineering.

Engineering why? Because there is going to be a lot of course correction.

Okay. So in order to do these things, in order to understand whether our geni application is working well or not, we have something called as evaluations. So

what evaluations do is they check the output for correct syntax and formats for bias for correctness relevance and other unnecessary checks. Okay. So for

example if on notebook LM al on granular we generated a summary we want to understand whether this summary is actually aligned with whatever has been

said in the meeting or not. So how do we test this? We test this with the help of

test this? We test this with the help of large language models. Okay. So what we do is to help you understand I'm going to give

you a case study on the AI first job website. Okay. So tell me in the chat

website. Okay. So tell me in the chat how many people are looking for jobs.

Tell me in the chat yes or no if you're looking for a job.

Yes guys even if you don't know your colleagues and your bosses might be present in this meeting. Now

I will give you an idea. Okay. And I'm

sure many people would have faced this.

Okay. So let's say I am creating a new kind of job website where what I'll do is I will crawl all the websites from the all I will crawl all the jobs from

major websites from the internet. So

I'll go to hired, I'll go to LinkedIn, I'll go to no, I'll go to Instaire, I'll go to multiple websites. I'll go to indeed and other websites in order to crawl jobs. scroll as in I will go to

crawl jobs. scroll as in I will go to this website automatically I will get the content and I'll scrape the jobs so now I have a job database I have

database of all the job descriptions titles companies and everything right now what I'll do is I will enhance this

job with LLM I will enhance this job with LLM and by enhancing what I mean is I will give it a prompt and then what I'll do is I will generate this output

so I will enter a job description and then the LLM will give me summary of the job description possible interview question. So from the job description in the company it will

try to understand what are the question that can be asked skills that are needed to do this job learning guide for this job and quiz for assessment which is a small quiz that

will help me understand whether I'm ready to fill or apply for this job or not. Okay, this is a simple website that

not. Okay, this is a simple website that I need to create. Okay, tell me everyone will this kind of website will be helpful for you or not? People are

looking for jobs.

Yes. So, it will give you some more context about the job description.

Right. So, now and this can be created.

You can go ahead and create this website today also like by using labable or something. Okay. But now there is an

something. Okay. But now there is an issue. How do I know this job

issue. How do I know this job description, interview question, skill needed, learning guide, everything is good or not. Okay. So I'm going to introduce one method of so understand

these things that I have told you which is all of these things they are done with multiple methods. One of the method

to run your evaluations is called as LLM as a judge. So we run our outputs through another more intelligent LLM in

order to understand whether it is correct output or not. Okay. So here the issue is that although I'm generating good information but this information can be wrong.

Okay. So what I'll do is I'll do this.

Okay. So I have crawled a job from the internet. I have stored them in my

internet. I have stored them in my database. I have ran them through

database. I have ran them through LLMs and then I have created this content.

I have created this content.

Now what I'll do is I will get all of these things again go through another LLM in order to understand which is called as evaluation

that whether this content is correctly created or not. This is called as LLM as a judge and then I will find out whether the

content is correct or not. This is a simple method of evaluation. Let me show you how it works.

Yes. So this is the evaluation. Look at

this very carefully guys. This is the prompt that I'll go to the I'll give to the another LM. Okay. You are an AI content quality evaluator for product management job listings.

Given the following, this is the original job description. This is the AI generated summary. This is the interview

generated summary. This is the interview question. This is the output from the

question. This is the output from the other LLM. Right? These are the skills.

other LLM. Right? These are the skills.

These are the concept. This is a quiz.

Evaluate the content using the following checklist. Is the summary accurate?

checklist. Is the summary accurate?

Interview questions correct? Listed

skills aligned? Are the concept useful?

Is the quiz challenging? is any

information correct or hell needing and then give me the response in this particular format okay then I'll have a dashboard where for every job description this score

would be mentioned then I would know whether my LLM is working correctly or not okay so let's say if I get poor or fair or low or weak what I'll do is I

will go back I will again try to tweak these things if my quality is add then I will maybe choose change the model or I'll change

the rag instrumentation or I'll go ahead and change the context engineering which is the prompt that I have given okay that is how you improve upon AI product with the help of evaluation this is a

super simplistic example of evaluations okay there are codebased evaluations as well there are LLM based evaluations as well generally this evaluation LLM that

you use so this is LLM1 and this is LLM2. This LLM is generally a more intelligent LLM which is able to assess.

Right? So this is an example of evaluation. Everyone a quick yes or no

evaluation. Everyone a quick yes or no if you're able to understand this.

Yes, there are many open source LLM guys. OpenAI OSS is an open source LLM.

guys. OpenAI OSS is an open source LLM.

Lama is an open source LLM.

Right? So this is there. Now we have almost completed the session. This is

some set of tools. If you guys are curious, you can go ahead and start using this in your product management workflow. There is also a detailed video

workflow. There is also a detailed video on this on our YouTube channel. Okay. AI

for PM toolkit discovery for discovery for discovering what to build chat GP notebook perplexity mix panel textsql.

Okay. for delivering at intelligence which is nothing but Jira, Asana, Vzero, Kurser, Postbot, Figma which you can use to create wireframes prototypes very quickly. Okay. And then for distribution

quickly. Okay. And then for distribution you have craftable, chatbased, notion, genex, moage that can help you create marketing materials, collaborate with your team and look at analytics.

Okay. So try to explore these tools at your own time so that you are able to become an AI enabled PM as well.

Whatever we have discussed so far was for AI like applied AIPM, right? And

then one more important thing guys that in this program like so far what I have done is I have given you information

I have given you information about evaluations about agents about various tools about rack about prompt engineering about finetuning but information is not

helpful for you information is just a couple of prompts away what you need is knowledge.

And how do you convert information into knowledge? With the help of action

knowledge? With the help of action and with the help of feedback and reflection.

Right? So what you need to do is please do not limit yourself to only this knowledge.

Go ahead and do something. Doing as in go ahead and build something. Okay? You

have so much information available across the internet. Utilize and try to go ahead and build something right. If

you guys are looking for mentorship then we also have a very detailed program. So

understand whatever I have told you in this last two session that is not even 10% of what we teach in our bigger program. So if you really found this

program. So if you really found this information genuine useful I teach a bigger program which is a 15week program on AI product management. Okay you can go to the website you can get all the

details out there. But if you have to ask me let's say three reasons why you should join LOPM program against other programs is very simple.

First it is the most detailed AI product management program. We first talk about

management program. We first talk about the traditional product management then we talk about AI product management. You

can go to the website you can compare the curriculum. You can compare

the curriculum. You can compare curriculum to any other course in the world. You will not be able to find a

world. You will not be able to find a more detailed program. First point.

Second point is everything is practical.

We believe in doing things. So in the end of the program, you're actually going to go ahead and build your own product. Okay. So if you go ahead and go

product. Okay. So if you go ahead and go to the website, if you go ahead and go to the website hellom.co, you should be able to find in the curriculum section that this is the

whole curriculum. Everything is

whole curriculum. Everything is supported with a lot of case studies and tools.

Plus in the end of the sessions, we are also going to help you build your own AI product.

This helps you prove to other people and to yourself that you are not an information seeker. You are actually

information seeker. You are actually someone who understands how to build products and this is a big opportunity for you. Right. And we also help you

for you. Right. And we also help you prepare for interviews and all that is given. And then

given. And then yes and then the third part is we are the only program that supports you for the long term. Even if you have

completed the program, we understand that some people are able to get the job within the program, some people get the job after the program. We support you for a period of two years. So in these two years, if you have any questions, any feedback, anything that you want to

get help from us, you should be able to talk to us. Okay. And every week, we keep on doing sessions to make sure that you are able to learn. Okay. The

Yes.

Yes. And you can go to the website in order to check all the other details that are out there. The cost of the program stands at 75,000 rupees or $1,000 if you are outside from India.

Okay. But it is my assurance that if you go ahead and take this program, this is going to go ahead and give you a lot of content, right? We don't give any kind of

right? We don't give any kind of discounts, but this is the context. You

if you want to get more information about this cohort, just go to the website hello.co and then click on this video. You should be able to get all the

video. You should be able to get all the information right now an important part. Okay, that is why I've given you this context.

Yes, classes are conducted on Saturday and Sunday evening 8:30 to 10:30 for ISD and morning for the US batches. Okay. So now

everyone understand if you really want to get into AI product management you have to build a strong portfolio so that people understand that you have what it takes.

Okay. You are actually not a talker but a doer. Okay. So what you should do is

a doer. Okay. So what you should do is in order to create a portfolio you can start very small. You start by commenting and start by creating content

about geni. Understand for yourself.

about geni. Understand for yourself.

Okay. Then do tear down in case studies.

Then do product improvements and go ahead and build some like try to go ahead and build some hypothesis of the products. After that build some side

products. After that build some side projects and then if you have already done some past work you can include all of these things and then you can use in your portfolio. You can go to the hello

your portfolio. You can go to the hello channel there we have created. So if you go to the hello channel on YouTube

we have an amazing playlist called as summer of product. Okay. Or this is the playlist getting started with product management or summer of product where I have explained how to create a portfolio.

Okay. And then the last part is this.

This is the most important slide. What

you should do is please do not limit this information that I have told you so far to just this session. Okay? Don't

forget about everything as soon as you leave the session. Please take action.

What you should do is reverse engineer top AI products in the same way that I have told you. Understand gaps,

frustrations in your personal and professional workflow. Build AI tools

professional workflow. Build AI tools and agents to solve these problems. Start writing what you are learning. Tag

me. I'll be able to like and post and depost. Right? And then keep a track of

depost. Right? And then keep a track of AI companies what they are doing. Help

them improve, enhance, expand and then follow up. Try to find companies who are

follow up. Try to find companies who are do really doing good and then understand their product. Reverse engineer the

their product. Reverse engineer the product. Suggest improvements. reach out

product. Suggest improvements. reach out

to the founders and then follow up with the founders. Most of the time you are

the founders. Most of the time you are going to get some appreciation, attention and maybe a job.

Okay, there are many people that we have helped came from completely non- tech background, some senior background, some junior background and they were able to become product managers. You can go ahead and look at this playlist called as transition to product management

where you can read their stories.

Okay. So this is it about today's session guys. Yes. Uh the flagship

session guys. Yes. Uh the flagship program is four weeks of foundation PM content which is strategy and product sense.

8 weeks of going ahead and creating like understanding AI in detail. So

understand if I can give you this much information in 2 days what can I do in 8 weeks or maybe 15 weeks and then there are three weeks of building your own

product.

So flagship program covers the entire AIPM syllabus.

If some of you wants to check how does a hello cohort looks like, let me show you something. Give me a moment.

Yes, I have shared a link in the chat. This

is the detailed video where we have shared the kickstarting call for a cohort. So at hello PM, we try to keep things as transparent as

possible. We love that. We love being

possible. We love that. We love being honest in this really corrupted tech world. So I have gone ahead and shared a

world. So I have gone ahead and shared a link where you can understand who are the kind of people who join, what are their backgrounds so that you are able to go ahead and take a better call.

Yes. And there is a lot of free content that is available on our YouTube channel. You guys should absolutely go

channel. You guys should absolutely go ahead and check it out. The placement

support that we offer is including four things. We offer you we help you create

things. We offer you we help you create a better resume.

We help you build a strong portfolio.

We help you prepare better for the interviews and then we also have an internal job board where alumni or network companies and other companies that we seek out to also list your jobs

and for many of these jobs referrals and recommendations might also be available.

We do not guarantee placements but we help you out in all the possible ways that we can and we also uh are available across the gloes. So we have a lot of

people who are going to join from UK, from the US, from Canada, Australia, New Zealand and as well from India.

And this happens because of a large alumni network that we have created.

You can use self-hosted LLMs without having internet, but you should have a powerful computer.

Cool. Thank you everyone. I'll just go ahead and end the session now. Thanks a

lot for choosing to spend your weekend with me and with Hello PM. I believe I was able to go ahead and add some kind of value as available from the feedback and the LinkedIn comments. We are very

soon going to conduct more sessions. The

next batch is going to start from 9th of November. So take your time. Understand

November. So take your time. Understand

you do not have to take this decision in FOMO. Go ahead, take the right decision

FOMO. Go ahead, take the right decision and understand it is not just a time invest. It is not just a money

invest. It is not just a money investment but it's an important time investment. Okay? So take your decision

investment. Okay? So take your decision by understanding everything. We want you to win but once you come in the session we make sure that you are working hard.

Yes. Cool. Yes. Everyone if you have any custom questions you can go to the website. There's a shoulduling call

website. There's a shoulduling call link. You can go ahead and schedule a

link. You can go ahead and schedule a call and then the team should be able to help. And we keep on doing these

help. And we keep on doing these counseling sessions so that you can ask your questions as well. Okay. Cool.

Yes. Cool. Thank you everyone. Thank you

everyone. Bye. Take care. You have been a great audience. Thanks again for choosing to spend your time with me. I

really feel good that I was able to be somewhat helpful for you guys. Thank you

everyone. Bye and take care. It was my pleasure hosting you guys. Thank you

guys. Take care.