Observability in the Age of AI with Jeetu Patel, Atin Sanyal and Yash Sheth
By Cisco
Summary
Topics Covered
- Observability Is the Prerequisite for Agent Security
- A Loose Prompt Can Deny Patients Medication
- Agent Reliability Unlocks Enterprise-Scale Adoption
- Token Spend Must Map ROI, Not Just Track Costs
- Route Tasks to the Cheapest Accurate Model
Full Transcript
JEETU PATEL: Hello again, and welcome to the Inside Products Podcast.
I have with me today two very special guests from a recent acquisition that we made of a company called Galileo, Yash and Atin.
And we have Vikram, who is the third musketeer who is not here today because he has had a new addition to his family.
So he's enjoying well-deserved time with his family.
So we'll make sure that we have him on the next pod.
But welcome to both of you.
ATIN SANYAL: Thank you.
YASH SHETH: Thank you.
JEETU PATEL: Before I get started, I actually-- you folks have had an amazing background.
You were at Google.
You worked at DeepMind.
Talk to us a little bit about that.
What did you do over there?
And what learnings did you have before we go into how did you think about founding Galileo and what happened from there?
YASH SHETH: Yeah, so before we started Galileo, I spent about a decade working at Google, building conversational AI systems. We started with very rudimentary speech-to-text systems. And there was a time when we were able to convince Larry Sergey and the business that conversational AI is going to be extremely, extremely important.
That's going to be the main way by which the world-- JEETU PATEL: People interface with tech.
YASH SHETH: --consumes technology.
Yeah.
And so we've gone over a whole host of models over the last decade, whether it's right from Gaussian mixture models to RNNs to DNNs to transformers and then to large transformers.
And I think the big unlock that LLMs, which is the large transformer architectures, have had is we didn't have to really hard-code the natural language understanding parts into a hard-coded workflow that initially these assistants were built on.
And on top of that, these models became multimodal.
And so we were part of this early team that was building out these models and scaling them.
We worked with Denison and team as well.
And on top of that, our team powered about 20-plus products at Google, whether it's the Google Assistant, Google Cloud and Google Meets and the cars and the home devices and the phones.
So-- JEETU PATEL: Yeah, the same voice engine that you would actually-- YASH SHETH: The same conversational AI engine.
So it's the voice and natural language Understanding along with actions that you can do in different modalities.
So I think that's-- JEETU PATEL: Were you in charge also?
Did you also think deeply about tone and-- YASH SHETH: Yes.
JEETU PATEL: --tonality, all that?
YASH SHETH: Tone, punctuations, and then even quality of the generated voice.
How can we make it-- JEETU PATEL: How can you make it more human-like?
YASH SHETH: --sound more natural?
Exactly.
And so that was adopted by Contact Center businesses and the meeting-- in COVID, we saw a huge spike in these Google Meets and Webex and all of these products.
And then conversational AI, transcripts, and now-- we take that all for granted now, but that was a tough time for us to scale this technology to long-form speech and long-form conversations.
So that's where I come from.
And towards the end of my student-- JEETU PATEL: You currently run research and core engineering-- YASH SHETH: Yes.
JEETU PATEL: --in this team.
And then often you run the product and the engineering side.
YASH SHETH: That's correct, yeah.
JEETU PATEL: And so Atin, what was your background?
ATIN SANYAL: So I started my career in the earliest days of Siri, where I worked on some of the first versions.
This is-- JEETU PATEL: Tom Gruber.
ATIN SANYAL: Tom Gruber, yes.
And this is still in the ImageNet, AlexNet days, where deep learning is still kind of in the labs.
But really learned how to build end-to-end productionized language systems before deep learning models were a thing.
Spend a few years there, and then really joined a very interesting team at Uber back when Uber was on a tear.
They were growing.
And they had barely any AI footprint.
It was a team of six or seven people.
It's known as Michelangelo.
And we literally built out their entire AI system.
JEETU PATEL: Was this before Monic or during Monic's time?
ATIN SANYAL: This was before and enduring.
JEETU PATEL: OK.
ATIN SANYAL: Yes.
But we really were a few of the folks who built out end-to-end AI infra and the foundations and blueprints which have been adopted pretty much across the globe.
We were one of the first to build feature stores, which are data systems for machine learning models, as well as monitoring systems, where I really saw the battle scars of how do you make AI models do the right thing and how do you measure them?
And then met Yash and Vikram, who were solving the same problems at Google.
And we sought out together and we said, how do you solve the trust problem in AI?
YASH SHETH: For unstructured data and language models in particular.
That was the-- yeah.
ATIN SANYAL: So unstructured was one of our key pillars.
And we said that we want to build for the next 10 years of AI, not the past 10 years.
Because we had seen deep learning models get cheaper and cheaper to productionize.
So we saw that coming.
We didn't, of course, foresee exactly how it would manifest.
But we kind of feel like we hit the nail on the head on the unstructured vision.
And not only was it exciting for us because it was future-facing, but we feel like we spent the first couple of years before ChatGPT happened really building the foundations of understanding AI models.
JEETU PATEL: Yash and Atin, what I'm going to do is I'm going to take a step back and say, if I were to think about how observability for AI is broken down in my mind, the three distinct components to it, number 1 is the observability for the infrastructure itself.
So how are you going out and measuring resilience of the infrastructure?
GPU utilization, uptime, all of those pieces.
YASH SHETH: Networks.
JEETU PATEL: Networks.
The second area then is agent behavior observability, where you can actually see how the agent is behaving, seeing if they're behaving in a way that you don't want it to behave.
And when that is happening, can you dynamically at runtime intercept to alter that behavior?
And then the third area is around tokenomics.
Is the consumption pattern of tokens for these agents within the boundary conditions that you want to actually have, or is it actually going out completely in a different stratosphere, which could then create a very, very different economic impact for you?
YASH SHETH: Yeah.
JEETU PATEL: Am I missing anything in that?
And then how are you folks thinking about how Splunk fits into that entire picture?
YASH SHETH: I mean, I think, Jeetu, that these three things literally cover the entire agentic stack.
While infrastructure seems-- we sometimes ignore infrastructure and think that it's commoditized.
But as agents are scaling, we are seeing customers hit infrastructure bottlenecks very quickly.
As we all know, just the networking impact of agents is just 450 times that of a human.
And so our customers are actually asking us for an end-to-end observability across not just the agent behavior, which we talked about in quite depth already with evals and enforcement, but how are the infrastructure components dealing with agents?
What network accesses do these agents have?
What are the security layers that we can put on-- because without observability, you can't have security.
And so if we want to run agents, whether it's in a desk side computer or it's in a server rack or it's in the cloud, without end-to-end observability, we can't enforce the right guardrails around the agentic processes.
And the final piece is tokenomics, where tokenomics is actually bubbling up quite a bit now as something that's of importance.
But we've always tied the end outcome, the ROI of agents to the spend that we're making.
When we talk to customers in the past in the first couple of years post-ChatGPT, we've had so many dinner table conversations with CIOs and saying, hey, I spent $15 million on LLMs. What's the ROI?
Are we shipping more product?
Are we actually moving the needle on our revenue?
And the answer has been no in the past.
But now I'm starting to see that change, where there is measurable revenue.
There is measurable impact.
There's measurable productivity.
But the thing that's missing is the infrastructure to measure that itself.
How can we quantify and tie the agent's behavior and the tokens spent to the business impact or the ROI.
JEETU PATEL: To the value that you are going to receive.
YASH SHETH: Because what leaders want to actually do is we don't want to spend less on AI.
We want to spend more on the right pieces that are giving the right ROI.
There are thousand things going on with AI.
There's a lot of tinkering going on.
JEETU PATEL: So give me a concrete example that a customer can say, if I have these class of problems, I should be thinking about observability for AI with Splunk.
YASH SHETH: So there are three classes of problems. When we think about who needs observability for AI, most enterprises today are having some form of agentic behavior in their workflows.
These are tools and productivity tools and coding agents for their developers.
That's the biggest spend today, and that's the biggest amount of agentic activity.
That being said, the first-party and third-party agents, those are the other two categories, where first-party agents are the ones that enterprises build themselves.
And that could be around their internal workflows, but also consumer-facing products.
And third-party agents are the entire agent sprawl we are seeing in every SaaS provider having their own agents.
JEETU PATEL: And you want to make sure that you provide observability for the entirety of the estate of all agents.
YASH SHETH: It has to be.
We cannot just have blind spots around, oh, I measure my first-party agents, but what about the rest?
But the point I was trying to make is the agentic behavior in products and services is only growing.
And today is just the early days.
As more and more workflows in our tools, in our products become agentic, the spend and the observability needs for agentic on first party, third party will increase.
But today, as we stand, we cannot ignore the coding agents and the tools that vertical as well.
JEETU PATEL: So I'll come to you, Atin, on the product side of the house because there are so many products that actually use these terms that sometimes it's hard for a customer to say, I can't tell the difference.
So here's my question to you.
If I'm a customer at a bank or an insurance company or a pharmaceutical company, pick an industry or a vertical, what is a concrete example where what you do and what we do with Splunk Observability for AI can meaningfully change the game on the kind of value
they harness from AI?
ATIN SANYAL: Absolutely.
I'll give you a concrete example, say, of a healthcare company.
They're building workflows to make critical decisions on, say, prior authorizations of expensive drugs.
They're using GenAI to solve it, and they're building agents to automate this whole gnarly process.
And it's ripe for disruption because these workflows are just filled with unstructured data, PDFs, documents.
And LLMs are-- for the first time, we have technology that can make sense of it and take actions for it.
So we have the ingredients.
Once you build an agentic workflow, a whole host of things can go wrong at many layers, including the three layers that you mentioned.
I'll talk about the first layer, which is tokenomics, and second layer, which is the observer.
JEETU PATEL: Agent behavior.
ATIN SANYAL: The behavior.
Starting with the behavior, one wrong mistake that the agent makes because of a loose prompt or a wrong instruction can lead to a bad action, which in this case would be the denial of a drug for a patient in need.
So the results can be catastrophic because of wrong behavior.
Then there's the cost aspect to this where tokenomics comes in.
And I'll tell you how layer 2 and layer 3 are related.
There's the absolute measurement of tokens that will be needed to take this action, including parsing all the PDFs, the docs, the insurance docs.
All this requires tokens, and we can't escape that.
But the question is, how do you do it efficiently?
Because these models are probabilistic.
They can go wrong, and they can spin into loops.
So it can be very expensive to get this action done.
So the problem to solve is, one, measure the total number of tokens it took to take the action.
And two, whether the action was taken or not, which is also a key part of tokenomics, but the insight and the signal for whether the action was done right or wrong comes from the layer below.
So that's a-- JEETU PATEL: OK, so I measure the key actions taken with the use of tokens.
What do I do if I find out that-- what's a good scenario and a bad scenario in that particular instance?
And what does our technology help with on that front?
ATIN SANYAL: A good scenario is that there's action completion happening, which is the agents are doing yep.
What the instructor action is, which is to approve or deny the drug and do it with high saliency or high accuracy.
The bad scenario is, of course, doing the wrong action.
But the bad scenario could also be your rewrite action.
JEETU PATEL: Very consumptive on tokens, while you're taking-- ATIN SANYAL: Extremely consumptive, extremely slow, because whether it's a healthcare workflow or an e-commerce application that can deliver goods to you and issue refunds, the architecture of agents behind the scenes is the same.
And they have the same proclivity to the errors that they can make.
JEETU PATEL: So why did you folks feel like being part of Splunk would actually be additive to your mission?
What was it that Splunk brings to the table that you felt like you didn't have?
YASH SHETH: I think to my point earlier, the thing that we're super excited about is that Cisco is building the critical infrastructure for the AI era.
Splunk, as a business overall, is looking at solving the digital resiliency problem.
And agentic or AI resiliency is the next chapter in this entire business.
We also heard from our largest customers, some of the largest Fortune 100 customers, that they have an entire digital resiliency stack, whether for application monitoring, for enterprise security.
And then as more and more services start becoming agentic, they asked us as, how is your platform going to tackle all of it?
Because we want the entire digital resiliency to be in one end-to-end flow, in one end-to-end solution.
And so from a product and technology perspective, what we've built is extremely additive as well as strategic from Splunk's customers, as well as Galileo's customers.
JEETU PATEL: This is truly a full-stack observability solution from infrastructure observability, application observability-- YASH SHETH: And tokenomics.
JEETU PATEL: And then going into agent behavior, tokenomics and pulling it all together in a way that's one cohesive interface in Cisco Cloud Control.
YASH SHETH: Absolutely.
And apart from Splunk's capabilities also, Cisco has the AI Defense aspects, the DefenseClaw and AI Defense pieces that actually fit in very well with the story, as well as ThousandEyes and security solutions.
With all of these capabilities, Cisco can truly provide the solution for critical infrastructure for AI agents.
I'm not saying that lightly because one bespoke point in time solution is not going to solve the overall problem to build.
And all of this allows us to build the fabric for digital resiliency, agent resiliency across agent software.
ATIN SANYAL: And whether you take the example that took Fable down or the healthcare example I just gave, these are anecdotal examples of singular workflows that, of course, have the same failure patterns.
But really, when you imagine an agentic fleet running in production, that happens across multiple devices, multiple data centers, multiple availability zones in the cloud.
And to detect a security issue in an agentic flow, the data can come from anywhere.
So providing, number 1, the unified data fabric to be able to-- a unified interface across all your data.
You need that to solve the security problem.
You need observability on all this data to solve the security problem.
It all blends into one another.
So having this full stack really is saying that I have the key ingredients to solve agent reliability.
And Cisco, from what I know AI is the only company in the world that has that.
YASH SHETH: And to your point, Cloud Control stitches it all together.
And we're super excited about where we're going with the direction.
Every product in Cisco can be operated-- JEETU PATEL: You folks integrated into Cloud Control in like 2 or 3 days or something like that.
YASH SHETH: Yes.
And-- JEETU PATEL: It's ridiculously fast.
YASH SHETH: --we joined Cisco, and in a week at Cisco Live, we demoed Cloud Control with agent observability and tokenomics.
And so that's been an incredible journey as well, because it's so easy to build Cloud Control apps now that every product in Cisco can be operated through one interface.
JEETU PATEL: So let me try to summarize and see if we can actually-- and correct me if I'm wrong on any of this stuff.
In order for us to have agents do work on our behalf, what do agents need?
They need to make sure that they're treated as digital coworkers and have tools access.
In order to have a pattern where we can delegate to the agents in a trusted way rather than an untrusted way, what you have to do is protect the agent from the world so that if there's prompt injection attacks, there's data poisoning, all of that stuff's happening, we can figure out a way to intercept it-- ATIN SANYAL: Yes.
JEETU PATEL: --with things like AI Defense.
And number 2, we can also make sure that we protect the world from agents that are misbehaving.
YASH SHETH: Yes.
JEETU PATEL: And all of that has to be done with detection that happens at machine speed and scale.
ATIN SANYAL: Absolutely.
JEETU PATEL: So what you folks have built is the full stack and the observability apparatus to ensure that all of this gets stitched together.
But the key components that Galileo brings are agent behavior observability, which includes evals.
And then tokenomics so that you can continue to keep refining how the agents are behaving in the consumption pattern of tokens, and marry that with the infrastructure observability and the application observability that we have with OLI Cloud and App'd.
And the reason this is really pertinent to be part of Splunk is because Splunk has time series machine data.
And observability is fundamentally a time series problem.
And so you want to make sure that you can actually go out and utilize that data in a way that's effective.
ATIN SANYAL: Even security.
Security is now a series of events, like the adversarial attacks I mentioned.
It's all time series.
JEETU PATEL: And if a customer does this, what have we seen as the primary ROI that the customer gets?
What does a customer go to bed feeling peace of mind with?
YASH SHETH: I just give out an actual data point.
One of our largest enterprise customers went from one agent in production to 75 agents in production in a matter of 6 to 7 months.
If you have the right agent resiliency stack, then the enterprise can feel confident.
All the teams building different agents and even onboarding agents from different solutions feel confident that they have the right playbook in place.
And that's extremely important.
If we can not only deliver the right products, but the right solution is the most important thing.
When we go to customers, we can say, now you have the end-to-end playbook of what every team needs to do.
What is your governance and compliance strategy?
And our products support all of that.
That becomes magic to our customers, yours, because we're coming with this end-to-end solution that will help them accelerate.
JEETU PATEL: And by the way, you folks have built your small language models, SLMs, so that we can actually lower the cost of inference as well for the observability stack that we're using.
It's like down by like-- we can reduce it by 95% YASH SHETH: That's huge.
To the point that the price that our customers pay, the savings that they get from the SLMs actually make up for the price as well and actually end up saving more.
JEETU PATEL: So your inference cost is de minimis in this thing.
ATIN SANYAL: That's correct.
JEETU PATEL: So given all of this, are there questions I should have asked you folks that I didn't that you want to convey to our customers?
YASH SHETH: Yeah, so one of the things that we're hearing all the time these days is, how are we solving the tokenomics problem?
Our customers have been seeing a huge unexpected spend on tokens.
And they don't have the ability to measure the right ROI from their reference.
JEETU PATEL: And they don't have the ability to fund all of that either, because it's hundreds of millions of dollars.
And people don't just have that lying around.
YASH SHETH: And we're are always competing on what to fund, what to invest in.
And so I think there are a few things that we're looking at deeply from building the right product and then also helping customers with insights.
I think the key problems that we're trying to solve with tokenomics is, firstly, how do we give customers the deep observability into, where is the spend-- JEETU PATEL: Happening.
YASH SHETH: --happening?
JEETU PATEL: By which agents.
For what activities and what tasks?
YASH SHETH: What activities, what projects, what tasks, which personnel?
What are the budgets that are going off?
All of the measurement part, but that's not enough.
What customers have been asking us is, how can you help us determine what's the ROI from these agents?
And how do we map the spend to the ROI?
And then second is, can we actually do some powerful forecasting on these spends?
And Splunk with its time series data, the Cisco Data Fabric getting all of the time series information, we have our time series foundation model that was built from scratch on time series data.
We're leveraging these capabilities to give customers better forecasting on their token spend.
That is music to people's ears-- JEETU PATEL: I think there's going to be a third area, which I'm not sure if we have fully cracked the code on, but we're going to need to, which is the dynamic interception when agents start to go rogue, so that we can actually stop-- ATIN SANYAL: Stop-- JEETU PATEL: --the blockage.
YASH SHETH: Actually, there is a rudimentary solution today where-- JEETU PATEL: But it's a very blunt instrument right now.
YASH SHETH: It's a very blunt instrument.
Any single prompt cannot be more than these tokens.
If an agent-- JEETU PATEL: That's right.
YASH SHETH: --session goes beyond these many tokens, we saw-- JEETU PATEL: But over time, if you were to think about from a product hygiene standpoint-- YASH SHETH: Absolutely.
JEETU PATEL: --how would you want to build this?
You would want to make sure that you almost, in runtime, start to instruct the agent to pull back and not have that level of consumptive pattern being exhibited for the job that they're doing.
ATIN SANYAL: Absolutely.
I mean, runtime is really the only way to curb cost is to detect rogue agent behavior or anomalous behavior at runtime and prevent the-- JEETU PATEL: And not just intercept, but also influence and redirect the agent there.
YASH SHETH: Redirect, exactly.
ATIN SANYAL: That's the actionability bit.
JEETU PATEL: Yeah, exactly.
That's a key part of tokenomics, which is detecting the issue and taking the action and curbing the wrong action, which, at scale, would lead to millions of dollars of wastage.
YASH SHETH: Well, and speaking of blunt instruments, there's a fourth aspect to tokenomics is today, speaking of blunt instruments, most users are just using the most powerful model for every single thing that they're doing.
And this is another ask that keeps coming up, is like using-- JEETU PATEL: You have intelligent routing.
YASH SHETH: Using our eval capabilities, can we measure that-- you could do the same job with the next cheaper model and give that proactive insight-- JEETU PATEL: Post facto.
YASH SHETH: --post facto, but still proactive.
JEETU PATEL: So that you can then go back and actually reapply it.
YASH SHETH: Yeah, because most teams are not investing the time to measure which-- can there be a cheaper model that can do an equivalent amount of job?
JEETU PATEL: Sure, sure, sure, sure, sure.
So that actually is a very interesting point, which is like over time, be able to say that this is how much this task cost you.
But this was the distribution of the task between a frontier model and an SLM.
But if you had done it with this level of routing intelligence, you would have actually seen a fraction of the cost.
And here's what the cost would have been.
And so that's what we're going to incorporate the next time we do something.
YASH SHETH: And while not sacrificing accuracy, because that is the most costly.
JEETU PATEL: Absolutely, absolutely.
YASH SHETH: Yeah.
JEETU PATEL: Gentlemen, thank you for being at Cisco.
Thank you for actually doing all the amazing stuff that you're doing.
It's a pleasure.
And I'm sure we'll have many, many more of these conversations.
YASH SHETH: Thank you for-- ATIN SANYAL: Thank you.
YASH SHETH: --bringing us here, and it's an exciting time.
Let's go!
JEETU PATEL: It is an exciting time.
Loading video analysis...