AI Agents Acting on Stale Data Make Bad Decisions…Fast

AI agents act at the clock speed of computers. Enterprise data architecture was tuned to the clock speed of humans: collect data in batches, centralize it, and review a report in the morning and again in the afternoon. When agents act on stale or incorrect data, they keep deciding, and taking incorrect steps faster than any human can fix.

Brad Shimmin, VP and Practice Lead for Data Intelligence, Analytics, and Infrastructure at Futurum, talks with Shaun Clowes, Genral Manager of Confluent and Data at IBM, about Streamhouse, an emerging architecture category built to keep agents working from the current state of the business.

The lakehouse stays. Clowes credits it with teaching enterprises to model and see across the business, but the decision for data architects is which processing moves left, upstream into a Streamhouse. His method is to start with a customer experience agents could transform in real time, then identify the data it needs continuously.

Key Takeaways:

🔹 Agents change the clock speed data has to meet. Batch architecture assumed a human would read the output, act, and check again later. Agents act at computer speed, and a business may point thousands of them at one problem.

🔹 Stale data misdirects agents. Agents acting on outdated or incorrect data keep deciding and make wrong calls faster than people can correct them. The business loses the upside of automation and absorbs the risk.

🔹 Current state needs continuous, decentralized, always-on data. A Streamhouse processes changes as they happen, works with data where it’s created instead of through brittle pipelines into one store, and stays up under the load of thousands of agents.

🔹 Lakehouses model the business, but agents need that model current. An airline rebooking agent needs seats, gate status, weather, and crew at once. Clowes says lakehouses aren’t fast or reliable enough for that, so those workloads move left.

🔹 Streamhouse ends the federated-versus-centralized trade-off. Data fabric, data mesh, and data products all aimed to separate data from the systems that produce it. Clowes says moving, processing, and querying data in motion is now cheap enough to do that continuously.

Clowes’s examples, from flight rebooking and crew rescheduling to logistics routing, share one requirement: every agent can see the current state of the business. That leaves people to handle the exceptions.

Watch the full video at sixfivemedia.com, and subscribe to our YouTube channel so you never miss an episode.

Learn more about Confluent at confluent.io.

Disclaimer: Six Five Media is for information and entertainment purposes only. Over the course of this video, we may discuss companies that are publicly traded, and we may reference their equity share prices. Nothing discussed during this webcast should be considered investment advice or a recommendation to buy or sell any security. We are not investment advisors, and you should not rely on this content as financial advice. Six Five Media collaborates with technology companies and industry leaders to produce research-driven interviews and multimedia programming for enterprise technology audiences.

Shaun Clowes:

The Streamhouse is the middle ground. The Streamhouse lets you think about data separate from its systems and think about it in a reusable, continuous sense. They're not forced to make a decision between federated data access and centralized data access. There is a middle ground, and that middle ground turns out to be incredible.

‍

Brad Shimmin: 

Hello and welcome to this Six Five virtual webcast. My name is Brad Shimmin. I'm with the Futurum Group and today we're here to talk about something called Streamhouse and why AI agents and applications need continuously current business context. I know that's a mouthful, but I am joined by Sean Klaus, who's the general manager of Confluent, who everyone here I'm sure knows. And he's also a manager for data at IBM. And we're going to talk about Streamhouse. So Sean, thank you so much and welcome to Six Five. Hey, Brad, it's honestly awesome to be here.

‍

Shaun Clowes: 

Looking forward to this.

‍

Brad Shimmin: 

Yeah, so it is, you know, everyone knows streaming data. Everyone knows Kafka and Confluence and how important a role those play in data estates across all businesses. But it's interesting to me because, you know, when we think about data estates now and how those are being used for agentic AI in particular, you can see that there's a lot of stress happening at the corners there, because a lot of those estates are really built for batch processing of data. So, you know, we collect the data, we try to centralize it, and we kind of analyze it for later, but Once you're using agentic systems, that puts a huge strain on the ability of those agents to sort of act as the business events themselves unfold if you're using that stale batch data. So I wanna ask you, what is it about this moment that makes this new architectural category? Because I think we can call it that, can't we? Streamhouse is a category. What makes that necessary right now?

‍

Shaun Clowes: 

Yeah, I mean, I think that's a fantastic question. Broadly speaking, we're kind of in this incredible transformation of technology. It's like we're clearly living the next era of technology and we suddenly have these incredible brains, honestly, brains, LLMs that can take incredible action. So instead of just whatever we can declaratively code, write in imperative code, we now have the ability to literally ask a computer to do something and have it actually go and successfully do that, which is kind of mind-blowing when you think about it. But you don't want to be able to ask this agent things that it was trained on, things about Wikipedia, things about the world that it knew. You want to be able to apply it to your business problems. And when you think about that, the challenge is that mostly all of the different technology we have built and our data architecture in the enterprise was really designed for humans. It was designed for the idea that the purpose of the technology is to bring the data and the capabilities to the human. The human will interpret the world, understand the world, take action back, then interpret the world again. And so it's kind of, it's tuned to the clock speed of humans. And it turns out that humans actually aren't that fast. We might look at a report in the morning, we might look at the same report in the afternoon, we might process some inbound requests. But realistically speaking, humans move very slowly. We sense the world, we act on the world at the speed of humans. And these AI brains are going to be acting much more quickly. They're going to be acting at the clock speed of computers. They're fundamentally able to act much more quickly than any one of us can. And what's even more significant than that is not only are these engines more capable than us or able to move faster than us, we might have many thousands of them on any particular business problem. So each individual agent is faster and armies of agents are obviously much, much faster against whatever problems we put them against. So that means that we need to be, that'll only be as good as the quality and the speed and the accuracy and reliability of the data that we can bring. They can't sense anything about the world without that.

‍

Brad Shimmin: 

Yeah, no doubt. And it is interesting, isn't it, about the clock speed of this, because you're right. We're used to dashboards. That's where you get your insight from business data. And if something's not right, you might like, oh, I'll have to ring up Steve over in the data department for data scientists, and he'll know what's going on. Agents don't wait for that. They will take action because they're goal oriented, not task oriented. And that means like big ramifications. And actually, you know what? Let's talk about that for a second, because it's an inconvenience when a human is using data that's like an hour behind. But for an agent itself, when it's using stale data, what's the real business risk there? What does it look like for architecture that needs to kind of keep those applications and agents working in the actual state of the business? It seems like a big risk to me.

‍

Shaun Clowes: 

Yeah, totally agree, Brad. I think ultimately, if the agents end up acting on data that's out of date, it's stale or it's incorrect, they're going to continue making decisions, and those decisions are going to be wrong. And they're going to make decisions at the speed of computers. So they'll make bad decisions extremely quickly, and potentially much faster than any human can come back behind them and try and fix all of the mistakes that they're making. So not only are you going to miss the opportunity, without a clear, coherent view of the world, the ability to understand what's going on in the business quickly, without that ability, not only will you get the value of the agents, you'll get all the risks, you'll get all of the worst outcomes that are potential from these agents. And so when you think about what you're really trying to do, you're really trying to provide a data architecture that means that every agent and every system can see the current state of the business. And that may sound really simple. But what I mean is like literally to be able to see across the business, all of its kind of thorny systems of record and all the different systems that are holding, you know, information about different business processes and things that business cares about, you need it to be able to see across all of those things. And you need it to be able to see across it in a continuous sense. So, however the business operates and updates and the clock cycle of the business, you need that to be evident to the agents so that they can sense it and react to it. And if you think about that, this Dreamhouse architecture that we're talking about is designed to meet that capability. So Streamhouse is designed to be a type of data architecture that delivers three things. Firstly, it's continuously up to date. So as things change in the business, there's a continuous processing so that as change happens, that is then manifested out into the world and it becomes part of the state that's then accessible. You need it to be decentralized, so you don't have the time or the scope for brittle pipelines to be able to suck data out of systems and try and land it in one big central place. You need to be able to work with data as it is being created, as things are changing inside the business, you need a decentralized architecture. And finally, at the end of the day, you need this architecture to be business critical. So basically, it's, you know, if you think about these agents are running around making actual business decisions, the fabric, the data system underneath them also has to be able to meet the scale of those agents, there might be 1000s of them, their demands for data, like the load that they put on the relevant systems. And it has to always be up, because if it goes down, then clearly the agents are going to stop working. And they're going to end up making terrible decisions, like we talked about a moment ago.

‍

Brad Shimmin: 

Yeah, we don't want that. And when I try to visualize that, I try to think of like an airline booking system like Sabre. And it has to be highly concurrent. And it has to make sure to the microsecond that the exact state of that airplane is reflected in there. That's not an easy task. And imagine doing that in the complexity of a business like a supply chain. you know, trying to track where all the goods are for for whichever you're building. That's an incredibly complex problem. So I can see why. I can see why we need an idea like the stream house. But we have, you know, an idea that companies have been investing in for years already in the lake house. And it sounds, you know, the way you described it, if you take the real, you know, the real time off the table there, what the architecture you described for the stream house sounds a lot like the lake house. that we have. So my question to you, Sean, is, you know, how should leaders really kind of decide on which workloads belong in the lake house versus those which should be in the stream house? Do they need to decide?

‍

Shaun Clowes: 

I think, honestly, we've learned a lot about modeling and understanding businesses from all of the work we've all been doing in lake houses over the years. And it's been transformative. Like we've been able to see across businesses in a way we haven't been able to before. And we now need an architecture that can meet this moment. And the problem with the Lakehouse architecture is, like I said, it's not fast enough, it's not reliable enough to really give that current business state right now. So when organizations are looking to, for example, deliver, you know, real transformation with autonomous AI agents, you brought up a really good one. Like when it comes to, say, for example, travel booking or airlines, You know, if I'm if I have a flight disruption, I want the agent to be able to or I miss my flight, I want my agent to be able to rebook me and to just like, let me know. And remember that I won't be the only one having that experience, I will be one of the many customers at that airline who's been disrupted. So in the process of the agent rebooking me, it needs to make certain it's not booking me in a seat that you've you've now taken. It needs to know, is the plane still at the gate? Has the plane literally left the gate? I really need the agent to know the current state of many different things all at once. The state of the weather, the state of the planes, the number of available seats, whether or not the crew is on board, all these interesting questions. And so when you're trying to deliver these, what I would say, honestly, you're going to be incredible customer experiences that are going to drive customer loyalty, drive customer revenue, and reduce costs. You basically find those types of things where you can really deliver an outrageous experience, like an incredibly good experience. And then you just go left, you go, okay, now what data would I need? to be available continuously, up-to-date, mission critical, decentralized, you go, okay, what data would I need to deliver that to truly make it incredible? And then you go, okay, no worries, I will move that processing left out of the lake house into the stream house where I can get that exact view and I could really make the experience work and work at scale.

‍

Brad Shimmin: 

So let's talk about the decentralization. Okay, because, you know, it is very tempting in the enterprise. And I remember the data warehousing days and building data marts, and the pressure to try to centralize all of your information. I'd love to ask you, because you've seen this in action for some years now, what changes for both the data architects that are building these systems out and for the companies that they work for when they can basically query data that's in place instead of having to put everything in a mart somewhere?

‍

Shaun Clowes: 

Yeah, honestly, I agree with you. There's always been this temptation to get the data from these source systems and try and centralize it. And we've had movements like Data Fabric, Data Mesh, Data Products, and we've had all these different movements. But what they all have in common is we're trying to be able to separate the data from the system that produced it. into a reusable thing that is rich and valuable that I can then use other places. And honestly, I think that's exactly what we're saying in the Streamhouse too. But we're saying that we can't do that by trying to… It can't be federated across all the source systems and it can't be centralized. So what's the alternative? What's the middle ground? The Streamhouse is the middle ground. The Streamhouse lets you think about data separate from its systems and think about it in a reusable continuous sense. And the reason that that's now possible is we've seen a huge amount of advancing in the underlying technology. It's now very inexpensive and easy to move data continuously. It's very inexpensive and easy to work with it as it's moving. It's very inexpensive and easy to query it in its kind of moving sense. And so what that really means for architects and people building systems all over the world, is that they now have a new opportunity. They have a new way of working with data. They're not forced to make a decision between federated data access and centralized data access. There is a middle ground, and that middle ground turns out to be incredible. It's actually the exact fabric or foundation that you really need for this agentic transformation.

‍

Brad Shimmin: 

I love that because we did all see this like, let's build a data mesh. And it's these pie in the sky ideals are very hard to achieve in real life. And so the middle ground, I will always go for.

‍

Shaun Clowes: 

every time.

‍

Brad Shimmin: 

So to sort of conclude our chat, I would love to get real for a second, okay, Sean, and talk about what businesses, let's say businesses that aren't accustomed to this, that don't use Confluent Kafka, that don't understand streaming, Saying to them, yeah, your data doesn't have to just tell you things. You can actually do things with this data once you're able to see it as it unfolds in real time. So if you can, give our listeners some insight into what you've seen in terms of what organizations can actually achieve when they sort of have agents that can act on the current state of the business.

‍

Shaun Clowes: 

Yeah, I mean, this is where AI goes from cool technology to incredible customer outcomes. This is where you really unlock the ability to transform your business in really meaningful ways. So obviously, in the travel industry, you heard examples of automated flight rebooking agents. There's actually crew rescheduling agents. There's baggage management agents. that can really transform the operations of an element. You have people building agents in the hospitality world. So for example, helping frequent travelers find the perfect hotel for them, helping them manage multiple stays across multiple different properties while their travel experience is changing, helping the staff at hotels and manage the hotel much more effectively without having to you know, deal with customer inquiries and customer complaints all the time. In logistics, you have examples where people are able to treat the whole logistics network and understand what's happening with weather, with vehicles, with drivers, and effectively optimize the movement of goods and services and vehicles, you know, and really just deliver incredible outcomes to customers getting their packages faster, getting the things they need faster, while also reducing costs. And so what these use cases kind of all have in common is that if you can think about your business as the things you care about, like customer, in financial services might be customers and accounts and payments and mortgages and policies, like when you get access to that, the things that your business cares about, and you can have that data available to you, like at your fingertips to every agent, then you can effectively present everything your business does, you can automate almost anything that you can think of, and you can make it just incredibly compelling to customers to be able to get at everything you do, and even for your own staff to be able to remove all of the hard work of optimizing, sensing, reacting to things going on inside your business. You don't need humans in the loop anymore. Instead, the humans just deal with any exceptions or they deal with directly interacting with customers around exceptions.

‍

Brad Shimmin: 

Yeah, I think I think every human would agree that and take, for example, a small company that's a restaurant and that typically takes a full time person that's doing nothing but managing the staff and staffing schedules and managing the purchasing that morning for the food that they're going to serve. I think they would all say yes, please. Sign me up for this. Sean, thank you so much for this chat today. I really appreciated the conversation and enjoyed it and for your insights. Thank you. It's been a pleasure, Brad. Thank you very much for having me. And everyone out there, thank you for tuning in. We appreciate you. And so this has been the Six Five Virtual Webcast. And that's where you are. Don't forget to hit subscribe and you can follow us on the socials, the typical And we would invite you to check us out on the sixfivemedia.com for more. We publish a lot of material there. So we invite you to come take a look. And until then, we'll see you.

‍