Home

Why Most AI Agents Never Make It Past the Demo (MongoDB)

Why Most AI Agents Never Make It Past the Demo (MongoDB)

MongoDB Principal Technologist Devin Dickerson joins Jason Andersen to explain why AI agents fail on the path from demo to production, as data moving across disconnected systems adds latency, drift, and cost. For enterprise platform leaders, collapsing operational data, retrieval, and memory into fewer systems creates the trust and cost predictability that production agents require.

AI agents are easy to prototype but hard to run in production, and the gap usually shows up in the data layer. Every handoff between operational stores, vector databases, orchestration layers, and tools adds latency, drift, retries, and costs that a demo never exposes.

Jason Andersen, Vice President and Principal Analyst at Moor Insights & Strategy, is joined by Devin Dickerson, Principal Technologist at MongoDB, to discuss what has to be true before an agent earns a permanent place in production. Dickerson works with enterprise customers on these deployments every day. He argues that an organization's differentiated value in agentic AI comes from its own data, processes, and context, much of which lives inside the application stack and changes as the business changes.

That argument puts data architecture at the center of agent readiness. Dickerson's case is that trust, cost, and performance problems compound at the seams between systems. In his view, collapsing operational data, vector and hybrid search, and memory into one platform reduces synchronization work and keeps permissions consistent. It also makes clear who owns a failure when one happens.

Key Takeaways:

🔹 Prototyping agents is no longer the hard part. Almost anyone with a coding agent can wrap a model in a prompt and connect a few tools. The harder work is building an agent that runs on trusted data, delivers business value, and can be used responsibly in production.

🔹 Enterprise differentiation lives in operational data. Most organizations won't create value by building custom models. Their advantage is the data, processes, relationships, and context inside their application stack, which keeps changing as the business does.

🔹 Autonomy scales on trust. Dickerson compares agents to DevOps, where many teams reached continuous integration but stopped short of continuous deployment because removing the human required maximum trust in the system. For agents, that trust comes from fresh data, relevant retrieval, permissions, traceable actions, and continuous evaluation.

🔹 Governance is the mechanism that makes autonomy possible. For most organizations, the mission comes first, so security has to be balanced with delivery. Developers need visibility into what an agent can access, what actions it takes, and what data shaped a response, because accountability can't be delegated.

🔹 Agent costs compound at the seams between systems. Model calls are a small part of the bill next to context retrieval, token volume, data movement, synchronization, tool calls, retries, and human review. Bursty AI workloads spread across many dependencies are also harder to forecast, so average utilization falls short for capacity planning.

Enterprises that consolidate the data, retrieval, and governance layers beneath their agents will spend less effort managing handoffs. They'll also move more agents into production with predictable costs and clear accountability.

Watch the full video at sixfivemedia.com, and subscribe to our YouTube channel so you never miss an episode.

Learn more about MongoDB at mongodb.com.

Disclaimer: Six Five Media is for information and entertainment purposes only. Over the course of this video, we may discuss companies that are publicly traded, and we may reference their equity share prices. Nothing discussed during this webcast should be considered investment advice or a recommendation to buy or sell any security. We are not investment advisors, and you should not rely on this content as financial advice. Six Five Media collaborates with technology companies and industry leaders to produce research-driven interviews and multimedia programming for enterprise technology audiences.

Transcript

Devin Dickerson:

If an agent needs to move data between operational stores, a special vector database, orchestration layers, and tools, all of those are little failure points, little seams that cause retries, that cause context pollution, that could actually reduce the quality of a response while also raising costs.

Jason Andersen:

Hello and welcome to The Six Five. I’m Jason Andersen. And really, over the last year or so, what we’ve seen is enterprises struggling to move from kind of that great demo or that great pilot project with an agent and bring it all the way up to production scale. And our finding is that along the way, one of the key areas where the process breaks down is around the handling of data and context. Help better understand what’s working and what’s not in the field and get a feel for also what Mongo has to offer. I’m joined by Devin Dickerson. He’s the principal technologist at MongoDB, and he works with customers every day solving these problems. So over the next 20 minutes or so, we’re going to be walking through some of the most common questions that we’re both seeing in the field and hearing from our customers and clients, and talk about how we can get better at building durable and trustworthy agents. Devin, welcome to Six Five.

Devin Dickerson:

Hey, thanks for having me. Really excited for the talk today.

Jason Andersen:

Great, great. Well, why don’t we just dive right in, right? So, you know, how do we move from that kind of killer demo that the board sees, right? Maybe from somebody like yourself when you’re trying to position Mongo and turn that into what will ultimately become a very durable and usable production agent.

Devin Dickerson:

Yeah, so out in the field today, there is a ton of excitement around agents. I’m sure I’m not telling you anything you don’t know on that one. And the barrier to entry has actually gotten really, really low on the prototyping side of this. We’re at a point where basically anybody with a with their favorite coding agent account could wrap a model in a prompt connected to a few tools and maybe even call the result an agent. I think the hard part is not creating something that looks like an agent, but creating something that delivers actual business value and that can operate on trusted data and be used responsibly in production. So I think we’re moving past the part where it was actually really cool to be able to create something that looks or appears agentic because almost anyone can do that now. And it’s really about being able to create differentiated value for the business when the stakes are really high and really real. And I think that the organizations that I work with, they all have something unique to contribute to the agentic ecosystem. Because most organizations are not actually going to deliver their value in this space by creating custom models or things like that. What they actually have to contribute is data, processes, relationships, context. That’s where the value is, and that’s what organizations bring to the table uniquely. That’s why we can’t just use publicly trained models for everything, right? Because we need that unique operational context. And a lot of that data is inside the application stack. And if you want the agent to be operating on real-time information, the data inside the application stack is actually uniquely continuously changing. It changes when the business changes. And so I think the real key to unlocking agentic value really starts in the application data that organizations already have and can build on top of.

Jason Andersen:

It’s an interesting point because you mentioned applications, and you mentioned outcomes and timing- a lot of factors here. Any of those factors can have a big impact on the overall accuracy and the outcome of the agent. What does it really take to trust the agent’s answers? What’s that quality assurance that happens as an agent flows, especially when you have this constantly changing dataset?

Devin Dickerson:

That’s actually a really good question because if we want customers to build on their most up-to-date data, then we have to start solving problems about building on top of something that is constantly evolving and changing. And I think the reason that, first of all that we have to do this is because agent reasoning: we ultimately want correct answers, and we don’t get correct answers if the information is stale. We saw this a few years ago before we even had things like RAG, where if you asked ChatGPT for information about today’s stock price, well, the knowledge cutoff would have been before that. So that’s when we started to get smart about connecting to more up-to-date data sources, more up-to-date tools, and things like that. I think part of how organizations solve this is by reducing complexity. So if we think about all of the things that are required to bring fresh data into an agent’s reasoning, into its context window, we can really quickly start to think about a lot of different systems where all of this might take place. If you have, like, unique point solutions for all of the different ways that you might want to do that, you can very quickly end up in a situation where you’re creating latency; you’ve got all of these individual seams to manage. You also start to think about things like drift. So if I’ve got two separate systems where I’ve got the data that my agents have access to, and then I’ve got my application operational data, well, who’s going to keep those synchronized? That’s another seam we have to manage. That’s another headache that we have to deal with. I think one of the easiest ways to reduce complexity for customers is to collapse those seams. So if we can bring operational data and retrieval together, like vector and hybrid search capabilities, that’s sort of MongoDB’s approach. We add context, memory. Looking to the future, we’re thinking about things like runtime capabilities, all the things that you need to use that data responsibly. And I think that trust, and by the way, autonomy scales on trust. We saw this with DevOps a few years ago. Lots of organizations got to continuous integration. We never really got to continuous deployment. The reason for that is because continuous deployment takes the human out of the loop and requires maximum trust on the system. So we also have to figure out where those humans sit in. And if we acknowledge that autonomy scales on trust, then we think about where trust comes from. Data, relevant retrieval, permissions, traceable actions, and the ability to continuously evaluate all of these outcomes.

Jason Andersen:

Yeah, it’s funny because you mentioned the thing about everybody being able to build these agents. And it just kind of strikes me that, like in the early days of like web platforms and web apps, everybody was a UX designer because they all wanted their own things. And they never knew that what sat behind what that visualization had could have been remarkably complex, or even in some cases, impossible to produce without the steps you just mentioned, right? It was just kind of a massive gap, right? People, I don’t want it to look like this. I remember I was in the services organization, a services organization back then. And they would be, I would say, well, you know, that part of your web screen probably costs like $2 million because, you know, we have to do all this work to actually present it that way, the way you want it for that dashboard or that view. But, I mean, and that’s a good point. The backend stuff’s a good point because these folks, you know, they’re sitting there, they’re seeing the slick stuff, they’re getting excited, they’re feeling, oh wow, we’re now relevant because we’re doing AI stuff, right? And then you start having to talk about, like you were just saying, privacy, security, governance. And that kind of gets to be like a wet blanket, right? It’s like, people go, oh, you know. So how do you kind of provide that type of capability and not stifle that innovation, right? Like, how do you kind of, you know, don’t bring the party down, let’s keep it going, right- kind of thing is the vibe. But the truth is, is that, you know, none of those folks are going to want to hear They may get excited, but they also want to make sure they’re secure. They want to make sure that they don’t have a data breach or any types of those issues. So how do you kind of manage that balance, right?

Devin Dickerson:

That balances everything. So I used to work for the government. And if I went around and talked to, if I go talk to two dozen government program managers in charge of modernizing their application estate in these agencies, and I ask them what their top priority is, many of them will say security. I would argue that most organizations that that say security and governance don’t really mean it, because, and it’s because they don't actually have the luxury for that to be true. The truth is, is that for almost every organization private or public, their mission is their top priority. And their, their mission and their strategic objectives, and their biggest risk- the thing that can hold them back from achieving those objectives is actually security or governance shortfalls. And so, you can’t just say that security is the biggest priority. Nothing else matters. No, like you alluded to, you have to balance between innovation and control. Because if we don’t actually deliver on the mission, it doesn’t matter if you deliver the most secure product in the world that nobody ever uses, right? So we do need our developers to be able to move quickly. We and we need them to be able to build these agentic experiences and automate processes and deliver value. And that means they need to know what an agent can access, what actions it can take, what data it shared, what data was used to generate this response, right? You can delegate anything in this world, except for accountability. So we need we still need some sort of accountability for if something goes wrong, what happened? When did it happen? How can we remediate? And so I think that autonomy and trust are inseparable. And innovation, security, governance, also inseparable. Because very few organizations have the luxury of saying, all we care about is security, right? We have to be able to deliver that while still enabling our developers to build things.

Jason Andersen:

I mean, right. And if you think of what you just were talking about is absolutely foundational, then we get to the next kind of order of needs, and which can also be a barrier is cost and performance, right? I mean, you know, you can, you can kind of, you have to balance those two things in or mix those two things in as well as you get to scale, right? And, you know, AI is not like deterministic transactional applications;  very different, right? So, you know, how do you kind of, where do you bring that into the picture? And also, like, you know, how do you kind of bring that control into place?

Devin Dickerson:

There are a lot of different ways to think about complexity, because whenever you take these types of requirements on, I mean, one, the difference between a prototype and a production system in an enterprise is scale, governance, and security, foundationally, right? So that really makes things like governance the very mechanism that makes autonomy even possible. And I think getting there often requires figuring out a platform strategy that combines capabilities and pushes the complexity of those things down into the platform. So that the organization and their engineers and their builders and their product people, that they can focus on what they actually do best. From a bank, I want to deliver amazing banking experiences for my customers in a really secure way. I don’t necessarily want to have to be an expert on all of these platform trade-offs that can maybe comply with my multi-cloud strategy, that can help me evaluate in a model-agnostic way because we can’t go all in on a provider. A lot of it is about getting enterprise capability, things like secure agent runtimes, orchestration, identity, access controls, isolation, like observability, evaluation, all of these things from the platform. And not having to stitch it all together and have all of these seams that we have to manage because really, if I’m an enterprise, I want to focus on that sliver at the top, which is what we do best and how either how we make money or how we achieve our mission, not so much all of those in-the-weeds platform trade-offs.

Jason Andersen:

You know, it’s funny you bring this up because one of the things I’ve talked with other vendors like Mongo about is, you know, whether we like it or not, the AI unicorns, if you will, you know, your OpenAI's, your Anthropic's, they kind of set a price point, right? They go, oh, you can kind of come in, and it’s 20 bucks per user per month, or if you’re a super user, maybe it’s 100 to 200. Whatever that is. And the reality is, is that that’s kind of a, you know, as you mentioned before, it’s general-purpose. It’s not, it’s not tailored to the business needs. It’s not; it isn’t, doesn’t take into consideration all these other components you just mentioned; they’re missing, right? So, you know, so I guess the question I’m asking, you know, on behalf of customers, and customers asked us all the time is, Well, I clearly don’t want to just give everybody $20 a month general-purpose plans. So how do we kind of talk about the cost story in a more enterprise-centric way? And what I guess I’m getting at is what’s the real cost of kind of stitching together all these things? Versus maybe just doing it in one platform, like, you know, a platform like Atlas or MongoDB is a great starting point, right? So can we talk about that, how those costs, those kind of vital yet kind of unaccounted for in the public space or the consumer space, get accounted for into a platform strategy or the alternative, which is trying to do it yourself?

Devin Dickerson:

Yeah, I think the first thing that customers realize when they start building their own agents is how just how much of this ecosystem exists outside the model. And for cost, this is also true. Yeah, like if we were to if we were to paint a picture of the entire agentic ecosystem and what it takes to build an agent, models would be a surprisingly small part of it, like the actual model choice. And the same is true for cost. So the cost of an agent is a lot more than just the model call. Includes context retrieval, token volume, data movement, data going across certain boundaries that impose a tax, not to name any names, synchronization, tool calls can drastically compound costs, especially inefficient tool calls that are crossing into different platforms. Infrastructure retries and then of course there’s the just the time and labor tax of human review. And so, I always like to say that performance and costs- all of these risks compound at the seams between systems, at the boundaries between things. And so if an agent needs to move data between operational stores, a special vector database, orchestration layers, and tools, all of those are little failure points, little seams that cause retries, that cause context pollution, that could actually reduce the quality of a response while also raising costs. And I would also say enterprises do care about being able to predict costs. And when you have all of these workloads spread out across all of these different dependencies, it becomes even more difficult to forecast. And we know AI workloads are super bursty. Anyone who has used even the coding tools knows just how fast you can make these workloads really sweat. And so average utilization is probably not going to cut it for capacity planning either.

Jason Andersen:

You know, the flip side, it’s funny you bring that up because even the flip side of that is you’re even starting to see again with the consumer-oriented type of stuff where they’re coming back and saying, hey, you should use our scheduler. That’s not convenient for you. That’s helping them because you’re able to kind of shift some of your requests. So they want you to batch your jobs. And it’s kind of like, It’s almost like going back all the way back to, you know, 100 years ago when people were doing time-sharing on mainframes. It’s like, well, this is crazy. Now I’m setting up my jobs to batch at certain times of the day and run by themselves. Right. So it’s kind of your point’s really well taken. But I guess, you know, further is I think I’m seeing a trend, and I definitely see it when I look at what you guys have been working on and working towards at Mongo. The idea of, you know, just being the platform being able to do as much as possible as it can for that customer, right, versus, you know, having to go, well, geez, I really want to use this harness with this model with, you know, with this database with these tools, you know, and then, and then, you know, forgetting the entire part about, you know, evaluation and making sure correctness happens, which is a whole different set of tools, right? So. I mean, it really stands out with me that that’s an opportunity customers haven’t always thought about. So I guess maybe to wrap up the call is like when customers like how how are customers finding you? How are customers coming to you and approaching you with these problems? Are they coming to you when something breaks? Are they coming to you? Are they smart enough to come to you proactively and say, hey, we’re already using Mongo, so now we want to do… What’s that journey look like?

Devin Dickerson:

We see the whole gamut, but I will say that a common thread really arises from the legacy stack. And that’s where you really see the cost of the fragmented agent stack. And what. I mean, by legacy stack in this context is like, look, we don’t always have the luxury of starting from scratch. If we want, we talked about it earlier, if we want to deliver value in the agentic ecosystem, most organizations are going to do that with their own data. If I need that data to be real-time, do milliseconds matter? How important is timeliness? All of that kind of leads me to needing to operationalize my application data in the agentic stack. And I think that when you start to work in those types of use cases, that’s when you really see it. And it’s not just your procurement spreadsheet that this shows up in. It’s also in the complexity that people have to deal with and how long things take. It shows up in integration work, synchronization pipelines, the latency between systems. The failed security review, the operational handoffs from one team to another. How many teams does it take to screw in a light bulb, right? Well, quite a few if it’s an agentic light bulb. And so I think from a Mongo perspective, when the operational data lives in one system, the vector database lives in that alongside that operational data, the memory is alongside that operational data. And you can leverage an agentic platform with observability and evaluation, and governance. Each of these boundaries that we are collapsing is an opportunity for better context with consistent permissions, better traceability, and clear ownership when something does fail, because these are newer workloads, so things are going to fail.

Jason Andersen:

That’s a great answer. Well, listen, this has been super informative, like, you know, really getting your perspective. I appreciate it.

Devin Dickerson:

Enjoyed the convo. Thanks.

Jason Andersen:

I’m just going to say to everybody else, thanks for tuning in today on The Six Five, and don’t forget to hit subscribe. We’re also very active on the socials, so you can check out this and all of our videos there. And if you want to look at everything, you can look at sixfivemedia.com.

MORE VIDEOS

How Far Away Is Quantum, Really? Timelines, Encryption Risk, and the Talent Gap

On The Main Scoop with Daniel Newman and Greg Lotko, quantum industry veteran Denise Ruffner explains why 2030 is an aggressive timeline for quantum computing, why post-quantum security can’t wait, and how the industry can widen its talent pipeline. Organizations that start their quantum-safe transition now will be ready when quantum moves from research to advantage.

Intel on Why Agents Need More CPUs, Not Just More GPUs

At OCP Global Summit 2026, Intel’s Jeff McVeigh explains how two years of x86 ecosystem collaboration and open standards are shaping AI infrastructure as agentic workloads put CPUs back at the center. For enterprises, shared interfaces make it possible to deploy AI inference inside existing data centers.

The Operations Problem Agentic AI Just Made Urgent

HPE’s Sylvia Hooks joins Tiffani Bova to explain how agentic AI is pushing IT operations toward self-driving infrastructure, and why trust at machine speed depends on observability, guardrails, and recoverability. For enterprises, how fast operations can automate and govern agents sets how far agentic AI scales.

See more

Other Categories

CYBERSECURITY

QUANTUM