Inside Signal65 PINNACLE: How Claude Fable 5.1 Broke the Fabrication-vs-Capability Tradeoff

A new AI benchmark just caught something nobody expected. Two days after Signal65 launched PINNACLE, a fresh model release flipped one of its most consistent early findings.

Ryan Shrout, President of Signal65, walked through the results with JV Roig, Senior Generative AI Platform Engineer at Kamiwaza.ai, and Mitch Lewis, Senior Performance Analyst at Signal65, in a conversation about what Pinnacle is measuring and why it matters.

PINNACLE is built to score what Shrout calls “correct work,” by looking at whether an AI agent actually finishes an enterprise task correctly. That's a different question than raw token throughput, and Shrout says it's the one enterprises actually care about.

Early results across 44 models found a pattern that held until this week. The most capable frontier models scored highest on raw agentic performance, and also fabricated more often. Anthropic's Claude Opus and Claude Sonnet showed higher fabrication rates than expected, and OpenAI's GPT-5.6 family scored a full tier lower.

Roig traces the pattern to over-helpfulness baked in during post-training. When a model can't find data it needs, it substitutes its closest guess for the missing data, a habit that looks harmless in an interactive chat and becomes dangerous in an unsupervised, agent-run batch process.

Claude Fable 5.1, tested within days of its release, broke that pattern. It kept frontier-level agentic performance and posted a fabrication rate under 1 percent, one of only two models to clear that bar and roughly a tenfold improvement over Opus. It landed as the most expensive model per completed task.

Key Takeaways:
🔹 PINNACLE scores tasks with deterministic, pure Python code that checks generated CSVs, JSON files, and reports against exact, pre-defined correct answers. Roig built it this way specifically to avoid stacking one probabilistic AI system on top of another as a grader.
🔹 Each test scenario runs 40 times with randomized data to resist contamination and measure consistency. A model that scores 80 percent across 40 runs of the same scenario reveals how consistently it performs, something single-pass leaderboard scores don't capture.
🔹 Clean and messy data produce very different results. Of the 44 models tested, nine scored above 95 percent on PINNACLE's organized-data scenario. On the messier, more realistic enterprise scenario, that dropped to two.
🔹 GPU generation made a bigger difference to agent capacity than raw throughput. Signal65 found a single NVIDIA B300 node supported 320 instances of one Qwen 3.5 model, compared to 88 on an H200 node, a sixfold jump in how many agents the hardware could run at once.

Shrout says Signal65 plans to re-run PINNACLE as new models ship, treating it as a living benchmark that updates in near real-time.

For the full PINNACLE methodology white paper, live model-to-model and GPU-to-GPU results, and the TCO calculator, visit pinnacle.signal65.com.

Watch the full video at sixfivemedia.com, and subscribe to our YouTube channel so you never miss an episode.

Disclaimer: Six Five Media is for information and entertainment purposes only. Over the course of this video, we may discuss companies that are publicly traded, and we may reference their equity share prices. Nothing discussed during this webcast should be considered investment advice or a recommendation to buy or sell any security. We are not investment advisors, and you should not rely on this content as financial advice. Six Five Media collaborates with technology companies and industry leaders to produce research-driven interviews and multimedia programming for enterprise technology audiences.

Ryan Shrout:

Hey, everybody, welcome to a Signal65 video insights. I'm Ryan Shrout, president at Signal65, and I've got some great guys with me here today. I'm joined by Mitch Lewis. Mitch, how are you today?

‍

Mitch Lewis: 

Doing good. Always happy to be here.

‍

Ryan Shrout: 

And our engineer extraordinaire, J.V. Roy. J.V., how is it over there early in the morning?

‍

JV Roig: 

Yeah, super fine, 6 a.m., but I'm very excited, yeah.

‍

Ryan Shrout: 

Thanks for getting up for us, I appreciate it. So what we wanted to do here today is there's no third-party guests, there's no vendors on the call, right? This is us, and we wanna talk about Signal 65 Pinnacle. And we have been working on this project for what seems like a very long time. But in the course of software development, maybe it's not that long, but in the era of AI and kind of how fast all that stuff moves, it's definitely very quick. Maybe the first thing to do is kind of give an overview of what Signal 65 Pinnacle is and what it does, and kind of why we wanted to build it, right? I think the very short managerial position for this, from my perspective, is that Signal65 Pinnacle is a new AI benchmark focused on agentic functionality. And what's unique about it is that it measures correct work. And that's a phrase you'll probably hear a lot from us. I've been writing it a lot. I've been using it a lot in social media as we kind of talk about the release of this test. And this is about looking at model performance and model capability. You know, which models finish these enterprise agentic jobs with the most token efficiency or with the most accuracy and the least fabrication. And I'll let JV talk a little bit about what all that means. But it's also about looking at you know, node-to-node performance. How does one GPU compare to another GPU? How does one platform compare to another platform in terms of your token throughput? But more importantly, what is your agentic throughput? How many agents can these different GPUs and nodes and configurations really support with all these different models? Because that's really what enterprises are after. They're not after a tokens-per-second rate. You know, that's an important piece of the puzzle. They're after how much work can I get done consistently on this? And then the third piece of it is really, you know, how do we bring in the TCO and the ROI metric of it. So we can talk about what is the cost per piece of correct work. And then we can talk about how to frontier model APIs compared to on-prem solutions when you're renting a node versus you're buying tokens off of Anthropic or OpenAI. And Pinnacle allows us to do all of that. And it's an incredibly awesome tool that we're just getting started with just launching this week, but already lots of big things are happening. So let me do this. Let me start with a question for you, JV. When we started talking about this, we weren't gonna build a new benchmark. We were gonna use benchmarks. And we kind of wanted to talk about what were the right tools to use. Every other, Benchmark has leaderboards and grading systems, but what did you see that was missing from what existed in the world before to what we have been able to build?

‍

JV Roig: 

So, you know, this was a very core problem we had, especially, you know, here at Kamiwaza AI, where we deal with customers. And when I say we deal with customers, so we have customers where we connect, you know, we handle their AI orchestration, connecting all of their enterprise data, different regions, different places. Of course, the success criteria is when all of the piping is done, the customer workflow, the use case actually has to succeed. The customer has to be happy. So we had to figure out, okay, for particular use cases, what kinds of models, which models in particular, given budget constraints or size constraints, which models would actually succeed? And what we found was, looking at leaderboards, like all of these traditional leaderboards, especially, oh, what's your MMLU score? Those things don't have any connection, any relevance, it's not actually causal to any success indicator. So what we needed to do was, okay, so how do we know? How do we know? And so that's what led us to develop alongside our orchestration technologies to develop the foundational agentic AI simulator technology like Picard, Kami, and Riker, which we use to create the foundations for what we've now released as PINA.

‍

Ryan Shrout: 

Yeah. And it's something actually that I think is important, right? The company that you work for, Kamiwaza, is deeply embedded in the agentic deployments of AI. Right. And so you see firsthand, what are the stress points? What are the breaking points? Uh, what works and what doesn't for all these enterprises. And I think there's, there's a, there's a very valuable angle there about what pulls through in the actual enterprise deployment side of things. that maybe a generic benchmark or, you know, some replay that somebody found online to play back and kind of get maximum throughput on tokens per second kind of misses. And Mitch, this is also something that you see all the time in our work on other projects, right? Is that at the end of the day, people aren't looking for, you know, maximum throughput, just like in CPU performance. We don't just talk about the frequency or your memory bandwidth. We want to get to actual kind of business outcomes.

‍

Mitch Lewis: 

It's a little bit of an issue with benchmarks in general, right? They're simulated, they don't always translate exactly to real work. But I think when you get into AI and specifically agentic AI, it throws a whole nother wrench into it because there's this level of intelligence. And so maybe you're just looking at tokens because that's a nice number, it's a nice stat that you can kind of say, hey, look, we have this many tokens per second. But did the model actually complete the task? Are you updating databases? Are you getting correct answers out of CSV files? All those things that you might have an agent go do, even if your tokens are really efficient, if you're not doing that right, an enterprise doesn't care about that model or shouldn't care about that model, right? So it's all these things. It's not just the performance. but it's the performance and the cost and the intelligence, but all kind of have to link together. And I think that's kind of the goal here at Pinnacle.

‍

Ryan Shrout: 

One of the things that I'm really proud of in the early pieces of this is getting the support of AMD and NVIDIA kind of very early into the release of Pinnacle, right? They've been able to supply us hardware. They're supplying us access to engineering. They're very interested in the methodologies and the processes that we've built, and they see how Signal 65 Pentacle is kind of differentiated from other tests. And this is not meant to slam against other benchmarks or say that there's not value in what other pieces of what other benchmarks are looking at, but that what we're building and what the goals are and what our roadmap looks like is interesting. So it was great to get the support from NVIDIA and AMD for that. JV, what I want to ask you, and this is a difficult one, because I know how deeply you are invested in the software that you have put together for Kami and Riker and Pinnacle. But in like three minutes, can we talk about what actually happens when a model gets tested? Like, what is the flow of what happens in Pinnacle?

‍

JV Roig: 

When a test starts, a model will see itself in its own sandbox. Sandbox is just a Docker container. This is a Docker container that's instantiated strictly for the test. What's in that sandbox is we have certain scenarios that we have created for LLM, which mirror certain professional realities. By professional realities, I mean, these are tasks, very long running tasks that were modeled according to the US Department of Labor Occupational Informational Network. So these are activities that the Department of Labor has created a taxonomy for. so that several different levels of abstractions of activities lead to different occupations. So we take as many of these different abstracted tasks We model all these different capabilities into the scenarios so that we cover as much of them as we can, so that we know the kinds of common sense and task capability that we ask of the LLM are things that we know will translate into enterprise, meaning for example, they will have to look at CSVs, they will have to look at sales reports, they will have to look at surveys, they will have to look at product defects, they will have to create reports, they will have to go create communications to vendors and to suppliers and to executive and so on and so forth. So now these scenarios and one of the key design constraints we make from our framework is that we want things to be contamination resistant. Now, we can't ever guarantee there will not be contamination, but we are as contamination resistant as we could possibly do right now. And the way we do that is when the scenario is instantiated, So for example, the task is step one, read this giant CSV, see how many employees are compliant according to rules X, Y, and Z. So maybe that's just step one. That CSV will always exist in the sandbox. But for each instantiation of that test, the contents of the CSV will be different. So we randomize the data, thresholds, and other information in the test so that every instantiation of the test is going to be unique. So the answers will be different depending on the instantiation. Plus, we don't ask them one question. When they go through the task, they have 7, 8, 9, or 10 deliverables. And a certain deliverable might be a giant JSON file, like a hundred names, because maybe that deliverable was how many employees are going to get promotions or bonuses, or how many employees are going to have a PIP. Now, because these were all dynamically generated, at the very end of it, another key design constraint we had was we absolutely did not want LLM as a judge to be the scorer. Because there are just so many documented in research biases about LLM as a judge. That's the whole trouble of you know, if you start with an LLM at this end because, oh, let me evaluate an LLM because, you know, they're kind of stochastic probabilistic. And then at the very end you say, okay, the way I'm going to evaluate this noise is by, at the end, adding another stochastic probabilistic. It just doesn't fly for enterprise, right? So in these scenarios, since we control the generation of all of the data and the scenarios and the problems, at the very end, we have a deterministic code path that checks for correctness, meaning we absolutely guarantee 100% correctness in, oh, this CSV is correct, he gets a point. This JSON is correct, he gets a point. The text file that he's submitting should have 100 names. Those 100 names are the exact 100 names that are needed. So our scoring is absolutely deterministic, pure Python. There is no LMS adjustment, I guess. So all in all, we run multiple, multiple samples. So we have about seven general scenarios that are just jam-packed with all of the generalized work activities from the Department of Labor informational network. Those seven scenarios get pre-run over and over. with different instantiations. So we call them samples. We sample them at least 40 times. And that strictness is needed because again, if the environment also changes, then we have to see how does the LLM cope when, yeah, maybe in that instantiation, which was kind of lucky, maybe he got that right. What happens when it's a different set of randomization? What if it happens that there is only one person who needs to get promoted or what if there aren't any people to get promoted? Is he suddenly going to make things up? So we run those scenarios multiple times and we get to see not just, hey, what's your overall score? We actually get to see, wow, you run the same scenario 40 times, you got it like 80% of the time, which itself is a different indicator. right, where it's not just, oh, you got 80% out of a test, where you got 80 questions out of 100. In this case, what we see is, wow, this is the exact same scenario. We just kept running you. So is it because as the data gets randomized, you have different capabilities, or it's just because stochasticity, which itself, you know, is a different thing.

‍

Ryan Shrout: 

So, I mean, that's kind of the overview of how the model itself gets tested. And there's For anybody watching, having been a part of this development process forever, JV would talk about this stuff for hours if I let him. I would encourage everybody to go to pinnacle.signal65.com, download the methodology white paper that goes through all the details about what the testing process is, what the Kami testing process is, what the Riker testing process is that tests you know, accuracy versus fabrication and all these other things. It's awesome, right? And we also dive into how do we measure performance? How do we measure the throughput capability of agentic models on say, you know, B300 versus H200, right? And some of that is included in there, right? I wanna spend a few minutes at least talking about kind of like what, were a couple of the key things that we walked away with in these first sets of results that were kind of interesting. The first one is, you know, this line of Clean data, one of the things we didn't talk about is we kind of run it in two scenarios, right? One where data is clean slash, you know, organized in a nice tidy pile in this fictional world of where data is all where it's supposed to be and everything is organized and named and numbered correctly. And then in a kind of a messy environment, which we believe better simulates realistic enterprise deployments where We've got data in some piles over here, some over here, some people have organized it very well. Sometimes it's in a flat file, sometimes it's in a database, right? Like there's all kinds of ways that this stuff could look up and we run some of the scenarios in both ways, right? We'll call it clean and enterprise or messy. It seems like for many higher end models, like this clean data scenario is, getting close to solved. I don't want to say solved because nothing's hitting 100%, but the enterprise kind of realistic messy configurations are far from it. Is that kind of what we saw, JV, in those initial data points?

‍

JV Roig: 

So in an organized scenario, what's interesting here is this is one of those nuances in our testing by design aimed towards helping enterprise understand what model to use for a different scenario, right? So when we're talking about the enterprise and messy environment, fewer handholding available. So you can imagine if you were in an enterprise and you were just using some general agent for some ad hoc tasks, whether you're using something like Cloud Work, Cowork or OpenClore or whatever, right? If they let you use OpenClore in the Enterprise Windows, right? So it's just, you have this agent, it's somehow connected to all of the data that you're allowed to access, whether it's like some SharePoint with bunches of like stretches or whatever, and some other different databases. And then you just ask it to do stuff. So there has not been prompt engineering, there has not been guardrailing, there really hasn't been workflow optimization and architecture in that scenario. It's just, here's the OpenCloud agent, here's the real world, go do this thing that I just told you about right now in one minute without really explaining to you all of the rules. Right. It has to go handle all of that. That's a super hard scenario.

‍

Ryan Shrout: 

It is. And I think, Mitch, you know, again, kind of looking at it, our world of other projects, you know, even when we're working on, you know, maybe even things that are using AI tangentially or they're not directly related to testing models. Right. What we normally are provided with target data sets are these kind of messy, noisy type configurations. Right.

‍

Mitch Lewis: 

Yeah, and I think it gets back to like, you know, how you open this up, like measuring real work. So not only are, you know, agents, you know, they're actually doing real tasks, but we're testing them in different environments, because not everyone's going to have a perfectly nice, clean environment. So I think to put some numbers by, I think what we found is out of the 44 models that we had run in the governance scenario, the nice clean scenario, nine of them were scoring above 95%, 95% or more. Once things get messy, I think it dropped down to two. So it's, you know, there's a clearly, you know, a change there. And it's like, okay, that's not real work, that real environment, when things aren't laid out perfectly for you.

‍

Ryan Shrout: 

It's also worth noting, you said 40, was it 44 models that we had gone through We added another one today as we record this, right? Fable 5.1 came out, ran the Signal 65 Pinnacle Benchmark across it, spent the day today kind of analyzing those results and writing that up. And that was interesting because literally on day two, day three of the benchmark being live, now we have a new crowned leader on the model board, right? And, you know, It was very interesting to see kind of one of the main differences is that, you know, we had found that the high end models, the smartest models, mostly the frontier models, the API based models, were the smartest, but made stuff up the most. They kind of fabricated more frequently. Fable 5.1, I think JVA in our results kind of reversed that, right? Or corrected it, right?

‍

JV Roig: 

Right. Yeah. So just to put the nuance on it, they weren't necessarily fabricating the most. So there were definitely non-frontier and open models that had like much worse fabrication. But the observation was that Contrary to expectation, because we have seen models that would have fabrication rates of 1 to 3% already, we were expecting that, of course, the frontier models would also be somewhere around that, but we've seen that Opus, Sonnet for example, would be a little worse than that. And then the GPT 5.6 family were also just a whole tier worse than the Cloud family, right? So both of them are trailing the open models, which is surprising, right? But what we saw when we analyzed the traces was this seemed to be a trade-off that was done during post-training, where because of over-helpfulness, the fabrication was because of over helpfulness. When one of the things being asked of them was not in the knowledge base, they decided, oh, let me just find the closest match and this is going to be the answer that I'm going to say, which in an interactive session does not look harmful. So you might say, but JV, that kind of sounds like that's bad measurement. Because if I see Claude Code say, oh, but I don't see Ryan Shrout in my database, do you mean Ryan someone else? Here's his data. And then you can just correct him and say, no, no, no, no, Ryan Shrout. If that's not the data, let's figure out where the data is so you can answer me. but the enterprise is not going to be like that all the time. Imagine you were running a gigantic AI augmented batch script overnight that sends this consolidated report to the executives. So let's say supply planner in the morning, the supply planner just sees a CSV with 10,000 rows or a thousand rows, all of which have like valid data. If the LLM assembling that decided to take a substitution, the supply planner will not see it by eyeballing the CSV because everything has a valid figure. The only way he would see that he would eyeball and realize something was wrong if there was not a real figure and it said there was a blank or an abstention. Yeah.

‍

Ryan Shrout: 

It's a different goal, right? If the mission is to build agentic systems that can operate without a human in the loop, right? Without a human guard, right? And that is your point about kind of like if in your interactive chat session, Maybe the best option is to supply data that you're 80% sure is right and let them correct it. But when the next decision that that agent has to make is based on that data and it's fabricated, that's a real problem.

‍

JV Roig: 

And at Fable, where we were headed was Fable was the one who reversed this trend. Yes. So before that, it felt like because of the increase in agenticness and there was this measurable gap between most of the open models versus frontier AI from OpenAI and Anthropic, there was a real measurable gap in the frontier enterprise scenario. So it felt like, oh, this huge increase in agenticness came at the expense of over-helpfulness, which ends up degrading their fabrication resistance scores. But Fable yesterday came out and when we tested it, not only did it retain the expectation for frontier performance in agenticness, its fabrication rate was basically one of the best out of two models. This is like the second best model. This is one of only two models with a fabrication rate of less than 1%, which means from Opus, this is basically almost a 10x improvement from where we were.

‍

Ryan Shrout: 

But it also turned out to be one of the most expensive, or I think it is the most expensive cost per completed task as well, right? So that's the trade-off in some of these instances, right? Especially with the API models, is that you pay for that higher intelligence, you pay for that higher accuracy and capability throughput. A couple of other quick things, like we do have some GPU measurements in there, looking at H200 versus B300, showing some generational performance improvements. I think our best example there was we had one result that showed a 6x improvement in the capability or the saturation level of agents supported by those two nodes, right? So you could basically run 88 agents, what was it? One of the Quin 3.5 models on an H200 node, but you could run 320 of them on a single B300 node. And that's, you know, we've got a lot more work to do in terms of, you know, testing more precisions, looking at other inference engines and really trying to optimize the GPU node measurements that we're going to do. And AMD and NVIDIA have been engaged with us on that and they've been super helpful. But already we can see some of the benefits that this real world testing is providing that I think are really unique to what Signal 65 Pinnacle can do. I think we're going to do this several times as new releases come out. As we see new models and we we iterate on things. I kind of want to wrap up with a little bit of a, like, what comes next, like, we're gonna add more models, we're gonna improve our GPU coverage, our performance coverage. But JV, what is maybe one thing that you are most excited to be able to add to Pinnacle or that people should expect to see from Pinnacle in the coming weeks or months?

‍

JV Roig: 

Yeah, so we're hard at work at it. There are like very two major things that are coming across the horizon. One is we're improving the way we measure silicon. And this improvement will allow us not just to measure GP versus GPU, but also the rest of the system. Meaning we can also characterize how much capability or contribution particular CPUs versus other CPUs will bring to the work since agentic AI is all about tool calling and that's outside of the GPU. The second thing is we are expanding also the intelligence frameworks underneath the benchmark itself. We are doing expansion there too. As the models get smarter, we give them more and more harder and complex tasks. So we're actually expanding what they're going to tackle in both in the Kami and the Ryker subsets.

‍

Ryan Shrout: 

Mitch, anything stand out to you as something that you want to see, that we know is coming, that you think will be most useful in some of the projects and enterprises and partners that we work with typically?

‍

Mitch Lewis: 

Yeah, I mean, I think Jamie covered all what's coming, but I just think, you know, we've already two days into it, we've already seen it be, you know, this live living kind of benchmark. So I'm really just excited to see how it evolves, how it kind of, you know, evolves with new models, but continues to diverge from, you know, these existing leader boards. that, you know, we've already seen our results are different than some of the things that are out there, which is really interesting. And then I think it just gives, you know, these different looks of like what an enterprise actually cares about, what models are, you know, really completing tasks, you know, what does it look like if I run them on a you know, hardware A versus hardware B, you know, how does that, how many agents can I run, all those things I think are really interesting and we can. you know, dive deeper into a lot of businesses.

‍

Ryan Shrout: 

One of the things I want to see is are any of these benchmarks going to show us the pathway to having more of the agents that can help us keep up with the changing models and iterations of Signal 65 Pinnacle itself is something that I've clearly seen a need for as well. I mean, I think we've seen two or three models released this week, and I think more of that is coming in still. Mitch, JV, thank you for joining me today to kind of give our quick overview of this. Like I said, I think we're going to do this a few more times as we dive into more of the details and more of the changes that we have coming. For everybody who's watching, I'd encourage you to go to pinnacle.signal65.com. That's our landing page. You'll see our results that compare model to model, results that compare GPU to GPU and even get our kind of TCO cost per completed work calculator and Pareto curve kind of visually out there for you. Then we'll have update articles and everything as we go. So I encourage everybody to stop by and we'll be back with another iteration of this exact same conversation with new goodies to talk about soon. Thanks everybody.

‍