Building the Right Data Architecture: Learn how to fine-tune AI models, host them securely, and train them on proprietary data sets.

AI has tremendous potential – but taking the right approach to architecture, data management, and your partner stack is critical. Host Keith Townsend, is joined by Doug Oathout, Director Enterprise AI Solutions at Lenovo, and Peter Lillian, Director of Product Management, Enterprise Platforms and Solutions at NVIDIA on this segment from Six Five Media. Their conversation covers the challenges and opportunities surrounding enterprise AI implementation, including model training, data governance, and predictions for the future of AI in business.

‍

Key takeaways include:

‍

🔹Right-sizing AI solutions is crucial: Lenovo and NVIDIA are focused on modular designs that scale to fit specific business needs, whether it's a small chatbot for a hotel chain or a large language model for complex process optimization.

🔹Data governance and security are paramount. 

🔹Smaller models, big impact: While LLMs grab headlines, smaller, domain-specific models can offer faster ROI and quicker implementation.

🔹The power of partnership: Nvidia's collaboration with Lenovo provides a comprehensive platform, from edge devices to data center solutions, and a robust software suite.

🔹Agentic AI is the future: AI will increasingly act as an assistant, augmenting human capabilities and automating mundane tasks – revolutionizing how enterprises operate, improving efficiency and productivity.

‍

Learn more at Lenovo and NVIDIA.

*Reference: Boost Efficiencies and Optimize Performance with Turnkey AI Enterprise Solutions

‍

Watch the video below at Six Five Media, and be sure to subscribe to our YouTube channel, so you never miss an episode.

‍

Or listen to the audio here:

IMPORTANT: Note to Uploader: Libsyn AUDIO EMBED - added by uploader.

‍

Disclaimer: Six Five On The Road is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded, and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors, and we ask that you do not treat us as such.

‍

Keith Townsend: All right, you are watching another exciting webcast from Six Five Media. We have this series, Lenovo AI for All, in which we're exploring the complex issues associated with building AI systems, and how they're used in the enterprise. We have joined us today, Doug, and Peter respectively, from Lenovo, and Nvidia to dive deep into this important topic. I want to have a conversation around this topic of training models. There's a lot of challenges, fun, and well, not so fun, specifically around power, efficiency, et cetera. Let's talk about how Lenovo, and Nvidia is helping customers overcome some of these challenges. So, let's start with Doug. Let's talk about what's happening at Lenovo to help customers overcome some of the challenges around power.

‍

Doug Oathout: So, Keith, that's a great question. So, we're really focused on what the end users want to do, our IT decision makers, what they want to create for their customers, and we're really focused on making sure they have the right size solution to fit their install base to fit their customers, right? So, if I'm an IT manager in a hotel chain, I want to make sure I'm creating a chatbot, or a model that fits my business to solve a particular problem. So, I need to make sure that I have a modular infrastructure that supports the size, the number of users I need, the data, how much data do I want to input into the model, and then also machine that can prescribe decisions really quickly to the end users who are using the system. So, it's really about a modular design that scales up, and down with the different use cases for the customers. And my goal within Lenovo really is to bring those solutions with Nvidia to our customers so they can see what others have done so they can learn from others, and make it a more likeable outcome, or higher probability outcome that they will deliver a solution with quick, fast ROI, and they can get back to the next AI project they have.

‍

Keith Townsend: Peter, how is Nvidia partnering with Lenovo to help that come to fruition?

‍

Peter Lillian: Nvidia is very well aligned, and partnered with Lenovo on AI solutions. We are providing infrastructure, networking, GPUs, CPUs in some cases, and a full platform of software for developers, and enterprises to go productize their AI solution sitting on top of relative to right-sizing. I like the term that you're using, Doug, around modularization, or right-sizing for the enterprise. Our Nvidia A enterprise software suite includes a plethora of tools for the developer if they want to start from scratch, and want to develop with just SDKs, and build their applications from scratch all the way up into containerized optimized solutions for GPUs that have been accelerated out of the box. And then we have this new concept called NIM, which is a microservice that we launched at our GTC conference earlier this spring, which is a fully vetted microservice, which includes large language models, a bunch of different domain-specific applications as well. So, the developers don't have to put the bits, and bytes, if you will, and manage the bits, and bytes inside of that microservice so they can just point their application at the microservice, and just get it to run.

‍

Keith Townsend: So, I'm hearing a combination of software, services, hardware coming together to offer this platform, but many of the challenges that we see in the industry are not necessarily related to getting the hardware, and services right. That's a challenge in itself. Talk to me about data, and data governance, because I think this is the number one challenge is customers are running into is they're trying to get their culture, or capture that culture, and get it into their models in an efficient but compliant manner.

‍

Doug Oathout: Yeah, so Keith, as we roll out these solutions, and we call our solutions, either they're a solution brief with an ISV, or they're a reference architecture, or a Lenovo validated designs, we incorporate everything from the software to the services to the hardware. But in every piece of hardware we build, there is security, right? So, we're securing the hardware, and then as we build on the software, we're relying on our software to not interject any malicious malware, or any malicious content into any of the models we're building. And then we take those industry standards, and then we apply them to the customer's data. And the customer's data is really the challenge of the whole thing. Which data do I want to incorporate in the model? Which data do I want to keep on-premise so that nobody else can see it? And that's really where data modeling, and data setup is incredibly important. And that's where this private AI that we're developing solution blueprints for, and reference architecture for really kick off with the combination of Nvidia software, Nvidia hardware, and then Lenovo services software, and hardware to create a model that runs these applications, and run this data through a very efficient, very secure model.

‍

Keith Townsend: So, we've established there's a platform, there's governance around that platform to get the data into your platform. Let's talk some detail about that, getting the data into the platform. How are customers approaching this model challenge? Are they looking at large models? Are they looking at smaller models? What's some of the learnings that you're seeing across customers?

‍

Doug Oathout: Yeah, to me you have the, depending on the customer type, you see them starting with smaller models to start, they want to build either an agent that can ingest a lot of company's data, and produce outcomes. So, for example, a company might want to ingest a lot of their legal documents, or a lot of their technical documents to either help their call center, or help their legal department be better, and more efficient at answering customer's questions, or answering legal questions for their businesses. And then you have call centers who are developing ingest models so they can answer customer's questions more fluently, more efficiently, and more accurately so they can produce better results for their end users. And then you have a large language model, or larger customers really imploring larger language models to solve business critical issues. So, it's a combination of two, if you're dealing with end users, you're dealing with internal users, you tend to start with smaller models. If you're dealing with complex issues like process, and other things, you tend to start with a large language model.

‍

Keith Townsend: And are we seeing any domain specific advantages with smaller models over larger models, or is this more of, again, based on the size?

‍

Doug Oathout: I would say we're seeing a lot more success with small language models, because you can be domain specific for a retail environment, you can be domain specific for a call center environment. You can be domain specific into a service company. So, if a company's out there doing services on washing machines, or appliances at home, it's much easier to create a small language model so they can interact with their customers as they go to debug the problem before they get there. So, it's much easier to deal with that kind of domain in a small language model than it is to take one of these large language models, and try to do it for a wide variety of industries with a lot of different inputs, and a large data set.

‍

Keith Townsend: So, Peter, we're seeing these, even the smaller models grow in size, the larger models, there's these complex business processes that Doug mentioned earlier. We want to start putting this on device. How is Nvidia helping to get this capability closer to that edge?

‍

Peter Lillian: There's a couple of things to address it. So, the open-source models of AI, it's an open-source wild wild west if you'll, so I wanted to go back briefly to touch on your security question as well. Because it's open-source, there's a lot of opportunities to ingest some unnecessary things inside of the software stacks. And so, one of the things we're working with Lenovo on in particular from a security perspective is as part of our AI enterprise software suite, we have what's called a production branch, which has full-fledged CVE, security scanning to make sure that that open-source software is clean on daily, hourly, monthly basis to ensure that those customers that are using those applications, using those large language models, that AI software suite is secure, and safe from unknown entities coming in the open-source. And that takes a lot of effort, and lots of time to go run through that.

‍

And if you are a system admin, and you're focusing on developing applications, you don't have to be worrying about all the different elements, and the thousands of different sources that open-source can kind of feed into there. So, we're trying to take care of that for you. Because it's all open-source, and because it's, again, referring back to the wild wild west as it were, there's lots of choices. And every month, literally every month, there's a new model that it comes out as well. And one of the things that Nvidia is working on both on the large scale, so the largest scale large language models, and saying large, large language models are supposed to large, small, large language models... They're all large to begin with.

‍

Keith Townsend: They're all big. Seven billion, and above is pretty big.

‍

Peter Lillian: Lots, and lots of parameters, and complex items. So, going back to the NIM statement, and doing optimizations both from a Harvard perspective, from a precision perspective, and ensuring that those models run, and optimized, and have the best accuracy, which is super important for enterprises, because you don't want something spinning out the wrong answer. We're supporting those smaller models as they come out putting a NIM wrapper around it so that the customers have one of the assurances from a security perspective. And secondarily, it's accurate, it's easy to deploy, and easy to manage, again, all the way from those small models all the way up into the 400 billion working all the way up into the 1.1 trillion size models as well. And so working with Lenovo for those on-prem models, how do you go deploy those using a standard Kubernetes stack to ensure that they're easy to deploy, manage, scale up, scale them as need be, and again, all the way up to larger scales as well.

‍

Keith Townsend: So, let's go into some level of detail around skill. I hinted at the beginning that retraining, and training models is probably the best way to get the model to operate in a way that's consistent with your organization. There's the RAG process, and there's retraining. This isn't an either or, this is a both scenario, but let's dig into the training challenge. We're looking in the news where we're seeing these massive clusters being built to train the largest models. Where are customers selling, and how is Lenovo, and Nvidia helping to downsize, or right size is a better term that you used earlier, Doug, these infrastructures to do training?

‍

Doug Oathout: Well, one of the best ways to downsize your infrastructure, or to get the right infrastructure to do the job is to understand how big the data is, and how much data you need to process. A lot of customers start with a wide scope, and a lot of parameters. And the truth is they don't need to start that wide. They can start with their domain expertise, they can start with their own data, and build a smaller model which would indicate how things are going to go, and then add more data to it, add more parameters to it. A lot of customers start at the high end, high parameters, and work their way down, which is a very inefficient way to do it. If you start with a smaller model, smaller data, and you prove what you're doing first, and then you scale up. So, starting that way is a different approach, and it is more effective.

‍

Secondly, we have energy efficiency built into all of our designs. We have industry leading water cooling called Neptune, which allow us to do it a lot more efficiently from a heating, and cooling, and energy efficiency standpoint in the data center. And then also we have high performance storage architectures, which allow you to do the data management in a smaller footprint than what you have in general on a database, or an SAP background where you have lots of copies, lots of duplication of data, you can slim that down as you move to a better data management structure. So, starting smaller, proving it works, and then expanding from there. You start with a smaller model, and then having the right infrastructure that can scale that's efficient to begin with, and then have the secure data management process in place will allow you to be more efficient in your footprint.

‍

Keith Townsend: Peter, talk to me about the Nvidia approach to this because I'm picturing in my mind different sized, Lenovo solutions that range from the edge all the way to the data center, and this validated design that Lenovo offers for their AI platforms. How's Nvidia supporting these different form factors specifically around this need to go from maybe training a smaller model all the way up to the larger?

‍

Peter Lillian: It's a very good question, and something we have customers ask us routinely about. Lenovo has a wonderful full-fledged portfolio that goes all the way from the edge. So, the SE350, SE450 class platforms, which are again, classic edge, very low power. We have smaller GPUs that fit into those systems that allow customers to go do inferencing near the data. So, if you're thinking about retail, thinking about classic IVA type applications where you have a lot of video streams coming in, is that a suitcase? Is that, classic, is that a cat, or is that a dog type thing. But going even further into know IVA security aspect of, "Okay, somebody left a suitcase behind an airport, I need to identify that there's nobody around that suitcase", and identifying security to go off, and go resolve that. So, those smaller systems, again, it's smaller applications, but a large number of uses, and a large number of instances running at the edge, multiple different locations, and everything else, speeding cameras, I'm sorry, smaller GPUs had to go accommodate that.

‍

And then as you get closer into what we used to call near edge, far edge, as you get closer into the data center side of the world, the GPU support gets larger. The GPU communication between the GPUs gets larger because as you're getting these models that are getting larger, and larger, and larger, as we're talking about billions of parameters, hundreds of billions of parameters, and then trillions of parameters, they take a lot of inter GPU communication, a lot of internode communication. And so working with Lenovo on their, like the SR 675V3 is a wonderful example of a platform that Lenovo brought to market earlier in the space, which allows up to eight GPUs interconnected through PCIe. But that is from a frame buffer perspective is a fairly large instance. And then you connect it with Nvidia networking at scale to these larger model training. So, you connect both the GPUs, interconnecting the GPUs, and then the node to node communication through Nvidia networking tied together with software stacks. So, you can train these larger models at scale as well. Again, anywhere from the larger scale, HGX based platforms all the way down into the smaller edge based platforms, Lenovo has solutions for those in a power efficient manner as well.

‍

Keith Townsend: Yeah, I think this is the complexity of solving this challenge. Most customers don't know where to start on the journey. There's plenty of options on the market, especially between providers, and partners such as Lenovo, and Nvidia. So, even with all the content we've created over the past few days helping to dispel some of the FUD, and form, talk to me about the practical first step, Doug. Where should customers start their journey?

‍

Doug Oathout: Well, customer journey really starts with their priorities. What are they trying to do, and what are they trying to solve? So, generally we would start a customer in an advisory workshop. We'll probably look at five, or six things they want to go do, and we'll say, "Here's the two that will generate the best, fastest ROI, and fastest implementation. Here are two that are very well-known use cases. We know how to do them. We'll tee them up as projects to go do. And here's two that need to be researched because they are net new, net new thoughts, net new capabilities, which will take a little longer to develop." So, one of the best things that Lenovo can do with Nvidia's help is to help give the best practices to the IT staff, to the data scientist, and say, "Here's what other people are doing", and advise them, here's the fastest way to implement these, and show the company that we're generating real returns before we move to the next set of projects. A lot of times in the past couple of years, companies have kicked off projects, and they've gone to pilot, and they didn't succeed in their pilot because they didn't achieve the results. So, using our advisory services, using some of our professional skills to tell them what other people have done, and where it's worked has really helped us get from the starting gate to at least phase one, phase two of their implementation of these solutions. And then they can really look at how do I solve some of the larger problems with some of the large language models?

‍

Keith Townsend: And again, I don't want to leave this conversation without talking about data. Can you give us maybe three top recommendations around data infrastructure?

‍

Doug Oathout: Yeah, the key about data infrastructure is to identify what data is going to be associated with the model, and making sure it's tagged correctly, and it's marked correctly so that it can be pulled into the model. And secondly, you want to make sure your data is on a secure platform, and is being integrated into a secure piece of software, because as you're building, whether it's a reference, a RAG model, or just an inferencing model, you want to make sure that the data is clean, and is also secure so that your results are accurate, and tell the truth versus having somebody put something malicious in there, or having a false positive. So, we want to make sure that the data is clean, it's secure, and it's simple to manage. And then secondly, what you want to really do with your data is make sure that you're on a platform that can not only support the current use case, but is available for the next use case. So, as we're cleaning up data, as we're doing the thing with the data management, that's correct, for one model, you want to enable your data to be used for the next solution as well. So, we see this a lot manufacturing, the same data will be used either for the SAP solution, or for the manufacturing floor solution. So, we'll clean up the data so that you're ready to use it for the first model, but also the second implementation as well.

‍

Keith Townsend: And last question for both of you. Its prediction time. We're near the end of the year, we're recording this at the end of the year. How do you see enterprises using AI in the coming years?

‍

Doug Oathout: I think the easiest way is agents, customers can use agents internally. So, you can have a chatbot for your human resources. You can have chatbot for your services, your IT services. So, using agents is going to be the fastest way, and it's a harmless model. If your IT bot gives you the wrong answer, there's no harm, no foul, because you're basically going to call up the IT staff anyway, and get the right answer. The same thing with a lot of self-service portals for your call centers, et cetera. There's no harm, no foul if it doesn't give the right answer, because they'll follow up with the call. The other thing we're going to see is a lot of applications, containerized applications will become hardened, so we'll be able to roll out more of these off-the-shelf prepackaged solutions that didn't exist 18, 12 months ago. They will start to exist, and so we'll be able to take containers, and NIMs off the shelf, and being able to implement those very, very quickly for customers. So, I think the chatbots, and then these predefined ISVs, or predefined NIMs will allow us to implement more use cases in a faster way, which we haven't had the capability to do before in the last 12 months.

‍

Peter Lillian: And I fully agree, agentic AI is the way that everybody's going to go look at this, especially in the enterprise. And if you read some of the news that's out there of, "AI is going to go steal my job. AI is going to go remove my ability to go do my job." The intent here, and the focus is augmenting, and helping you do your job faster, and better, and removing those mundane tasks that you have to go, where did I put that PDF? Where is that file? Where's that piece of data I'm going looking for? Instead, you just ask for it, and it will automatically go off, and go find it. The other point I want to say, fa lot of things in the news around Chat GPT these large scale training models. The majority of things that you've seen in the news today for AI adoption has been on the large scale initial implementation in the creation of those models. The actual execution, and adoption through enterprise is coming soon. Lenovo is uniquely positioned to go take advantage of both that training infrastructure from large scale data centers all the way out to the edge, huge, wonderful workstation business as well. So, thinks from a business edge to data center, desktop to data center. Lenovo is well positioned to go drive that, and you're going to see a large adoption of AI inside of the enterprise leading in with the agentic side of the world, taking advantage of these multiple years of AI training that's been going on in the industry to date.

‍

Keith Townsend: Doug, that's a full throw to endorsement from Peter. How do people reach out? What's the next steps? How do people reach out?

‍

Doug Oathout: Well, the easiest way to reach out to us really is contact us through your sales rep, contact us through your business partner. Lenovo is not Lenovo without our business partners. So, all of our business partners are well-tuned, and well-thought-of from an AI perspective. And then our sales reps are the second-best way to get ahold of us. And then we have a lot of collateral online at Lenovo.com. We have workbooks, we have best recipes, we have thought leadership pieces with Nvidia. So, there's a ton of different ways to get the information to you.

‍

Keith Townsend: Doug, Peter, thanks for joining me. This has been an amazing conversation. I'm excited about the future of AI, agentic AI. Specifically, please visit the link below for more information about Lenovo's hybrid AI strategy.

‍