In the evolving world of AI, inferencing is the new hotness. Here’s what IT leaders need to know about it (and how it may impact their business).
With a new year underway, narratives around generative AI are as varied as they are contentious. Perhaps none more so than those concerning the continuing growth of AI models.
Since OpenAI made ChatGPT available to the masses, AI experts have promoted the notion of model scaling, in which AI models grow larger and more powerful as they are trained on datasets.
However, with many of those same AI experts admitting that model scaling is seeing diminishing returns, owing to a shortage of data, scaling laws and other issues, the narrative has pivoted to the rise of inferencing scaling.
This entails applying more compute power to the critical inferencing stage of AI models, in which large language models (LLMs) “think,” or reason their way through problems.
Inferencing helps digital assistants serve up information to those who prompt GenAI models. Advanced inferencing and reasoning are also foundational for autonomous AI agents.
Inferencing 101, Hell’s Kitchen Style
And training is foundational to inferencing. It helps to think of it this way: Suppose you want to be a chef.
Chances are you’ll train at a culinary school, learning the fundamentals of cooking, such as knife skills, cooking techniques, ingredient preparation and recipe construction. The culinary school helps institute a solid foundation of knowledge—just like an LLM is trained on a vast amount of text data.
Fast forward a bit. You’ve earned your culinary degree and you’re the lead chef, maybe even at your own restaurant. In addition to leaning on your cooking training, you’re using that knowledge as inspiration for new dishes. You’re making hundreds of quick decisions, such as altering menu items based on seasonal availability as well as evolving customer tastes. You are making inferences.
Similarly, when an LLM is deployed in the wild it begins inferencing, relying on the knowledge with which it’s been pre-trained to generate text from prompts and make predictions on the fly. The model is taking the skills it learned during training and applying them to new, unseen data, just like you’re applying your culinary skills to craft new menu items and navigate new challenges.
The Reasoning Race and Why it Matters
Inferencing is great, but you know what’s better than inferencing? Reasoning, which encompasses a range of cognitive processes that includes inferencing but uses additional layers of logic to solve problems. Hierarchical reasoning, knowledge graphs and attention mechanisms comprise a few of the reasoning components that help AI models connect bits of information together, ferret out relationships and apply higher order thinking to make and prioritize conclusions.
Breakthroughs in reasoning are why many AI experts waxed ecstatic over OpenAI’s release of its o3 and o3 mini models last month. The models showcased major enhancements in visual reasoning, abstract concepts in math and software coding.
The software coding enhancements are key. Code is the lingua franca of AI agents—software programmed to solve problems on their own. Reasoning is the key ingredient that will help set AI agents best challenges and accomplish their goals. And the tech industry is demonstrating great strides in advancing reasoning capabilities forward in the application layer.
IT’s Role in the Reasoning Race
These developments—along with those from Google and other builders on the bleeding edge of GenAI innovation—will help shape the AI revolution for months and years to come.
As an IT leader, you may not be ready to deploy agents, let alone other technologies that employee advanced reasoning. Nevertheless, it’s important that you know the role reasoning plays in their evolution and ultimate success.
Reasoning will help agents better surface information and help answer questions customers may have about your company’s products. And as agents become more prevalent, they will augment digital experiences for employees and customers alike.
GenAI technologies will become embedded in every IT organization’s digital product or software footprint by 2027, automating code reviews and infrastructure management, among other manual, labor-intensive practices, according to Deloitte’s Tech Trends 2025 report.
This future presents both opportunities and challenges. For your organization to adapt and thrive, you’ll need to address the technology, people and process shifts that accompany these new tools.
If you haven’t already, confer with your C-suite peers and line of business stakeholders, even your board to hash out an AI strategy that fortifies your organization for the future. Avoid siloes that accompany such tectonic shifts and be sure to communicate and collaborate early and often as well as stand up GenAI education and reskilling opportunities at all levels of the organization.
IT leaders should also seek help to operate in this turbulent terrain. Fortunately, trusted advisors can help organizations execute their AI strategies, counseling on anything from the selection and deployment of AI-enhanced technologies to modern frameworks with which to bring them to bear.
Is your organization ready to navigate this brave new world?
Learn more about the Dell AI Factory.