Large Language Models (LLMs) have made significant strides in recent years, boasting remarkable capabilities such as passing the Uniform Bar Exam and coding video games. However, their abilities remain primarily linguistic. A new approach from researchers at Rutgers University suggests that LLMs can improve their performance by outsourcing tasks to specialized AI systems, thus creating a more comprehensive generalist system.

The Stickle-Brick Approach to AI, as proposed by the Rutgers University team, allows a human to describe a task using natural language, which an LLM then analyses. The LLM pieces together several specialist AIs to provide a solution, combining the breakthroughs made in AI over the decades into a single, generalist system. This approach leverages the expressive power of human language to explain various problems or model capabilities.

The researchers have developed a software platform called OpenAGI that connects pretrained LLMs and domain-specific AI models. They conducted experiments using three LLMs—OpenAI’s GPT-3.5, Meta’s LLaMA, and Google’s FLAN-T5—and several smaller models specialised in tasks like sentiment analysis, translation, image classification, image deblurring, image captioning, and text-to-image generation.

With OpenAGI, the user provides the LLM with a natural-language description of a task and the relevant data set. The LLM analyzes the task, devises a step-by-step plan, and selects the appropriate specialist AIs to solve the problem. This approach unifies different challenges into a single data format, with language serving as the medium.

Researchers from Microsoft and Zhejiang University in China have also explored a similar concept with their system, HuggingGPT. This system connects OpenAI’s ChatGPT service to Hugging Face’s repository of AI models. Users provide a natural-language explanation of their desired task, and ChatGPT devises a plan, selects and runs the required models, and compiles the results into a natural-language response.

One significant difference between OpenAGI and HuggingGPT is that the former is LLM-agnostic and open source, allowing for improved task-planning training using human-devised examples or performance feedback.

Both OpenAGI and HuggingGPT represent a growing trend in AI research: connecting LLMs to other AI models and digital tools, often through APIs. Examples of such efforts include Microsoft’s TaskMatrix.AI, Meta’s Toolformer, and the Allen Institute’s VISPROG. Mahesh Sathiamoorthy, a software engineer at Google Brain, believes this approach to be more promising for enhancing future AI capabilities than multimodal training.

While the Stickle-Brick Approach to AI has the potential to significantly improve LLM capabilities, some critics argue that the use of the term “artificial general intelligence” (AGI) in the context of this approach is misleading. David Schlangen, a professor of computational linguistics at the University of Potsdam, Germany, maintains that these new models are interesting experiments but do not provide a solution to key flaws, such as the tendency to fabricate facts.

The Stickle-Brick Approach to AI demonstrates the potential of combining specialized AI systems with Large Language Models to create more capable generalist systems. By leveraging the power of human language and outsourcing tasks to domain-specific models, AI researchers can continue to push the boundaries of what these systems can achieve. While critics caution against overhyping the connection to artificial general intelligence, the Stickle-Brick Approach undeniably represents a promising direction for AI research.

As researchers continue to explore the potential of AI collaboration and domain-specific models, the development of platforms such as OpenAGI and HuggingGPT can open new possibilities for AI applications across various industries. By integrating the wealth of specialized AI systems with the linguistic capabilities of LLMs, we can create more efficient and versatile AI solutions.

Moreover, the open-source nature of some of these platforms encourages collaboration and innovation within the AI community. This collaborative spirit can lead to the development of even more powerful AI systems that can tackle increasingly complex tasks.

In the coming years, AI research will likely continue to focus on optimizing the integration of LLMs and specialized AI models. As the Stickle-Brick Approach gains traction, it may pave the way for further advancements in AI capabilities, allowing for more practical and useful AI-driven solutions in various sectors, including healthcare, finance, education and beyond.

The Stickle-Brick Approach to AI showcases the importance of leveraging the strengths of both Large Language Models and specialized AI systems. By embracing collaboration and harnessing the power of human language, we can work towards a future where AI is more capable, efficient and versatile in solving a wide range of problems, ultimately benefiting society as a whole.