We look forward to presenting Transform 2022 in person again on July 19 and virtually from July 20 to 28. Join us for insightful conversations and exciting networking opportunities. Register today!
The history of artificial intelligence (AI) development over the past five years has been dominated by scale. Huge strides have been made in natural language processing (NLP), image understanding, speech recognition and more, adopting strategies developed in the mid-2010s and putting more computing power and more data behind them. This has led to an interesting power dynamic in the use and proliferation of AI systems; one that the AI sees very similar to the power grid.
For NLP, bigger really is better
The current state of the art in NLP is powered by neural networks with billions of parameters trained on terabytes of text. Simply keeping these networks in memory requires multiple state-of-the-art GPUs, and training these networks requires supercomputing clusters well beyond the reach of all but the largest organizations.
One could train a significantly smaller neural network with significantly less text using the same techniques, but performance would be significantly worse. So much worse, in fact, that it becomes a difference of kind instead of a difference of degree; There are tasks such as text classification, summarization, and entity extraction where large language models perform excellently and small language models only coincidentally.
As someone who has been working with neural networks for about a decade, I am really surprised by this development. From a technical point of view, it is not obvious that increasing the number of parameters in a neural network would result in such a drastic improvement in performance. However, here we are in 2022, training neural networks almost identical to architectures first released in 2017, but with orders of magnitude more processing power and better results.
This points to a new and interesting dynamic in this area. State-of-the-art models are too computationally intensive for almost any organization – let alone an individual – to create or even deploy. For a company to use such models, it must use one created and hosted by someone else – much like how electricity is generated and distributed today.
Share AI like it’s a metered utility
Every office building needs electricity, but no office building can accommodate the necessary infrastructure to generate its own electricity. Instead, they connect to a central power grid and pay for the electricity they use.
In the same way, a wide range of companies can benefit from integrating NLP into their operations, although few have the resources to create their own AI models. This is exactly why companies have built large AI models and made them available through an easy-to-use API. By providing a way for companies to plug into the proverbial NLP power grid, the cost of training these state-of-the-art models at scale is amortized for various customers, giving them access to this cutting-edge technology without the most modern infrastructure.
As a specific example, suppose a company that stores legal documents wants to display a summary of every document in its possession. You could hire a couple of law students to read and summarize each document individually, or they could use a neural network. Large neural networks working in tandem with a law student’s workflow would dramatically increase summarization efficiency. However, training one from scratch would cost orders of magnitude more than simply hiring more law students, but if the company had access to a state-of-the-art neural network via a network-based API, they could hook up to the AI “power grid.” ‘ and pay for summary usage.
This analogy has some interesting implications if we take it to its logical extreme. Electricity is a utility, like water and transportation infrastructure. These services are so critical to the functioning of our society that in Ontario (where I am writing from) they are successfully maintained by crown corporations (owned and regulated by federal or provincial governments). These crown corporations are not only responsible for infrastructure and distribution, but also for assessment and quality assurance, such as B. the water quality test.
Regulating the use of AI is also of central importance
Also, like electricity, this technology can be misused. It has also been shown to have several limitations and potential abuses. Much research has been done on how these models can potentially cause harm through astroturfing and the spread of prejudice. Given how this technology will fundamentally change the way we work, it is important to consider its governing body and regulation. Several vendors of these NLP APIs have recently published a set of best practices for deploying these models, but this is obviously just a first step, building on this previous work.
Andrew Ng famously said, “AI is the new electricity”. I believe he meant that it will drive a wave of progress and innovation that will be vital to the functioning of our economy, with the same implications as the introduction of electricity. The statement is perhaps a bit exaggerated, but perhaps more accurate than I originally thought. If AI is the new electricity, then it must be enabled by a new set of power plants.
Nick Frosst is co-founder of Cohere.
data decision maker
Welcome to the VentureBeat community!
DataDecisionMakers is the place where experts, including technical staff, working with data can share data-related insights and innovations.
If you want to read about innovative ideas and up-to-date information, best practices and the future of data and data technology, visit us at DataDecisionMakers.
You might even consider contributing an article of your own!
Read more from DataDecisionMakers
https://venturebeat.com/2022/07/15/what-ai-and-power-plants-have-in-common/ What AI and power plants have in common