Are we Heading towards the Next AI-Winter?

AI-Winter             picture credit:bdtechtalks.com

The History of AI Waves and AI Winters

At the Dartmouth Summer Research Project on Artificial Intelligence (DSRPAI) hosted by John McCarthy and Marvin Minsky in 1956,  McCarthy brought together top researchers from various fields for an open-ended discussion on artificial intelligence, the term which he coined at the very event. In the following decades AI has gone through a roller-coaster of ups and downs dubbed as AI waves and AI winters:

First Wave (1956–1974)

Programs that were developed during this time solved mathematical problems, proved theorems in geometry or were used for language translation. The first simple concept of an artificial neural network was introduced in 1958 by Frank Rosenblatt predicting that “a perceptron may eventually be able to learn, make decisions, and translate languages”. In 1970 Marvin Minsky told Life Magazine, “from three to eight years we will have a machine with the general intelligence of an average human being”.

First Winter (1974–1980)

In the 1970s, AI was subject to critiques and financial setbacks. AI researchers had failed to appreciate the difficulty of the problems they faced. Their tremendous optimism had raised impossibly high expectations and when the promised results failed to materialize, funding for AI disappeared.

Second Wave (1980–1987)

The belief was that human intelligence could be programmed based on rules, replicating  our thoughts. This “if-then-else” reasoning set the foundation of the first generation of so-called expert systems or symbolic AI, which could be  used for configuring a product to meet certain specifications.

Second Winter (1987–1993)

The expert systems soon proved to be unmanageable as the exponentially increasing complexity caused by  the increasing number of rules to be programmed lead to a total misalignment of human and financial resources. As a result, funding for AI projects again was abolished.

Third Wave (1993–present)

The availability of huge data resources combined with high performance/low cost compute power and enhanced by innovations such as ‘Deep Learning’ has revolutionized the AI landscape till today. Also referred to as ‘Connectionist AI’, it significantly differs from the approach taken by ‘Symbolic AI’ that was prevalent during the second wave of AI.

State-of-the-Art in AI

AI has become a widely used tool for advancing knowledge, optimizing the results of pattern recognition from data used to solve specific problems. Yet Artificial Neural Networks (ANNs) and algorithms applied for making predictions often cannot replicate or explain the predictions offered. Moreover, the  quality of the training data required for supervised learning is often biased, resulting in erroneous decisions as has happened in credit applications by consumers. Hence a significant part of AI-research is focused on reducing the effort and improving the data-quality for training ANNs. Modelled on the way a child learns, new self-learning algorithms might provide a computational model reducing time and costs for solving problems. Invented about three years ago,  so-called ‘Generative Adversarial Networks (GANs)’ can construct entirely new objects, leading to the development of previously unimaginable applications and ideas. We refer to these as ‘thinking machines’ even though they don’t think in the way that humans do. They perceive their environments, but they lack awareness and consciousness. AI systems can anticipate or predict based on informational input, but a machine is only capable of implementing them in extremely narrow forms. For example, a highly advanced machine-learning algorithm designed to make predictions from road-traffic patterns cannot repurpose its intelligence to have a conversation or play a game. To resolve this  limitation, spurred by exponentially growing computer resources, the size of ANNs  has grown to the point where the content of the entire internet is being mapped and cross-indexed for applications such as language generation or search. GPT-3 from OpenAI has started to license the access to this huge repository of knowledge. Its practical usefulness is currently one of the hottest topics discussed within the AI-community.

Transdisciplinary Neuroscience to overcome current limits of AI

Francis Crick, Nobel laureate and co-inventor of DNA once stated: “There is no scientific study more vital to man than the study of his own brain. Our entire view of the universe depends on it.”

The models of the brain that dominate modern neuroscience depict the brain as an autonomously active, dynamic, plastic and variable organ that has evolved in biological‐socio‐cultural symbiosis. These models highlight the significance of social and cultural impacts on the brain’s architecture; notably, through the impact of the cultural imprints epigenetically stored in our brains. The brain is culture‐bound; it exists in contexts and cannot be understood independently of them. Recognition that human individuals as well as human societies are products of both their cerebral architectures and the environments in which they have evolved, suggests a constructive transdisciplinary approach to brain research. The realization of the extent to which the human brain develops in response to learning and experience, shaped by cultural influences has profoundly influenced contemporary neuroscientific models of the brain. While the mutual relevance of humanities, social sciences and neuroscience has increased significantly, the complexity to analyse brain functionality with current AI technology has reached limits. Expanding AI with new insights from neuroscience is required to overcome these barriers.

An example of how to empower AI with Neuroscience

An AI research team at the University of Massachusetts Amherst and the Baylor College of Medicine report that they have successfully addressed what they call a “major, long-standing obstacle to increasing AI capabilities” by drawing inspiration from a human brain memory mechanism known as “replay.” They have developed a new method to protect deep neural networks from “catastrophic forgetting” as today’s ANNs forget the results of previous computation. Unlike ANNs, humans can continuously accumulate information throughout their life, building on earlier lessons. An important mechanism in the brain believed to protect memories against forgetting is the replay of neuronal activity patterns representing those memories. The team’s major insight is in “recognizing that replay in the brain does not store data. Rather, the brain generates representations of memories at a high, more abstract level with no need to generate detailed memories”. Inspired by this, the research team created an artificial brain-like replay, in which no data is stored. Instead, similar to the brain, the network generates high-level representations of what it has seen before. This AI-concept of  “generative replay” proved extremely efficient, and the team showed that replaying just a few generated representations is sufficient to remember older memories while learning new ones. Generative replay not only prevents catastrophic forgetting and provides a new, more streamlined path for system learning, it allows the system to generalize learning from one situation to another. For example, if a network with generative replay first learns to separate cats from dogs, and then to separate bears from foxes, it will also be able to differentiate cats from foxes without being trained to do so.

The Non-Computational Universe

Abstract concepts like creativity, human desires, social cognition, meaning and free will are necessary components of any conscious being but have proven to be extremely difficult to formalize mathematically. They are practically impossible to translate to computer code and therefore impossible to be transferred to an intelligent machine. These concepts cannot be explained or recreated using machine-learning or deep-learning algorithms; no matter how large the data set, the software will not be able to understand or acquire uniquely human traits like empathy or sensitivity. The development of knowledge structures is one area where AI researchers are improving AI efficiency to further complement increasingly massive datasets. However, scientists are a long way off accurately rendering intangible, emotionally advanced phenomena as formal instances. The potential capacity of modern computing can be summed up with the phrase: “If you can understand a task, you can program it”. This suggests that virtually everything knowable can be formalised mathematically and programmed. However, this limits our explorations to unambiguous concepts rooted in the material world, while the realms of metaphysics and philosophy remain untouched.

Going from Digital to Analog

Nature uses digital coding, embodied in strings of DNA, for the storage, replication, modification, and error correction of instructions conveyed from one generation to the next. However, it relies on analog coding and analog computing, embodied in brains and nervous systems, for real time intelligence and control. Coded sequences in neural cells store the instructions to grow a brain, but the brain itself does not operate, like a digital computer, by storing and processing digital code. In a digital computer, one thing happens at a time. In an analog computer, everything happens at once. Brains process three-dimensional maps continuously, instead of processing one-dimensional algorithms step by step. Information is pulse-frequency coded, embodied in the topology of what connects where, not digitally coded by precise sequences of logical events as processed by digital computers. “The nervous system of even a very simple animal contains computing paradigms that are orders of magnitude more effective than are those found in systems built by humans,” argued Carver Mead, a pioneer of the digital microprocessor, urging a reinvention of analog processing back in 1989. Hence, considerable research is under way to build so-called neuromorphic systems, replicating the spiking of neural processes of the brain. So far, due to their low power consumption and spurred by experimental chipsets build by Intel and IBM, this technology is being introduced in continuous image classification tasks with decentralized IoT devices. To some researchers neuromorphic systems might one day resolve the mystery of consciousness as systems based on neural spiking represent the functionality of the brain much better compared to digital computations applied in ‘deep learning’ applications.

From Thought to Knowledge to Wisdom and no AI-Winter in sight

As our bodies develop, genes are passed from previous generations, an evolutionary process covering many generations. Human thoughts, however, can be transferred to anybody, anytime, whether they are from the previous, current or next generation. Humans can think about the purposes and aims for everything and can frame this approach with previous experiences. AI is limited to correlations between causes and effects while a human can transform events towards a purpose and invent the effect (causation vs. correlation). Wisdom is different from Intelligence. Intelligence seeks knowledge and seeks to eliminate ambiguity. Wisdom on the other hand, resists automatic thinking, seeks to understand ambiguity better and grasps a deeper meaning of what is known but also comprehends the limits of AI-generated knowledge. Reaching wisdom involves a process of human development which can benefit from AI by structuring information, thought and knowledge. Keeping human thought and its relationship to individual non-computational experiences alive, human-level AI will remain and support us as a building block for the exploration of new frontiers. Hence a third AI-winter is highly unlikely to occur.