Dealing with Complexity? Try Collective Intelligence & AI

Posted by Peter Rudin on 29. January 2021 in Essay

Picture Credit:


Human intelligence rests on billions of years of evolution. The unifying feature of all natural-intelligence systems is that they have evolved under strong selection pressures to solve the problems of surviving and thriving sufficiently to reproduce better than their competitors. Unlike the evolution of faster speed, sharper teeth, more efficient harvesting or better camouflage, all of which improve physical capabilities, the evolution of intelligence enables better choices to be made as to how and when to employ those capabilities by processing relevant sensed and stored information. From this perspective we can define intelligence as the ability to produce effective responses or courses of action that are solutions to complex problems. Obviously, this becomes more difficult as the number of possible choices increases, and the number of relevant factors multiply due to the exponential growth of scientific discovery. Research in dealing with complex problems provokes a discussion about the potential application of Artificial Intelligence (AI), well-beyond the current range of its capabilities and applications. Just as Darwinism moved us from a preeminent position over all life forms into a branch of the evolutionary tree of life, the current and recent sweep of advances in understanding of neuroscience, cognition, behavioural science, evolutionary psychology and related fields call for yet another round of reappraisal of AI’s potential in dealing with  the exponential growth of complexity in our daily lives.

Human limitations in solving complex problems

One immediate conclusion that can be drawn for the successful tackling of complex problems is that there is a massive requirement for cognitive bandwidth—not only to keep all the relevant aspects at all the relevant scales in mind as one seeks to understand the nature of the problem, but even more challenging, to model and simulate the non-linear dynamics of natural life. Given the well-known limitations of human working memory, short-term memory and attention span, this is an obvious area for future AI support. However, there is a fundamental problem that needs to be addressed first: how to acquire the necessary relevant information and data about the composition, structure and dynamics of the complex problem and its context. This requires an effort of continuous learning and updating across all the dimensions and scales of the problem. Because the human brain evolved to deal with the problems of surviving that our ancestors faced, modern humans are still equipped with the same heuristics, behavioural tendencies and biases that worked well enough in the distant past. These hardwired shortcuts based on rules of thumb, operating automatically below conscious awareness and so permitting very rapid adaptive responses to various simple conditions enabled our ancestors to cope with the level of complexity that existed then. Today’s complexity, our way of life with modern economy as our source of income, far exceeds what evolution has taught us in adapting to change. Within just a few hundred years, science and knowledge have become the critical success factor for survival.

Half-life of knowledge as a socioeconomic challenge

The half-life of knowledge is a phrase attributed to Fritz Machlup, former Professor at Princeton University, who studied the economic value of knowledge back in the 1960s. It is defined as the amount of time that has to elapse before half of the knowledge  of a given discipline is superseded or shown to be untrue. Machlup and others were trying to figure out how much the acquisition of certain types of knowledge were worth. To do so, they needed to understand how long the knowledge we paid to acquire would remain relevant. Focused on engineering as a baseline profession, they discovered that while an engineering degree acquired in 1930 was useful for about 35 years, an engineering degree earned in 1960 was only relevant for about 10 years. In 2002, the president of the National Academy of Engineering, William Wolf, said that the half-life of engineering knowledge was somewhere between seven and two-and-a-half years. In other words: over the course of a 40-year career, a typical engineer would need to invest 9’600 hours just to keep up with the basics of the field, which is the equivalent of having to earn two additional undergraduate degrees. Lifelong learning is not simply nice to have but instead something we should include as part of to our educational system. While we might be able to maintain an individual’s value-contribution over a longer period of time due to factors such as experience or growth of emotional intelligence, the question looms how corporations and organisations can keep-up with the increasing complexity of scientific research and its potential for creating value. As an indication of this trend, the following diagram documents the exponential growth of scientific publications  between 1960 and 2010:

Picture Credit:

From collaboration to collective intelligence

Humans’ ability to collaborate in order to achieve otherwise inaccessible goals may be one main cause for our success as a species. We all possess intelligence; however, intelligence is not something that arises within individuals alone – it also arises in groups of people. It has long been recognised that when groups of people come together to solve a problem, they can become more than the sum of their parts. The ‘wisdom of the crowd’ was acknowledged as far back as Ancient Greece, when Aristotle noted that many unremarkable men often make better collective judgments than great individuals. For mutually beneficial collaboration, individuals need cognitive mechanisms to coordinate actions with partners and mechanisms to distribute the acquired resources in a way that incentivizes them to continue collaborating. All major communication platforms such as Microsoft-Teams or Slack have in common the ability to connect individuals for exchanging information, providing administrative support to carry out tasks or to search for matching interests. One of the most important principles to help groups to achieve more than the sum of their parts is diversity. Anita Woolley, a leading expert in organisational behaviour at Carnegie Mellon University, has demonstrated that gender diversity helps smaller groups to improve their problem-solving ability and that cognitive diversity is vital for creativity and learning in workplace teams. This principle also applies when it relates to larger scale collective intelligence efforts, where it may intuitively seem that diverse contributions could make it harder to distinguish quality from quantity. Collective intelligence covers a wide range of participatory methods, including crowdsourcing, open innovation, market predictions, citizen science and deliberative democracy. Some of them rely on competition, while others are built on co-operation; some create a sense of community and teamwork, while others operate based on aggregating individual contributions or microtasks.

Adding AI to the equation

Generating ideas through discussions with employees, customers and other external parties represents one way in which collective intelligence is commonly used. Collecting ideas and feedback from a greater number of diverse individuals generates large data sets for training artificial neural networks. Incentives for participation can be provided by offering a reward for designing the best AI-algorithm for solving a specific problem, for example. AI has the biggest impact when it is applied by cross-functional teams with a mix of skills and perspectives. Having business and operational people working side by side with analytics experts ensures that initiatives for problem-solving address broad organizational priorities, not just isolated business issues. Diversified teams can also think through the operational changes new applications may require, for example that the introduction of an algorithm that predicts maintenance intervals should be accompanied by an overhaul of maintenance workflows. When AI is adopted broadly, employees up and down the organisational hierarchy will augment their own judgment and intuition with algorithms’ recommendations to arrive at better answers than either humans or machines could reach on their own. But for this approach to work, people at all levels have to trust the algorithms’ suggestions and feel empowered to make decisions,  abandoning the traditional top-down management approach.

The rise of the GIG Economy

Broadly speaking, the Gig Economy is a free-market system in which companies work with independent contractors or freelancers as opposed to hiring full-time workers. Internet connectivity and mobile network access have accelerated this trend and are challenging conventional employment models. ‘Gig’ refers to a model of work where workers are contracted and paid per hour or completed project milestones. The rise of this economy is closely connected to advances in technology and the trend of working remotely as so-called digital nomads. In the US about 36% of all workers are now involved in the Gig Economy and  according to the US Bureau of Labor Statistics, more than 50% of the US workforce will participate in it by 2027. According to a study conducted by MasterCard in 2019, globally the Gig Economy is growing at an average rate of 17% per year between now and 2023. Currently about 40% of this workforce consists of young people between the age of 18 and 34. The majority work in low-income positions, providing delivery services (taxi, food, documents etc), but the trend is that students and graduates from the STEM segment (Science, Technology, Engineering, Mathematics) prefer to work as freelancers rather than being employed. This stipulates a cultural shift towards a ‘flexible’ work-life environment, altering the working population’s conventional expectations of a typical 9-to-5 workday.

Knowledge exchanges and human resource management

Digital transformation has caused disruption to businesses that were unable to adapt in time to a data-driven economy. Fuelled by the abundant availability of venture capital, Start-ups are flourishing with a rate of about 8% of sustainable success, while many eventually fail. Collective intelligence is part of the Start-up culture; however, its potential reach encompasses our entire socioeconomic landscape. The combination of collective internet-connectivity with knowledge will introduce new IT-platforms supporting AI-focused value generation. Small and Medium Sized  Enterprises (SMEs), challenged to adapt their traditional product and service portfolio to new scientific breakthroughs, can benefit from these cloud-based knowledge exchanges. Facing increasing complexity, AI-empowered human resource management will be key in this transformation process. The capacity to build and manage teams (employed and/or contracted) with diverse competencies and distinct behavioural profiles to optimize man-machine interactions, will define a new competitive landscape for the ‘survival of the fittest’. Moreover, adhering to values related to a positive, sustainable work-life balance in family settings will foster regional development away from overcrowded centralized hubs. Competing for the best brains, companies might be well advised to move their headquarters and manufacturing facilities to areas where individuals and families find the best work-life environment. High-performance teams excel in real (not virtual) collaborative work-environments. Video conferences cannot replace human’s basic social needs, a lesson we all learned during the Covid-19 crisis.

Leave a Reply

Your email address will not be published. Required fields are marked *