Learning From Spies: What the Insights Industry Can Absorb From the Intelligence Community
By Andrew Grenville, Chief Research Officer | July 25, 2019
The intelligence community is charged with making sense of large amounts of information and presenting that information in such a way that it is easily understood by decision makers.
If that sounds like the job we do in the insights industry, it is because it is. We can learn a lot from the intelligence community. A new report on the future of spying has important lessons for the insight industry and where it should be headed.
A time of change
The volume and types of data available to both the intelligence community and insights industry has radically and rapidly changed.
Principal Deputy Director for National Intelligence Sue Gordon recently said: “We’ve moved from a world of data scarcity to data abundance. Where we used to go looking for single pieces of information that no one else had . . . now what we have is a world that is so much information that we have to make sense of it. How do you take advantage of the data that is now available and do something special with it so we know something a little more, a little sooner? We have to help the community build more capacity, whether that’s in artificial intelligence, or in cyber, capacity and capability . . . to make use of all the data that exists.”
Being a member of the intelligence community is a high stakes occupation. The U.S. intelligence community’s charge is to “conduct intelligence activities necessary for the conduct of foreign relations and the protection of the national security of the United States.” Their successes are hidden, but their failures are public and often excruciating. They strive to do the best job possible and there is much we can learn from them.
Intelligence analysis is big business
The spend on the intelligence community in the U.S. is roughly double the amount of money that is spent on all market research analysis and insights in America. They are very serious about excellence in methods and are constantly looking for ways to improve.
The National Academies of Sciences, Engineering and Medicine recently released a report with recommendations for where the intelligence community should be heading. While the report has much to offer insights professionals, these are the two key takeaways:
We need look outside our own industry for ideas for improving the insights we produce.
We need to think beyond the survey and go much deeper in the analysis of other types of data and alternative types of analysis.
The report, entitled A Decadal Survey of the Social and Behavioral Sciences: A Research Agenda for Advancing Intelligence Analysis points out ways for the intelligence community (IC) to improve its effectiveness. In many aspects of the report, you could replace “intelligence community” with “insights industry” and wouldn’t think twice.
“Ten years from now, the job of the intelligence analyst will be transformed,” the report begins. “Intelligence analysts themselves will have new resources based on advances in data processing and other technologies. Technologies including artificial intelligence (AI), large dataset analytics, dynamic search tools, and interactive technologies are already allowing analysts to process and integrate multiple sources of data and intelligence far more quickly and efficiently than ever before. They are also dramatically expanding opportunities for collaboration involving personnel and technology, and integration of new types of data.” The parallels to other types of insights professionals are not lost on the report’s authors.
“The challenge of sensemaking for intelligence analysts is akin to challenges faced by analysts in other sectors, such as business, risk analysis, or urban planning. The need to find meaning in complex, data-rich environments and to understand especially challenging—’wicked’—problems is not unique to the IC. The activities and responsibilities associated with intelligence analysis are similar to those needed for any planning or analytical task that requires expertise, the consideration of multiple variables across time and space, and the assessment of human behaviors and actions. Analysts seek to recognize patterns in behaviors, trends, and relationships among actors.”
The report looked “across the current landscape of SBS [social and behavioral sciences] research to highlight new and emerging insights with potential to improve sensemaking, including from areas that have not been consistently applied in a national security or intelligence context.” “Our review,” they found, “of a wide range of ideas and trends in SBS research revealed four areas with the potential to be particularly fruitful in supporting analysts’ sensemaking efforts—the study of narrative, the study of social networks, the study of complex systems, and the affective sciences. We see in these four areas a combination of emerging developments and relevance to core analytic challenges that highlights the power of SBS research to strengthen intelligence analysis.”
This focus on narrative, social networks and affect and emotion represent an extension of the types of intelligence information and analysis they currently focus on (see sidebar).
Narratives and context
“Understanding narratives—from the meaning of cultural traditions, to political themes in press coverage, to trends in social media communications” the report says, “is fundamental for the intelligence analyst, who must understand the content of communications and how and why they are conveyed.” They note that “developments in the study of narratives have been fueled by the exponential growth in data created by social media such as Facebook and Twitter and by the fact that vast amounts of content are now stored digitally, which has made the study of narratives at a large scale much more practical. These capabilities offer new frontiers for applying the study of narratives to intelligence analysis.” They also note that “new technologies are providing more sophisticated ways of using these and other measures.”
Narrative analysis has typically not been part of an insights professional’s scope, but it will be in the future. People have long recognized the power of studying narrative, but it’s been too difficult to do—beyond qualitative reads of small samples—because the amount of text can be overwhelming.
That’s why we’ve built text analytics, and machine learning powered by IBM’s Watson into Maru HUB, our technology platform. We know the future of consumer insights involves thinking outside the survey and want to equip researchers with the tools they need to tame unruly textual information in order to better understand the dynamics of the markets they are studying.
Social networks
The analysis of social networks is not new to the intelligence community. Where they see the opportunity for growth is in the adoption of new and extended techniques from other disciplines. They say “the utility of this research for intelligence analysis rests on interdisciplinary work with other SBS disciplines. It has been demonstrated that social networks have profound influences on many human sentiments (an aspect of affective sciences, which are discussed below), behaviors, and actions that are of interest to the IC.” “Methods of social network analysis”, the report noted, “can also be applied to texts and used to understand features of narratives, how narratives have changed, and the rhetorical strategies (strategic framing) they use. Recent work combines these approaches with the study of emotion to examine identity and influence.”
The insights industry too could benefit from greater use of network analysis. We’ve made a start at it by developing a social media listening tool that allows you to identify influencers who are likely to matter most in a community on Twitter. Identifying and analysing the effect of influencers allows you to see how ideas spread and understand how opinions are shaped.
Affect and emotion
“Emotion and affect are complex phenomena, and many fields—particularly branches of psychology, psychiatry, neuroscience, and biology, but also others, including sociology and anthropology—have contributed to a growing understanding of these human phenomena. Indeed, many topics within the affective sciences are studied across disciplines.”
This is perhaps one area where the insights industry has been making some strides ahead of the intelligence community. Tools that allow you to assess emotion or affect indirectly have gained favor in recent years and have become increasingly affordable and accessible. We have, for example, built Implicit Association Testing (IAT) into Maru HUB. IAT allows you to understand what consumers feel subconsciously about your brand and what influences their behavior.
Our video capability, also built into Maru HUB, allows for facial coding of emotions. This coding can be very helpful in revealing—moment by moment—how people are reacting to content. We’ve even used it to measure emotional reaction to long boring surveys.
Learning from spies
This fascinating report on the future of intelligence analysis is chockful of inspiration for the insights industry. But here, again, are the two big takeaways:
We should look outside our own industry for ideas for improving the insights we produce.
We need to think beyond the survey and go much deeper in the analysis of other forms of data and alternative types of analysis.
Look outside
Researchers tend to think we have unique skill sets and unusual challenges. That may have been true when insights were largely interview driven. But market research is no longer the sole source of truth. The explosion of available information means we need to expand our horizons and absorb data from more sources.
What is important is the insight we generate, not how we obtain or analyse data.
As we look to move the insights industry forward, we should take inspiration from the intelligence community. They are looking to be more multidisciplinary and to absorb ideas from adjacent fields. And there are many fields to inspire us.
Prosecutors, historians, investigative reporters, archeologists, detectives, and physicians are all challenged to absorb varieties of qualitative and quantitative data—some of it conflicting and unreliable—and turn it into insights that fuels decision making. They too must figure out how to distill a myriad of information into a story that will convince a judge, sell subscriptions, encourage a patient to be compliant, or define history. There is much we can learn from how they approach the challenges we all share.
Think beyond the survey
The skillsets the insights industry needed in the past are not sufficient for tomorrow.
With better and more information coming from everywhere, the abilities needed to conduct a survey or moderate a focus group are not enough. As the ESOMAR Director General Finn Raben told me when I interviewed him for The Insights Revolution: Questioning Everything “The demands of the insights profession, moving forward, are very different from what was needed when we all came into the industry.”
We need to learn from multiple disciplines and become information omnivores. Let’s learn from spies and embrace the future of insights.