“Chance favors the connected mind” says Steven B. Johnson to those looking for the next big idea in his book “Where Ideas Come From: The Natural History of Innovation”, a book that looks at the macro trends on how innovation evolves.

“It is easy to talk about great ideas as if they were light-bulb moments, sudden epiphanies where everything comes together at once…but that’s rarely how it works.”

Great Ideas Generally Evolve over Time as Slow Hunches Rather Than Sudden Breakthroughs

Johnson believes that good ideas don’t usually come from a lone genius working in a lab, but more often are the product of a connected environment, and are the result from interactions between geniuses, or even just normal people working together.

Although in retrospect great discoveries may seem like single eureka-moments, in most cases they are gradually maturing slow hunches, which need time and cultivation to bloom.

History’s famous eureka moments are overrated, and Johnson believes that half-baked ideas can often be the most powerful: half-baked ideas are Slow Hunches.

Basically, innovation comes from accumulated hunches over time, and more often than not, people don’t really know where the hunch is going to take them, they’re not sure why it’s important, but they’re drawn to it.

How to Foster an Environment that Nurtures Slow Hunches

“The snap judgements of intuition … are rarities in the history of world-changing ideas. Most hunches that turn into important innovations unfold over much longer time frames. They start with a vague, hard-to-describe sense that there’s an interesting solution to a problem that hasn’t yet been proposed, and they linger in the shadows of the mind, sometimes for decades, assembling new connections and gaining strength.”

Because these slow hunches need so much time to develop, they are fragile creatures, easily lost to the more pressing needs of day-to-day issues.

One of the problem of hunches is that they are the first things people forget, because there are so many things to do daily, like going to work, picking up children from school and so on. This is why is so important to write every random hunch down.

But to write them down is not enough. We need to revisit those hunches over time, because suddenly those hunches can combine with others, and spark a new idea or direction.

Darwin’s Natural Selection

Johnson also notices that many great inventors and scientists tend to forget or distort the history of their own discoveries. For example, according to Darwin, the theory of natural selection simply popped into his head when he was considering Malthus’ writings on population growth.

However, Darwin’s own notebooks show that a long time before his “sudden idea”, he had already described an almost complete theory of natural selection. In other words, his final theory was the product of slow hunches that matured over time.

Since Darwin’s idea seems so obvious in retrospect to many, then the conclusion is that it must have come in a flash of insight, but this is now how it happened.

The Creation of the World Wide Web

Another example of slow hunches is the way the World Wide Web was born. In his own account of the Web’s origins, differently from Darwin, Tim Berners-Lee makes no attempt to reduce the evolution of his marvelous idea into a single epiphany. The Web came into being as a classic slow hunch. It started first with a child’s exploration of a Victorian-era how-to book.

Tim was fascinated by the “portal of information” that he had found in the the old encyclopedia. Later, the idea grew thanks to a freelancer’s idle side project designed to help him keep track of his colleagues, and finally he started a deliberate attempt to build a new information platform that could connect computers across the planet. Berners-Lee’s idea needed time to mature. He said:

“Journalists have always asked me what the crucial idea was, or what the singular event was, that allowed the Web to exist one day when it hadn’t the day before. They are frustrated when I tell them there was no “Eureka!” moment . . . Inventing the World Wide Web involved my growing realization that there was a power in arranging ideas in an unconstrained, weblike way. And that awareness came to me through precisely that kind of process. The Web arose as the answer to an open challenge, through the swirling together of influences, ideas, and realizations from many sides, until, by the wondrous offices of the human mind, a new concept jelled. It was a process of accretion, not the linear solving of one problem after another.”

A Failure: The Phoenix Memo and 9/11

The importance of the nurturing the proper environment, in our minds and in our work space, to help slow hunches to develop into breakthrough, can be also illustrated by a dramatic failure, like the Phoenix Memo in connection with the tragedy of September 11, 2001.

Because the ideas that led to Darwin’s natural selection theory and the creation of the World Wide Web were by definition successful ones, it’s easy to attribute their success to intrinsic causes: the brilliance of the idea itself, or the brilliance of the mind that came up with it.

But those intrinsic causes can easily overshadow the environmental role in the creation and spread of those ideas. This is why it is useful to look at the sparks that failed, the ideas that found their way to a promising region of the adjacent possible but somehow collapsed there.

Johnson writes:

“On July 10, 2001, an Arizona-based FBI field agent named Ken Williams filed an “electronic communication” with his superiors in Washington and New York, using the Bureau’s Automated Case Support system, the antiquated electronic repository through which the Bureau shared information about ongoing investigations. The six-page document began with this prophetic sentence: “The purpose of this communication is to advise the Bureau and New York of the possibility of a coordinated effort by USAMA BIN LADEN (UBL) to send students to the United States to attend civil aviation universities and colleges.” This was the now legendary “Phoenix memo,” a warning shot fired—and largely ignored—during the lazy summer months leading up to 9/11.”

Williams had been inspired to write the memo by a pattern he had detected over the preceding year: an “inordinate number” of people of “investigative interest” who had registered for various flight schools and other civil aviation colleges in Arizona.

Why was it ignored?

Because even if it had been addressed to several high-level offices, Williams’s memo quickly entered what investigators later called a “black hole” at FBI headquarters. It remained in limbo, until it was assigned to an analyst to who labeled it “routine” instead of “urgent.” Another agent called it “speculative and not very significant.”

But only a months after the Phoenix Memo was filed,

“Zacarias Moussaoui enrolled at Pan Am International Flight Academy on the outskirts of St. Paul, Minnesota, where he began training to fly a Boeing 747-400 on a simulator. Instructors and other employees at the flight school were immediately suspicious about their new pupil, who paid his entire $8,300 fee in cash. Moussaoui possessed an inordinate amount of interest in the operation of the cockpit doors and flight tour communications, despite the fact that he claimed no interest in ever flying a real plane. The Pan Am employees contacted the FBI, and after a quick background check, Moussaoui was arrested on immigration violations at a motel on August 16.”

Unfortunately, even if an interrogation convinced the field agents that Moussaoui posed an active threat and could be part of a wider conspiracy, the request to seek a search warrant for his laptop was formally denied on the grounds that the evidence for probable cause was “shaky.”

Connecting the dots between the Phoenix Memo and Moussaoui’s training could have certainly supplied enough probable cause to justify examining the contents of Zacarias Moussaoui’s laptop before it was too late: but it didn’t happen.

The Phoenix memo was precisely one of those failed sparks. It contained great wisdom and foresight, but that information proved to be ultimately useless, because the environment was not conducive to make the needed connections.

“The problem with Ken Williams’s hunch was environmental: instead of circulating through a dense network, the Phoenix memo was dropped into the black hole of the Automated Case Support system. Instead of seeking out new connections, the Phoenix memo was deposited in the equivalent of a locked file cabinet. Hunches that don’t connect are doomed to stay hunches.”

“The failed spark of the Phoenix memo suggests an answer to the mystery of superlinear scaling in cities and on the Web. A metropolis shares one key characteristic with the Web: both environments are dense, liquid networks where information easily flows along multiple unpredictable paths. Those interconnections nurture great ideas, because most great ideas come into the world half-baked, more hunch than revelation. Genuine insights are hard to come by…”

“And so, most great ideas first take shape in a partial, incomplete form. They have the seeds of something profound, but they lack a key element that can turn the hunch into something truly powerful. And more often than not, that missing element is somewhere else, living as another hunch in another person’s head. Liquid networks create an environment where those partial ideas can connect; they provide a kind of dating service for promising hunches. They make it easier to disseminate good ideas, of course, but they also do something more sublime: they help complete ideas.”

For more interesting details read Where Ideas Come From: The Natural History of Innovation”