Search, Discovery, and the Engineering of Serendipity

Dall-e | Angel Armendariz

"Stepping stones are portals to the next level of possibility. Before we get there, we have to find the stepping stones....computer scientists even have a term, search space, that refers to this very concept - it's the general idea that creation and discovery happen within a space of possibilities that contains stepping stones leading from one discovery to another." - Kenneth O. Stanley & Joel Lehman, Why Greatness Cannot Be Planned

"...all is not a question of rigor, but rather, at the start, of reasoned intuition and imagination, and, also, repeated guessing. After all, most thinking is a synthesis or juxtaposition of advances along a line of syllogisms - perhaps in a continuous and persistent 'forward' movement, with searching, so to speak 'sideways,' in directions which are not necessarily present from the very beginning and which I describe as 'sending out exploratory patrols' and trying alternative routes." - Stanislaw Ulam, Adventures of a Mathematician

Thanks for reading Loaded & Layered! Subscribe for free to receive new posts and support my work.

Everyone is always searching for something. This fundamental truth about human nature led to the creation of my second company - a social media platform built on the premise that everyone is always searching for someone. The catalyst was personal: the "discovery" of my wife through random nearby partner search mobile applications (dating apps).

The app I created acknowledged a deeper truth: we search for people continuously throughout our lives, though our reasons evolve. It enabled users to search and explore people nearby and connect based on their specific needs - whether for business collaboration, socializing, or learning/studying together. Since the iPhone's debut, such applications have proliferated as people seek connection. This constant search for friends, partners, clients, and more remains an unchanging principle.

Yet search and exploration extend far beyond finding people - they are, in fact, the fundamental actions of life itself. Our ability to search and explore largely determines our health, growth, income, intelligence, and happiness. Through searching, we explore the field of possibilities, though we often encounter limiting steps within this process that shape our success or failure.

Consider this: isn't the value of searching and exploring determined by what you know? If I stumbled upon a large diamond, the value of such an encounter would depend on whether I knew both its worth and appearance. 

Similarly, if I discovered an incredible opportunity to purchase property at 50% of market value, I would be limited by my knowledge and understanding. To "exploit" this opportunity, I would need to recognize its value and know how to execute on value extraction - through acquisition, brokering, or other means.

Here we reach the crux of search, exploration, and discovery - it's limited by what you know. The more you know, the bigger your field of exploration and possibilities becomes. In a profound way, the field of all that is possible in your life expands based on your knowledge and mental database, because only through understanding can you recognize valuable opportunities, insights, or treasures. Fundamentally, information becomes the limiting factor in the performance and development of most human-influenced systems. It follows that having some knowledge/information growth function is necessary for continual discovery and invention/innovation.

It goes deeper - because you cannot pre-determine what you will know, how that will change you, and thus what you will be able to search for and discover, you cannot predict your future path to success or the discoveries you will make. Fundamentally, you don't know what you don't know. These are Donald Rumsfeld's proverbial "unknown unknowns."

Artificial Intelligence: A New Lens on Search

This paradox of knowledge and discovery has found an unexpected ally in artificial intelligence research. In "Why Greatness Cannot Be Planned," researchers Kenneth O. Stanley and Joel Lehman present a counterintuitive thesis: the path to success is guided not by predetermined goals, but by novelty. In their world of building artificially intelligent systems, less is more regarding directives. An agent guided simply by seeking novelty outperforms a more "sophisticated" agent given more instruction and human knowledge rules. This counter-intuitive premise seems like an odd thesis for scientific researchers to promote.

Among the research discussed is "Picbreeder," a program that allowed individuals to "mate" pictures and create "offspring" pictures, much like horse breeding. The fundamental insights were twofold: 1) Simple "blob" images can produce remarkable life-like images in as few as 74 generations, and 2) Life-like images emerge only when no goal is attempted. Simply breeding interesting images leads to worthwhile results. In fact, lacking a goal better predicts meaningful image creation than deliberate plans. This approach proves most useful when the solution or answer is unknown - the realm of invention, innovation, and novel creation.

Novelty Leads to Magic: The Biological Connection

The power of novelty extends beyond artificial systems into our own biology. Just as AI systems thrive on novel experiences, our brains are literally shaped by them. Novelty stands as one of three known pillars of neurogenesis and neuroplasticity (alongside exercise and enriched environments). This biological foundation helps explain why effective searching and learning are so deeply intertwined.

Joseph Altman and colleague Gopal Das first discovered neuroplasticity in the 1960s, though it didn't gain widespread awareness until the 1990s through Elizabeth Gould's work. Altman and Das discovered that the mammalian brain (rats) generated new neurons into adulthood. Using thymidine labeled with tritium, a radioactive isotope, they observed its presence in new cells. While this indicated neurogenesis occurred, the mechanism remained unknown until the discovery of stem cells. Further research revealed fascinating discoveries deeper in the causal chain, including epigenetic processes - whereby existing genes can be turned on (expressed) or off, leading to physiological and physical changes.

One implication of neurogenesis and neuroplasticity is that inducing these mechanisms improves learning. Studies show this also preserves existing neural networks and memories.

From Biology to Business: Search in Practice

"If everyone is thinking alike, then someone's not thinking." - George S. Patton

These insights about novelty and learning find practical application in the business world, particularly in sales and business development. Here, the search for opportunities and connections becomes a structured practice, though one that still relies on our fundamental capacity for exploration and discovery.

In business, the search for insights, strategies, and ideas never ends. Consider business development and sales. In this world, sellers and business development reps (BDRs) look to engage prospects and make them clients. They engage in a perpetual search & discovery motion. The model breaks down as follows:

BDRs - maximize outreach to find qualified leads for sellers

Sellers - explore the needs of customers to find the right product/service fit & convert leads into paying customers.

For BDRs, the best ones search effectively - they know where to search, understand prospects' potential needs, and recognize what good looks like. They identify where to play (prospect domain & medium) and how to hook them into a conversation or meeting.

For sellers, the best ones know how to explore customer worlds - they ask the right questions and discover the constraints and relevant processes & steps needed to close the deal. The initial part of all sales is called discovery - the seller tries to understand and discover the customer's current unmet needs and match their offering.

Here, searching, exploring, and discovery distinguish a terrible seller from a superstar rainmaker. It also differentiates a Fortune 500 company from an "also-ran" company.

Optimizing the Search: Learning from Success

Understanding how search works naturally leads us to ask: how can we get better at it? The answer begins in childhood, where we first master the art of learning through observation and mimicry. This fundamental human capability - what the Greeks called mimesis - becomes more sophisticated as we age, evolving into what we now call modeling.

Anthony Robbins, in his landmark book Unlimited Power, delves deep into modeling's power, using Neuro-Linguistic Programming to specifically model the behavior and mental state of those achieving the results he hopes to clone.

Using the NLP model, theoretically, one can achieve the same results by recreating the mental state and actions of the person one wishes to model. By using sub-modalities (visual, auditory, and kinesthetic inputs) to create the same mental mind state, and then repeating the behavior identically, we can achieve the same result. While this approach can yield impressive results, the more interesting aspect is that by searching for someone currently executing the desired behavior and achieving the desired result, you can discover a successful approach to the problem you wish to solve.

I first deliberately used this approach as a high school telemarketer. My goal was three sales per hour, selling long-distance calling services. After failing in my initial approach, through the way I had been trained, I searched for better-performing peers to find out what worked. I found colleagues regularly exceeding three sales per hour. By "shadowing" them, I identified patterns in their approach. Instead of the full "NLP" Tony Robbins approach, I simply mimicked their cadence, volume, tone, and word tracks. Almost instantly, I performed similarly. This hack can be called - search for what good looks like and clone it. It works in environments with known targets or benchmarks regularly achieved by some cohort.

The Quality Question: Defining Good Search

As our understanding of search deepens, we face a crucial question: how do we know what's worth finding? This challenge becomes particularly acute in a world of limited time and energy, where choosing where to search becomes as important as how to search.

Given our finite time and limited energy, we must limit our search domain. Put another way, we must choose what we pay attention to, on our terms. Failing to deliberately choose your path puts your life in others' hands. As we age, making trade-offs in our treasure search becomes increasingly necessary. In modeling individuals, having a good sense for pattern recognition, i.e., knowing what good looks like is useful. Becoming a good explorer requires discerning legitimate skill or achievement from accidental achievement. Several principles help effectively discern worthwhile exploration approaches. When searching for something good to copy or replicate, we can be deceived by associating results with efficacy.

Business provides the clearest example. During good economic times, winners abound. A typical economic cycle has particular characteristics. First, low interest rates create an easy money environment, where debt is cheap. This leads to business and consumer borrowing for investment or purchases. While the initial wave leads to smart investments, copycat investors/buyers perpetuate this cycle. When strong enough, almost all boats float - everyone makes money. This further drives fear of missing out (FOMO), creating an even greater wealth effect. Everyone participating believes themselves smart - equating wealth with intelligence & competence - where mostly there is none. Deciphering true competence from mere mimesis becomes challenging. When the tide turns, the less competent become exposed and fall victim to their ignorance, usually blaming external factors rather than their incompetence.

To discover success's true causal elements worthy of copying, one must ask: was the result 'because of' or 'in spite of'? When everyone wins, this becomes very difficult. To boil down an effective approach, we must focus more on process, and secondarily on results. This explains why anecdotes can mislead - small sample sizes with many variables make discovering a causal chain extremely challenging. These stories contain biases and subjectivity, usually mistaking correlation for causation. For instance, I have a friend who eats chocolate cake regularly, doesn't exercise, and remains thin. Is this a worthwhile strategy for those seeking thinness? Two approaches add rigor and truly discover valuable approaches:

1. Increase the sample size - find more thin people and determine their diet and activities. A larger sample size likely reveals more frequent, likely successful patterns.

2. Try it yourself - eating cake regularly without exercise will yield results, likely poor ones requiring course correction.

Given limited time, learning from others instead of trying everything ourselves generally proves smarter, saving time and accelerating results. However, in truly novel realms, as mentioned earlier, you cannot increase your sample size as no sample exists... although there's an additional caveat...

Beyond Trial and Error: The Simulation Revolution

The evolution of search strategies took a dramatic turn with the advent of computers, introducing a new dimension to how we explore possibilities. This technological leap allowed us to move beyond simple trial and error into sophisticated simulation, while still preserving the value of traditional axiomatic thinking.

Exploration via simulation became possible with the computer's advent. By creating virtual worlds, researchers began simulating realistic situations and modeling scenarios. In fact, one of the first major simulation milestones occurred with the Manhattan Project at Los Alamos.

The simple version of how simulation works contrasts with the axiomatic method. Simulation is simply trial and error on steroids - try many different things until you land at a best answer.

The axiomatic method seeks to find the one correct answer or way of doing things through logic and principles - usually mathematics. This traditional approach has been incredibly important in science, mathematics, and physics evolution. Together they encompass a broad range of strategies for discovery and search.

This also explains an interesting dilemma most of us have noticed: Many rich people are dumb and many smart people are not. The explanation lies in our simulation and axiomatic models. Typically, the not-so-intelligent person who becomes wealthy tries doing something - start a business, etc. They don't spend time trying to find the right answer before beginning. They participate in real simulation, aka trial and error. Eventually, this cohort, with its bias for action, has more attempts, leading to increased probability of success through mere trying. Others, perhaps more intelligent, look for the test answers or the one answer needed before beginning; abundant data seeking might also discourage them from trying as they learn their success odds are poor. They use the slower axiomatic method. Repeat this scenario among a large population and due to the volume of "less smart" simulators versus "smarter" axiomatics, we arrive at the population's wealth skew. This is, of course, a gross generalization, but directionally correct.

Engineering Serendipity: The Strategic Approach

All these insights - from artificial intelligence to biology, from business to simulation - culminate in a practical question: how do we make search work for us? The answer lies in what I call engineering serendipity, a deliberate approach to creating conditions where valuable discoveries become more likely.

For effective search, we must define a domain. For instance, if you want to become a banker doing big deals for Fortune 500 companies, what should your geographical domain be? Should it be Tulsa, Oklahoma, or New York City? The answer is obvious, but important. When deciding how we search for treasures, whether insights or opportunities, we need to think rigorously about the domain.

In business strategy, we would seek to answer 'where do we play?' This helps define our chosen market, including demographics, geography, segment, industry, etc. A common tell of a business or seller lacking domain definition is a willingness to work in all markets/industries/geographies. For instance, if a realtor says they work everywhere in the city, run for the hills. What this really means is that they're simply opportunistic and haven't mastered any particular area well enough to be truly valuable.

By defining the domain, you limit your search space but conversely multiply your opportunities to encounter something meaningful. I call this process engineering serendipity. With a defined domain, combined with a learning function to continually augment your mental database, you hone, over time, a serendipity engine primed to rendezvous with Caerus - the Greek god of opportunity.