#106 - Chasing Greatness: Follow Your Curiosity (Part 1 of 2)
Hello friends👋
I should warn you right up. This is not your usual fare. This is a leap. It is also a refusal of pretty much the only narrative I hear about today.
I’m talking about popular advice on goals. Goal setting and goal achievement is an industry in itself, be it weight loss, weight gain, or how to get to your first million whatever in how-many-ever days. In the shadow of such advice, I believe, some insights catch mold.
I also think that when we take such advice and run with it to every possible avenue, we hit dead ends. When we treat the pursuit of our deepest ambitions the way we approach a beach body program, we close down possibilities when we should be opening them up.
So, how should we chase greatness? Is there a right way to chase a completely-nuts idea?
There may be two. Or none, if you choose to disagree. As I said, this is going to be a leap. Here’s the first. Part 2 next week will present the second theory, and tie up the two.
Greatness cannot be planned.
I’m tempted to add true before greatness so that you get that here I’m talking about ideas high on their out there-ness. Things that haven’t been done before. Pursuits so ripe with ambition they are for loonies. A few specimens off this list: the microwave oven (seems regular now but go back a hundred years), Amazon Web Services (AWS), Stanley Kubrick’s film career (critically and commercially recognized across 10 genres), rock and roll.
Last year I read a fascinating book. It questioned the value in setting objectives. It argued against what it called objective-oriented search. It suggested something you don’t hear much about: Greatness cannot be planned. It is un-orchestrated. It is inherently uncertain.
Now before you call me a lover of chaos, here are two facts:
The book was written by two AI researchers, one of whom has been with OpenAI and Uber AI.
I’m just coming off a 12-week plan at work that is the very definition of objective-oriented pursuit.
What follows here is not a how-to piece. It is a why-to piece. My belief is that once you see the sense in aimless wandering, you will be tempted to find your own way to aimlessly wander.
Here are the highlights from this week:
Objectives are for things that have been done before (losing weight, running a race). Not for radical breakthroughs (the modern computer).
No invention has reached its end objective. We continue to build on them like they’re stepping stones (vacuum tubes to computers to AI to…).
Popular advice on goals makes it contrarian to follow your curiosity. But the biggest innovations across history didn’t happen because their pursuits were allowed. They happened because someone was comfortable wandering aimlessly. True innovations come to you when you don’t go looking for them.
Stepping stones are the unit of innovation. A discovery one stepping stone away is close to realization. It makes sense to set objectives for it. Anything further and the path is foggy. (Yet, the path will look obvious looking back.)
On the surface objective-less pursuits might make life feel futile. But that view ignores our knack for smelling potential. Freeing ourselves from the tight grip of objectives could be the gateway to finding new meaning in our lives.
Follow your curiosity
In 1945, a bar of candy melts in the pocket of a scientist as he stands next to an active magnetron, used to generate radio signals in early radars. Curious, he sends for some popcorn seeds. In front of the magnetron, those pop.
Not much later, Percy Spencer, the fifth-grade-educated scientist who in the words of a colleague ‘doesn’t know what can’t be done,’ goes on to use the magnetron as tubes that are the centerpiece of the microwave oven.
Quoting from Why Greatness Cannot be Planned: The Myth of the Objective, authored by AI researchers Kenneth Stanley and Joel Lehman:
Only when Percy Spencer first noticed the magnetron melt a chocolate bar in his pocket in 1946 did it become clear that microwaves are stepping stones to ovens.
Stepping stones. A phrase Stanley and Lehman keep coming back to in the book. We can find it elsewhere as well.
Around 2000, Amazon was growing fast, hiring big, but not building applications any quicker. They were taking months just to build out the compute or storage component for applications. They had teams building their own resources for individual projects, with no thought to scale or reuse. That’s when the smart people at Amazon realized they needed a set of common infrastructure services everyone within could access.
And you would think one day someone bright got the idea for AWS and its transformative potential. Not quite.
If stepping stones are a unit of innovation, a discovery that is one stepping stone away is close to realization. It makes sense to set objectives for it. Anything further and the path is foggy. Yet, the path may seem obvious looking back.
A TechCrunch piece points out that the core systems in AWS developed ‘without any real planning’ over 2000-2003. No one realized in those early years that they had the stepping stone to a business that would become AWS. Andy Jassy, AWS CEO, who was there from the start, says: ‘In retrospect it seems fairly obvious, but at the time I don’t think we had ever really internalized that’.
Start with the end in mind is great advice but is it possible to always know the end? Should you then refuse to start anything for which you do not know the end? Should you only allow yourself to follow a path where it’s been already charted?
From the origin stories of the microwave oven and AWS, it seems there’s more to it.
Can not trying be better than trying?
When the ingredients don’t resemble the final dish, the dish cannot guide us to itself. When the whole is utterly different from the parts, any kind of objective function used to measure progress is a false compass.
The seduction of linear progress en route to the accomplishment of a neatly defined yet audacious objective is what Stanley and Lehman call a false compass in their book. They call it so because they prove that progress in the pursuit of new isn’t linear.
A maze is a good search space to see the chicanery of the false compass. Stanley and Lehman, both AI researchers, tested two algorithms:
1) where the goal was to get a wheeled robot to learn to find its way out of a maze (called objective search), and
2) to have the same robot try only ‘new’ behaviors in the maze with no specific objective (called non-objective or novelty search).
They repeated the experiment 40 times for each algorithm. In 39/40 non-objective tries, the robot solved the maze. Only in 3/40 objective tries, the robot solved the maze.
Why did the robot fail almost every time it was given a goal and hardly ever when it wasn’t?
In the objective search, when the robot came closest to the goal it found itself at a dead end. But it continued with the behavior because that was ‘good’ by the objective function of the algorithm (how far it was from the exit). So, the robot kept ramming into dead ends.
In the non-objective search, the algorithm simply tried to come up with new paths for the robot each time. Novelty was the only thing that mattered. For this, the robot did more and more new things. The book explains the actions of the aimless robot:
So, if the robot goes around a wall it’s never gone around before, then chances are that a slight modification of that behavior might go even further. On the other hand, if the robot does something it’s done many times before, like crash into a wall, then that behavior is ignored and not explored further.
This points to two surprising findings:
1️⃣Following curiosity can be more productive than trying to reach a goal. The robot made its best discovery (come out of the maze) when making any discovery was not its objective, but trying new things was.
2️⃣New ideas are the stepping stones to newer ideas. With each new try, the robot went for a slight variation to the last try. Variation is the key idea here. A stepping stone isn’t the same as adding one more floor to a single-storied house and then one more until it becomes a skyscraper. It is fundamentally different the way a clay hut is different from a skyscraper. If we only ever tried to make slightly better clay huts, we would not end up with a skyscraper.
The paradox in scientific progress
This paradox of discovering when not trying to discover comes up in science again and again. Anyone familiar with scientific research may have heard of blue skies research or basic research. Blue skies or basic research is done without a hypothesis to justify it. It is curiosity-driven. It is pursued as a way of pure investigation to deepen understanding, not for real-world applications.
In recent decades, as the volume of scientific research has snowballed, the blue skies variety has lost its sheen within the scientific establishment. At many levels, that makes sense. The biggest scientific projects should promise bang for the buck. Paradoxically, emphasis on well-defined objectives has contributed to the problem of diminishing returns for science, as Patrick Collison, CEO of Stripe, and Michael Nielsen argue.
In 2018, Collision and Nielsen ran a survey that asked physicists from top-ranked physics departments globally to compare Nobel Prize-winning discoveries in their fields across decades. The duo used those rankings to determine how physicists thought the quality of Nobel Prize-winning discoveries had changed over the decades.
‘Physicists judged every decade from the 1940s through the 1980s as worse than the worst decade from the 1910s through 1930s’, Collison writes.
The results point to a shift toward incremental advances, away from the groundbreaking advances from the first half of the 20th century. Much like the objective-search robot, it appears that modern scientists view progress as a series of incremental improvements that inch-by-inch approach the objective.
But we know what happened to the objective-search robot more often than not? It hit a dead end.
If you, like me, are wondering why this graph ends in the 1980s, that’s because the Nobel Committee has only deemed three discoveries made post-1990 worthy of Nobel Prizes—a statistic small enough to be excluded from the analysis.
These numbers suggest something significant. Even the most free-thinking among us can be driven by the false compass. Objective-led thinking dictates the path of modern science. What cannot be measured should not be funded.
Making your own luck
‘If you discover an ancient artifact while exploring an uncharted river,’ write Stanley and Lehman, ‘it’s not by complete chance—you found it because you were exploring, even if you didn’t know what you would find.’
And if you have spent weeks and months by the river bed, it is likely that you have by now not just come up with ways to find artifacts but you also have honed the thinking to redirect currents and build dams and irrigate surrounding areas.
Scientist Percy Spencer tried enough things with magnetron to find out that it could be useful in more ways than just sending signals to radars. But if his objective was only to find a use for radars, he would probably have found any other potential use a distraction. In that sense, he made his own luck by staying curious.
Goals make us happy when we achieve them, and anxious until that happens. Following our curiosity means we choose happiness all along, not momentarily. There’s no guarantee of a destination but the fun is in the journey: A true explorer is never lost because even getting lost leads to discovery. Destination blindness is a wonderful condition for the eternally curious.
Putting an end point to all pursuits stops us from attempting interesting things. It holds us back from chasing novelty, from looking for stepping stones, from mixing and matching what is there to come up with something that has never been there.
On the surface, objective-less pursuits might make life feel futile. But that view ignores our knack for smelling potential. Freedom from objectives frees us to search for and find new meaning.
Thank you for your attention 🙏 What did you make of this issue? Let me know in the comments!
📢Finally, an end-of-the-week call.
Last week, some of you responded yes to my ask for a show of interest in a private community for keen learners of decision-making. The novice me didn’t realize that Substack doesn’t (or I don’t yet know how) give away subscribers who respond to polls. So now I know a bunch of you are interested, just not who all. May I request those interested to confirm at satyajit@satyajitrout.com. That way I can be assured your details don’t slip out into the ether. The plan is to set up a Zoom call in a couple of weeks and kick things off. I’m excited!
COMING NEXT: Part 2 of Chasing Greatness
Great piece...as always! I wouldn't see it as a refusal of the goal setting narrative. I see it as another truth. Maybe an opposing truth. The genius of the AND might apply here. Sometimes the destination is fixed AND sometimes it's not. Allow for journeys without fixed destinations, and see which gold mine you strike.