Competition

How to Talk About Algorithms

This is part of The Algorithmic Society, an ongoing series about how algorithms and algorithmic thinking have taken over the world. Want to know when new content shows up? Sign up for my newsletter here.

This invasion of one’s mind by ready-made phrases can only be prevented if one is constantly on guard against them, and every such phrase anaesthetizes a portion of one’s brain.
-George Orwell, Politics and the English Language

The term “algorithm” has become the latest linguistic tool for sounding sophisticated when talking about technology, a sort of TED Talk-esque shortcut to identifying with the Silicon Valley set. There’s something about the word itself that mystifies the average mind and imbues the user with an air of sophistication.

When someone at a cocktail party starts using the world “algorithm,” it becomes evident that this person is well-read and keeping up with the times. In other words, it’s become another way to signal status by leveraging (the appearance) of technical knowledge.

The word itself has become a stand-in for the god-like power that the largest internet platforms hold over our daily lives, a device for describing the black boxes behind behemoths like Google and Facebook. How we use the term “algorithm” hints at a sort of cowed awe at the sheer magnitude of their impact on the modern world.

We don’t know how they operate or who really pulls the levers (I imagine someone thinking are there levers on algorithms? as they read this) behind the curtain. Owners of algorithms are fine with this, because it makes their lives easier — they get to keep trade secrets to themselves, and they’re given a veneer of respectability in the process.

Ian Bogost described this dynamic best:

The next time you hear someone talking about algorithms, replace the term with ‘God’ and ask yourself if the meaning changes.
-Ian Bogost

Part of this has to do with the connection between algorithms and the gargantuan, public fortunes they’ve created in the era of high technology. There’s a new sort of American dream associated with algorithms, most famously captured in the movie The Social Network.

There’s a scene in that film where Eduardo Saverin and Mark Zuckerberg (played by Andrew Garfield and Jesse Eisenberg, respectively) are working on an algorithm by writing on a window.

*Eduardo Saverin (Andrew Garfield) works on an algorithm with Mark Zuckerberg (Jesse Eisenberg)*

While the algorithm in this scene is for a hacked-together pet project, the implication of the context is clear: algorithms with humble beginnings can conquer the world. Multiple billion-dollar fortunes were spawned by this primordial bit of mathematics that got translated into code.

Now algorithms are all over our cultural landscape, which is itself dominated by business narratives and the investors who drive them.

For example, it is now unimaginable to create a startup that didn’t offer some kind of algorithm that’s at least a minor improvement over another, existing algorithm. Not only is it not cool, but venture capitalists tend to shy away from such businesses because they lack scale.

In short, algorithms are ever-present in our social media, on our phones, and in every domain imaginable that involves what could be considered “technology.” Algorithms are everywhere, algorithms are in everything, algorithms are everything.

Marc Andreesen famously said “software is eating the world,” but I would argue that the more correct phrasing is “algorithms have already eaten the world.”

Defining Algorithms

There’s an odd contradiction at the heart of our views on algorithms. On the one hand, we all seem to understand (to varying degrees) that algorithms play an outsized role in our lives. On the other, nobody seems to know what the word “algorithm” means.

What’s even weirder about this is that this definitional problem isn’t confined to the tech-illiterate: even computer scientists don’t have a generally-accepted definition of what an algorithm is.

It’s a long-standing debate, and some even say it’s not possible to sort algorithms from non-algorithms because of a computer science concept known as the halting problem.

This ambiguity can be viewed as a positive or a negative, depending on the algorithm and how you relate to it. But that’s all part of a larger exploration that I’ll get into later. For now, we need to come up with some kind of starting point for discussing algorithms that makes sense.

Since there isn’t a single, unified definition of an algorithm, we’ll have to use a simplified (and therefore flawed) one for now:

A set of well-defined steps for solving a specific class of problems.

What’s nice about this definition is that it gives us the ability to generalize the idea of an algorithm beyond its mathematical and computational origins. We can use it to describe any process that’s designed to operate in a repeatable, predictable manner.

We could say, for example, that low-tech activities like cooking involve the use of algorithms. After all, a recipe is a set of unambiguous steps (add ½ cup sugar, bake for 25 minutes, etc.) for “solving” specific food-related problems (how to convert ingredients into food, which in turn solves the problem of being hungry, etc.).

Bureaucratic procedures at a large company or government organization are also algorithmic if we run with this definition.

An employee can also be seen as a sort of algorithm as well — their whole job is to perform a specific set of tasks in order to accomplish goals for the organization. They do this by performing algorithms for each problem they’re presented with.

A Transformative Force

It’s also worth adding another dimension to this definition:

Algorithms transform some set of input values into a desired output or set of outputs.

This is key to understanding all thing algorithmic. Algorithms transform what they take in and generate something novel in the process.

In some sense, this is the most important way to look at algorithms. It makes you realize that there’s some objective involved, that what the algorithms are creating isn’t just math — they’re machines of creation.

Algorithms aren’t just transforming inputs from computer systems, either. As they integrate more and more with physical objects (including people), they are using the real world itself as a set of inputs and creating new landscapes in their wake.

Now we can start to unravel the linguistic consequences of using the word “algorithm” the way we do. By talking about algorithms as our digital overlords — never to be questioned or examined in a meaningful way — we hand them power they don’t deserve.

Even though they’re designed by high-caliber computer scientists, these algorithms are still designed and operated by humans. This means they’re flawed in countless ways, and even our most powerful computers can’t rid themselves of their designer’s human errors.

When you embrace that fact, it becomes clear that talking about algorithms with a glint of admiration in our eyes is often a mistake. While some are worthy of praise, quite a few are far more fragile, inaccurate and exploitable than their owners would like you to know.

More than anything, we need to get rid of this idea that algorithms are simply hand-waving mechanisms for explaining new technology. Algorithms are real, they serve specific purposes and it’s possible to at least begin to understand how they operate if you equip yourself to probe them.

Their impact, despite their flawed nature, is large. We’ve built, and continue to build, an algorithmic society, and it’s simply irresponsible to treat algorithms in such a haphazard way. It is the duty of every intelligent, capable adult in this new world to get a handle on what algorithms are and how they are shaping our world.

And this starts by learning to talk about algorithms not as magical code-driven dragons. It starts by seeing how they’re infiltrating not just our computers, but our very identities, our everyday existence.

They are fractal, spinning themselves through increasing levels of abstraction as they generate billions of dollars and shift our personal lives in ways that even their creators often don’t understand.

Algorithms are, in short, the most important topic of the modern era. It is my goal with this series to give you a glimpse into just how large of an impact they’re having, and then provide you with the tools you need to navigate this world more intelligently.

There will be more to learn, but consider this the starting point.

Losing Games

Ace Eddleman

This is part of my 5 Minute Concepts series, which is designed to help you understand fundamental concepts about subjects like learning, memory and competition in the shortest time possible. Each episode is available in video format on my YouTube channel and audio via my podcast. If you prefer to read, the transcript is below.

Want to know when new content shows up? Sign up for my newsletter here.

Much of what I create revolves around the idea of competing intelligently. My overall hypothesis about competition is that most people do it haphazardly, and expect their own intuitions — mixed with a recognition of incentives — to carry them to victory.

Thinking this way is a serious error. It leads to making the same mistakes over and over again, and creates situations where meaningful learning takes much longer than it should.

What’s more important, in my opinion, is that we recognize just how competitive the world is. Competition exists at all levels of life, all the way down to single-celled organisms. Competition is a key component of evolutionary biology, and there isn’t any form of life on earth that can escape this dynamic.

There’s competition for money, competition for status, competition for relationships. There’s competition everywhere.

Life is competition and competition is life.

With all that being said, there’s an idea that’s just as important to keep in mind: sometimes, you’re playing a game you can’t win — no matter how intelligent you are or how hard you work. When you find yourself in this kind of game, what can be called a losing game, you need to exit that game as fast as you can.

Recognizing losing games like this is a skill in and of itself, one that many people find hard to develop. It’s particularly common in American culture, where we’re constantly told that hard work is the answer to all of life’s problems.

An extreme example I like to use is professional basketball. The first requirement for playing basketball at the pro level is to be very tall, which is something you can’t train for. You’re either born with tall genetics or you aren’t.

This can be a hard pill to swallow for people who don’t hit those height requirements but love the game enough to dedicate their lives to it. Someone who is only average height can spend every waking hour refining their game, doing everything they can to get better, and still come up short.

The problem in this situation isn’t that the player isn’t committed or intelligent enough. It’s just not a game they can win — and there’s nothing they can do about that.

Instead, this same player could find another way to be involved with the game. Maybe they could find work as a talent scout, or a commentator, or a sports writer, and still play for fun in recreational leagues.

Maybe they’re an exceptional programmer or mathematician, and that could allow them to build some kind of technological product that is intertwined with basketball.

Those are all winnable games for this fictional person: they offer odds with large payoffs and none of them have requirements that are impossible to train for.

What tends to drive people like this crazy is the search for glory. They want to do what’s most admired in society, like playing a sport professionally.

The irony is that wasting time on paths like this more often than not generates an excessive amount of unnecessary misery. Most people don’t really know what they want, they just think they know, so they waste their time pursuing goals that other people or society set for them.

That same player who wants to be a player more than anything might find more fulfillment in an auxiliary role than they can estimate.

Instead of wasting years pursuing a professional playing career that ends badly, they should find something else that might even end up being more fulfilling (or even lucrative).

Sometimes you can create the game yourself, and sometimes you have to go play someone else’s game. Either way, you should be doing this kind of analysis on a regular basis. Every now and then, stop and ask yourself: Is this a game I can win?

I’ve failed at this more times than I can count, and it’s cost me dearly on a few occasions. Hopefully you can heed my words and not make the same mistakes I have. Don’t play games you can’t win — find a place where you can play with favorable odds, and then throw yourself into that.

Pattern Recognition

Ace Eddleman

Want to know when new content shows up? Sign up for my newsletter here.

Transcript:

What separates experts from novices? More than anything, it’s pattern recognition. The expert has the knowledge and experience to recognize the most important patterns in any given situation.

While it is possible to leverage pattern recognition in solitary pursuits, and there’s quite a bit of literature on the role of pattern recognition in overall intelligence, my focus today is on how it impacts competition.

Pattern recognition provides two important advantages. First of all, it makes processing your environment much more efficient.

When you’ve seen a pattern a million times and know it’s significant, you automatically tune out other, less relevant cues in your environment in order to focus on that pattern.

This is a mental shortcut, because it means your brain doesn’t have to work hard to find signals within the noise that surrounds you — it just happens by default.

Novices struggle in large part because they lack that ability. They get overwhelmed by the cues around them and try to address all of them at once. The expert knows what to focus on, and therefore can allocate their mental and physical resources on what matters the most.

In this context, you can view pattern recognition as a sort of built-in risk management mechanism. When we see things we recognize, we can more easily classify them as “safe” or “dangerous.” This is true not just in competition, but in life in general.

Secondly, pattern recognition gives you the ability to anticipate what’s going to happen in the future. If you’ve spotted a pattern in an opponent and they don’t show signs of changing it, you can exploit that.

Much of competition is this dance of figuring out patterns, exploiting them, and then adapting to replacement patterns.

This dance happens in donkeyspace, with both sides trying to figure out how to get maximum exploitation out of their opponent’s patterns without creating too many exploitable patterns themselves.

Poker is an excellent example of how pattern recognition can dominate an entire game’s competitive spaces. Everyone sitting at the table is watching and learning, trying to spot patterns (which are referred to as “tells”) that can be used for profit.

Maybe one player always scratches their nose when they’ve got a good hand, so you choose to fold rather than bet against them whenever that pattern shows up. Or maybe they start playing with their chips in a certain way when they’re bluffing.

Top-level professionals are watching those physical cues, plus they’re paying even closer attention to betting patterns. They watch how every player pushes their chips around at every stage of the game, looking to spot ways they can infer an overall player “personality.”

There’s even a common language players use to describe specific types of players, which is itself a mechanism for aiding pattern recognition. For example, those who only play the best cards in a hyper-conservative manner are labeled “Rocks,” and the players who are intensely aggressive are called “Maniacs.”

It’s a general principle that the skill of spotting meaningful patterns in opponents is one of the most sustainable competitive advantages you can create for yourself. Being able to see and exploit the holes in your opponent’s game better than they can means you’re always a step ahead of them.

But there are dangers to be found in our pattern recognition capabilities. To start with, there’s always the risk that we’re focusing on the wrong patterns. There’s a whole metagame of creating the patterns designed to deceive opponents, and if you aren’t aware of that metagame then you can find yourself in hot water very fast.

On top of that, you might see the right pattern but come up with the wrong method for exploiting it. It might be because you lack the experience or execution skill to take advantage of it, or it might be because your analysis of that hole was just off the mark. Just because you can take advantage of a pattern doesn’t mean you will.

To use the poker example again, another player might be aware of their own tells and use that against you. They might know they have a tendency to scratch their nose when they bluff, and deliberately scratch it when they have a strong hand just to bait you into playing against them.

Anyone can (within reason) get better at pattern recognition with practice. But not everyone is capable of getting good enough at it to hijack the pattern recognition habits of other people. This is, in my opinion, the mark of the master competitor.

Think about this next time you have to square off against an opponent in some competitive arena. It’s easy to spot patterns for yourself, but what are you doing to set traps with the patterns you give off? Practicing this will pay gigantic dividends in the long-run if you stick with it.

Specialization

Ace Eddleman

Want to know when new content shows up? Sign up for my newsletter here.

A real-world manifestation of the exploration-exploitation dilemma can be found in the choice we all have to make between being a specialist, which is a path dominated by exploitation, or a generalist, which involves a focus on exploration.

In my course on competition, I tell students that specialists tend to win in competitive games. I still stand by that, but only because I was talking about the specific context of formal games. The real world is much more complex and therefore the answer is murkier.

The problem is that the world is a fascinating place, and there’s so much to learn. Even if you spend all day learning and probing around, you’ll continue to uncover knowledge that makes you feel like you’ve been wandering in ignorance your whole life. This tends to pull us towards becoming generalists.

On the other hand, endless exploration is not a feasible choice for those of us living within a capitalist framework. Nobody is going to pay you to read and think all day, unless you already have a pool of resources to work with.

This is why people who spend their whole lives immersed in creative activities tend to either be rich already, or benefit from some kind of patronage (the Medicis financed many of the great Italian Renaissance artists, for example).

That means most people are expected to become specialists, which is why there’s such an emphasis on professional training.

Going to college and spending time climbing a corporate ladder is the common path for demonstrating expertise within a given domain. Kids in school aren’t encouraged to explore much, and instead are told it’s best to pick a pathway that’ll guarantee a healthy income in the future.

There are some problems with this. First of all, a contradiction lives at the very core of all this: you can’t find the best exploitation paths without exploring first. I’ve covered this before in great detail, but it’s worth reiterating that exploration and exploitation feed off of one another.

By telling people to simply take the path of least resistance — to exploit a known payoff schedule, like going into investment banking — employers, teachers and other authority figures are nipping potential in the bud.

As a society, we’ve decided that securing an income in an established industry matters more than developing new ways forward.

Secondly, there’s a serious risk associated with deep levels of specialization: path dependence. If you go far enough down a path, it becomes near-impossible to turn around and reorient yourself in the world.

For example, if you’d spent 20 years becoming an expert in typewriters right as the personal computer boom began, you’d be in trouble. The payoff for your expertise would have evaporated, and adjusting to this new world would be a tall order (to put it lightly).

This obsession with specialization helps people who are already part of the status quo, and hurts those who don’t fit into the molds those with power hold sacred. It’ll get you a job, but it might also shut off your capacity to live the way you want to.

I remember one time I saw a job ad for a company in the credit card industry, and one of the “Nice to Haves” was, no kidding, “A deep passion for credit cards.” All I could think when I read that was, “What kind of psychopath has a deep passion for credit cards?”

The answer is simple: people who want to get jobs in the many modern industries can’t just be knowledgeable about credit cards (or whatever domain they want to work in), they have to be so specialized that it seeps deep into the fabric of their personal lives.

To put it another way: the world wants you to not only know something about what you do for a living, it wants you to be dominated by that subject until it consumes your whole identity. I can’t help but see this as a problem.

But, again, we come to a quandary. This is the world we live in, and it won’t change overnight. What can a single person do to give themselves a chance at success?

My suggestion here — and it is only a suggestion, because I don’t think this can be fully fleshed out and resolved in a single 5 Minute Concepts episode — is to think about the intersections of your interests. Where do different skills and ideas overlap to create something that could be viewed as “useful” to others?

It’s not an easy task to accomplish, as I can tell you from experience. But if you can come up with some kind of mix that’s both unique and valuable, you’ll find that you’ve flipped the whole formula on its head.

You will be specialized because of your generalist tendencies, and not specialized in spite of them.

Failure & Learning

Ace Eddleman

Want to know when new content shows up? Sign up for my newsletter here.

Transcript:

Failure is a subject that I have a love-hate relationship with. On the one hand, failure is an important feedback mechanism that can drive learning. On the other hand, it’s been fetishized by the self-help crowd and turned into some glorious path to success.

Yes, it is true failures are inevitable and cannot be divorced from the expertise-gathering process. Everyone, even the best in the world in any given field, has experienced failure at one point or another along the way.

Sometimes a particular type of failure — usually some kind of dramatic departure from what you wanted to happen — can act as a beacon for a beneficial change that leads to a big win. That is a very real side of failure, and I don’t want to discount the value of failure in certain situations.

And yet there are all kinds of memes that lead people to view failure in the wrong light, or get used as a means of not doing the hard (but necessary) work of examining why they failed.

The most common meme around failure is the “Man in the arena” quote by Teddy Roosevelt. Now, I’m not knocking the quote — there’s a lot to be said for giving something a shot when others aren’t willing to — but I am knocking how it’s used as a means of hand-waving away failures.

Every time I see or hear someone using that quote, I want to ask that person, “Is failure always meaningful?” The hard truth is that it usually isn’t, and sometimes critical failures completely derail us. Sometimes that derailing is permanent.

It’s important to confront this uncomfortable truth head-on. There are, in fact, events in life that have serious consequences. There are situations where it’s not possible to recover from a failure, as much as we might want to play the role of the eternal optimist.

Why do we fail? More often than not, it’s because we underestimate the problem we’re trying to solve, and overestimate our ability to solve it. Or we don’t even understand the problem in the first place, and as a result we apply an inappropriate solution.

It’s common to lean too much on our own psychology as well. We think that belief or motivation alone are enough, when more often than not they’re only small, insignificant pieces of the puzzle.

We tend to have high opinions of ourselves, so it’s not so shocking that we step on landmines which could have been spotted if we had our egos a little more in check. When it becomes clear that failure is inevitable, we’re shocked at our own hubris — but alas, it’s too late.

After the fact, we’re prone to pointing fingers in the wrong directions. Not to say that there aren’t ever external factors to blame (I’m personally not a fan of the “absolutely everything is the fault of the individual” school of thought), but there are usually signs that mistakes were made ahead of time.

So, now that we have that dark side out of the way, when does failure help us instead of hurt us? What are the heuristics we can all use to figure out if we’re failing in the right direction, or just failing?

While there isn’t a simple algorithm for all situations, there are some guidelines you can use to make an educated guess.

First of all, ask yourself if said failure was unexpected. If it was, what kind of probability did you assign to it? If it wasn’t, what could you have done to stop it? Asking these questions will force you to confront your own estimations of yourself and what you were trying to accomplish.

Second, was something meaningful learned from the failure? It’s critical to be honest here — don’t just make up some sorry, generic phrase to describe the failure like “I learned that I have a long way to go.” That may be true, but you need to be specific — what actions led to this outcome? What could you have done differently?

Lastly, what’s your experience level in this domain? If you’re brand new, it’s normal to fail a ton, and it’s equally normal to learn quite a bit from each failure.

As your experience increases, you’ll hit diminishing returns on each failure because you should be collecting enough data to fix the biggest mistakes over time.

If you possess a deep level of expertise and you’re making big mistakes, then it’s time to go back to the drawing board. The world you’re in may have changed in some fundamental way, and you might have to rethink how you approach solving that set of problems.

You’ll never escape failure entirely — even the best still screw up sometimes — but you should see a decreasing frequency in the biggest failures if you’re learning the right way.

In general, it’s worth pondering the overall value of failure. In many cases it’s not helpful, despite what the gurus would lead you to believe. But it’s also a crucial part of any successful learning plan, especially for beginners.

You could say that failure is both underrated and overrated, it just depends on the nature of the failure itself and who it’s affecting. Think about that next time you hit a snag in your learning.