Be Good at Being Wrong

by Brian Toomey, JB Analytics CEO

At JB Analytics we work hard to be right, to give good advice and strategy, and to do so from day one. But we try even harder to be good at being wrong. We aim to learn quickly, limit the damage/downside, and to be friendly and humble about it.

Here are some key principles we’ve picked up along the way. They’ve proven to be central in building successful businesses–both our own and our clients’.

We are looking for organizations that have a similar culture and approach, and thrive with those that do. If, on reading this essay, you think we would work well together, we’d love to hear from you.

1. Have the crucial conversations

The first sentence of Tolstoy’s Anna Karenina is:

“Happy families are all alike; every unhappy family is unhappy in its own way.”

So too with businesses; every happy one is the same. Happy, thriving businesses all:

Provide an excellent product or service
In an environment that people love working at
At the lowest cost they can without sacrificing quality or team morale
With a focus on adaptive and continuous improvement

And yet. And yet…

So often at JBA we are hired to help optimize a business only to have things go sideways because of subterranean objectives and misaligned incentive structures. There are always tradeoffs and attempting to deny this can make your website and business into a bloated Rube Goldberg nightmare.

Examples are unique, but patterns include:

Business Group A is vying for power and resources against Group B instead of serving the larger whole.
A team(s) (design / product / marketing / etc.) is trying to build what they want (brand/website/product/etc.), not the one their customers want.
A dev team is trying to build The Perfect Mouse Trap. The inherent limitations of systems and the shifting needs of an evolving business create irreconcilable tensions with their quest and the organization more broadly.
Time horizons are mixed. Some people want to scale as fast as possible today, while others want to conserve resources until improvements have been made or more funding secured.
No one can agree on who / what the website site is actually for. (e.g. investors vs users, alumni vs prospective students, displaying social value vs driving sales or leads, etc.)
Institutional forces prevail, when either:
- Inertia prevents admitting where a change is needed
- Momentum favors of the Next Big Thing and fails to keep what is working
  - see: Chesterton’s Fence (tldr; don’t destroy the fence until you understand why it was built and what its value is/was.)

These and other such conflicts are the silent heaviness lurking in the corner of every meeting, draining morale and profits. Internally, we often refer to the Artax of such unacknowledged forces and tensions underlying work.

Perhaps the most challenging thing is that often both sides of these simmering cold wars have good points and a role to play. But each side is tragically unable to admit the value of their organizational adversaries.

At JBA we:

Strive to have the hard conversations upfront and as soon as we see them, guided by equal parts humility, directness, and warmth.
Seek to facilitate conscious discussions to transform unconsciousness tensions into consciously navigated tradeoffs.

It’s always better this way.

2. Remember, it’s not about you

Yes, “the customer always comes first” is a platitude. It’s also fundamental to success and difficult to really put in practice.

Making the customer first doesn’t only mean trying hard to serve and delight them. It also means working relentlessly to see the world through their eyes.

We had a client in the jewelry business. Their family had owned the business for 4 generations. They insisted, despite actual data to the contrary, that we should call necklaces “neckpieces” on their website.

Absurd, right? You don’t need data to know that customers themselves shop for “necklaces,” while industry insiders call their inventory “neckpieces.”

We can guarantee you are making at least one subtle mistake of this kind. Seeing through your clients eyes is hard, continuous work. It can also be humbling and unpleasant. There are likely some crucial conversations to be had.

3. Be a Scout not a Soldier

Although testing is hard (c.f. the current replication crisis in science), and there are a million ways to fool yourself with data–there are even more ways to fool yourself without it.

As Richard Feynman has famously said: “The first principle is that you must not fool yourself and you are the easiest person to fool.”

The unconscious power of motivated reasoning–trying to make some ideas win/lose or to attack/defend ideas–is a critical force for fooling ourselves. It comprises what Julia Galef, co-founder of the Center for Applied Rationality, calls the Soldier mindset. Its counterpoint is the Scout mindset: working to get an accurate picture of reality, even when it’s unpleasant or inconvenient. Researchers have found traits that support the good judgment of this mindset include:

Curious: More likely to say they feel pleasure when they learn new information or feel an itch to solve a puzzle.
Intrigued: More likely to feel intrigued when they encounter something that contradicts their expectations.
Value Flexibility: More like to say it’s virtuous to test your own beliefs and less likely to say someone who changes their mind seems weak.
Grounded: Do not identify their self-worth (good/bad, smart/stupid, etc.) with being right/wrong about a given topic or idea.

Orienting to this mindset can be as simple as asking if our effort and work are focused on gathering information or defending / attacking a given position. Organizations and decision-makers can support the shift toward making better decisions with testing.

Formal testing provides a statistically valid and impersonal means for gathering intelligence and resolving conflict among differing opinions. Sadly, most organizations don’t test enough.

The Islamic Sufi scholar Al-Tirmidhi said “trust God, but tie your camel up.” So too with our precious ideas about how to improve the business. We can place our faith in expertise, intuition, etc. and also, be pragmatic enough to test our ideas in the real world. Of course, this can be humbling if you’re certain you’re right and end up on the losing side!

We work hard, read the relevant academic and business literature closely, have multiple PhDs on our team in pure and applied mathematics, have run hundreds of tests, and are still wrong about 35% of the time in our formal testing.

We also can’t test everything. Testing, as noted, is hard to do. Tests take time to produce insight, which translates to a combination of money and traffic/volume. This constraint offers an excellent reason to think carefully about what we test so our testing budget gives the best ROI. We need to pick our shots.

Even in cases where strict A/B testing is not possible for technical reasons, additional forms of analysis may be appropriate and chart a path toward being more scout-minded about a given problem or question. Once we’ve adopted a more curious approach, tested our ideas, and generated actionable insights–changes need to be implemented. Without follow through, we are only paying lip-service to better decision-making as leaders and organizations. What we learn may be bracing, may contradict assumptions or prompt inconvenient changes. But that’s part of being a scout and not a soldier. Informed change is central to positive, sustainable improvements.

4. Stay humble

Think back 3 years. Certainly you were wrong about something huge. Goodness knows we were. It’s simple to understand and even easier to forget that this is true right now as you read this very sentence.

Let’s act like it! Always keep in mind that you are definitely wrong about something big and you can be more humble, not in the sense of false modesty or meekness, but in the sense that you are pliable, and eager to learn.

5. Don’t reinvent mistakes

You are making a website to help sell a good or service; you are not Tim Berners Lee inventing the internet. This is not a new task, many before you have done this and left clues on how to do it well.

You (and we) are not the first team to make faceted navigation, stand up ads, track visitors, or build a checkout. Your product or service needs to have a unique selling position; you very, very likely do not need your website itself to stand out by rethinking conventions like logo top left or how navigation works. Instead:

Do extensive research
Call on expert knowledge
Start from best practices as a base of iteration

This will help save everyone time (and money).

6. Don’t depend on a broken ruler

Perhaps the simplest path to being wrong is a broken ruler.

More than half of the digital analytics we audit are misconfigured and spitting out data that is wrong. Most of the rest are failing to record business-critical metrics. As a consequence reporting and team decision making is over reliant on secondary or proxy figures.

Don’t do this.

Get a good, working ruler and measure what matters.

7. Admit you don’t know the future

Everyone knows that no one knows the future. Everyone also knows we don’t act like it. Implicit in big projects are project plans and forecasting. Lurking within them is the fatal assumption that we can know what is going to happen in the future confidently enough to build a reliable structure inside of it with no mid-course corrections or external feedback.

If you really could soundly plan a whole month/quarter/year in advance why mess with Google Analytics and incoming data at all? To put it starkly; if you truly value data, then plan to use it to change on the fly.

Of course, we understand that goal setting and plans of action are necessary to orient teams, allocate resources, and so on. However, failing to acknowledge the inherent flaws and weaknesses in forecasting can leave organizations vulnerable to a range of costly errors and conflicts.

The truth is: forecasting is easy, but forecasting well is incredibly hard. Be prepared to have the crucial conversations about:

Implicit assumptions about your market and customer base
Exposure to sudden (and gradual) changes to your market and customer base
How every organization lies to itself about how long a Big Project will take; it’s certainly more than you are planning.
- For the interested reader, there is extensive scientific literature on this
How to make use of ongoing learning to inform, adjust, and even scrap The Plan.

8. Remember the act of measuring changes things

The economist Horst Siebert coined the phrase “Cobra Effect” to encapsulate a story from the British Raj in India. The colonial British government was concerned about poisonous snakes in the capital city of Delhi so they offered a bounty for dead cobras. While this was initially effective it had the perverse effect of incentivizing people to breed cobras, which in turn caused the program to be canceled and then the surplus cobras to be released. Now there were more snakes. This is also known as Goodhart’s Law:

“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

Examples abound, both serious and humorous:

The Taiwanese government has pleaded with people to stop changing their name to “salmon” after dozens made the unusual move to take advantage of a restaurant promotion. (source)
Farmers in Afghanistan pretended to set up Taliban camps that the US would then bomb. The farmers would collect the remnants of millions of dollars worth of bombs and sell them for $100 as scrap metal. (source)
Spotify pays by the song. Two three minute songs are roughly twice as profitable as one six minute song. So songs are getting shorter. (source)
Tom Steyer reportedly spent $2.9 million in 30 days on Facebook ads begging for $1 donations so he could get 130,000 small-dollar donors and qualify for the presidential debates. (source)

This Cobra Effect pattern is ubiquitous in organizations driven by a shallow approach to data. For example, if you set your KPI as:

Leads: May incentivize cheap, but poor quality leads to drive that number.
Sessions: Tempting to buy cheap clicks from uninterested users to drive that metric
Sales conversions: If “more sales” == “better,” you may well be incentivizing your team to focus on the least profitable sales.

The antidote to this is to have serious conversations about not gaming metrics, putting guardrails on your strategy, and staying focused on sustainable growth.

9. Prefer small, reversible changes (options matter)

A confession: We hate website redesigns.

They take months, are almost always late and over budget, there’s often been much wailing and gnashing of teeth, 100 things have changed, and the net result to the business is usually unclear if not notably minor.

But wait, there’s more! It’s actually worse than that– some of the 100 changes were positive, and some were negative. But now no one knows which ones were which.

What’s the alternative? Change the smallest thing you can that you think will meaningfully improve the business and have tighter iteration cycles and continuous improvement.

We realize that sometimes you need to scrap the code and start again. But the need for this is much rarer than people think (remember Chesterton’s Fence?). Small changes are much more cost effective, reversible, and you can learn along the way.

Smaller, faster changes beat slower, bigger ones virtually every time.

10. Navigate between extremes

Aristotle argued that virtue exists as an average between extremes. We find that framing useful for thinking about when to act and when to measure.

Too much measurement? Accept complexity and imperfection.

Some organizations try to measure too much. They pour resources into staring at hundreds or thousands of desiccated numbers, bloat their sites with too much tracking and are generally paralyzed. Perfect attribution is impossible.

You are not going to solve problems of measurement and inference that have been with us since the Scientific Revolution (and always will be.) Admit you can’t measure everything, and that’s fine because no one can.

Too little measurement? Embrace informed intuition.

We’ve all worked with organizations where people demand things based on their guts when good data is available to inform decisions (either within the business or in general best practices). Let available data inform instinct and experience.

We’ve all also seen and possibly worked in places where people failed to track things like the costs of or return on investment for a Big Project when they easily could have. Measure when you can. An absence of metrics is an absence of quantifiable results and feedback. In turn this can lead to an absence of concrete accountability both for wins and losses, diminishing the capacity to discern where to focus effort and improve.

Too biased to action? Let knowledge develop.

Many organizations have a bias toward action. Sometimes the best option is to leave something be. In the interim, gather data and focus effort elsewhere. But the temptation to meddle is strong, especially when a given person or department feels the need to justify themselves by generating “results” or “taking action”.

Once again, don’t forget Chesterton’s Fence. Instead, take seriously the idea that, “we don’t know yet, so let’s either wait or make small changes,” is often the path to wisdom and sanity.

Too biased to inaction? Do the next right thing.

Fear and insecurity are real and perfectly understandable human hindrances. Some environments don’t support making mistakes. People want to avoid looking bad. Perfectionism rears its ugly head and someone wants something to be “just right.” The reasons for inaction are many. But by the same token, sometimes swift action is called for. There are easy fixes, low-hanging fruit, and houses on fire that need attention.

Opportunity costs are real but often go unacknowledged. Delaying action to get something “right” may cost valuable opportunities to learn; it may also cost current revenue or leave significant revenue on the table. Sometimes the best thing to do is to do the next important/necessary thing, now.

Clever readers will notice this is hard! There isn’t a single structure or set of principles to resolve each tension. Instead, we need to navigate between extremes of action/inaction and too much/little reliance on data. No way around it. That’s a conversation and a balance that needs continual vigilance and renegotiation.

We’ve heard leaders in other organizations sum some of these principles up using a few short maxims. “Don’t try to boil the ocean” serves as a warning against doing too much to the point of doing nothing. “Don’t let the great be the enemy of the good” serves as a warning against pursuing dysfunctional forms of excellence at the expense of genuine improvements. In both cases we see the wisdom of moderating against extremes at work.

11. Don’t forget to fly the plane

On December 29, 1972 Eastern Airlines Flight 401 from NYC to Miami crashed. The reason? The entire flight crew was trying to figure out if they had a faulty landing gear light and failed to notice the autopilot had been accidentally deactivated. When they realized it, it was too late. The plane crashed killing 101 of 163 persons aboard.

While the above example is admittedly dark, we include it because if it can happen to a well-trained, hard-working flight crew where mistakes can lead to the loss of human life, then certainly it can happen to us. Take a moment and ask — who’s flying the plane?

For websites this means asking: What is the one, core thing that your website does? If/when you have a clear answer, spend 90%+ of your effort doing this thing better and be sure to get out of your own way. Hat tip to our friend and colleague Dilo on this one.

12. Exercise & Eat Your Vegetables

We know, it’s boring. But…get your title tags and meta descriptions in the right character count. Be benefit-focused rather than clever in your copy. Use photos of smiling people to engender trust, not art & photography that looks cool. Stop using the carousel and any of its unholy progeny.

The unspoken truth in our line of work is that most of digital excellence is the unsexy work of not getting anything basic wrong. This is also why so much of it goes neglected. Pursuing excellence means following through on recommendations & implementing best practices–even, maybe especially, when they’re unexciting.

It’s radically underrated to just not get an obvious F on anything on your organization’s report card. This is, in part, the famed Theory of Constraints from Lean Management, e.g. ‘just fix the weakest link’ repeatedly. In truth though refining and improving the digital infrastructure can be complex, and needs to be approached with both subtlety and appreciation for detail.

Businesses and websites are often more like ecosystems than tightly controlled production lines. So we need to think holistically and systemically about the core meta question:

Given the real constraints of the world, how do we realistically forge a sustainable path from here to there while minimizing risk and maximizing upside?

This might mean taking on the weakest link first, but it might also mean knocking out a few easier wins to build morale and resources before committing to face the real dragon. To be sure, there are exciting, complex conversations to be had here.

Reasonable professionals will certainly have good faith disagreements. Time and diligent observation may reveal the best path in time. But all the while, one thing that doesn’t need exhaustive debate is that a big part of being healthy is getting the basics right.

Looking for a Growth Partner to Support Being Good at Being Wrong?

Although there’s so much more to say, we hope this set of 12 key learnings is helpful and resonates with you. If you’d like help making and learning from mistakes to help your business flourish–let’s talk. We’re friendly, and getting better every day at being wrong.