My sense is that the market generally prefers CEOs who are predictable, and this much of an about face is neither a good look nor particularly encouraging wrt leadership.
I'm intrigued by the counterfactual : if he had claimed it was an experiment, would he have gotten as much press? I would be willing to bet yes!
"If you are shipping features without running a *controlled* experiment, you are really running an *inefficient* experiment, where analysis is done by looking at a time-series graph"
I love the statistics about failures in books, drugs, and social programs, but you're missing the one closest to home: for online controlled experiments, or A/B tests, the median success rate is 10%: https://bit.ly/ABTestingIntuitionBusters Table 2.
Indeed. I presumed I was writing for an audience familiar with the success rates of A/B testing programs but fair point - I should have a more optimistic presumption of who this might reach and should include this for clarity! I'll update the post.
Good piece!
My sense is that the market generally prefers CEOs who are predictable, and this much of an about face is neither a good look nor particularly encouraging wrt leadership.
I'm intrigued by the counterfactual : if he had claimed it was an experiment, would he have gotten as much press? I would be willing to bet yes!
"If you are shipping features without running a *controlled* experiment, you are really running an *inefficient* experiment, where analysis is done by looking at a time-series graph"
-- https://www.linkedin.com/posts/ronnyk_abtest-featureflags-experimentguide-activity-6897828685114675201-fJyX
I love the statistics about failures in books, drugs, and social programs, but you're missing the one closest to home: for online controlled experiments, or A/B tests, the median success rate is 10%: https://bit.ly/ABTestingIntuitionBusters Table 2.
Indeed. I presumed I was writing for an audience familiar with the success rates of A/B testing programs but fair point - I should have a more optimistic presumption of who this might reach and should include this for clarity! I'll update the post.