Do Incremental Improvements Add, Multiply, or Something Else?

Suppose you make an x% improvement followed by a y% improvement. Together do they make an (x + y)% improvement? Maybe. The business principle of kaizen, based on the Japanese ?? for improvement, is based on the assumption that incremental improvements accumulate. But quantifying how improvements accumulate takes some care. Add or Multiply? Two successive 1% improvements amount to a 2% […]

More Details

Web Analytics and Statistical Bias

Sample Bias The people who block ads and cookies differ from those who don’t by more than just their aversion to ads and cookies. Around 30% of users actively block ads. Sometimes this also means blocking the JavaScript that drives analytics. Technologically savvy, privacy-conscious users are more likely to use VPNs, ad blockers, etc. This group may […]

More Details

HIPAA Considerations for Google Analytics and Alternatives

PII and Data Thresholding It is explicitly against Google Analytics terms and conditions to supply Google with personally identifiable information (PII). Fore example, neither unhashed usernames nor email addresses can be used as User IDs or supplied as custom dimensions. When exploring Google Analytics 4 (and you have Google Signals turned on to allow inclusion […]

More Details

Browser Cookies and Privacy Laws

What is a Cookie? A cookie is a small chunk of data placed on a user’s computer when they visit a website. Cookies are not new—they’ve been around since the beta version of the first web browser—but there has been more discussion of cookies lately due to privacy concerns and legislation to address privacy concerns. […]

More Details

Early Stopping and the Bayesian Formulation

The basic question that A/B testing answers is which of two options is better, such as which of two page designs customers prefer. You may run with whichever option seems better, even if it is only slightly better. But sometimes simply asking whether A is better than B, or vice versa, is not the right […]

More Details

Can You Look at Experimental Results Along the Way or Not?

Suppose you’re running an A/B test to determine whether a web page produces more sales with one graphic versus another. You plan to randomly assign image A or B to 1,000 visitors to the page, but after only randomizing 500 visitors you want to look at the data. Is this OK or not? Of course […]

More Details

Client-Side Versus Server-Side Testing

There are two different ways to test a page. The most common approach, client-side testing, sends all users the same the page, but then JavaScript running on the client loads changes that alter the page for some users. This has the advantage of being simple to set up. The other way is server-side testing. With […]

More Details