Online panels are a major source of participants for both market research and UX research studies. This article looks at the accuracy and variability of estimates from online panels and offers takeaways such as:

1. UX metric estimates vary, but not as much as expected
2. How precise do you need to be? For some studies, a 3% to 20% amount of uncertainty (on top of sampling error) is enough accuracy; in other cases, differing by 10%, 5%, or even 3% can mean making different decisions. Know how much uncertainty you can tolerate.
3. Don’t change panels. Continuing the recommendation from the earlier research: Pick a panel and stick with it when tracking data over time, such as likelihood to recommend a product or brand attitudes. Differences between panels can in many cases exceed real differences in the population.

How Accurate are UX Metrics From Online Panels