The impact of distribution properties on sampling behavior

Objective: People often have their decisions influenced by rare outcomes, such
as buying a lottery and believing they will win, or not buying a product because
of a few negative reviews. Previous research has pointed out that this tendency
is due to cognitive issues such as flaws in probability weighting. In this study we
examine an alternative hypothesis: that people’s search behavior is biased by
rare outcomes, and they can adjust the estimation of option value to be closer
to the true mean, reflecting cognitive processes to adjust for sampling bias.

Methods: We recruited 180 participants through Prolific to take part in an
online shopping task. On each trial, participants saw a histogram with five bins,
representing the percentage of one- to five-star ratings of previous customers
on a product. They could click on each bin of the histogram to examine an
individual review that gave that product the corresponding star; the review was
represented using a number from 0–100 called the positivity score. The goal
of the participants was to sample the bins so that they could get the closest
estimate of the average positivity score as possible, and they were incentivized
... mehr
based on accuracy of estimation. We varied the shape of the histograms within
subject and the number of samples they had between subjects to examine
how rare outcomes in skewed distributions influenced sampling behavior and
whether having more samples would help people adjust their estimation to be
closer to the true mean.

Results: Binomial tests confirmed sampling biases toward rare outcomes.
Compared with 1% expected under unbiased sampling, participants allocated
11% and 12% of samples to the rarest outcome bin in the negatively and
positively skewed conditions, respectively (ps < 0.001). A Bayesian linear mixed-
effects analysis examined the effect of skewness and samples on estimation
adjustment, defined as the difference between experienced /observed means
and participants’ estimates. In the negatively skewed distribution, estimates were
on average 7% closer to the true mean compared with the observed means
(10-sample ∆ = −0.07, 95% CI [−0.08, −0.06]; 20-sample ∆ = −0.07, 95% CI
[−0.08, −0.06]). In the positively skewed condition, estimates also moved closer
to the true mean (10-sample ∆ = 0.02, 95% CI [0.01, 0.04]; 20-sample ∆ = 0.03,
95% CI [0.02, 0.04]). Still, participants’ estimates deviated from the true mean by
about 9.3% on average, underscoring the persistent influence of sampling bias.

Conclusion: These findings demonstrate how search biases systematically
affect distributional judgments and how cognitive processes interact with biased
sampling. The results have implications for human–algorithm interactions in
areas such as e-commerce, social media, and politically sensitive decision-making contexts.