Thinking About The Consequences: Design Knowledge For Adherence On Pricing Systems

AI systems increasingly support decisions, yet users frequently override recommendations. To address this, research suggests providing explanations to increase trust and adherence. However, users still override recommendations even when explanations are provided. We study why this occurs through a design science study with a provider of a hotel pricing system. We identify two factors driving excessive overrides: misaligned fairness perceptions and lack of consequence awareness. Drawing on Anchoring and Adjustment Heuristic and Social Comparison Theory, we develop two design principles: (1) Fairness Calibration provides reference points like competitor prices to align fairness judgments with market realities, and (2) Consequence Understanding quantifies potential revenue losses to clarify override impacts. We evaluate a prototype through think-aloud studies with hotel managers. Results demonstrate user acceptance and adherence intention. Together, our findings highlight that effective human-AI collaboration requires addressing also the decision context, offering actionable design knowledge for practitioners and expanding XAI research.