"I did use it!" - Assessing subjective vs objective cognitive artifact usage

For decades, IS researchers have discussed the reliability of subjective
measurements to assess actual artifact usage. Especially in experimental settings,
as in the design science context for instance, the participants’ usage data of the
evaluated artifact represents an important point of analysis. However, collecting
objective usage data, (i.e. logfiles) is often not feasible depending on the artifact.
In this paper, we present the theoretical grounding of collecting cognitive artifact
usage data using eye-tracking technology. Grounded in immediacy and eye-mind
assumption the participants’ artifact fixations are used as objective usage
measurement. The question remains if in comparison, the collection of subjective
(e.g. perceptual) usage data is sufficient and reliable for such experiments. The
results of our comparative analysis indicate that researchers could use subjective
measurements when comparing different artifact designs and should rely on
objective measurements when testing the effect of an artifact compared to a
control group without artifacts.