When Awareness Fades and There Is No Support, the Phisher Has an Easy Game

Background: Phishing has plagued our digital communication for many years now. Every day
more phishing attacks are perpetrated and more reports of phishing infiltration in businesses or
private lives make headlines. Phishing presents a particularly insidious threat to communication
because the difference between phishing and legitimate emails can be all but imperceptible, even to
users who have received training in phishing prevention. Therefore, phishing awareness measures
and tools need to improve in design and thus enable users to decide more effectively between
legitimate and phishing massages. However, one major challenge to the effective prevention of
phishing is the long-term impact of awareness interventions. Previously, there has been a gap
in knowledge about how long phishing awareness measures last and what refresher awareness
measures are potentially effective. Furthermore, more information is needed, whether phishing
awareness measures or tool support is more effective and if a combination of both could further
improve the user’s decision process. This thesis tries to close these research gaps and is therefore
... mehr
separated in two parts: Part I is systematically analyzing the effectiveness of three phishing
awareness measures over different periods of time, including refreshment and Part II is a systematic
evaluation of a phishing awareness measure against and in combination with a tool support.
Additionally, the strengths and weaknesses with regard to specific phishing tricks are analyzed
for all interventions of Part I and II.
Methods: Overall, four online user studies have been conducted to evaluate three different phishing
awareness measures (video, e-learning, and workshop), four refreshers (video, short text, long
text, interactive e-mail example), and one tool support (link-centric warning). In Part I, three
retention studies have been conducted, whereby in study 1 a short awareness measure (video) was
implemented and tested over a period of eight weeks (N = 89) and in study 2, a more extensive elearning
was conducted and tested over a period of five months (N = 46), both as repeated measure
design studies. Additionally, in study 3, a between-subject design was chosen to systematically
evaluate a workshop over a period of twelve months with five retention time points (after 4,
6, 8, and 12 months) in a sample of N = 439 employees. Additionally, four refreshers (video,
short text/poster, long text/leaflet, interactive e-mail example) were evaluated at the point in time,
where the awareness gained through the workshop had no significant improvement anymore.
In Part II, a phishing awareness measure (NoPhish video) and a tool support (TORPEDO linkcentric
warning) was systematically evaluated in a sample of N = 420 clickworker participants as individual and combined interventions over eight groups: two control groups (status quo: status
bar and tooltip), two intervention groups with awareness measure NoPhish video (video + status
bar, video + tooltip), two interventions groups with tool support (TORPEDO with and without
tutorial), and two combined interventions (NoPhish video + TORPEDO with and NoPhish video
+ TORPEDO without tutorial). Across all studies, signal detection theory was applied to evaluate
the effectiveness of the interventions according to the values sensitivity d′ and criterion C.
Statistical analyses were dependent on the study design as repeated or single measures ANOVAs
conducted with statistical program R.
Results: Overall, n = 856 participants were analyzed (n = 22 in study 1, n = 16 in study 2, n =
409 in study 3, and n = 409 in study 4). As a result of Part I, it was found in study 1 that the
trained awareness of the video was still significant after eight weeks and that in study 2, the gained
awareness through the e-learningwas still significantly improved after five months. The evaluation
in study 3 revealed that the trained awareness by the workshops drops back to the starting level
between the fourth and sixth month. However, it could be found that three out of four refreshers
(video, long text/leaflet, and interactive e-mail example, but not the short text/poster) had been
successful to refresh the lost awareness after six months. With the help of the three refreshers, the
increased awareness lasted another 6 months until the evaluation after 12 months. In addition,
different phishing tricks were evaluated separately across the three studies. In particular, it was
found that the phishing trick "Small Deviations in the Domain“ was difficult to detect across
different phishing awareness measures. The results of Part II showed that a video as a phishing
awareness measure and also a link-centric warning as a tool support both significantly improved
sensitivity when compared to status-quo measures and tools alone. The link-centric warning and
the accompanying tutorial outperformed the video, while the tutorial played an essential role for
the link-centric warning to be most effective. The combination of tool support with tutorial and
video achieved the best sensitivity values overall. Additionally, it is only this combination that
achieves near-optimal effectiveness with regard to the dangerous phishing trick "Small Deviations
in the Domain“ category.
Conclusion: This work provides important insights into the effectiveness of phishing awareness
measures over time and how the effect of phishing awareness measures could be maintained by
refreshers or further improved by the combinationwith phishing tool support. This evidence-based
recommendations will help users to make informed decisions in the face of phishing threats and
help to decide when to plan refresher training and in which form to give that training: Firstly, the
workshop and the e-learning as interactive measurements performed very well and the awareness
lasted for a period of four to six months till a refresher was needed. Secondly, a combination of
a short video with a phishing tool support provided excellent results. The success of the video
must be particularly emphasized, as it is a very short five-minute measure that is therefore easy
to implement. Additionally, the phishing trick “Small Deviations in the Domain” seemed to be the most difficult to detect and should therefore be given greater consideration in phishing
awareness measures. Overall, these findings can make a difference for both the advancement of
the research about phishing prevention as well as for organizations and everyday users coping
with the ever-evolving threat of phishing.