ChatGPT’s Potential for Quantitative Content Analysis: Categorizing Actors in German News Articles

We assessed ChatGPT’s ability to identify and categorize actors in German news media
articles into societal groups. Through three experiments, we evaluated various models and
prompting strategies. In experiment 1, we found that providing ChatGPT with codebooks
designed for manual content analysis was insufficient. However, combining Named Entity
Recognition with an optimized prompt for actor Classification (NERC pipeline) yielded
acceptable results. In experiment 2, we compared the performance of gpt-3.5-turbo, gpt-4o,
and gpt-4-turbo, with the latter performing best, though challenges remained in classifying
... mehr
nuanced actor categories. In experiment 3, we demonstrated that repeating the classification
with the same model produced highly reliable results, even across different release versions.

We assessed ChatGPT's ability to identify and categorize actors in German news media articles into societal groups. Through three experiments, we evaluated various models and prompting strategies. In experiment 1, we found that providing ChatGPT with codebooks designed for manual content analysis was insufficient. However, combining Named Entity Recognition with an optimized prompt for actor Classification (NERC pipeline) yielded acceptable results. In experiment 2, we compared the performance of gpt-3.5-turbo, gpt-4o, and gpt-4-turbo, with the latter performing best, though challenges remained in classifying nuanced actor categories. ... mehrIn experiment 3, we demonstrated that repeating the classification with the same model produced highly reliable results, even across different release versions.