In my master thesis research I am looking at comments on news articles on news websites and on Facebook. During this process I have had to find a way to categorize comments. This has been a long, and sometimes frustrating process. I started out, before beginning the research, with a few categories that I thought would cover all comments, but that I expected would have to be changed. And they were.
I have implemented a heuristic research approach. The qualitative heuristic approach is an exploratory research method where the data affects the categories. This means that the researcher should be “open to new concepts and change his preconceptions if the data are not in agreement with them” (Kleining and Witt 2000). This lead to the creation of 5 broad categories, that I for a long time was quite happy with.
But then I decided to be thorough (stupid me), so I did a reliability test. I gave a person not involved with my master thesis a list of 60 comments to categorize, and then compared them with my own categorization. The results were not good: 60% agreement. After tweaking the definitions of my categories, creating a more procedural method of categorization, and coding the comments again, a second and third reliability test gave me a score of 79 and 75%.
A reliability score of 70% or more is often used as a criterion for exploratory research. But a score of 80-90% would be considered more acceptable in most situations, and above 90% is considered acceptable in all situations (Lombard, Snyder-Duch and Bracken 2002, 593).
Things were looking better, but I was still not happy. So I decided to start completely from the beginning with a new method for finding categories. In stead of coding the comments into one category or another, I wrote any word that came to mind when reading a comment. After doing this with enough comments (about 100) a pattern emerged. Some words were repeated, and these word became the foundation of my new and final categories. And with these categories, another three reliability tests gave me a score of 82-92%. And the categories, which I now am quite confident about, are:
Argumentative comments: Arguments contain a proposition that can either be true or false (Blair 2009, 44). These propositions should be testable. They are also formulated for the purposes of persuasion. This means that there needs to be a point of view that the commentator wants someone to adopt, that is backed up by a proposition. Argumentative comments can also contain proofs from classical rhetoric: Ethos, Logos or Pathos.
Opinions: Opinions are comments that are not necessarily meant to persuade, but function as a direct or indirect statement of what the commentator thinks and believes about an issue.
Reactionary comments: Reactionary comments are short expressions of emotions with little or no informative value. They can also be unspecific statements – statements that are not specific enough for the reader to accurately interpret what the commentator is writing about.
Reactionary comments often contain a set of punctuation marks, especially the exclamation mark, or sets of emoticons. Reactionary comments can also be non-verbal. In these cases, the comments contain either only emoticons or written non-verbal expressions, such as “Haha!!”, indicating laughing or joy.
Informative comments: These comments do not directly argue for or against something, although they can be used in discussions to build a case for a point of view. They are meant to provide relevant information – whether or not that information is factual. Informative comments, with the exception of those classified as personal experience, contain testable factual information that can be either true or false.
Derogatory comments: These are comments that uses some form of critique or potentially hurtful discourse, usually directed at another commentator. Put they can also be directed at public figures or the subject of the article.
Humorous comments: These are comments that, with the intention to be funny, brings together two disparate ideas, concepts or situations in a surprising or unexpected manner, or that contains a play of words or self-deprecating, humorous statements.
Tagging comments: Comments containing only a tagged name. They are usually found on Facebook, and seem to be written to direct the attention of the person being tagged to the article.
Suggestions: This is one of those categories where there is not much more to say than the name. Comments containing suggestions.
Questions: Also a category where the name says it all.
Supportive comments: Supportive comments are comments made in defense of someone, including the commentator himself.
Speculative comments: Speculative comments are comments where the commentator is making speculative assumptions, for which there is no real evidence, and making conclusions that cannot reasonably be verified.
Image comments: Sometimes comments contain only an image. Images can contain relevant information, and can even be considered argumentative – a view dating back to classical rhetoric (LaGrandeur 2003, 119).
Links: These are comments that only contain a link to another website. I have not found any of these comments in the data I have analyzed myself, but I’ve made it a category because I did observe these comments while preparing for my research.
Arbitrary comments: Not really a category, just a collection of comments that made me think “WTF??”. Arbitrary comments are comments that are grammatically or contextually difficult to understand.
Blair, J. Anthony. 2009. “The Rhetoric of Visual Arguments” In Defining Visual Rhetorics, edited by Charles A. Hill and Marguerite Helmers, 41-62. New York: Routledge
Kleining, Gerhard, and Harold Witt. 2000. “The Qualitative Heuristic Approach: A Methodology for Discovery in Psychology and the Social Sciences. Rediscovering the Method of Introspection as an Example” Forum: Qualitative Social Research. 1, no 1, Art 13 – January 2000.
LaGrandeur, Kevin. 2003. “Digital Images and Classical Persusion.” In Eloquent Images: Word and Image in the Age of New Media, edited by Mary E. Hocks and Michelle R. Kendrick, 117-136. Cambridge: MIT Press
Lombard, Matthew, Jennifer Snyder-Duch, and Cheryl Campanella Bracken. 2002. “Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability” Human Communication Research 28, no 4. DOI: 10.1111/j.1468-2958.2002.tb00826.x