Strategies for ensuring validity and reliability in ethnographies of communication

When you are engaged in doing an ethnography of communication, how do you ensure that you are assessing your key concepts accurately?  How do you make certain that your readings of the data are correct? There are a number of strategies that can be used to test the validity and reliability of work done under the aegis of the ethnography of communication.

First, does the researcher make it a point to use key terms, concepts, descriptions, and explanations used and/or provided by the people under study?  (Philipsen, 1982, p. 49) One excellent model for this approach is an article on an Osage community by Pratt and Wieder (1993).  In this article, Pratt and Wieder provide readers with a step-by-step description of the speech events under analysis, including detailed information on who (gender, age, experience, role) can speak for others (and why); who participates in these speech events (speaking for others) and how; how these events start, proceed, and end; how they are regulated; how/where people sit/arrange themselves; how stages of the events are ordered; the underlying reasons for the events; how people prepare for the events; what expectations govern the events; may and may not be said; how listeners comport themselves; and the delivery (eye contact, gaze, volume, tone) of speakers.  In this way Pratt and Wieder use informants’ terms and also describe very carefully, down to the smallest details, how informants see these speech events playing out.

Second, does the report expound on something that the people under study would actually acknowledge as a facet of their lives?  In other words, would community members recognize the findings as something real and true about their world? (Philipsen, 1982, p. 49) One great example of this is Manning’s (2008) analysis of online forums for Starbucks baristas.  In the forums the baristas let off steam about “SCOWs” (stupid customer of the week), a local concept.  Manning uses the baristas’ own words to elucidate what, exactly, stupid customers are (what they do, say, etc.)

Third, does the researcher produce an analysis that actually helps people from the community under study to “better to understand [their] own social world?” (Philipsen, 1982, p. 49)  When sharing your work with your informants, do they report back that it helps them to analyze, understand, or deal with the issues better, more effectively, or more successfully?

Fourth, (how) does the researcher seek out checks and/or validation of the findings from members of the community under study?  (Lindlof & Taylor, 2002)  Such checks can do a great deal to validate the accuracy of the findings, because ultimately an ethnographer of communication seeks to discover the meanings and understandings of the informants themselves.  Two model studies of this are Baxter (1993) and Bailey (1997).  In both of these cases, the researchers test their findings by sharing them with informants, asking them if they got things right, and having them provide further explanations where necessary.

Fifth, having multiple researchers on the project can function to ensure validity.  Pratt and Wieder (1993) are a good example of this, with Pratt’s inside knowledge as a member of the Osage community working in combination with the experience of Wieder.

Sixth, another good strategy is to have comparative data at hand. Bailey’s (1997) article is a good model for this because he shares transcripts of typical service interactions from multiple perspectives.

Finally, intercoder reliability checks can be a very effective way of ensuring reliability in the data analysis.  To do this, one must engage a second (or third, etc.) person to look over the data not only to make sure that the codes and categories seem logical, but also to test whether or not they code it consistently with the primary researchers.

What methods do you use to ensure the validity and reliability of your qualitative work?

References

Bailey, B. (1997). Communication of respect in interethnic service encounters. Language in Society, 26(3), 327-356.

Baxter, L. (1993). “Talking things through” and “putting it in writing”: Two codes of communication in an academic institution. Journal of Applied Communication Research, 21, 313-326.

Lindlof, T. R., & Taylor, B. C. (2002). Qualitative communication research methods (2nd ed.). Thousand Oaks, CA: Sage.

Philipsen, G. (1982). Linearity of reserach design in ethnographic studies of speaking. Communication Quarterly, 25(3), 42-50.

Pratt, S., & Wieder, D. L. (1993). The case of saying a few words and talking for another among the Osage people: ‘public speaking’ as an object of ethnography. Research on Language and Social Interaction 26(4), 353-408.