Analysing free text objectively

From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU] On Behalf Of Laurie Moseley

Sent: Friday, 24 March 2006 2:23 AM

To: SPSSX-L@LISTSERV.UGA.EDU

Subject: Analysing free text data

For qualitative research there are at least two elements.

(1) The data. I don't call them qualitative data, largely because the quality is often low. I prefer to call them call free text, volunteered data,  which is less tendentious and prejudicial. They can be analysed by anyone who  can get hold of them, although they are rarely made available to independent researchers. My own analyses of transcripts from other centres have found that they frequently contain sufficient errors to misrepresent what was actually said. In such cases, even the validity of the data is in doubt.

(2) The analysis. The common weakness is that this is rarely replicable. I regularly read verbatim quotations and disagree with the original  researchers' coding and interpretation. When the work is about a topic in which  I have some first-hand knowledge, the interpretations which I have noted down  are often at variance with what the original researchers claim to have observed. It very often, perhaps even usually, represents the researcher's opinion of what was meant by respondents or subjects of observation. It is even rare to find multiple blind coding, which I would regard as the minimum safeguard against researcher bias.

Therefore, such 'methods' are not replicable and not trustworthy They may represent hunches, impressions, hypotheses, guesses or the like, but they rarely  provide anything that could be called knowledge. The commonest form of "evidence" used is verbatim quotations. Even when they are accurately recorded and reported, all that they tell the readers is that one respondent said one thing on one occasion - no more, and no less.

With regard to SPSS, there is now a product called Text Analysis for Surveys. It takes the free text volunteered data and permits its analysis (a) by statistical means and (b) by linguistic methods.  I have talked to three  people who have used it. All said that they were able to gain reliable results  and insights which they believed could not have been achieved by using a  pre-coded format, or by a less replicable free text analysis method. I have not  yet used it myself, but I have a PhD student who is in the early stages of  becoming comfortable with it. Our first impressions are favourable.

Keep your friends close, and your enemies closer.

I find this a fascinating article.  Well-argued, witty, intriguing... but wrong.  These folks decide there is no bottom line.  A world without universal morals or material truths?  Maybe toothache doesn't hurt in some social contexts.  More likely someone has been told to stop complaining or they'll be given something that will REALLY give them something to complain about.  But being oppressive is OK, as it's all relative.

Edwards, D., Ashmore, M. and Potter, J., (1995). Death and furniture: The rhetoric, politics and theology of bottom line arguments against relativism, History of the Human Sciences, 8, 25-49.

Why undergraduates over-rate qualtiative methods.

Murtonen, M.  (2005).  University Students' Research Orientations: Do negative attitudes exist toward quantitative methods?  Scandinavian Journal of Educational Research, 49, 263 - 280

This paper examines university social science and education students' views of research methodology, especially asking whether a negative research orientation towards quantitative methods exists. Finnish (n = 196) and US (n  = 122) students answered a questionnaire concerning their views on quantitative, qualitative, empirical, and theoretical methods, their readiness to use quantitative and qualitative methods in their own research, and the difficulties they experienced in quantitative methods' learning. Students were clustered in groups according to their views. Students had varying combinations of views on the methods, that is different research orientations towards methods were found in both countries. Some of the students had a dichotic attitude towards quantitative and qualitative methods; they seemed to “choose their side” between these methods. In both countries a negative research orientation towards quantitative methods was found. It was connected with either difficulties in quantitative methods' learning or with a lower appreciation of empirical methods than that of other students. Major subject and study year had no effect, so the views were not discipline-specific and students seemed to already have them on entering university. Views were quite stable during the course. A reduction in difficulties experienced with quantitative methods' learning was connected with a lowered over-appreciation of qualitative methods at the end of the course

Focus groups

You know why governments like focus groups?  Because you can sample 7 people and extrapolate from that the the country, strategically loading the groups with sympathisers.  Don't tell me it doesn't happen. 

Semioticists declare that "the text is the thing", and those studying 'discourse' will equally say that what someone says is the basic information, hence the need for a very high level of transcription (including pauses, malaprops, etc.). 

But as we all know, what someone SAYS is not what they DO.

Focus group researchers are starting to get the idea.

People say that self-report questionnaire research is subject to bias, deceit, and social desirability.

So are interviews and focus groups: .  

Why methods, self-criticism, and peer review are necessary

Science and Pseudoscience in Law Enforcement

A User-Friendly Primer

1. Scott O. Lilienfeld
   1. Emory University, Atlanta, Georgia, slilien@emory.edu

1. Kristin Landfield
   1. Emory University, Atlanta, Georgia

Abstract

Pseudoscience and questionable science are largely neglected problems in police and other law enforcement work. In this primer, the authors delineate the key differences between science and pseudoscience, presenting 10 probabilistic indicators or warning signs, such as lack of falsifiability, absence of safeguards against confirmation bias, and lack of self-correction, that can help consumers of the police literature to distinguish scientific from pseudoscientific claims. Each of these warning signs is illustrated with an example from law enforcement. By attending to the differences between scientific and pseudoscientific assertions, police officers and other law enforcement officials can minimize their risk of errors and make better real-world decisions. 

Criminal Justice and Behavior October 2008 vol. 35 no. 10 1215-1230

Qualitiative approaches can cure stammering?

Beware of those who want power and authority without responsibility.  Knowledge gives you power and influence, and as such you need to acquire it with integrity, and avoid making false conclusions about causality, generality, and validity.  Many qualitative researchers are explicit about this, and accept they are researching focussed subjective experience in an area that may not lend itself to quantifiable research.  Others want the authority which validated clinical search attracts, though without havign to go through the rigorous truth criteria that it demands (often to the despair of those working in the field).

Decide for yourself whether a qualitative method cures stammering: it all reads a bit "Clever Hans" to me.

some papers to get you going

Mike Morgan's combatative challenge: http://cranepsych2.edublogs.org/files/2009/08/Qualitative_research_science.pdf

This is why I have 'issues' with much qualitative research:
http://www.badscience.net/2006/08/deconstructing-the-evidence-based-discourse-in-health-sciences-truth-power-and-fascism/