Why the Internet is on the verge of blowing up all of our methods courses

March 20th, 2010 by Christian

(or: Methodologists, atone!)

By far my favorite book on the research methods, Unobtrusive Measures (first published in 1966), is a skeptical romp through social science where the authors take the position that most of what we call social science is wrong.  The theme of the book is that research is likely wrong because research design is very difficult and researchers too easily substitute received wisdom and procedure for hard thinking about designing studies, experiments, measures, tests, and so on.  Scientific conduct has a rote character that extensive training and preparation (e.g., making you get a Ph.D.) can reinforce.  Peer review and the tenure system can be engines of conservatism.

So you perform a survey in which you ask a particular question of a particular group not because it means something as evidence or it is a particularly good idea.  You do it because your advisor did it that way, or someone else (cite, year) did it that way and it is therefore respectable. And if someone did it before, it’s comparable.  This is perfectly reasonable.  It’s likely you are interested in a particular problem, but not really in the methods or statistics relevant to tests related to that problem, so you offload all of the thinking about statistics by performing the methods and statistics that everyone else does.  It’s efficient.

Yet when you stop and actually think about the intricacies of any particular research design, it gets ugly.  Einstein said, “Theory is something nobody believes, except the person who made it. An experiment is something everybody believes, except the person who made it.”  For decades (since even before Webb in 1966), various cranky types have been alarmed at the misuse of quantitative research.

My own struggles with the topic led me to design a graduate course called Unorthodox Research Methods.  The premise is this:  Most research courses teach procedure, but we need to train our students to think about research design and evidence first and we are not doing a good job of that.  (I’m revising the syllabus for this course and so I’m thinking about these issues again, hence this post.)  The Internet is making us rethink many of our research methods, and Webb’s 1966 critique has never been more apt.

A blog post is only big enough for one example, so here’s a big one:  A huge pitfall in our procedure-based methods education is the use of statistical significance.  Even non-quants are familiar with those nagging asterisks that appear after all sorts of columns in all sorts of journal articles across the social sciences.  Statistical significance is the end of conversation about method in many research projects.  Once p < .05, you pack up your kit and go home. Why do you test significance this way?  Because it’s a step in your list of steps. I think it is fair to say that most researchers have internalized this approach despite the fact that it is totally wrong and the statistics literature has railed against it for decades.

Just so we are clear: statistical significance is often useless – it’s not even a hint toward the right answer for your research project in many situations.  Luckily for the truth, the rise of the Internet is about to cause this test to blow up in our face.  We have taught statistics so badly in the social sciences that most researchers do not appear to realize that the test of significance is about sampling.  (Bam!)  It is a test that helps you figure out if you are being excessively skeptical because of the small size of the sample that you’ve got.  And our samples are now changing.

Thanks to the Internet and our ongoing revolution in computing we are entering the era that the UK calls e-Social Science, and here in the US we call Computational Social Science.  Fast processors. Big iron. Big datasets. Many variables.

Data from the cloud now potentially lets us test all kinds of social science questions (particularly if you are interested in human communication) that before would have by necessity sat in a small sample questionnaire. As social scientists turn toward “big data” they are going to trip over their bad habit of significance testing.  The fact is, most methods courses and research procedures in wide use are obsessed with errors caused by sampling, especially small sample sizes. (Bam!) But as a sea of digital data opens up to the horizon, our problems are increasingly about specification error and not sample sizes, just as measures are increasingly unobtrusive and not self-reports.

Remember, statistical significance is about sampling.  ”Except in the limiting case of literally zero correlation, if the sample were large enough all of the coefficients would be significantly different from everything.” (McCloskey, p. 202).  Take your study of communication patterns from 60 paper-and-pencil questionnaires replicate it with a random sample of a million Facebook accounts (if you can get access… see this editorial).  You’ll find that statistical significance — particularly at the arbitrary point of p < .05 – tells you zip.

(click for more shirts like this.)

I think most of the solution is to de-emphasize procedure, as social science procedure is becoming much more volatile as information technology improves.  We need to get people to understand that research design is a creative act, not the boring part of the research process.  Students need to write new procedures, not memorize old ones.  To that end, we need classes about evidence and research design.  Figuring out how to do that is a challenge but we’ve got to step up to it.  (If you’ve got ideas for revising the syllabus for my last attempt, send me an email or a comment.)

Chant it with me:  Statistical significance does not equal substantive significance.  Please chant it with me.  This is something we ought to know already but it may take the big datasets of the Internet to teach us.  What other lessons are in store?

Read more:

Deirdre McCloskey: Rhetoric Within the Citadel: Statistics (http://deirdremccloskey.org/docs/pdf/Article_181.pdf) and Why Economic Historians Should Stop Relying on Statistical Tests of Significance (http://deirdremccloskey.org/docs/pdf/Article_180.pdf)

John P. A. Ioannidis: Why Most Published Research Findings Are False (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/)

Jonathan A C Sterne and George Davey Smith: Sifting the Evidence: What’s Wrong with Significance Tests? (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1119478/?tool=pubmed)

Creative Commons License
This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Share Alike 3.0 License.

14 Responses to “Why the Internet is on the verge of blowing up all of our methods courses”

  1. uberVU - social comments Says:

    Social comments and analytics for this post…

    This post was mentioned on Twitter by niftyc: Why the Internet is on the verge of blowing up all of our research methods courses — http://bit.ly/9IFIFe #fb…

  2. HASTAC Says:

    This post has been re-blogged by Cathy Davidson at HASTAC. http://www.hastac.org/blogs/cathy-davidson/statistical-significance-v-substantive-significance

  3. Sheryl Grant Says:

    What a great blog post — I saw your work re-blogged over at Hastac.org by Cathy Davidson, and shared the link to your post with my doctoral cohort.

    And then I had a full-circle doofus moment when I went to recommend Research Confidential for your class, and then realized that you were one of the contributors :)

    I’m going to see if I can find enough doc students to do an independent study and follow your syllabus. I’m sure we could find a methods nerd professor who would like to participate in seminar discussions. Thanks for posting!

  4. multicast » Blog Archive » Why the Internet is on the verge of blowing up all of our methods courses | Think Networks Says:

    [...] multicast » Blog Archive » Why the Internet is on the verge of blowing up all of our methods cours…. Posted by Noshir Contractor / Filed under:Networks and Technology [...]

  5. niftyc Says:

    Thanks Sheryl, Yes, let me also recommend Research Confidential to everyone (http://www.press.umich.edu/titleDetailDesc.do?id=268873)! Christian

  6. Kurt Richter Says:

    In the design of your class, you should probably take a look at the following work:

    Reigeluth, C. M., & Frick, T. W. (1999). Formative research: A methodology for improving design theories. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory (2 ed.). Mahwah, NJ: Lawrence-Erlbaum Associates.

    If you haven’t looked at a copy of “The Green Book” yet, the work referenced in part 2 of the citation, you might find it very helpful. Please feel free to contact me for information on how formative research has been used for the past 8 years on research in a school district on the interstate ring around Indianapolis. Always happy to share…

    -Kurt-

  7. Josh Joseph Says:

    Nice post and spot on about statistical “significance” not having much to do with importance or meaningfulness. Case in point: we look at a lot of federal data…surveys with upwards of 200,000 employees. With a sample size that big, just about every 2% point difference is statistically significant…but doesn’t mean much.

    Only thought is you may want to choose your words more precisely on this. Saying that “stat. significance is about sampling” may confuse some people. I know you mean it’s about “the size of your sample,” but some might interpret the term sampling as being about ‘who you sample,’ an important but different issue.
    – Josh

  8. reconsidering methods in research on and about the Internet « [day in the life] Says:

    [...] on and about the Internet Jump to Comments Fascinating post by Christian Sandvig – Why the Internet is on the verge of blowing up all of our methods courses. Of particular interest is that it’s written from a quant perspective, while much of the [...]

  9. jebyrnes Says:

    Great stuff. You might also enjoy Hurlbert and Lombardi’s 2009 piece in Annales Zoologici Fennici [pdf] or my short review of the paper. This is becoming a huge problem in Ecology as well as our environmental monitoring programs grow by leaps and bounds. It’s particularly toxic when combined with researchers who know little of the natural history of the systems they are examining.

  10. TwittLink - Your headlines on Twitter Says:

    [...] Tweets about this great post on TwittLink.com [...]

  11. Juan Espinosa Says:

    Thanks you very much for this post.

    All the best,

  12. Chronic Disease in Data and Narrative | e-Patients.net Says:

    [...] surveys allow us to paint with a broad brush, but they are limited in what they can tell us about a population. Living with a chronic disease is associated with being [...]

  13. Alison Powell Says:

    I have been having some very interesting conversations with colleagues about research design recently. They are the direct result of the phenomena that you’ve described here. As someone trained as a qualitative researcher I’ve always been focused on design, and had “transparency about design is essential” drilled into me. But I still sometimes hear the argument that “more data is better” – as if the sample size itself could address the failures of procedure.

    Could your class experiment with ways of building transparency into quantitative research design? In the methods seminars I led this year, we explored this a bit by identifying points of contact between different types of methods, and the kind of transparency required at those points.

  14. multicast » Blog Archive » Future Social Science On and With Digital Media Says:

    [...] Newish Comments A Premature Obituary of Online Video Sharing and Amateur Video Culture | Watching YouTube on The Television Cannot Be RevolutionizedP2P Foundation » Blog Archive » A premature obituary of online video sharing and amateur video culture on The Television Cannot Be RevolutionizedDoc Searls Weblog · How Infrastructure Learns on The Television Cannot Be RevolutionizedFerril Irham M on Google v. China: Principled, Brave, or Business As Usual?Alison Powell on Why the Internet is on the verge of blowing up all of our methods courses [...]

Leave a Reply

Bad Behavior has blocked 251 access attempts in the last 7 days.