Tuesday, January 11, 2011

There’s a Research App for That (the App Just Doesn’t Know it Yet)


            A not uncommon difficulty faced by psychology research is generalizing results from studies using undergraduate students as participants to the broader population of people. Undergraduate students, often enrolled in an introductory psychology course, are often used because, well, they’re readily available and will often participate as part of a course component, unlike the general public, who often (but not always) expect some sort of compensation for their participation. Needless to say, most psychologists aren’t made of money, so they tend to make use of the somewhat captive pool of undergraduate students for their participants. Which, with respect to generalization, is a problem.
Before going any further, I need to explain some terms. First, the population is the overall group that you’re interested in studying. With respect to humans, this might be all the people on the planet. Or, if you’re really interested just in Albertan five year-olds, then the population is all the five-year-olds in Alberta. In an ideal world you’d run your study on every single individual in the population; obviously, unless you’re interested in a really specific population of only a few hundred people, this isn’t feasible for all kinds of reasons. Therefore, studies are carried out with smaller samples of the population. A sample is the subset of the population that is actually used in the study. Any study using a sample has to generalize (that is, “extrapolate”) its results to the larger population that the sample was taken from.
Now, as I said, psychologists often use undergraduate students as subjects. Unless the study is specifically interested in some psychological phenomena particular to undergraduate students, these students are being used as a sample for a larger population. But, how representative, by which I mean “indicative” or “typical”, of the greater population are undergraduate students? In other words, how generalizable are the study’s results going to be? Good studies generalize from the sample to the population very well. But therein lies the problem with undergraduate students as study participants. While relatively available, 18-22 year old undergraduate students may not generalize well to, for example, 40-55 year olds, or even to non-undergraduate 18-22 year olds. Given the aforementioned scarcity of funds available to most psychologists, paying for a more idealized sample of subjects for their studies is not usually an option. What is the unfortunate psychologist looking for good generalization of results to do? (I encourage you to insert a melodramatic sigh here.)
            Fortunately, recent technological developments are providing solutions. A variety of new on-line data collection tools, produced principally for graphic designers and on-line business analytics, can be utilized for at least some types of psychology research. I’m only going to discuss couple of these new tools here, but keep in mind that there are a number of others available. Part of the reason I’m focusing these particular applications is because they can be used, albeit in a restricted format, for free. This could be a real advantage for pilot studies and class projects, or even for psychologists looking to try the application out to see if it might meet their future research needs.
            Just to be clear, I am in no way associated with or being paid by the providers of the services I’m going to be discussing below. I just personally think that these are great applications with excellent potential for research applications.
            First, a snazzy duo of on-line data collection services, developed to let web and graphic designers upload sample pages and get feedback before putting the final product on-line, called the fivesecondtest (http://fivesecondtest.com/) and clicktest (http://theclicktest.com/). To use the service for free you volunteer some time and take other users’ tests; this gains you “karma points” that you can then use to have people take your tests. Alternatively, you can pay for the service and not have to take tests yourself (and gain other benefits as well). Fivesecondtest has subjects view an on-screen image for five seconds then answer one or more questions about what they remember while clicktest has subjects view an on-screen image and use the mouse to click on a feature related to a previously asked question.


            Ok, great, but who are the participants for a test? Participants come in two types. First, anyone who has come to the site and is willing to volunteer some time to take a test. Second, if you subscribe to the service (costing $20-200 per month; the higher costs give more responses per month and some other perks) participants can be limited to individuals given prior permission to take a specific test. The first option seems great: subjects could be from anywhere in the world, they could be anyone. How much more representative of the general population could you get? Well, actually, most of the people volunteering to take tests are probably building karma points to be able to use the service themselves, so they are probably web or graphic designers who are starting out in their career or are looking to get feedback on a project without spending much money. So, there’s likely a bias here, but still, it is fairly unlikely that they are first year undergraduate students in an introductory psychology course. Fortunately, the second option gives the researcher a lot more control over who is a participant. However, we’re back to the problem of getting participants in the first place. Still, given that the tests take seconds to complete and can be carried out from any internet capable computer, smart researchers, perhaps using some clever e-mail, Twitter, or Facebook promotion of their research, should be able to get at least some non-undergraduate participants; very savvy researchers should be able to get increasingly specific samples of the population for participants. (Getting into the use of new social media and word-of-mouth marketing to reach research participants is a whole different topic, best left for a future blog.) While these two tests are inexpensive it must be noted that they are fairly specific data collection tools, limited to static images and fairly simple questions. That being said, a single test takes, literally, seconds, making the cost in terms of time commitment for a participant in a research study minimal. I can see a variety of great uses for fivesecondtest and clicktest in, for example, perceptual, social, cognitive, and advertising psychology research.
            The second on-line data collection tool I’m going to talk about is SurveyMonkey (http://www.surveymonkey.com/). Again, there are free and subscription options with this service. The free service restricts the number of questions on the survey and the number of surveys you can collect in a month, but there are still good possibilities for psychological research here, especially with respect to pilot studies or class projects. The subscription fees are relatively small. At the time of writing, the most expensive package is $25 Canadian per month, which provides up to 1000 questionnaire results per month; note that this is much less expensive than providing even a small honorarium to 1000 research participants as motivation for taking part in a study. Subscription offer the researcher many more options and controls. With a subscription one can construct longer questionnaires, use “skip logic”, and have total control over participants by limiting access to specific individuals. One of the things that I think is terrific about SurveyMonkey’s online questionnaires is the “skip logic” function, that is, the use of customized question paths. How a subject answers a particular question determines the subsequent questions that are asked, meaning that subjects don’t have to waste time on questions that aren't relevant, thereby reducing subjects’ frustration with the questionnaire process and, hopefully, leading to more accurate and representative answers.

As with fivesecondtest and clicktest, SurveyMonkey is highly accessible: it is available 24/7 from any internet capable computer, allowing subjects to participate in a questionnaire study at their convenience. From the researcher’s perspective, it can also be a time and money saver: a research assistant doesn’t need to be on hand to distribute and administer questionnaires to participants. SurveyMonkey could readily be utilized by researchers in any area of psychology that utilizes questionnaires for data collection. Sure, the same issues of recruiting subjects raised earlier are still here, but with the Internet and on-line sampling tools researchers really do have access to the world’s population (and all the subset samples thereof), providing that they are a little proactive and savvy about networking with prospective participants. For example, I’ve seen  a Twitter request by a psychology graduate student for participants to take her SurveyMonkey questionnaire; I re-tweeted the request, advertising this student’s study to an even larger collection of possible participants.
            Are these on-line testing applications the be-all and end-all solutions for the issue of getting away from the use of undergraduates as psychology research participants and increasing the generalizability of the samples to the population? Of course not. Do they have limitations? Obviously. But for what they do, these are excellent testing platforms for certain areas of psychology research. Yes, there are a variety of other issues that I haven’t gone into here that researchers utilizing them need to take into account, for example, the need to debrief participants post-test (although it is relatively easy when utilizing the subscription options on these services to have participants take a link to the test from a study-specific website that provides introductory and debriefing content for participants). Regardless, there are an increasing number of on-line utilities and tools becoming available for research. Yes, there’s an app for that, as savvy psychologists willing to step a bit outside the traditional data-collection box are discovering.

No comments:

Post a Comment