Download this articleOver the span of the last decade technological advances have created new ways to test case strategies and develop jury profiles. One development, which has been hotly debated, surrounds the increasing use of online surveys as an alternative to traditional modes of jury research. In 2002, nearly $500 million was spent on online surveys. By 2009, that number exploded to $2 billion (AAPOR). The shift away from traditional telephone methodologies can be attributed, in part, to the low cost and high degree of flexibility that online survey research offers.
The Benefits of Conducting Online Research
In addition to its low relative cost compared to telephone options, online survey platforms offer a high degree of flexibility and control over the presentation of information. For example, response bias can easily be addressed by rotating response options across participants, complex skip patterns can be built into the survey and verdict form, questions can be presented in a manner that precludes participants from reading ahead and can be tagged as “mandatory” to prevent missing data. Trial stories can be designed to be highly engaging and can incorporate photographs, diagrams, and videos. Furthermore, unlike telephone surveys participants can complete online research at a time that is most convenient to them. The convenience factor may help to improve response rates.
For decades, attorneys and trial consultants have employed telephone surveys as a means for devising jury profiles and refining strategies for jury selection. This approach can be traced all the way back to the 1972 trial of the Harrisburg Seven when social scientists used a telephone survey to identify important attitudes and experiences that proved invaluable during voir dire.
While telephone surveys have proved useful over the years, the method has several limitations. The number of background questions that can be asked over the telephone is limited and case descriptions must be kept brief. In contrast, online survey platforms afford the opportunity to collect a significant amount of case-relevant attitudinal, experiential, and demographic data; present an engaging and detailed trial story; and provide key jury instructions and verdict questions with built in skip patterns. Furthermore, due to the convenience factor, online surveys can be considerably longer than the telephone option. Respondents often report positive experiences with well-crafted online surveys, which can take more than 20 minutes to complete: “I was actually very impressed with the quality of this type of survey!”
The Limitations of Online Research
Internet-based research has the potential to be a valuable tool for developing jury profiles and identifying complex relationships between verdict preferences, attitudes, and life experiences. However, there are also serious concerns about the validity of online survey research and the generalizability of results to the jury pool as a whole. Most telephone surveys employ a simple random sampling method, meaning that everyone in the population of interest (e.g., the jury pool) has an equal and known probability of being selected. As a result, inferences can be made beyond the sample. In contrast, most online surveys use a non-probability sampling technique based on “opt-in” panels. These panels are comprised of people who choose to participate in online survey research. When the individuals who join such panels are different in important ways from those who do not, samples are not representative of the jury pool.
Under the Hood: How Are Opt-In Panels Built?
The increase in online survey research has led to a growing demand for participants willing to take surveys. Companies are constantly working to build and maintain large opt-in panels to meet clients’ survey needs. When a panel is too small to meet demand, or is comprised of a large number of inactive members (i.e., those who do not respond to requests to complete a survey) vendors are forced to rely on a cohort of experts – known in the industry as survey crack heads – to complete survey projects. This can lead to biased results. As such, online survey vendors compete with one another to find participants willing to join their panels and take surveys.
Several different approaches are employed to build opt-in panels. Visitors can go directly to a survey provider’s website and join a panel. Online survey providers also advertise on websites with high traffic volume (e.g., news, special interests, and social media sites) and e-currency sites (e.g., PrizeRebel.com). Many of the larger vendors competitively bid on key words on search engines (e.g., Google) and place sponsored text ads in hopes of recruiting participants.
Borrowing from the online advertising model, vendors also develop networks of “publishers.” Depending on their agreements, publishers are paid a fee whenever a visitor joins a panel, clicks on an ad, or completes a survey. Agencies and affiliate networks offer an additional source of potential recruits. Agencies are hired to identify and place ads on websites expected to maximize returns.
Several vendors appear to recruit heavily from e-currency and online gaming websites. Farmville serves as a prime example. Players who are willing to take surveys are able to accrue e-currency, which can be used to purchase items on the Farmville game platform.
This approach has several drawbacks. When hardcore gamers discover easy opportunities to earn e-currency, they may devise ways to take advantage of the offer, which can lead to bad data. In the age of blogging, this concern can cause serious problems:
Yo guys listen up!!! for those who havent heard of it yet, you got an option to take a survey and then receive free G coins. Gunz are doing it because they get paid for that. Take survey BUT when your about to start the survey ALLWAYS MARK YES IN THE FIRST QUESTIONS cuz if not , you will not be able to qualify for the survey. You can also make some new accounts so that u can earn more and more coins and then GIFT it to your self from the other account lol…
Websites have also cropped up that attempt to monetize panel recruitment. Many of these websites advertise that visitors can earn hundreds of dollars by taking online surveys: "I have paid for all of my Christmas presents this year from the money I’ve made taking surveys. Thank you SurveyClub!" Survey Club claims to have over 15 million members and encourages new recruits to be honest about their lifestyle so that “you can qualify for as many surveys as possible…”
Survey respondents from sites such as Survey Club who sign up for multiple panels and take several surveys a week are not likely to be representative of the general public. This small faction of the online community actively searches for opportunities to profit by taking surveys. These individuals even post reviews of the competing online panel providers on the basis of incentives offered, timeliness for payment, and survey opportunities available. As one reviewer commented: “The rewards/incentive offered for completing a survey is fairly good compared to other survey sites…I would love to have the opportunity to participate in a survey at least once a week…”
Shopping for Survey Panel Providers
When assessing competing online survey vendors, it is important to get a good understanding of how panel members are recruited and compensated for completing surveys. Panels with purported millions of members should be evaluated with caution. A large percentage of the panel may be inactive, meaning only a small percentage of the panel actually responds to survey invitations. Vendors also tend to emphasize the many sources their panels are drawn from. This too can be misleading. While a vendor can recruit from a large number of partners and websites, a significant percentage of the panel may be built from just one or two sources. Panels built largely from social media or online gaming websites should be viewed with caution.
The European Society for Opinion and Market Research (ESOMAR) has published the 26 Questions to Help Research Buyers of Online Samples. This is a valuable resource, which provides guidelines for assessing the quality of opt-in panels.
How to Maximize the Value of Online Survey Research
Attorneys and trial consultants can take advantage of the flexibility offered by online survey platforms and limit threats to validity by using a simple random sampling technique to recruit participants. This approach has been successful in the past. For example, Trial Innovations recruited a random sample of jury eligible community members over the telephone and asked them to take a survey online for compensation. Participants were provided with a link that directed them to a survey, which included a juror eligibility section and lengthy “voir dire” component. Participants then read a detailed trial story, reviewed jury instructions, and rendered their verdicts. This approach helped to ensure that our sample was representative of the jury pool.
How Should Jury Consultants and Attorneys use Online Surveys?
Organizations such as the American Association for Public Opinion Research (AAPOR) and ESOMAR have made an effort to assess the strengths and weaknesses of the online survey method. AAPOR’s Report on Online Research concludes that results derived from non-probability samples (e.g., opt-in panels) should not be used to estimate population values. However, the results can be used to improve understanding of how “personal characteristics interact with other survey variables such as attitudes, behaviors, and intentions.”
These relationships can have practical value to jury consultants and attorneys. Online survey research can provide an affordable means for identifying important relationships between attitudes, experiences, and verdict preferences. These relationships can be used to develop jury profiles, jury questionnaires and oral voir dire scripts.
Bryan Edelman Ph.D. is a co-founder of Trial Innovations, a trial consulting firm with offices based in Northern and Southern California. He has experience working in venues across the country on both civil and criminal matters. You can read more about Dr. Edelman at the Trial Innovations website.
References
AAPOR, (2010). AAPOR Report on Online Panels
ESOMAR (2008). 26 Questions to Help Research Buyers of Online Samples
While I agree with Mr. Edelman's comments on the limitations of using a database panel for jury research, he does not explore the full benefits of conducting effective jury research in an online environment. The two limitations that he notes are the result of using an "opt in" panel – that the jurors are not vetted AND that they are "professional opinion givers" who are given "rewards" to participate. He is correct in that these panels are not an accurate reflection of the jury pool. However, they can be effectively addressed by using traditional recruiting methods and appropriate remuneration.
For example, we recruit surrogate jurors using seven characteristics matched to the venue (gender, age, race/ethnicity, education, income, marital status and employment status) – using call lists just as you would in traditional research. Also, we remunerate the jurors based on their time commitment and vet the jurors to make sure the person recruited is the person who completes the study.
Technology advances such as video streaming, internet access, focus group interaction and security have enabled jury consultants the ability to develop a rich online study that will provide valuable insight for their clients with some unique benefits to online (polling 100+ jurors effectively/efficiently, study length (1-2 hours), speed (results in 3-5 days) and cost).
Our clients are asking for more options and by using an online tool that is specifically designed for jury research, you can conduct a valuable study online – without the limitations noted by Mr. Edelman. And by being fulling informed about all the options you can better serve your clients and expand your practice as well.
I agree that the limitations inherent in opt-in panels can be addressed by using traditional recruiting methods. As I mentioned in the article, we have developed a method where we recruit jury eligible community members over the telephone and provide them with a link that directs them to the survey. Participants are compensated at the end of the survey.