Using Advance Letters in Surveys

In recent years, survey organizations have faced the problem of declining response rates for random-digit-dialed (RDD) telephone surveys. In an attempt to combat this problem, researchers have experimented with sending ‘advance letters’ explaining the purpose and importance of the study to selected households before phoning to survey them. Sending an advance letter may help to increase the legitimacy of a research study for potential participants and to distinguish survey research calls from telemarketing and sales calls. Advance letters can also improve the confidence of telephone interviewers, who are assured that they already have a ‘foot in the door.’

The results of experiments with advance letters have been inconsistent. Some studies report significant increases in response rates as a result of an advance mailing while others show no significant effects. The failure of advance letters to improve response rates may be because some letters do not reach or are not read by potential respondents. Others have speculated that an advance letter may give ‘timid’ participants more opportunity to prepare to say ‘no’ to the survey request.

These competing theories and results led the ISR, in cooperation with several Ontario health units, to implement our own advance letter experiment in the Rapid Risk Factor Surveillance System Study (RRFSS). Our study had two goals: first, to assess the effects of advance letters in general, and second, to compare the effects of advance letters sent from two different organizations: the local health unit and the ISR. We hypothesized that letters sent from the local health unit would provide the most substantial increase in response rates because the organization would be familiar to potential participants. We also speculated that letters sent from the ISR would provide some increase in response rates compared to households that were not sent an advance letter.

Methods

The Rapid Risk Factor Surveillance System, initiated in 2001, is an ongoing survey about health-related topics that is conducted in 21 health units in Ontario. Approximately 100 respondents from each health unit are interviewed each month. Five health units participated in the advance letter experiment: Halton, Durham, Waterloo, Middlesex-London, and Grey Bruce.¹

For three months (May-July 2004), selected RDD households from each of the participating health units were randomly assigned to one of three groups: one group was not contacted in advance, one group was sent an advance letter from their local health unit, and one group was sent an advance letter from the ISR. The text of the two advance letters was identical, only the letterhead, signatures and envelopes were different. There was some difficulty determining exact mailing addresses for all of the randomly selected telephone numbers, especially for households with rural route or general delivery addresses. Approximately 15-18% of the letters were returned as undeliverable.

For those households who were sent a letter, the telephone survey introduction included an additional sentence reminding them about the advance mailing. Four questions were added to the end of the telephone interview that asked participants whether they or their household received the letter, whether they personally had a chance to look at the letter, whether they sought additional information about the study, and how getting the letter affected their decision to complete the survey.

Results

Prior to the experiment, the average response rate for the six participating health units was 63%. For the three months of the advance letter project, the response rate for households who were not sent a letter was 62%, but for households who were sent an advance letter, the response rate increased to 68% (p<0.05). As expected, the response rate for households that were sent a letter from their health unit was significantly higher (71%) than for households that were sent a letter from the ISR (65%, p<0.05). For each individual health unit, sending an advance letter resulted in higher response rates than if no letter was sent, and sending a letter from the local health unit resulted in higher response rates than sending a letter from the ISR (see Graph 1).

Graph 1: Response rate (%) by type of advance letter

There is also some evidence that sending an advance letter results in better outcomes from the first call to each household (see Table 1). For example, the proportion of first-call refusals is significantly lower for households that were sent an advance letter (5.5% compared to 7.8%, p<0.05). Sending a letter from the health unit is associated with a higher first-call completion rate than sending a letter from the ISR, though interestingly, sending a health unit letter is also associated with a higher first-call refusal rate.

Call outcome	Health Unit Letter (n=750)	ISR Letter (n=750)	Any Letter (n=1500)	No Letter (n=1787)
Completion	11.2%	8.1%	9.7%	8.4%
Refusal	7.1%	4.0%	5.5%	7.8%
Callback	24.5%	22.1%	23.3%	24.0%
Other	57.2%	65.7%	61.5%	59.8%
Total	100.0%	100.0%	100.0%	100.0%

Among those who were sent a letter and who completed the survey, 30% volunteered that they had seen the letter when the interviewer mentioned it during the introduction to the survey. In addition, almost 20% said that they (or someone in their household) had seen the letter when they were specifically asked about it. People who recalled seeing the advance letter were significantly more likely to be women, to be older, and to have a higher level of education (college/university) than those who did not recall seeing the letter. The type of letterhead did not seem to affect participants’ recall of the letter. Although few respondents who had seen the letter reported seeking more information about the study (3%), more than half said that the letter had made ‘a lot’ of difference in their decision to complete the survey (54%).

Discussion

Although sending an advance letter generally improved response rates in this study, this effect is mediated by the type of organization sending the letter. As expected, letters from the local health unit appear to be more effective in soliciting people’s study participation than letters from the Institute for Social Research. People are likely more inclined to open and read a letter from their local health unit than from a research institute that is relatively unknown to the general population. These findings suggest that advance letters may not be universally useful in increasing response rates, and that a university affiliation may be less important to potential respondents than immediate, local relevance. For advance letters from organizations that may not have immediate relevance to potential respondents, it may be worthwhile for researchers to think about strategies for encouraging people to open and read the letters that are sent.

Despite finding that some types of people were more likely to recall receiving an advance letter, it is difficult to determine at this point if advance letters have more general effects on data quality. The demographic profile of those who recall receiving the advance letter (women, older, better educated) is often associated with differences in health status and behaviours. The results from this study, however, show few significant differences in reports of health status and behaviours between those who received an advance letter and those who did not.

Though somewhat puzzling, these results illustrate the need to better understand how using advance letters may affect the data collected in survey interviews. Are advance letters likely to recruit more ‘difficult-to-reach’ respondents or respondents who give ‘extreme’ answers into a study? Or, do they merely recruit a larger proportion of ‘easy-to-reach’ respondents or respondents who give ‘middle-of-the-road’ answers?

A key question is whether the increase in response rates justifies the additional labour and costs involved in sending advance letters. In this experiment, the cost of sending an advance letter was $3.62 per case (for postage, stationery, printing and labour), excluding the initial start-up costs. If advance letters result in needing fewer calls to achieve a target number of completed surveys, however, there is the potential for substantial savings in interviewing costs to offset the cost of using an advance letter. Although we found some evidence that advance letters may improve first-call outcomes, data on the average number of calls needed to get a completed interview do not show consistent effects. Average per interview costs of surveys completed with the advance letter were higher than those completed without the letter.

More specific research in this area is needed to help determine the potential benefits of sending advance letters. No doubt, this will be influenced by pre-letter response rates; for studies with initially high rates, sending advance letters may result in little or no improvement in response rates, whereas for studies with initially low rates, sending advance letters may have substantial effects. It is also worth considering that advance letters may have a value beyond that which can be easily measured in terms of response rates and ‘calls per completion’ in a specific study. From a more global perspective, advance letters may help to increase the profile and reputation of both the organizations supporting the research and the data collection agency, and help people to distinguish academic and policy research from other types of market research and telephone solicitation.

---

¹The Sudbury health unit also participated in the experiment but the use of ‘English-first’ letter presentation in an area with a substantial French-speaking population created a unique set of problems. Sudbury’s results are excluded from this analysis.

David Northrup is Associate Director at the Institute for Social Research, Andrea Noack is a PhD candidate in York’s Sociology Department, and Renée Elsbett-Koeppen is a Project Manager at ISR.