Sociology Survey Methods
by
David de Vaus
  • LAST MODIFIED: 26 August 2013
  • DOI: 10.1093/obo/9780199756384-0106

Introduction

Survey methods are some of the core methods for collecting and analyzing data in sociology. While survey methods have been used since the early days of sociology, they became a core method after World War II: they have increasingly found use in a wide range of other disciplines and have become a key tool in business, government, marketing, and many other applied areas. During the late 20th and early 21st centuries, survey research has become progressively more sophisticated and has benefited from developments in a wide range of disciplines including statistics, cognitive psychology, computer programming, and technological advances such as telephones and the Internet. Although Britain was the home of the earlier surveys, the United States fostered the explosion of survey research after World War II. Survey research has been used throughout the world and has become an important basis for comparative social analysis. While the fundamentals of survey research are well established—collections of structured data usually with a structured questionnaire and the statistical analysis of the interrelationships of variables—the practice of survey research is constantly changing. Our knowledge of how to ask good questions continues to develop, the means by which questionnaires are administered are being transformed as technology changes, and the challenges of obtaining good information increase as response rates decline. Our ability to analyze data continues to evolve as computer software enables survey researchers to develop complex survey designs and interrogate the data in ever more sophisticated ways.

General Overviews

Marsh 1982 provides an unusual overview of the survey method that focuses on its epistemological foundations. More typical overviews are texts aimed at different audiences. Moser and Kalton 1985 has stood the test of time since being first published in 1958. Babbie 2012 and Fowler 2009 have been widely used among undergraduates, while de Vaus 2002 and Fink 2003 are aimed more at postgraduate students. Groves, et al. 2011 is a more advanced treatment of the areas dealt with by these texts.

  • Babbie, E. R. 2012. The practice of social research. 13th ed. Belmont, CA: Wadsworth Cengage Learning.

    Save Citation »Export Citation »E-mail Citation »

    Widely used and highly accessible introduction to various social research methods including survey research. It provides a good context within which to understand surveys.

    Find this resource:

    • de Vaus, D. A. 2002. Surveys in social research. 5th ed. London: Routledge.

      Save Citation »Export Citation »E-mail Citation »

      Widely used text among postgraduate students. Provides an easy to understand and practical guide to conducting, analyzing, and critically evaluating surveys. A sixth edition is due in 2013.

      Find this resource:

      • Fink, A. 2003. The survey kit: How to design survey studies. 2d ed. Thousand Oaks, CA: SAGE.

        Save Citation »Export Citation »E-mail Citation »

        A set of ten volumes with each volume dedicated to topics that are typically a chapter in introductory texts. Volumes include question design, different methods of administering questionnaires, and sampling. Two volumes provide elementary introductions to survey analysis.

        Find this resource:

        • Fowler, F. J. 2009. Survey research methods. 4th ed. New York: SAGE.

          Save Citation »Export Citation »E-mail Citation »

          A well-tested and relatively brief introduction for undergraduates that provides chapter-length treatment on core topics of survey research, including one chapter on survey analysis.

          Find this resource:

          • Groves, Robert M., Floyd J. Fowler, Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2011. Survey methodology. 2d ed. Hoboken, NJ: Wiley.

            Save Citation »Export Citation »E-mail Citation »

            Provides an up-to-date revision of the popular first edition. This is an intermediate level overview of the main topic areas in conducting and evaluating survey data, but it does not venture into survey analysis. It is appropriate for postgraduate-level courses.

            Find this resource:

            • Marsh, Catherine. 1982. The survey method: The contribution of surveys to sociological explanation. London: Allen and Unwin.

              Save Citation »Export Citation »E-mail Citation »

              Marsh provides a strongly argued case for the contribution of survey-based explanations. She responds to the anti-survey perspective found in parts of sociology and argues that these critiques are ill conceived and misunderstand what survey research can accomplish in achieving sociological understanding.

              Find this resource:

              • Moser, C. A., and G. Kalton. 1985. Survey methods in social investigation. Aldershot, UK: Gower.

                Save Citation »Export Citation »E-mail Citation »

                This classic book first appeared in 1958 authored by Moser but subsequently co-authored by Kalton. It remains useful, although it is not as up to date regarding more recent survey method developments.

                Find this resource:

                Major Collections

                A great many of the advances in survey research are published in a wide range of journals and books, and it can be difficult to identify the best research available. A number of multi-volume collections of the key articles are being published; these provide easy access to some of the foundational work in survey research. Two such collections are de Vaus 2002 and de Vaus 2007 in which 167 articles are reproduced in eight volumes: both provide handpicked articles in these key areas of survey research. Bulmer 2004 is a four-volume collection on questionnaires, and Vogt 2011 is on quantitative methods. Alongside these collections of republished work are the collections of specially commissioned work. The highlights in this category include de Leeuw, et al. 2008 and Lavrakas 2008. But the outstanding special collection is the second edition of Marsden and Wright 2010 in which each paper provides a concise and authoritative, scholarly treatment of a wide range of topics.

                • Bulmer, M., ed. 2004. Questionnaires. 4 vols. London: SAGE.

                  Save Citation »Export Citation »E-mail Citation »

                  A treasure trove of the key articles on designing and administering survey questionnaires. Articles cover the format of questions, eliciting opinions, quantifying answers, question construction, sensitive questions, memory, validity, and the changing context of administering questionnaires.

                  Find this resource:

                  • de Leeuw E. D., J. J. Hox, and Don Dillman, eds. 2008. International handbook of survey methodology. New York: Lawrence Erlbaum.

                    Save Citation »Export Citation »E-mail Citation »

                    This comprehensive collection of chapters from leading experts covers similar territory to Marsden and Wright 2010. It is appropriate for more advanced students in postgraduate courses in survey methods and for survey professionals.

                    Find this resource:

                    • de Vaus, D. A., ed. 2002. Social surveys. 4 vols. London: SAGE.

                      Save Citation »Export Citation »E-mail Citation »

                      De Vaus brings together ninety-five previously published key articles and chapters. Articles cover areas such as ethics, epistemology, research designs, modes of data collection, sampling, and survey error. These papers provide detailed discussions that are impossible in texts or single-volume collections.

                      Find this resource:

                      • de Vaus, D. A., ed. 2007. Social surveys 2. 4 vols. London: SAGE.

                        Save Citation »Export Citation »E-mail Citation »

                        This four-volume collection extends the range of topics and articles available in the 2002 collection.

                        Find this resource:

                        • Lavrakas, Paul J. 2008. Encyclopaedia of survey research methods. New York: SAGE.

                          Save Citation »Export Citation »E-mail Citation »

                          This accessible treatment of surveys covers 640 topics in over a thousand pages. It is aimed at the undergraduate and postgraduate student audience and frames the treatment of surveys within the total survey error perspective.

                          Find this resource:

                          • Marsden, P. V., and J. D. Wright, eds. 2010. Handbook of survey research. 2d ed. Bingley, UK: Emerald.

                            Save Citation »Export Citation »E-mail Citation »

                            Marsden and Wright provide a complete rewrite and major expansion of the original work by Rossi, Wright, and Anderson. The authors of each of the twenty-eight chapters are leaders in their field and provide authoritative, non-basic, and contemporary treatments of the core topics.

                            Find this resource:

                            • Vogt, W. Paul, ed. 2011. SAGE quantitative research methods. 4 vols. London: SAGE.

                              Save Citation »Export Citation »E-mail Citation »

                              These four volumes provide a reference collection of previously published papers on a wide range of advanced topics on quantitative data analysis. Most articles relate to complex survey data and issues of testing causal models. The papers range from the relatively simple to those that require more statistical expertise.

                              Find this resource:

                              Journals

                              Research based on surveys is published in a wide range of journals. Sometimes these journals publish specialist methodological papers on survey methods. But a number of journals either specialize in papers on survey methodology or at least frequently publish some papers on survey methodology itself. Preeminent among these is Public Opinion Quarterly. The Swedish-based Journal of Official Statistics publishes papers on the statistical analysis of survey data as does the International Statistical Review. The Canadian-based Survey Methodology publishes papers on a wide range of methodological issues and findings in relation to survey methods. Sociological Methodology and Social Science Research publish papers on a range of research methods that include some significant papers on survey methods. With advances in computer-based methods of survey research and analysis the Social Science Computer Review is increasingly important for survey methodologists. Surveys are widely used in market research, so it comes as no surprise that the highly regarded Journal of the Market Research Society publishes very good articles about survey methods for improving market research.

                              History of Survey Research

                              Although some of the founding fathers of sociology (including Marx, Weber, and Durkheim) made some attempts at survey research, and the British social reformers conducted surveys of sorts, it was not until the 20th century that the most notable developments in survey research occurred. Marsh 1982 gives a concise account of the origins of survey research and the 20th-century developments. Bulmer, et al. 2011 provides a much fuller account of this history while Converse 1987 provides a detailed history of the growth of survey research in the United States. Abrams 1951 details the history of mainly British survey research up until the immediate postwar period. In contrast to these broader historical overviews, others have focused on the development of particular elements of survey research. Hansen 1987 provides an instructive overview of the development of survey sampling methods, while Converse 1984 outlines the development of attitude measurement in survey research. Although there have been more recent developments in survey analysis, the special issue of Public Opinion Quarterly (1987) provides an excellent collection of papers on the historical developments in survey research.

                              • Abrams, M. 1951. Social surveys and social action. London: Heinemann.

                                Save Citation »Export Citation »E-mail Citation »

                                Although only dealing with the history of social surveys up until the immediate postwar period, this account focuses primarily on the British story and demonstrates how survey research originated and was driven by a desire for reform and social action.

                                Find this resource:

                                • Bulmer, M., K. Bales, and K. K. Sklar. 2011. The social survey in historical perspective, 1880–1940. Cambridge, UK: Cambridge Univ. Press.

                                  Save Citation »Export Citation »E-mail Citation »

                                  This scholarly account of early social surveys demonstrates how social surveys were part of a social movement to improve society. It also shows how the partnership between the reformers and the “inquirers” eroded as reformers focused on social action and “inquirers” became more interested in social science, theories, and explanation.

                                  Find this resource:

                                  • Converse, J. M. 1984. Attitude measurement in psychology and sociology: The early years. In Surveying subjective phenomena. Vol. 2. Edited by C. F. Turner and E. Martin, 3–39. New York: Russell Sage Foundation.

                                    Save Citation »Export Citation »E-mail Citation »

                                    While many surveys have focused on the measurement of objective characteristics, an important component of many surveys and polls has been the measurement of subjective phenomena such as attitudes, opinions, and beliefs. Converse describes the emergence of this sometimes controversial aspect of survey measurement.

                                    Find this resource:

                                    • Converse, J. M. 1987. Survey research in the United States: Roots and emergence, 1890–1960. Berkeley: Univ. of California Press.

                                      Save Citation »Export Citation »E-mail Citation »

                                      While survey research drew its early impetus from Britain, the United States has been fundamental to the explosion of survey research. While this explosion has occurred from World War II onward, Converse provides a fascinating account of the earlier developments and immediate postwar growth of US survey research.

                                      Find this resource:

                                      • Hansen, Morris H. 1987. Some history and reminiscences on survey sampling. Statistical Science 2:180–190.

                                        DOI: 10.1214/ss/1177013352Save Citation »Export Citation »E-mail Citation »

                                        As one of the founders of scientific survey sampling in the United States, Hansen is well placed to highlight some of the key turning points in the development of survey sampling, especially those developments via the US Bureau of the Census.

                                        Find this resource:

                                        • Marsh, Catherine. 1982. History of the use of surveys in sociological research. In The survey method: The contribution of surveys to sociological explanation. By Catherine Marsh, 9–47. London: Allen and Unwin.

                                          Save Citation »Export Citation »E-mail Citation »

                                          Marsh provides a concise account of the origins of survey research in Britain, Europe, and the United States. She details the social reform origins of survey research and shows how broader social changes, as well as developments in statistical theory, shaped the emergence of contemporary survey practice.

                                          Find this resource:

                                          • Special Issue. 1987. Public Opinion Quarterly 51.4 part 2: s21–s191.

                                            Save Citation »Export Citation »E-mail Citation »

                                            Includes a wealth of papers on diverse developments in survey research including articles on Paul Lazarsfeld, public opinion polling, the organizational context of surveys, internationalization, developments in question wording, sampling, data-analysis techniques, and survey quality.

                                            Find this resource:

                                            Data Collection Modes

                                            Survey analysis relies on the collection of systematic data so that a variable by-case matrix of data is available for analysis. Essentially this means that information about the same characteristics (variables) needs to be collected for every case (typically, but not necessarily, people). This requirement has meant that instruments such as structured questionnaires and interviews have been the main methods of data collection. Over the years, survey researchers have experimented with a range of ways in which to administer these questionnaires. As technology has advanced, the methods have changed. Initially most questionnaires were completed either by face-to-face interviews conducted by trained interviewers or were self-administered by respondents. Technological advances have enabled the development of telephone-based interviews using computer-assisted interviewing tools. More recently the rapid growth of the Internet has led to the growth of web-based surveys and more recently mobile technologies are providing new opportunities for administering surveys. As the challenges to collecting quality information at an affordable cost mount and as technology advances, further changes in data collection methods can be anticipated. As one of the leaders in the development of web-based surveys, Couper 2011 anticipates what some of these methods of the future might be. A more comprehensive view of the future is canvassed in the book Conrad and Schobe 2007 (cited under Computer-Assisted Interviewing). The different methods of administering questionnaires are driven by a range of considerations, including maximizing sample quality and response rates, minimizing cost, and maximizing data quality. The different data collection methods have different advantages and disadvantages, and it is not possible to specify the “right” method. The choice of method will be a balance based on the particular type of survey and the trade-off between data quality and cost. Since different data collection methods have different attributes, the data collected is subject to mode effects—where the data is partly shaped by the method of collection rather than the underlying reality that the survey is seeking to capture. This means that it is important to assess the extent to which mode effects operate. More recently, some surveys are adopting mixed-mode administration methods, which add further complexity to assessing the quality of the survey data collected.

                                            • Couper, Mick P. 2011. The future of modes of data collection. Public Opinion Quarterly 75:889–908.

                                              DOI: 10.1093/poq/nfr046Save Citation »Export Citation »E-mail Citation »

                                              This is a well-balanced and well-considered review of the history and future of different modes of survey data collection that anticipates a plurality of modes and ongoing evolutionary developments.

                                              Find this resource:

                                              Mode Effects

                                              A great deal of research has been invested in the effectiveness and impact of different modes of survey data collection. Much of this research involves comparing one mode with another. But comparisons between modes are not simple, since the relative merits of each mode depend on the dimension considered. Biemer 2001 reports research that compares modes in terms of response rates and others in terms of sample coverage and bias; while others such as Aquilino 1998 and Jäckle, et al. 2010 consider the performance in terms of measurement error. Some of the research, such as that in de Leeuw and Collins 1997, focuses on very specific aspects of the survey enterprise, while other publications have a broader focus and still others, including Manfreda, et al. 2008, provide meta-analytic overviews.

                                              • Aquilino, William. 1998. Effects of interview mode on measuring depression in younger adults. Journal of Official Statistics 14:15–29.

                                                Save Citation »Export Citation »E-mail Citation »

                                                Compares reports of depression using three interview modes: self-administered, face-to-face, and telephone. The self-administered mode yielded higher rates of reported depression, and the conclusion drawn is that self-administered modes increase the rate of reporting of socially undesirable attributes and behaviors.

                                                Find this resource:

                                                • Biemer, Paul P. 2001. Nonresponse bias and measurement bias in a comparison of face to face and telephone interviewing. Journal of Official Statistics 17:295–320.

                                                  Save Citation »Export Citation »E-mail Citation »

                                                  Provides a methodology for comparing the level of measurement bias and non-response bias using face-to-face and telephone interview methods.

                                                  Find this resource:

                                                  • de Leeuw, Edith, and Martin Collins. 1997. Data collection methods and survey quality: An overview. In Survey measurement and process quality. Edited by L. Lyberg, P. Biemer, and M. Collins, 199–220. New York: Wiley.

                                                    DOI: 10.1002/9781118490013Save Citation »Export Citation »E-mail Citation »

                                                    An overview of administration mode on a variety of aspects of data quality is provided in this paper.

                                                    Find this resource:

                                                    • Jäckle, Annette, Caroline Roberts, and Peter Lynn. 2010. Assessing the effect of data collection mode on measurement. International Statistical Review 78:3–20.

                                                      DOI: 10.1111/j.1751-5823.2010.00102.xSave Citation »Export Citation »E-mail Citation »

                                                      As mixed-mode surveys become more common it is important to assess mode effects on survey quality. This paper outlines why it is difficult to assess these effects and then explores some ways forward.

                                                      Find this resource:

                                                      • Manfreda, K. L., M. Bosniak, J. Berzelak, I. Haas, and V. Vehovar. 2008. Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research 50:79–104.

                                                        Save Citation »Export Citation »E-mail Citation »

                                                        A meta-analytic analysis of forty-five experimental studies, this compares the response rates of web surveys with a range of other administration modes. Analysis indicates an average of 11 percent lower response rates with web surveys. The analysis does not assess whether differential response rates results in differential response error.

                                                        Find this resource:

                                                        Computer-Assisted Interviewing

                                                        Computer-assisted interviewing has become a key part of many modes of questionnaire administration—both self-administered surveys and interviewer-assisted surveys. Many face-to-face interviews are now administered using computer-assisted questionnaires that “personalize” the interview to individuals based on their characteristics and responses to earlier responses. Telephone interviews rely almost wholly on computer-assisted interviewing techniques. The advent of web-based surveys has seen the widespread use of computer software to generate self-administered computer-assisted questionnaires. Computer-assisted interviewing is an evolving field where the ability to implement computer-based methods has preceded knowledge of how they affect the quality of the data collected. There are particularities of computer-assisted techniques that are too specific to be discussed in this bibliography, but some broad overviews are certainly worth reviewing prior to exploring the detail related to particular modes. The collection of papers in Couper, et al. 1998 provides a snapshot of the state of the art some years ago. Conrad and Schobe 2007 reflects the changes since the Couper, et al. 1998 collection and anticipates the possibilities that lie ahead. De Leeuw, et al. 1995 offers a different way of viewing computer-based interviewing methods.

                                                        • Conrad, Frederick G., and Michael F. Schobe, eds. 2007. Envisioning the survey interview of the future. Hoboken, NJ: Wiley.

                                                          DOI: 10.1002/9780470183373Save Citation »Export Citation »E-mail Citation »

                                                          Looking to the future, this collection anticipates many of the potential uses of computer-based technologies in survey research. It provides an overview of what 21st-century technologies are likely to enable but also looks at broader principles related to computer-assisted interviewing.

                                                          Find this resource:

                                                          • Couper, M., R. Baker, J. Bethlehem, et al., eds. 1998. Computer assisted survey information collection. New York: Wiley.

                                                            Save Citation »Export Citation »E-mail Citation »

                                                            Although now a little dated, this wide-ranging collection of articles surveys a variety of ways in which computer-assisted data methods are used in survey practice. This includes both data collection and pre-analysis data processing such as coding and optical character recognition.

                                                            Find this resource:

                                                            • de Leeuw, Edith D., Joop J. Hox, and Ger Snijkers. 1995. The effect of computer-assisted interviewing on data quality: A review. Journal of the Market Research Society 37:325–344

                                                              Save Citation »Export Citation »E-mail Citation »

                                                              Rather than reporting research, this paper provides a framework for assessing the possible data quality effects of computer-assisted interview methods relative to traditional methods. It is a useful analysis despite the many computer-based developments since it was written.

                                                              Find this resource:

                                                              Telephone Surveys

                                                              Telephones provided the first major technological method for conducting interviews. Combining telephone use with computer-assisted interviewing software led to the widespread use of Computer-Assisted Telephone Interviewing (CATI). In those areas where there was high penetration of telephones, this method provided both a good sampling frame and a cost-effective method of conducting interviews. Over time this led to CATI interviews replacing face-to-face interviews for many surveys. A number of detailed books on telephone surveys have become available that cater to different audiences. Fink, et al. 2003 is an accessible introduction for those new to telephone surveys, while Groves, et al. 2001 and Lepkowski, et al. 2008 provide more advanced treatments for telephone methodologists. Gwartney 2007 is directed to telephone interviewers and those training and supervising telephone interviewers. Groves 1990 provides a succinct overview of telephone survey theory and practice. Over time, telephone interviewing has faced new challenges. Curtin, et al. 2005 highlights the problems of declining response rates, call screening, the wide use of mobile phones, and the deluge of telemarketing. Link, et al. 2007 pays special attention to the widespread use of cell phones and the need to develop telephone survey methodologies specifically directed to this mode of interviewing. Telephone interviewing continues to be a key mode of data collection and is constantly adapting to the new environment. It is likely that telephone surveying will continue to be important; but the ways it is used and the samples it is used for will change as the effectiveness of telephone surveys change and as alternatives (e.g., web-based surveys for some populations) become more attractive and feasible.

                                                              • Curtin, Richard, Stanley Presser, and Eleanor Singer. 2005. Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly 69:87–98

                                                                DOI: 10.1093/poq/nfi002Save Citation »Export Citation »E-mail Citation »

                                                                Using a long-running telephone survey, these authors document the steady decline in telephone response rates since 1979 and explore the components of the decline and the effect of incentives in arresting this decline.

                                                                Find this resource:

                                                                • Fink, A., L. B. Bourque, and E. P. Fielder. 2003. The survey kit: How to conduct telephone surveys. New York: SAGE.

                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                  A basic introduction to survey design with an emphasis on telephone administration, appropriate for an undergraduate audience. While useful for this audience, it lacks the depth required for a person actually designing and implementing a telephone survey.

                                                                  Find this resource:

                                                                  • Groves, Robert M. 1990. Theories and methods of telephone surveys. Annual Review of Sociology 16:221–240

                                                                    DOI: 10.1146/annurev.so.16.080190.001253Save Citation »Export Citation »E-mail Citation »

                                                                    Provides a succinct overview of theories to help understand the way in which telephone surveys succeed and fail. This discussion is supplemented by a summary of research on the way in which telephone surveys are subject to various types of error: coverage, sampling, non-response, and measurement error.

                                                                    Find this resource:

                                                                    • Groves, R. M., P. P. Biemer, L. E. Lyberg, J. T. Massey, W. L. Nicholls, and J. Waksberg, eds. 2001. Telephone survey methodology. New York: Wiley.

                                                                      Save Citation »Export Citation »E-mail Citation »

                                                                      Includes contributions from experts around the world and provides an in-depth treatment of telephone survey methodology in different parts of the world. Although not the latest treatment, it provides authoritative accounts of coverage issues, sampling, non-response, data quality, and methods of administration.

                                                                      Find this resource:

                                                                      • Gwartney, P. 2007. The telephone interviewer’s handbook: How to conduct standardized conversations. San Francisco: Jossey-Bass.

                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                        Designed for telephone interviewers and trainers, this essential volume covers the practical details and techniques required for successful telephone interviews in an increasingly difficult survey context. The advice is based on sociological and psychological research and a great deal of practical experience.

                                                                        Find this resource:

                                                                        • Lepkowski, J., C. Tucker, J. M. Brick, et al., eds. 2008. Advances in telephone survey methodology. Hoboken, NJ: Wiley.

                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                          This comprehensive collection from seventy-five experts and practitioners reflects 21st-century technological developments and the changing experience of telephone survey researchers. It discusses how to modify survey practices and how to use statistical techniques to adjust for the changing behavior of sample members in telephone survey study.

                                                                          Find this resource:

                                                                          • Link, Michael W., Michael P. Battaglia, Martin R. Frankel, Larry Osborn, and Ali H. Mokdad. 2007. Reaching the U.S. cell phone generation. Public Opinion Quarterly 71:814–839

                                                                            DOI: 10.1093/poq/nfm051Save Citation »Export Citation »E-mail Citation »

                                                                            Based on a study in which cell-phone and fixed-line telephone methodologies were used, this study urges considerable caution in implementing cell-phone surveys and points to the high cost and low response rates of cell-phone methodologies.

                                                                            Find this resource:

                                                                            Internet

                                                                            This mode of questionnaire administration is still relatively new and much is still being learned about the pros and cons of the method and how to maximize the efficiency and the quality of the data using the Internet. Some very useful book-length overviews and guides have become available in recent years that both provide practical, research-based advice about how to conduct useful web surveys and also point to some of the methodological challenges faced by web-based survey research. The most valuable of these are Couper 2008; Bethlehem and Biffignandi 2011; and Dillman, et al. 2008. As with much of the computer-based survey methods, the capacity to conduct Internet surveys has preceded our knowledge of what works and why it works (or does not work). Among the biggest challenges, especially for population-based investigations, is obtaining a scientific sample from the Internet. While some web survey companies have recruited probability-based samples, the bulk of Internet surveys appear to be based on non-probability samples. Baker, et al. 2010 overviews the most recent research on the value and quality of “opt in” online panels. The quality of web-based samples is a recurrent problem and the significance of this is addressed in Bethlehem 2010.

                                                                            • Baker, R., S. J. Blumberg, M. P. Couper, et al. 2010. AAPOR report on online panels. Public Opinion Quarterly 74:711–781.

                                                                              DOI: 10.1093/poq/nfq048Save Citation »Export Citation »E-mail Citation »

                                                                              Under commission from the American Association for Public Opinion Research, this group of survey experts assesses the value of survey panels—especially non-probability-based opt-in panels. It provides a clear overview of the sampling problems in Internet surveys and overviews the available research on Internet samples.

                                                                              Find this resource:

                                                                              • Bethlehem, J. 2010. Selection bias in web surveys. International Statistical Review 78:161–188

                                                                                DOI: 10.1111/j.1751-5823.2010.00112.xSave Citation »Export Citation »E-mail Citation »

                                                                                Internet coverage is incomplete and biased, and web recruitment is frequently based on self-selection. This paper discusses the methodological problems that flow from this and assesses whether good solutions are available.

                                                                                Find this resource:

                                                                                • Bethlehem, J., and S. Biffignandi. 2011. Handbook of web surveys. New York: Wiley.

                                                                                  DOI: 10.1002/9781118121757Save Citation »Export Citation »E-mail Citation »

                                                                                  An invaluable reference work on both the theoretical and practical aspects of web survey design and administration. In addition to detailing how to design web surveys, the handbook considers them in relation to other modes and proposes methods of minimizing some of the sampling-related problems.

                                                                                  Find this resource:

                                                                                  • Couper, M. 2008. Designing effective web surveys. New York: Cambridge Univ. Press.

                                                                                    DOI: 10.1017/CBO9780511499371Save Citation »Export Citation »E-mail Citation »

                                                                                    A comprehensive book written by one of the leading web survey methodologists. This is both a practical guide to designing web surveys and a good overview of the empirical evidence and theory relating to web surveys.

                                                                                    Find this resource:

                                                                                    • Dillman, D. A., J. D. Smyth, and L. M. Christian. 2008. Internet, mail, and mixed-mode surveys: The tailored design method. New York: Wiley.

                                                                                      Save Citation »Export Citation »E-mail Citation »

                                                                                      Dillman’s book on the “tailored design” method for mail surveys is a classic. This updated edition makes use of the most recent research evidence, and one chapter extends the principles to the Internet surveys.

                                                                                      Find this resource:

                                                                                      Mail, Self-Administered, and Paper

                                                                                      The Dillman 2007 tailored design method, initially designed for mail surveys but later extended to other forms of self-administered questionnaires, is the authoritative work on how to design self-administered mail surveys and to how to maximize response rates. More focused aspects of mail surveys are also provided in Harrison 2010, but most of the conclusions of these studies are incorporated in the more comprehensive treatments in Dillman 2007.

                                                                                      • Dillman, D. A. 2007. Mail and Internet surveys: The tailored design method. Hoboken, NJ: Wiley.

                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                        This highly influential updated book is the Bible of mail survey design. It is replete with practical, research-based advice on all aspects of the design and administration of postal surveys.

                                                                                        Find this resource:

                                                                                        • Harrison, C. H. 2010. Mail surveys and paper questionnaires. In Handbook of survey research. 2d ed. Edited by P. V. Marsden and J. D. Wright, 499–526. Bingley, UK: Emerald.

                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                          A concise but useful up-to-date summary of the core issues associated with mail questionnaires.

                                                                                          Find this resource:

                                                                                          Mixed Mode

                                                                                          As response rates have declined and alternative methods of questionnaire administration have become more common, survey methodologists such as the author of Dillman 2007 have increasingly explored the viability of mixed-mode methods of questionnaire administration. On the face of it, this makes sense. Different types of people in a sample may prefer different ways of responding, and the choice of response mode may maximize overall response rates and overcome sampling frame weaknesses (e.g., telephone or Internet access or seldom being at home for personal interviews) if just one response mode is available. But mixed-mode survey methods raise important questions, such as whether the additional effort in producing choice actually improves overall response rate. As well as pointing out how overall response rates are affected, Groves and Lepkowowski 1985 points to other advantages of mixed-mode surveys as well as potential disadvantages that are summarized in de Leeuw 2005.

                                                                                          • de Leeuw, Edith D. 2005. To mix or not to mix data collection modes in surveys. Journal of Official Statistics 21:233–255

                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                            De Leeuw provides an overview of common forms of mixed-mode designs and the advantages and disadvantages of these designs. She concludes by discussing practical issues in implementing mixed-mode designs and provides an agenda for research.

                                                                                            Find this resource:

                                                                                            • Dillman, D. A. 2007. Mixed mode surveys. In Mail and Internet surveys: The tailored design method. By D. A. Dillman, 217–244. Hoboken, NJ: Wiley.

                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                              A concise discussion of the advantages and disadvantages of mixed-mode surveys and some of the issues in implementing such surveys.

                                                                                              Find this resource:

                                                                                              • Groves, Robert M., and James M. Lepkowowski. 1985. Dual frame, mixed-mode survey designs. Journal of Official Statistics 1:263–286

                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                In reporting this US study that used a mixture of face-to-face and telephone interviews, Groves and Lepkowowski point to the cost and sample coverage advantages of the dual frame approach. The improvement in sample variance is seen to be a big gain of the dual mode approach.

                                                                                                Find this resource:

                                                                                                Questionnaires

                                                                                                Structured questionnaires are the most common instrument by which survey data are collected and are therefore potentially the key source of measurement error in surveys. Over the years a great deal has been written about the design and construction of questions and the design of questionnaires more broadly. One of the early gems remains Payne 1951. Oppenheim 2000 also provides an early and useful guide to question and questionnaire design (first published in 1966, fully revised in 1992). But Payne’s advice (and to a fair extent Oppenheim’s) was based on a lot of common sense and practice and less on systematic research. Valuable, research-based studies of question wording have appeared over the last thirty years and have added to our knowledge of the effects of question wording and design. Foddy 1994 produced an excellent and widely used summary of the theory and research relating to question design. Two further related books are Sudman, et al. 1996 (cited under Framing and Context Effects) and Bradburn, et al. 2004. For shorter introductions to question and questionnaire design it is difficult to go past Converse and Presser 1986 and Krosnick and Presser 2010.

                                                                                                • Bradburn, N. M., S. Sudman, and B. Wansink. 2004. Asking questions: The definitive guide to questionnaire design: For market research, political polls, and social and health questionnaires. Rev. ed. San Francisco: Jossey-Bass.

                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                  Since being first published in 1989, this book has become a classic. This updated edition provides advice that is appropriate for a range of different modes of questionnaire administration and a wide range of questions: from simple objective questions to sensitive and subjective questions.

                                                                                                  Find this resource:

                                                                                                  • Converse, J. M., and S. Presser. 1986. Survey questions: Handcrafting the standardized questionnaire. Quantitative Applications in the Social Sciences 63. Beverly Hills, CA: SAGE.

                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                    Although dated, this brief book continues to be a useful overview that provides research-based guidance regarding principles of question and questionnaire design.

                                                                                                    Find this resource:

                                                                                                    • Foddy, W. 1994. Constructing questions for interviews and questionnaires: Theory and practice in social research. Cambridge, UK: Cambridge Univ. Press.

                                                                                                      Save Citation »Export Citation »E-mail Citation »

                                                                                                      Based on research into question wording and construction, Foddy develops a theoretical framework for understanding the science and art of question construction. Using this framework he provides practical, research-based advice on how to produce effective questions.

                                                                                                      Find this resource:

                                                                                                      • Krosnick, J. A., and S. Presser. 2010. Question and questionnaire design. In Handbook of survey research. 2d ed. Edited by P. V. Marsden and J. D. Wright, 263–313. Bingley, UK: Emerald.

                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                        A relatively concise but authoritative overview of how to design high-quality questions and questionnaires. An ideal starting point for questionnaire design.

                                                                                                        Find this resource:

                                                                                                        • Oppenheim, A. N. 2000. Questionnaire design, interviewing and attitude measurement. 2d ed. London: Continuum.

                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                          Originally published in 1966, this revised and expanded book remains a very useful introduction and text.

                                                                                                          Find this resource:

                                                                                                          • Payne, S. L. 1951. The art of asking questions. Princeton, NJ: Princeton Univ. Press.

                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                            This classic has stood the test of time. Written in an engaging style, it urges the questionnaire designer to make questions brief, simple, relevant, and specific. Furthermore, he shows readers how to achieve these features in their questions.

                                                                                                            Find this resource:

                                                                                                            Framing and Context Effects

                                                                                                            The way people answer questions is not simply a function of question wording. The way questions are administered can affect answers. The design of the questionnaire itself and the context and order in which respondents answer questions can shape their answers. Sudman, et al. 1996 and Schuman and Presser 1996 were early authoritative accounts of the way question wording, framing, and context influenced responses. Different sorts of question order effects have been identified. Sudman, et al. 1996 distinguished between “contrast” and “consistency” effects while Moore 2002 introduced the concepts of “additive” and “subtractive” effects. Where people have ill-formed views, the particular context in which questions are asked or even minor changes of wording can affect responses—a phenomenon that is described by Chong and Druckman 2007 in their article on framing theory. Much of our knowledge of context and framing effects draws on the insights of cognitive psychology, which are well captured in Tourangeau, et al. 2000.

                                                                                                            • Chong, Dennis, and James Druckman. 2007. Framing theory. Annual Review of Political Science 10:103–126

                                                                                                              DOI: 10.1146/annurev.polisci.10.072805.103054Save Citation »Export Citation »E-mail Citation »

                                                                                                              Argues that where attitudes are not strongly formed, the context in which people answer questions can significantly shape responses. This context is referred to as the “framing effect.” Chong and Druckman outline framing theory and discuss how framing and priming affect public opinion research.

                                                                                                              Find this resource:

                                                                                                              • Moore, David W. 2002. Measuring new types of question-order effects: Additive and subtractive. Public Opinion Quarterly 66:80–91

                                                                                                                DOI: 10.1086/338631Save Citation »Export Citation »E-mail Citation »

                                                                                                                In addition to contrast and consistency effects, Moore identifies another way in which question order affects what he calls “additive” and “subtractive effects.” This dimension interacts with contrast and consistency effects to create eight types of reactions to the order in which judgment-based questions are asked.

                                                                                                                Find this resource:

                                                                                                                • Schuman, Howard, and Stanley Presser. 1996. Questions and answers in attitude surveys: Experiments on question form, wording and context. London: SAGE.

                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                  First published in 1981, this classic and influential book reports on well-designed experiments that demonstrate the way responses to attitude questions are affected by aspects of wording, the use of non-committal responses, question form, question order, and context. Chapter 2 focuses on question order effects.

                                                                                                                  Find this resource:

                                                                                                                  • Sudman, S., N. M. Bradburn, and N. Schwarz. 1996. Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.

                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                    When asked to form a judgment, respondents will compare an object to some standard of comparison. The judgment will depend on what other points of comparison a respondent is provided with and the order they are provided in. This phenomenon reflects contrast and consistency effects when making judgments.

                                                                                                                    Find this resource:

                                                                                                                    • Tourangeau, R., L. J. Rips, and K. A. Rasinski. 2000. The psychology of survey response. Cambridge, UK: Cambridge Univ. Press.

                                                                                                                      DOI: 10.1017/CBO9780511819322Save Citation »Export Citation »E-mail Citation »

                                                                                                                      Covers a wide range of psychological processes that affect the way in which people respond to different types of survey questions. Chapter 7 provides an accessible account of the way in which question context and order shape responses.

                                                                                                                      Find this resource:

                                                                                                                      Evaluating Questions

                                                                                                                      Survey researchers have often neglected rigorous pre-testing of survey questions. However, CASM (Cognitive Aspects of Survey Methodology) researchers have given prominence to the whole stage of pre-testing. Essentially this stage involves evaluating questions that have been developed using the guidelines that have been outlined in references already cited. This stage is designed to improve the reliability and the validity of responses and to ensure that the questions and answers mean what we think they mean. The collection Presser, et al. 2004 provides an authoritative collection of papers to evaluate questions. As seen in Beatty, et al. 2004, in-depth interviewing is a common part of pre-testing questions but Foddy 1996 points to the dangers in this approach. There is now a good collection of books on cognitive interviewing for pre-testing questions, including Sirken, et al. 1999 and Willis 2005. In addition to Sudman, et al. 1996 (cited under Framing and Context Effects), the books here are helpful overviews. Two brief introductory overviews of cognitive methods of pre-testing are Schwarz 2007 and Tourangeau and Bradburn 2010.

                                                                                                                      • Beatty, P., W. D. Hicks, E. Schmeidler, and C. Kirchner. 2004. Investigating question meaning and context through in-depth interviews. Quality and Quantity 38:367–379

                                                                                                                        DOI: 10.1023/B:QUQU.0000043133.61603.e9Save Citation »Export Citation »E-mail Citation »

                                                                                                                        Using data from actual surveys, this article seeks to demonstrate the value of in-depth interviewing as a way of understanding the meaning of survey responses.

                                                                                                                        Find this resource:

                                                                                                                        • Foddy, W. 1996. The in-depth testing of survey questions: A critical appraisal. Quality and Quantity 30:361–370

                                                                                                                          DOI: 10.1007/BF00170142Save Citation »Export Citation »E-mail Citation »

                                                                                                                          Argues that although in-depth pilot testing can help identify comprehension problems, most in-depth procedures are ineffective in identifying other problems with questions. Foddy proposes ways of making in-depth pilot testing more effective.

                                                                                                                          Find this resource:

                                                                                                                          • Presser, S., J. Rothgeb, M. P. Couper, et al., eds. 2004. Methods for testing and evaluating survey questionnaires. Hoboken, NJ: Wiley.

                                                                                                                            DOI: 10.1002/0471654728Save Citation »Export Citation »E-mail Citation »

                                                                                                                            Definitive and practical collection of papers from fifty world experts on questionnaire design, testing, and evaluation. Covers topics including cognitive interviewing, the importance of administration mode, how to pre-test, the use of experiments with question wording, and statistical modeling of pre-test data.

                                                                                                                            Find this resource:

                                                                                                                            • Schwarz, N. 2007. Cognitive aspects of survey methodology. Applied Cognitive Psychology 21:277–287

                                                                                                                              DOI: 10.1002/acp.1340Save Citation »Export Citation »E-mail Citation »

                                                                                                                              Reviews key themes in relation to the Cognitive Aspects of Survey Methods (CASM) and outlines the strengths and shortcomings of this methodology.

                                                                                                                              Find this resource:

                                                                                                                              • Sirken, M. D., S. Herrmann, N. Schechter, J. Tanur Schwarz, and R. Tourangeau, eds. 1999. Cognition and survey research. New York: Wiley.

                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                This collection of papers from a conference of CASM experts provides an excellent summary of the field: covers both what is known and what is yet to be learned.

                                                                                                                                Find this resource:

                                                                                                                                • Tourangeau, R., and N. M. Bradburn. 2010. The psychology of survey response. In Handbook of survey research. 2d ed. Edited by P. V. Marsden and J. D. Wright, 315–346. Bingley, UK: Emerald.

                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                  These two authorities in cognitive survey design provide an up-to-date, succinct summary of research and methods of cognitive methods in pre-testing for surveys.

                                                                                                                                  Find this resource:

                                                                                                                                  • Willis, G. B. 2005. Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: SAGE.

                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                    This readable and comprehensive book provides survey researchers with a detailed account of both the theory and practice of cognitive interviewing methods as they relate to questionnaire design.

                                                                                                                                    Find this resource:

                                                                                                                                    Interviews

                                                                                                                                    Because of the need to obtain structured data, survey interviews are mainly structured interviews based on structured questionnaires. References relating to Questionnaires are relevant to the construction of structured interview schedules. In this section, the references are restricted to those that deal with administering these questionnaires by a trained interviewer. In particular, it focuses on the techniques and dynamics of interviewing and on the problem of interviewer variability. Survey research seeks to obtain information by providing the same “stimulus” to each respondent. This means that the questions and interview format need to be the same for each respondent. Standardized questionnaires mean that the same questions are administered in the same order to all in-scope respondents. But interviewers can introduce variability and affect the way in which respondents both understand and choose to respond to the questions. As such the interviewer can be an important source of survey error. Cannell, et al. 1981 provides a useful but somewhat dated outline of some of the problems of interviewing and what is known about how to eliminate these. Fowler and Mangione 1990 provides a more recent and more comprehensive introduction to structured interviewing. Not all interviewer effects are due to interviewing behavior. Collins 1980 demonstrates that some effects can result from interviewer characteristics relative to the interviewee including gender and racial differences.

                                                                                                                                    • Cannell, Charles F., Peter V. Miller, and Lois Oksenberg. 1981. Research on interviewing techniques. Sociological Methodology 12:389–437

                                                                                                                                      DOI: 10.2307/270748Save Citation »Export Citation »E-mail Citation »

                                                                                                                                      Assembles and assesses the research evidence relating to the ways in which to conduct effective interviews and the effect of different approaches to data quality.

                                                                                                                                      Find this resource:

                                                                                                                                      • Collins, Martin. 1980. Interviewer variability: A review of the problem. Journal of the Market Research Society 22:77–95

                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                        Since interviewers are a key element of the interview process, they have the capacity to shape both response rates and the quality of the data. Collins assesses the evidence about the extent to which these behaviors and interviewer characteristics actually have an impact.

                                                                                                                                        Find this resource:

                                                                                                                                        • Fowler, F. J., and T. W. Mangione. 1990. Standardized survey interviewing: Minimizing interviewer-related error. New York: SAGE.

                                                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                                                          A good introduction to structured interviewing that deals with standardized techniques, concepts of interviewer-related error, interviewer selection, training and supervision of interviewers, and how to minimize interviewer-related error in surveys.

                                                                                                                                          Find this resource:

                                                                                                                                          Survey Error

                                                                                                                                          The total survey error paradigm is a framework for getting the best data for the least financial outlay. Error can be classified into various components including sampling error and measurement error. Sampling error can be further broken down into components such as coverage error and non-response error. Measurement error can consist of a range of elements including measurement bias (e.g., due to social desirability, acquiescence), non-response, poor reliability, and invalid measures. The goal of survey design is to minimize all these sources of error while keeping costs under control. At a certain point the gains (e.g., in sample quality and precision of estimates) are not worth the additional costs. A number of excellent overviews of survey error are available. Some of these adopt a total survey error perspective while others focus on particular elements of survey error. One of the earliest overviews is provided in the brief article Deming 1944. Three books—Groves 2004, Biemer and Lyberg 2003, and Weisberg 2005—provide comprehensive overviews of the total survey error approach and are excellent places to start. Finally, Biemer 2010 argues that total survey error is simply a subset of a broader concept—total survey quality.

                                                                                                                                          • Biemer, Paul P. 2010. Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly 74:817–848

                                                                                                                                            DOI: 10.1093/poq/nfq058Save Citation »Export Citation »E-mail Citation »

                                                                                                                                            Biemer places the concept of total survey error within the broader concept of total survey quality. He argues that survey quality is broader than survey error and includes “fitness for use” considerations such as relevance, accessibility, interpretability, comparability, coherence, and completeness.

                                                                                                                                            Find this resource:

                                                                                                                                            • Biemer, Paul P., and L. Lyberg. 2003. Introduction to survey quality. Hoboken, NJ: Wiley.

                                                                                                                                              DOI: 10.1002/0471458740Save Citation »Export Citation »E-mail Citation »

                                                                                                                                              Overlaps somewhat with the material covered in Groves 2004 and provides a comprehensive and contemporary treatment of total survey error and quality.

                                                                                                                                              Find this resource:

                                                                                                                                              • Deming, W. E. 1944. On errors in surveys. American Sociological Review 9:359–369

                                                                                                                                                DOI: 10.2307/2085979Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                Although he does not use the term “total survey error” this brief but remarkably comprehensive paper details thirteen broad sources of survey error. This paper remains a very good starting point for an understanding of the concept of total survey error.

                                                                                                                                                Find this resource:

                                                                                                                                                • Groves, R. M. 2004. Survey errors and survey costs. Hoboken, NJ: Wiley.

                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                  Despite being first published in 1989, this book is one of the most comprehensive treatments of total survey error (and survey costs) available. As well as dealing comprehensively with survey error it takes the question of survey cost seriously.

                                                                                                                                                  Find this resource:

                                                                                                                                                  • Weisberg, H. 2005. The total survey error approach: A guide to the new science of survey. Chicago: Univ. of Chicago Press.

                                                                                                                                                    DOI: 10.7208/chicago/9780226891293.001.0001Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                    Provides a contemporary examination of survey error, including a discussion of survey error theory, response accuracy issues, respondent selection issues, and matters of survey administration that affect survey error.

                                                                                                                                                    Find this resource:

                                                                                                                                                    Measurement Error

                                                                                                                                                    Measurement error is a huge potential source of survey error. Measurement error refers to the extent to which our measurement instruments fail to capture accurately the underlying reality they are designed to measure. Measurement error should be distinguished from sample-related error, which relates to inadequacies in the sample that compromise generalization from the sample to its population. There are many potential types and sources of measurement error. Sudman 1980 remains a good introduction to the way questionnaire items can be a source of measurement error. The following books provide more comprehensive treatments. Blasius and Thiesse 2012 provides a comprehensive overview of measurement types and sources of measurement error and offers some solutions. Alwin 2007 is a more advanced treatment focusing on reliability as a source of measurement error while Biemer, et al. 2004 produces a more advanced discussion of various forms of measurement error. At the heart of measurement error are the twin concepts of reliability and validity. Reliable measurement instruments are those that achieve the same measurement of the same state on different occasions. A valid instrument is one that accurately captures the underlying reality that it purports to measure. Obtaining reliable and valid instruments is a matter of how the instrument is designed and administered, and these matters are dealt with in Alwin 2007 and in other references in this article. The problem for researchers is in knowing how reliable and valid their instruments are. A number of introductions to the concepts of reliability and validity are available. Most of these discussions such as in Bohrnstedt 1970 and Carmines and Zeller 1979 also outline how the level of reliability and validity is assessed. A fuller and more recent discussion of these approaches is provided in Litwin 1995. Since many surveys rely on asking people about events in the past, the normal challenges of reliability and validity are compounded by problems associated with recalling and interpreting past events. Dex 1995 summarizes the research on the reliability of recall data.

                                                                                                                                                    • Alwin, D. F. 2007. Margins of error: A study of reliability in survey measurement. Hoboken, NJ: Wiley.

                                                                                                                                                      DOI: 10.1002/9780470146316Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                      Alwin examines six elements of the response process (question adequacy, comprehension, accessibility, retrieval, motivation, and communication), reliability estimation techniques, interviewer practices to enhance reliability, and question formats that maximize reliability. The book is appropriate for methodologists and graduate-level courses.

                                                                                                                                                      Find this resource:

                                                                                                                                                      • Biemer, Paul P., Robert M. Groves, Lars E. Lyberg, Nancy A. Mathiowetz, and Seymour Sudman, eds. 2004. Measurement errors in surveys. New York: Wiley.

                                                                                                                                                        DOI: 10.1002/9781118150382Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                        Collection of papers from eminent survey researchers that explores measurement error stemming from questionnaire content, administration mode, respondents, interviewers, as well as from other modes of data collection and from the interview process itself. The book concludes with a section on modeling and on estimating the effect of measurement error.

                                                                                                                                                        Find this resource:

                                                                                                                                                        • Blasius, Jörg, and Victor Thiesse. 2012. Assessing the quality of survey data. London: SAGE.

                                                                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                          Offers an introduction to methods of analysis to assess the quality and validity of survey responses. It explores error due to acquiescence, misunderstanding of questions, poor field work, fatigue, and faking answers.

                                                                                                                                                          Find this resource:

                                                                                                                                                          • Bohrnstedt, G. W. 1970. Reliability and validity assessment in attitude research. In Attitude measurement. Edited by G. F. Summers, 80–89. Chicago: Rand McNally.

                                                                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                            This classic statement is an excellent starting point regarding approaches to reliable and valid assessment when measuring attitudes.

                                                                                                                                                            Find this resource:

                                                                                                                                                            • Carmines, Edward G., and Richard A. Zeller. 1979. Reliability and validity assessment. Quantitative Applications in the Social Sciences 17. Beverly Hills, CA: SAGE.

                                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                              A good, accessible introduction to reliability and various forms of validity. In addition to discussing the concepts, this brief book outlines common ways of assessing both reliability and validity.

                                                                                                                                                              Find this resource:

                                                                                                                                                              • Dex, Shirley. 1995. The reliability of recall data: A literature review. Bulletin de Methodologie Sociologique 49:58–89

                                                                                                                                                                DOI: 10.1177/075910639504900105Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                Dex provides a useful summary of what is known about the conditions under which recall data are reliable and when they are unreliable.

                                                                                                                                                                Find this resource:

                                                                                                                                                                • Litwin, M. S. 1995. How to measure survey reliability and validity. New York: SAGE.

                                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                  This is a good introduction to the statistical methods commonly used to assess the reliability and validity of data that have been collected. Rather than showing how to avoid collecting unreliable or invalid data, the book focuses on detecting and measuring this form of measurement error.

                                                                                                                                                                  Find this resource:

                                                                                                                                                                  • Sudman, S. 1980. Reducing response error in surveys. Journal of the Royal Statistical Society Series D 29:237–273

                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                    Remains an accessible and remarkably undated account of ways of writing and presenting questions that will reduce the extent of measurement error.

                                                                                                                                                                    Find this resource:

                                                                                                                                                                    Sampling Error

                                                                                                                                                                    The second major source of survey error stems from sampling problems. While measurement error relates to the adequacy of the measurements collected in a survey, sampling error and related matters relate to the adequacy of the sample the measurements are obtained from. Good samples are necessary in order to generalize to a population from a sample. Even if perfect measurements were obtained, these are of limited value if they cannot be generalized confidently beyond the particular sample members. Sampling error can result from poor sampling coverage, which results from an inadequate sample design. The other main source of sample error is due to non-response from those selected to be part of the sample. This section focuses on non-response- related sampling error. Non-response and declining response rates in surveys are a major concern for survey researchers. From a sampling perspective non-response is a concern when it creates biases in the resulting sample. Since some people are less likely to respond to surveys, non-response typically is not random. To the extent that non-responders are different in relation to the focus of a survey (e.g., voting intention, attitudes) then the sample will be biased and will result in errors in estimating population parameters from the sample. The three main ways of dealing with non-response are: (1) to reduce non-response, (2) to estimate whether or not non-response is producing bias, and (3) using weighting procedures to adjust for non-response so that in some respects at least the biases will be corrected at the post-sampling phase. Non-response is different from sample non-coverage. Sample non-coverage occurs when the sampling frame excludes some sampling elements, while non-response occurs when selected sample elements do not participate in the survey. Both non-response and non-coverage can result in sample bias and therefore sample error.

                                                                                                                                                                    Causes of Non-Response

                                                                                                                                                                    An understanding of the causes of non-response is necessary if response rates are to be improved. The most comprehensive and up-to-date treatment of non-response (both causes and solutions) has been produced in Bethlehem, et al. 2011. Groves and Couper 1998 provides a conceptual framework for understanding non-response, and Groves, et al. 2002 outlines a number of theoretical approaches and also considers the effects of survey design and mode. Others such as Couper and Groves 1996 and Durrant, et al. 2010 consider the effects of interviewer and respondent characteristics on the decision to participate in surveys.

                                                                                                                                                                    • Bethlehem, J., F. Cobben, and B. Schouten. 2011. Handbook in nonresponse in household surveys. Hoboken, NJ: Wiley.

                                                                                                                                                                      DOI: 10.1002/9780470891056Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                      This comprehensive and recent guide to non-response covers both the causes of non-response, ways of reducing non-response, and methods of correcting for non-response bias.

                                                                                                                                                                      Find this resource:

                                                                                                                                                                      • Couper, Mick P., and Robert M. Groves. 1996. Household-level determinants of survey nonresponse. New Directions for Program Evaluation 70:63–79

                                                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                        Seeks to develop a theory of survey participation and focuses on one set of variables needed for such a theory—the socio-demographic characteristics of the household.

                                                                                                                                                                        Find this resource:

                                                                                                                                                                        • Durrant, Gabriele B., Robert M. Groves, Laura Staetsky, and Fiona Steele. 2010. Effects of interviewer attitudes and behaviors on refusal in household surveys. Public Opinion Quarterly 74:1–36

                                                                                                                                                                          DOI: 10.1093/poq/nfp098Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                          Shows that interviewer confidence and attitude toward persuading reluctant respondents is important in explaining variations in refusal rates. There are also interactions between respondent and interviewer characteristics such that greater similarity is linked to lower refusal rates.

                                                                                                                                                                          Find this resource:

                                                                                                                                                                          • Groves, Robert M., and Mick P. Couper. 1998. A conceptual framework for survey participation. In Nonresponse in household interview surveys. By Robert M. Groves and Mick P. Couper, 25–46. Chichester, UK: Wiley.

                                                                                                                                                                            DOI: 10.1002/9781118490082Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                            Argues that experienced interviewers achieve high response rates because of their ability to tailor their approaches by maximizing the gain/value of survey participation and minimizing the cost to that respondent.

                                                                                                                                                                            Find this resource:

                                                                                                                                                                            • Groves, Robert M., Don A. Dillman, John L. Eltinge, and Roderick Little, eds. 2002. Survey nonresponse. New York: Wiley.

                                                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                              This compendium of research on survey non-response includes chapters from the leading experts. Sections include theories of non-response, the impact of survey design on non-response, the effect of survey mode, and methods of statistically adjusting for the effects of non-response.

                                                                                                                                                                              Find this resource:

                                                                                                                                                                              Effects of Non-Response

                                                                                                                                                                              Non-response is a concern for a number of reasons. High non-response rates increase the cost of conducting a survey: achieving a specified sample size in the face of high non-response means that the cost of obtaining the sample will be increased. The extent of the additional cost will depend on the method of survey administration (e.g., web survey compared with a face-to-face survey). If the budget is capped, then high non-response will result in smaller sample sizes that can then lead to problems at the analysis stage. The other major danger of non-response is that it can affect data quality. If non-response is non-random then it will lead to biased samples and biased estimates. But response rates and bias are not always positively related. Groves and Peytcheva 2008 provides an overview of the extent to which samples that achieve high-response rates produce better samples than those with low response rates and Fricker and Tourangeau 2010 assesses whether the extra effort in improving response rates actually improves sample quality. One way of improving response rates is to use multi-mode designs; but the question with this approach is whether the introduction of mode effects outweighs the benefits of better response rates. Vogt and Saris 2005 assesses this trade-off.

                                                                                                                                                                              • Fricker, Scott, and Roger Tourangeau. 2010. Examining the relationship between nonresponse propensity and data quality in two national household surveys. Public Opinion Quarterly 74:934–955

                                                                                                                                                                                DOI: 10.1093/poq/nfq064Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                Does the effort in recruiting reluctant respondents warrant the expense in terms of reducing non-response error? Or does greater measurement error from reluctant respondents outweigh any reductions in response bias? This study addresses these questions.

                                                                                                                                                                                Find this resource:

                                                                                                                                                                                • Groves, R., and E. Peytcheva. 2008. The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly 72:167–189

                                                                                                                                                                                  DOI: 10.1093/poq/nfn011Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                  This meta-analysis of fifty-eight articles shows that although high-response rates can reduce non-response bias, this is not always the case and that non-response bias is attributable more to individual measures than a survey as a whole.

                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                  • Vogt, R. J. J., and W. E. Saris. 2005. Mixed mode designs: Finding the balance between nonresponse bias and mode effects. Journal of Official Statistics 21:367–387

                                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                    Mixed-mode designs are used to reduce non-response, but they run the risk of increasing measurement error due to mode effects. The data reported in this study indicates that mixed mode designs both improve response rates and reduce non-response bias.

                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                    Improving Response Rates

                                                                                                                                                                                    One way of improving response rates is to avoid the practices that lead to high non-response. Thus, avoiding Data Collection Modes that are subject to high non-response is one way of improving response rates. But there are positive actions that can improve response rates. Church 1993 and Singer, et al. 1999 show that providing incentives is effective in both postal- and interviewer-mediated surveys. But incentives are by no means the only way of improving response rates. Laurie, et al. 1999 describes a range of procedures used to maintain good response rates across the waves of a major panel survey. In face-to-face interviews there is a range of factors that can be employed by interviewers to maximize the chances of contacted people agreeing to participate. National Centre for Social Research 1999 provides useful and practical advice for interviewers.

                                                                                                                                                                                    • Church, Allan H. 1993. Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly 57:62–79

                                                                                                                                                                                      DOI: 10.1086/269355Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                      A meta-analysis of thirty-eight studies examines the effect of monetary versus non-monetary incentives and the effect of these when provided with the invitation to participate versus upon return of the questionnaire. Under certain conditions incentives had a moderate effect.

                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                      • Laurie, Heather, Rachel Smith, and Lynne Scott. 1999. Strategies for reducing nonresponse in a longitudinal panel survey. Journal of Official Statistics 15:269–282

                                                                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                        Non-response is a particular problem in panel surveys since it compounds with each wave. This paper describes the effectiveness of a range of strategies employed in the British Household Panel Survey.

                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                        • National Centre for Social Research. 1999. How to improve survey response rates: A guide for interviewers on the doorstep (non-commercial). London: SAGE.

                                                                                                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                          This guide from a major non-commercial survey center provides excellent practical and well-grounded advice for training face-to-face interviewers to gain cooperation.

                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                          • Singer, Eleanor, John van Hoewyk, and Nancy Gebler. 1999. The effect of incentives on response rates in interviewer-mediated surveys. Journal of Official Statistics 15:217–230

                                                                                                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                            While the positive impact of incentives in postal surveys is well established, this meta-analysis examines research on the effect of incentives in face-to-face surveys and telephone surveys. The effectiveness of cash incentives, both up front and promised (and for both low-burden and high-burden surveys) is demonstrated.

                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                            Weighting Samples

                                                                                                                                                                                            Since non-response can produce biased samples, it is necessary to minimize any effect of sample bias. Minimizing non-response should help, but high response rates do not necessarily mean non-biased samples. Apart from eliminating sample bias, the only effective way of reducing sample bias is to weight samples so that (on particular characteristics at least) the weighted sample reflects the population from which it is drawn. To the extent that the characteristics on which the sample is weighted are correlated with the sample items for which population estimates are sought, this reweighting should improve the accuracy of those estimates. Discussions of weighting can become very technical and theoretical. However, a number of accessible and practical overviews of approaches to weighting sample surveys are available. Crockett 2011, Lynn 1996, and Kalton and Flores-Cervantes 2003 all provide good introductory overviews from slightly different perspectives. Sarndal and Lundstrom 2005 provides a more in-depth and advanced discussion of various estimation procedures to deal with unit non-response. Kalton 1986 deals with some of the additional problems with non-response on panel surveys due to respondents participating in some waves and not others.

                                                                                                                                                                                            • Crockett, Alasdair. 2011. Weighting the social surveys. Manchester, UK: ESDS Government.

                                                                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                              Based upon the proceedings of the Weighting the Social Surveys event held by the Cathie Marsh Centre for Census and Survey Research at the Royal Statistical Society. This document explains the rationale for weighting and the different types of weighting approaches. Illustrates weighting from a number of British government surveys.

                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                              • Kalton, G. 1986. Handling wave non-response in panel surveys. Journal of Official Statistics 2:303–314

                                                                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                While wave non-response can be adjusted by weighting and imputation methods, the choice of compensation procedure is dependent on a range of factors reviewed in this paper.

                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                • Kalton, Graham, and Ismael Flores-Cervantes. 2003. Weighting methods. Journal of Official Statistics 19:81–97

                                                                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                  This concise overview of particular weighting methods provides a clear and very useful overview of the weighting options that are available and, in a non-statistical way, helps the reader understand both the merits of the various approaches and how to go about implementing the various types of reweighting.

                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                  • Lynn, Peter. 1996. Weighting for non-response. In Survey and statistical computing 1996: Proceedings of the second ASC International Conference, Imperial College, London, UK, September 11–13 1996. Edited by Randy Banks, et al., 205–214. Chesham, UK: Association for Survey Computing.

                                                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                    Rather than delving into detailed statistical matters this review outlines the different ways in which adjustments for non-response can be introduced during analysis. It helps with selecting between weighting methods and how to improve the capacity to weight.

                                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                                    • Sarndal, C., and S. Lundstrom. 2005. Estimation in surveys with nonresponse. Hoboken, NJ: Wiley.

                                                                                                                                                                                                      DOI: 10.1002/0470011351Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                      Sarndal and Lundstrom provide a comprehensive and more advanced discussion of a range of approaches to adjusting for sample non-response. It takes the reader beyond the more introductory overviews cited above.

                                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                                      Missing Data

                                                                                                                                                                                                      The references to date have focused on unit non-response—where selected sample elements do not participate in the survey at all. Another form of non-response that can produce its own biases is item non-response where, for one reason or another, a respondent declines to answer particular questions (e.g., income or particular behavioral questions). These missing data can introduce non-response bias. While it can be easier to estimate non-response bias when dealing with item non-response (since a great deal else is known about the non-responder), item non-response can nevertheless produce bias unless corrected and can lead to a substantial loss of cases for analysis. The best solution for dealing with item non-response is to reduce it at its source—to try to ensure that all questions are answered. While this is easier said than done de Leeuw 2001 provides many tips on how to do this. Even when methods of reducing item non-response have been employed, some missing data problems are likely to remain for some items. The main way in which item non-response is handled at the analysis stage is through some form of imputation—estimating what the person would most probably have responded on the basis of the their characteristics, their answers to other questions, and the pattern of responses in the sample. This is an area of statistics that has been developing through the late 20th and early 21st centuries. These advances mean that some of the older articles, while very useful, do not consider the latest methods of imputation. The discussions of imputation range from the relatively introductory overviews such as Allison 2001 and Hertel 1976 to the comprehensive and more statistical treatments in Little and Rubin 2002. A direct empirical comparison of some imputation techniques is provided in Huisman 2000, while some of the special issues of missing data in panel surveys are addressed in Kalton 1986 (cited under Weighting Samples).

                                                                                                                                                                                                      • Allison, P. 2001. Missing data. Quantitative Applications in the Social Sciences 136. Thousand Oaks, CA: SAGE.

                                                                                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                        Covers three main forms of imputation methods at a level suitable for postgraduate students. It deals with expectation-maximization (EM), multiple imputation (MI), and direct maximum likelihood (DML) methods.

                                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                                        • de Leeuw, E. D. 2001. Reducing missing data in surveys: An overview of methods. Quality and Quantity 35:147–160

                                                                                                                                                                                                          DOI: 10.1023/A:1010395805406Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                          Questionnaire optimization and survey design are the two main ways of reducing missing data. This paper provides a useful typology of missing data patterns and sources and discusses how to identify the mechanisms producing missing data and how to reduce item non-response.

                                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                                          • Hertel, B. 1976. Minimizing error variance introduced by missing data analysis in survey analysis. Sociological Methods and Research 4:459–474

                                                                                                                                                                                                            DOI: 10.1177/004912417600400404Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                            While dated, this article provides a useful and reasonably basic evaluation of the simpler forms of missing data treatments.

                                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                                            • Huisman, M. 2000. Imputation of missing item responses: Some simple techniques. Quality and Quantity 34:331–351

                                                                                                                                                                                                              DOI: 10.1023/A:1004782230065Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                              Compares the consequences of different methods of imputing missing data on items that are designed to form a scale. He demonstrates that methods that use information based on relationships between variables perform best.

                                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                                              • Little, R., and D. Rubin. 2002. Statistical analysis with missing data. 2d ed. Hoboken, NJ: Wiley.

                                                                                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                This is an authoritative and comprehensive book from two of the leading experts in missing data methods. It is a more advanced statistical treatment than the other references cited here.

                                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                                Sampling

                                                                                                                                                                                                                The references in this section on sampling cover the theory and practice of sampling. Sampling error and ways of dealing with problematic samples are discussed in the section Sampling Error.

                                                                                                                                                                                                                Probability Samples

                                                                                                                                                                                                                Quite a few book-length treatments of sampling theory, design, and practice are available. These range from simple introductions such as Fink 1995, which is directed at students and those new to survey research, to older and more advanced statistical treatments such as Kish 1965 and Deming 1950. Another category of sampling books includes those that take the reader well beyond basic introductions but focus on the realities of survey sampling. Sudman 1976 is one example. Another is Kish 1987 on sampling design, which provides an important emphasis on developing samples that are fit for purpose and on maximizing the quality of the sample within a limited budget. Of the most contemporary overviews of survey sampling Levy and Lemeshow 2008 is the most accessible of the comprehensive books.

                                                                                                                                                                                                                • Deming, W. E. 1950. Some theory of sampling. New York: Dover.

                                                                                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                  A fairly statistical and theoretical treatment of sampling that will be of less direct use to the typical survey researcher. Some of the more contemporary books on sampling are more accessible for most survey research practitioners.

                                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                                  • Fink, A. 1995. How to sample in surveys. New York: SAGE.

                                                                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                    A basic, readable introduction to survey sampling for those new to this field.

                                                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                                                    • Kish, L. 1965. Survey sampling. New York: Wiley.

                                                                                                                                                                                                                      Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                      A classic introduction to survey sampling for those with some understanding of statistics.

                                                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                                                      • Kish, L. 1987. Statistical design for research. New York: Wiley.

                                                                                                                                                                                                                        DOI: 10.1002/0471725196Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                        A compact treatment of core issues in sampling design that helps sample designers assess and develop the most efficient and effective sample designs. It focuses on how to maximize strengths and minimize weaknesses of sample designs within real cost constraints.

                                                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                                                        • Levy, P. S., and S. Lemeshow. 2008. Sampling of populations: Methods and applications. 4th ed. Hoboken, NJ: Wiley.

                                                                                                                                                                                                                          DOI: 10.1002/9780470374597Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                          This well-established text provides a comprehensive and largely non-statistical understanding of the basic and most current forms of survey sampling. It is well targeted to the modern environment in which perfect sampling is increasingly problematic.

                                                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                                                          • Sudman, Seymour. 1976. Applied sampling. New York: Academic Press.

                                                                                                                                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                            Rather than focusing heavily on the statistical foundations of sampling theory, Sudman addresses this book to researchers with more limited statistical knowledge who nevertheless need to understand the principles and practicalities of sampling. Using practical examples, Sudman covers a wide range of sampling methods including the simpler, less expensive designs.

                                                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                                                            Non-Probability Samples

                                                                                                                                                                                                                            While there are strong statistical arguments for probability sampling (at least in its pure form), the reality is that such sampling is not always possible. One problem with probability sampling is that it requires a known population and some form of sampling frame. But many sociological topics require samples for populations where it is impossible to know the size or membership of the population. These studies require samples of subgroups and rare populations where no adequate sample frame is available. Without such a frame it is impossible to estimate the probability of selection of any member of the population and this makes it impossible to obtain a probability sample from which one can generalize with a known margin of error. There is no ideal solution to this problem, but a number of approaches have been offered. The texts in the section above outline some of the non-probability methods of sampling. In addition, a number of sampling theorists and practitioners have proposed methods of producing samples from rare populations or those where there is no known sampling frame, as seen in Sudman and Kalton 1986; Sudman, et al. 1988; and Heckathorn 1997. A different approach both to obtaining samples where a sampling frame is unavailable or too expensive to obtain is to use a form of quota sampling. Although discredited in the United States in the 1948 presidential election campaign, the evidence in Marsh and Scarbrough 1990 shows that usable quota samples can be obtained.

                                                                                                                                                                                                                            • Heckathorn, D. D. 1997. Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems 44:174–199

                                                                                                                                                                                                                              DOI: 10.2307/3096941Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                              Heckathorn develops the notion of network sampling (Sudman, et al. 1988) using the concept of respondent-driven sampling. Using a variant of chain referral sampling, the author demonstrates how the biases associated with such samples can be reduced.

                                                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                                                              • Marsh, Catherine, and Elinor Scarbrough. 1990. Testing nine hypotheses about quota sampling. Journal of the Market Research Society 32:485–506

                                                                                                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                This experimental study demonstrates the extent to which quota samples can yield apparently good samples. It challenges some of the simple critiques of quota sampling and provides some arguments for using this form of sample design.

                                                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                                                • Sudman, S., and G. Kalton. 1986. New developments in the sampling of special populations. Annual Review of Sociology 12:401–429

                                                                                                                                                                                                                                  DOI: 10.1146/annurev.so.12.080186.002153Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                  A wide range of alternative methods for careful sampling of special populations are described. The authors observe that the appropriateness of the particular methods will depend on the characteristics of the special population.

                                                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                                                  • Sudman, S., Monroe G. Sirkin, and Charles D. Cowan. 1988. Sampling rare and elusive populations. Science 240:991–995

                                                                                                                                                                                                                                    DOI: 10.1126/science.240.4855.991Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                    Despite the overlap between this paper and Sudman and Kalton 1986 this brief article provides a good summary of probability methods for difficult to find populations. The discussion of network sampling is useful.

                                                                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                                                                    Survey Designs

                                                                                                                                                                                                                                    Surveys are developed for two main reasons. The first reason is to describe phenomena and to extrapolate a description from a sample to a population. This aim requires that the nature of the sample allows reliable inference from the sample to the population. References relating to this matter are discussed in the section on Sampling. The second main goal is to build or test causal propositions and theories. Since correlation does not demonstrate causation, the design and analysis of surveys should be conducted so that causal propositions can be tested. Experiments have often been regarded as the gold standard for testing causal propositions; but experimental work is typically inappropriate or not feasible in sociological research. Kish 1959 discusses various survey designs that provide researchers with the ability to draw causal conclusions without experimental data. Some innovative designs described in Sniderman and Grob 1996 are also developing in which the internal validity strengths of the experimental design are being combined with the external validity strengths of random sample surveys.

                                                                                                                                                                                                                                    • Kish, L. 1959. Some statistical problems in research design. American Sociological Review 24:328–338

                                                                                                                                                                                                                                      DOI: 10.2307/2089381Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                      A classic and readable paper on research design that addresses both the external validity and internal validity issues of design. While experiments can be strong in relation to internal validity, they can lack the strong external validity of surveys. Surveys have the opposite attributes.

                                                                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                                                                      • Sniderman, Paul M., and Douglas B. Grob. 1996. Innovations in experimental design in attitude surveys. Annual Review of Sociology 22:377–399

                                                                                                                                                                                                                                        DOI: 10.1146/annurev.soc.22.1.377Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                        Describes ways in which some of the strengths of the experimental design can be incorporated within sample surveys thus drawing on the strengths of both designs.

                                                                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                                                                        Cross-Sectional

                                                                                                                                                                                                                                        The most common form of survey design is the one-shot cross-sectional sample where causal analysis relies on modeling cross-sectional data for a diverse sample. While analytic techniques as discussed in Davies and Pickles 1985 can help with causal reasoning, cross-sectional designs are necessarily limited for causal analysis. Cross-sectional surveys that build in a time dimension can assist with causal analysis, since causal reasoning requires that time sequence can be established. Firebaugh 1997 outlines the use of trend designs that involve the same survey being conducted with different samples over time, which enables the tracking of macro or aggregate change over time and provides some way of establishing time sequence. Another variation of cross-sectional surveys with a time dimension is cohort analysis of cross-sectional data as described in Glenn 2005.

                                                                                                                                                                                                                                        • Davies, R. B., and A. R. Pickles. 1985. Longitudinal versus cross-sectional methods for behavioural research: A first round knockout. Environment and Planning A 17:1315–1329

                                                                                                                                                                                                                                          DOI: 10.1068/a171315Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                          This introductory comparison of the design strengths and weaknesses of cross-sectional and longitudinal designs makes the case strongly for the superiority of longitudinal designs for causal analysis.

                                                                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                                                                          • Firebaugh, Glenn. 1997. Analyzing repeated surveys. Quantitative Applications in the Social Sciences 115. Thousand Oaks, CA: SAGE.

                                                                                                                                                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                            Firebaugh discusses techniques for detecting and understanding aggregate trends in repeated cross-sectional surveys. This includes a discussion of cohort analysis, decomposition of aggregated trends, and a model for decomposing aggregate change.

                                                                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                                                                            • Glenn, Norval D. 2005. Cohort analysis. 2d ed. Quantitative Applications in the Social Sciences 5. Beverly Hills, CA: SAGE.

                                                                                                                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                              The second edition of this book is an entirely rewritten version of the first edition published in 1997. It deals with methodological basics of cohort analysis and some of the alternatives for estimating age, period, and cohort effects.

                                                                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                                                                              Retrospective

                                                                                                                                                                                                                                              Retrospective survey designs represent another form of cross-sectional design that seeks to build a time dimension into the survey so that the time sequencing of events can be established and the effects of past events on present attributes can be estimated. Retrospective designs involve asking respondents to recall past events. This approach is commonly used to build a life history of a person—it might be to establish a relationship history, a fertility history, an employment history, or some other set of past events. In establishing this history and the relative timing of events, it is possible to look at the effects of events on subsequent behavior (e.g., having a child on workforce participation; relationship type on fertility behavior). While prospective studies, where we wait for things to happen, could establish the same sequence, Scott and Alwin 1998 highlights the disadvantages, such as the large amounts of time and money required to conduct such studies. Schwarz and Sudman 1994 points to the reliability and validity challenges posed by using the recall method in retrospective designs. Bradburn, et al. 1994 and Dex 1995 (cited under Measurement Error) demonstrate that people can simply forget or misremember the details of past events, even quite significant events. This can lead to unreliable data.

                                                                                                                                                                                                                                              • Bradburn, N. M., J. Huttenlocher, and L. Hedges. 1994. Telescoping and temporal memory. In Autobiographical memory and the validity of retrospective reports. Edited by N. Schwarz and S. Sudman, 203–215. New York: Springer-Verlag.

                                                                                                                                                                                                                                                DOI: 10.1007/978-1-4612-2624-6Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                A theoretical model is presented to help understand backward and forward telescoping—i.e., recalling the event to have occurred later or earlier than in fact was the case.

                                                                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                                                                • Schwarz, Norbert, and Seymour Sudman, eds. 1994. Autobiographical memory and the validity of retrospective reports. New York: Springer-Verlag.

                                                                                                                                                                                                                                                  DOI: 10.1007/978-1-4612-2624-6Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                  Although the articles in this collection are not exclusively concerned with recall in the survey context, the findings have clear implications for the use of recall methods. While the articles identify some of the limits of recall methods, these methods still have their place if used with appropriate caution.

                                                                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                                                                  • Scott, Jacqueline, and Duane Alwin. 1998. Retrospective versus prospective measurement of life histories in longitudinal research. In Methods of life course research: Qualitative and quantitative approaches. Edited by J. Z. Giele and G. H. Elder, 98–127. Thousand Oaks, CA: SAGE.

                                                                                                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                    Retrospective methods provide a much faster and cheaper way of collecting life history data than do prospective data. But these speed and cost advantages come with a price. This chapter compares the strengths and weaknesses of these two approaches to collecting life history data.

                                                                                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                                                                                    Cross-National

                                                                                                                                                                                                                                                    Another version of cross-sectional survey design is the cross-national survey. Simple accounts of this design and some of the challenges they encounter are provided in Smith 2010. Although cross-national designs could employ panel designs, most cross-national surveys are cross sectional—i.e., they consist of similar cross-sectional surveys conducted in different nations. There are different conceptions of what constitutes cross-national survey designs. Indeed some argue that there is nothing distinctive about cross-national survey designs, since all survey research is comparative—the only difference with cross-national surveys is that the survey compares countries rather than subunits of a country or groups within the country. While there is some merit to this argument, Harkness, et al. 2003 and Harkness, et al. 2010 make the case for the special challenges and contributions of cross-national surveys. Heise 2010 tackles the concept of cross-cultural surveys in a quite distinctive way, arguing that they are not simply a series of population-based surveys conducted in multiple countries. Verba 1993 and Kuechler 1998 question the value of standard sample surveys in cross-national research and argue for a modification of the country-level approach.

                                                                                                                                                                                                                                                    • Harkness, J. A., M. Braun, B. Edwards, et al., eds. 2010. Survey methods in multicultural, multinational, and multiregional contexts. Hoboken, NJ: Wiley.

                                                                                                                                                                                                                                                      DOI: 10.1002/9780470609927Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                      Based on papers from leading survey researchers at a major conference, this book addresses issues relating to the need for a framework that maximizes quality in comparative surveys. It examines the ways in which cultural contexts add an extra dimension to assuring quality.

                                                                                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                                                                                      • Harkness, J. A., F. J. R. Vijver, and P. P. Mohler. 2003. Cross-cultural survey methods. Hoboken, NJ: Wiley.

                                                                                                                                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                        Provides a systematic, interdisciplinary account of the issues involved in cross-cultural research—especially equivalence of measurement and comparability of findings. As well as helping identify error and bias in such surveys, the authors outline approaches for minimizing problems at both the data collection and data analysis stages.

                                                                                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                                                                                        • Heise, David R. 2010. Surveying cultures: Discovering shared conceptions and sentiments. Hoboken, NJ: Wiley.

                                                                                                                                                                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                          Provides a useful and important different perspective about the way in which to survey sentiments within and between cultures. It goes well beyond the idea that cross-cultural surveys simply mean reproducing the standard population survey in a number of countries.

                                                                                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                                                                                          • Kuechler, Manfred. 1998. The survey method: An indispensable tool for social science research everywhere? American Behavioral Scientist 42:178–200

                                                                                                                                                                                                                                                            DOI: 10.1177/0002764298042002005Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                            Argues that extending national surveys to a large number of heterogeneous countries produces data of dubious value and that such data needs to be adjusted by non-quantifiable data.

                                                                                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                                                                                            • Smith, T. W. 2010. Surveying across nations and cultures. In Handbook of survey research. 2d ed. Edited by P. V. Marsden and J. D. Wright, 733–763. Bingley, UK: Emerald.

                                                                                                                                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                              Smith provides an up-to-date and wide-ranging overview of the methods and issues involved in conducting and interpreting cross-cultural surveys. This is an excellent starting point for understanding cross-cultural survey designs.

                                                                                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                                                                                              • Verba, Sidney. 1993. The uses of survey research in the study of comparative politics: Issues and strategies. Historical Social Research 18:55–103

                                                                                                                                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                First published in 1969, this paper discusses the limitations of traditional survey research in the context of macro comparative studies and argues for the inclusion of structural country characteristics into the survey design.

                                                                                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                                                                                Problems of Equivalence

                                                                                                                                                                                                                                                                The core challenge for comparative cross-national survey designs is that of comparability of survey results across nations. The logic of survey analysis assumes that all respondents are essentially subject to the same stimulus (i.e., asked the same questions in the same context). Given the impact of question wording, question order, mode effects, and other elements of administration, as well as the importance of sample designs it is important that data are collected in comparable ways for all respondents in all nations. The problem is, this is rarely the case. Even if all these elements could be standardized, Jowell 1998 argues there can be no certainty that the same questions mean the same thing in different contexts. Verba 1971 raises the question of whether the best way of achieving comparability of meaning in different countries is by using identical measures and questions or by using questions that are functionally equivalent across countries. While this is not a problem unique to cross-cultural research, it is probably more pronounced. Jowell, et al. 2007 provides useful practical advice about how to mitigate equivalence problems, and Lynn 2003 outlines ways in which survey standards can be addressed to help with the problems of cross-country comparability.

                                                                                                                                                                                                                                                                • Jowell, Roger. 1998. How comparative is comparative research? American Behavioral Scientist 42:168–177

                                                                                                                                                                                                                                                                  DOI: 10.1177/0002764298042002004Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                  After pointing to the difficulty in achieving true equivalence in multinational surveys, Jowell develops ten very useful rules of thumb for mitigating the problems of equivalence.

                                                                                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                                                                                  • Jowell, Roger, Caroline Roberts, Rory Fitzgerald, Gillian Eva, and Centre for Comparative Social Surveys. 2007. Measuring attitudes cross-nationally: Lessons from the European social survey. London: SAGE.

                                                                                                                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                    Based on their experience in the European Social Survey, Jowell and his colleagues have developed one of the most comprehensive and practical guides to cross-national survey design, data collection, and interpretation.

                                                                                                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                                                                                                    • Lynn, Peter. 2003. Developing quality standards for cross-national survey research: Five approaches. International Journal of Social Research Methodology 6:323–336

                                                                                                                                                                                                                                                                      DOI: 10.1080/13645570210132848Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                      Cross-national variations in survey implementation and the constraints on the way in which surveys are implemented are outlined. Five approaches to establishing survey standards in cross-national surveys to minimize these variations are discussed and evaluated.

                                                                                                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                                                                                                      • Verba, Sidney. 1971. Cross-national survey research: The problem of credibility. In Comparative methods in sociology: Essays and trends and applications. Edited by Ivan Vallier, 309–353. Berkeley: Univ. of California Press.

                                                                                                                                                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                        A good discussion of the concept of functional equivalence and on ways of analyzing data to address some of the issues of cross-cultural functional equivalence.

                                                                                                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                                                                                                        Longitudinal

                                                                                                                                                                                                                                                                        Sociological research seeks to achieve generalizable description and an understanding of social phenomena. One key type of understanding is achieving causal explanations of behavior and social change. Early surveys were cross-sectional and had to achieve causal explanations by means of the analysis of co-variation and the use of statistical controls. But such forms of causal explanation always suffer from the difficulty of establishing temporal sequence and actually demonstrating change—both important components of any causal explanation. Longitudinal surveys have helped overcome these shortcomings of cross-sectional designs. Kalton and Citro 1995 identifies a variety of longitudinal survey designs: they vary from trend studies that are simply a set of time-ordered cross-sectional studies through to panel surveys that track the same sample over time. Panel surveys are the gold standard of survey design in that they enable researchers to establish change at both the individual and aggregate levels and establish the time order in which events occur. Rose 2000 explains the rationale for panel surveys and provides authoritative reviews of the design challenges of panel surveys. However, despite the potential of longitudinal designs to deliver unique types of information, Boruch and Pearson 1988 points out that there are legitimate and important threats to their value, and any decision to implement a longitudinal design should consider these problems.

                                                                                                                                                                                                                                                                        • Boruch, R. F., and R. W. Pearson. 1988. Assessing the quality of longitudinal surveys. Evaluation Review 12:3–58

                                                                                                                                                                                                                                                                          DOI: 10.1177/0193841X8801200101Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                          This thoughtful critical evaluation of longitudinal designs raises considerations that are not widely canvassed in standard discussions of these designs. Following the critique the authors outline a number of ways of improving the quality and value of longitudinal studies—again raising ideas not normally raised in traditional discussions.

                                                                                                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                                                                                                          • Kalton, Graham, and Constance F. Citro. 1995. Panel surveys: Adding the fourth dimension. Innovation: The European Journal of Social Science Research 8:25–39

                                                                                                                                                                                                                                                                            DOI: 10.1080/13511610.1995.9968429Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                            The first half of this paper reviews the characteristics of a range of longitudinal survey designs, while the second half focuses on some design challenges that occur in panel designs.

                                                                                                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                                                                                                            • Rose, D., ed. 2000. Researching social and economic change: The uses of household panel studies. London: Routledge.

                                                                                                                                                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                              This important and readable edited collection consists of three sections. The first provides the case for the extra effort, cost, and complexity of conducting panel surveys. The second section deals with the challenges of conducting useful panel surveys, and the third demonstrates how a panel survey sheds new light on phenomena.

                                                                                                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                                                                                                              Panel Surveys

                                                                                                                                                                                                                                                                              Since panel designs are complex, there are many considerations in developing good panel surveys. Some very useful books both describe the features of various types of longitudinal surveys and also deal with the aspects of design that, if not dealt with properly, can undermine the potential of these surveys. Kasprzyk, et al. 1989 and Lynn 2009 provide a comprehensive set of authoritative chapters on panel designs. Menard 2002 offers a simple introduction, while Menard 2008 covers a wide range of material from design through to complex methods of analysis. While these book-length volumes cover most of the issues in longitudinal survey design, a number of journal papers deal with some of the specific issues. Non-response and panel attrition is a major challenge in panel designs, and Laurie, et al. 1999 provides a concise discussion of ways of reducing attrition. See also Jäckle 2006 and Understanding Society Innovation.

                                                                                                                                                                                                                                                                              • Jäckle, Annette. 2006. Dependent interviewing: A framework and application to current research. ISER Working Paper 2006-32. Colchester, UK: Univ. of Essex.

                                                                                                                                                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                Dependent interviewing is a method used in panel surveys to increase efficiency and reduce respondent burden by “feeding through” from a previous wave into the interview in subsequent waves. Jäckle provides a framework to help assess the value of dependent interviewing and the circumstances where it may not be effective.

                                                                                                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                                                                                                • Kasprzyk, D., G. Duncan, G. Kalton, and M. P. Singh, eds. 1989. Panel surveys. New York: Wiley.

                                                                                                                                                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                  Contains papers presented at the first International Symposium on Panel Surveys held in Washington in 1986 and brings together the state of knowledge as it had developed in the 1960s, 1970s, and early 1980s.

                                                                                                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                                                                                                  • Laurie, Heather, Rachel Smith, and Lynne Scott. 1999. Strategies for reducing nonresponse in a longitudinal panel survey. Journal of Official Statistics 15:269–282

                                                                                                                                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                    Based on fieldwork procedures used in the British Household Panel Survey, this paper evaluates the approaches used to minimize non-response and attrition. It also considers the changing methods that need to be employed as the panel matures.

                                                                                                                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                                                                                                                    • Lynn, P. 2009. Methodology of longitudinal surveys. Chichester, UK: Wiley.

                                                                                                                                                                                                                                                                                      DOI: 10.1002/9780470743874Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                      Contains papers presented at the 2006 International Conference of Methodology of Longitudinal Surveys and effectively brings together the developments in panel surveys since Kasprzyk 1989. It is an authoritative and comprehensive reference for longitudinal survey designers and analysts.

                                                                                                                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                                                                                                                      • Menard, S. W. 2002. Longitudinal research. Thousand Oaks, CA: SAGE.

                                                                                                                                                                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                        This book is a basic introduction to longitudinal research.

                                                                                                                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                                                                                                                        • Menard, S. W., ed. 2008. Handbook of longitudinal research: Design, measurement, and analysis. Burlington, MA: Elsevier/Academic Press.

                                                                                                                                                                                                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                          Menard has collected a wide range of work on key topics. This book surveys a range of longitudinal survey designs, considers a variety of measurement issues, and devotes a great deal of attention to both simple and particularly more advanced methods of analysis of longitudinal data.

                                                                                                                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                                                                                                                          • Understanding Society Innovation Panel.

                                                                                                                                                                                                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                            Understanding Society, a leading British panel household survey, contains a separate panel that is used to test questions, better understand respondent behavior (e.g., non-response), and undertake methodological experiments (e.g., impact of mixed-mode administration). Technical reports about the Understanding Society panel are also available.

                                                                                                                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                                                                                                                            Survey Analysis

                                                                                                                                                                                                                                                                                            It is not possible to cite a comprehensive selection of books that relate to survey analysis. Many excellent books are available that will assist survey researchers with the analysis of their survey data. Many of these books are not specifically written for survey researchers and relate to quantitative analysis more generally, regardless of the way in which the data were collected. This section therefore focuses on references that concentrate on survey analysis in particular and where the main emphasis is on the logic of the analytic method. Hyman 1955 wrote one of the first texts on survey analysis. Hirschi and Selvin 1967 and Rosenberg 1968 were responsible for explicating the logic of survey analysis for a generation of undergraduates. Davis 1971 is a brief but influential book that introduced students to a range of basic techniques and Davis 1985 is an important introduction to the logic of causal analysis. As more survey data became available through data archives, the opportunities for secondary data analysis increased. The special issues involved in secondary analysis are outlined at a basic level in Dale, et al. 1988. Another advance in survey research was the development of panel surveys and longitudinal data sets. The strength of these survey designs meant that a new and more complex range of analytic techniques became available to the survey researcher as outlined in Taris 2000.

                                                                                                                                                                                                                                                                                            • Dale, Angela, Sara Arber, and Michael Procter. 1988. Doing secondary analysis. London: Allen and Unwin.

                                                                                                                                                                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                              A book-length overview of the advantages and disadvantages of secondary data analysis that guides the analyst through some of the potential pitfalls.

                                                                                                                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                                                                                                                              • Davis, James A. 1971. Elementary survey analysis. Englewood Cliffs, NJ: Prentice-Hall.

                                                                                                                                                                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                A basic and relatively non-statistical introduction to survey analysis.

                                                                                                                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                                                                                                                • Davis, James A. 1985. The logic of causal order. Quantitative Applications in the Social Sciences 7. Beverly Hills, CA: SAGE.

                                                                                                                                                                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                  This is a brief but compelling explanation of the logic by which causal models can be built and tested using survey data. It moves beyond three variable models to multivariable models and provides the logical basis for understanding much subsequent causal modeling in the years that follow.

                                                                                                                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                                                                                                                  • Hirschi, Travis, and Hanan C. Selvin. 1967. Delinquency research: An appraisal of analytic methods. New York: Free Press.

                                                                                                                                                                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                    Written as a critique of the flawed thinking in delinquency research, this influential book was later published in 1973 as Principles of Survey Analysis. It introduced social scientists to the issues involved in drawing causal inferences from non-experimental data and helped clarify the logic of causal thinking and analysis.

                                                                                                                                                                                                                                                                                                    Find this resource:

                                                                                                                                                                                                                                                                                                    • Hyman, H. H. 1955. Survey design and analysis: Principles, cases, and procedure. New York: Free Press.

                                                                                                                                                                                                                                                                                                      Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                      One of the first texts to outline the logic and statistical techniques employed in survey research and to apply these techniques to survey data sets. Although basic by 21st-century standards, this was an influential book.

                                                                                                                                                                                                                                                                                                      Find this resource:

                                                                                                                                                                                                                                                                                                      • Rosenberg, M. 1968. The logic of survey analysis. New York: Basic Books.

                                                                                                                                                                                                                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                        Definitive book outlining the basic logic of survey analysis and the way in which survey analysts seek to draw causal conclusions without longitudinal data or experimental designs. Although this is an old book, it is still well worth reading because of its emphasis on the basic logic of survey analysis.

                                                                                                                                                                                                                                                                                                        Find this resource:

                                                                                                                                                                                                                                                                                                        • Taris, T. 2000. A primer in longitudinal data analysis. Thousand Oaks, CA: SAGE.

                                                                                                                                                                                                                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                          This is an accessible introduction to the issues involved in analyzing longitudinal data. It is good for those new to the subject and deals with some of the common methods of analysis (e.g., change scores, regression, event history analysis) but does not deal with the more complex modeling methods.

                                                                                                                                                                                                                                                                                                          Find this resource:

                                                                                                                                                                                                                                                                                                          Major Survey Data Sets

                                                                                                                                                                                                                                                                                                          A wealth of social survey data is available for secondary analysis by qualified users. These data sets provide access to data that would be beyond the capacity of any one individual or team to collect and, depending on the survey, enable the tracking of change over time, comparisons between nations, and focused analysis on a myriad of topics. The Internet provides a simple way of finding and (with permission) downloading these data sets for detailed analysis. This section points to some key studies and locations for accessing these data sets.

                                                                                                                                                                                                                                                                                                          Data Archives

                                                                                                                                                                                                                                                                                                          In many nations, centralized data archives have been established in which the main data sets are curated and made available to approved researchers. These are national data archives throughout the world, and particular attention is drawn to exemplary data archives in the United States (ICPSR), the United Kingdom, Australia (ADA), and Europe (CESSDA). See also ADA International Listing of Data Archives and UK Data Archives.

                                                                                                                                                                                                                                                                                                          Repeated Cross-Sectional

                                                                                                                                                                                                                                                                                                          Many national statistical agencies conduct regular specialized national surveys to collect data on employment, time use, health, housing, disabilities, education, and many other matters. These surveys are repeated at varying intervals and enable the assessment of these matters at a particular point of time and to examine trends over time. The Australian Bureau of Statistics provides links to a wide range of such agencies that conduct national repeated cross-sectional surveys. Two iconic repeated cross-national surveys are the US-based General Social Survey and the American National Election Study.

                                                                                                                                                                                                                                                                                                          Cross-National

                                                                                                                                                                                                                                                                                                          In addition to the types of annual surveys noted above, there are more than a few such surveys that, as well as being repeated, are also conducted in a number of countries. In addition to tracking changes over time, these surveys also enable comparisons of patterns and trends in a variety of countries. The most long-standing such program is the annual International Social Survey program. The World Values Survey is another example that is conducted every five years. The more recent but methodologically strongest cross-national survey is the European Social Survey.

                                                                                                                                                                                                                                                                                                          Panel Surveys

                                                                                                                                                                                                                                                                                                          Panel surveys include the Household, Income and Labour Dynamics in Australia Survey, the German Socio-Economic Panel, the Panel Study of Income Dynamics, and Understanding Society.

                                                                                                                                                                                                                                                                                                          Critics

                                                                                                                                                                                                                                                                                                          While survey methods are frequently employed in social research they are by no means the only method available. Nor are surveys the best method for all research questions. Schuman 2008 presents an “insiders” critique of surveys based on a deep knowledge and extensive experience. “Outsiders,” i.e., those who are not survey practitioners, frequently criticize surveys for a number of reasons, the most important of which is their critique of the variable-based approach that is fundamental to survey research. These critics argue that the variable approach of survey research prevents researchers from properly understanding behavior in context or holistically. They argue that when society is reduced to a set of variables and we seek to establish causal relationships between variables we miss out on the subjective and meaningful component to behavior (Blumer 1956, Esser 1996). Marsh 1982 analyzes and rejects these criticisms.

                                                                                                                                                                                                                                                                                                          • Blumer, H. 1956. Sociological analysis and the “variable.” American Sociological Review 21:683–690

                                                                                                                                                                                                                                                                                                            DOI: 10.2307/2088418Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                            This classic paper by Blumer critically evaluates sociological analysis that “seeks to reduce human life to variables and their relations”—one of the key characteristics of survey research.

                                                                                                                                                                                                                                                                                                            Find this resource:

                                                                                                                                                                                                                                                                                                            • Esser, Hartmut. 1996. What is wrong with “variable sociology”? European Sociological Review 12:159–166

                                                                                                                                                                                                                                                                                                              DOI: 10.1093/oxfordjournals.esr.a018183Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                              Argues that variable sociology, which is the basis of survey research, is characterized by a fundamental inability to explain phenomena. He argues that while variable sociology demonstrates relationships between variables, it is those relationships themselves that need to be explained.

                                                                                                                                                                                                                                                                                                              Find this resource:

                                                                                                                                                                                                                                                                                                              • Marsh, Catherine. 1982. The survey method: The contribution of surveys to sociological explanation. London: Allen and Unwin.

                                                                                                                                                                                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                                Marsh responds to the survey critics who argue that survey research is not only incapable of capturing the subjective and meaningful component of behavior but also cannot adequately establish causal relationships. Demonstrates that many of the criticisms are based on misunderstandings of what good survey research can achieve.

                                                                                                                                                                                                                                                                                                                Find this resource:

                                                                                                                                                                                                                                                                                                                • Schuman, Howard. 2008. Method and meaning in polls and surveys. Cambridge, MA: Harvard Univ. Press.

                                                                                                                                                                                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                                                                                                                                                                                  As an expert in survey research, Schuman presents a balanced approach to the results produced by surveys and polls. This readable book focuses on the meaning of survey responses and provides a nuanced appreciation of what surveys can and cannot do. It provides a judiciously critical evaluation of survey results.

                                                                                                                                                                                                                                                                                                                  Find this resource:

                                                                                                                                                                                                                                                                                                                  back to top

                                                                                                                                                                                                                                                                                                                  Article

                                                                                                                                                                                                                                                                                                                  Up

                                                                                                                                                                                                                                                                                                                  Down