Pub Date : 2022-10-25DOI: 10.1177/1525822X221132940
Barbara Felderer, J. Herzing
Even though the proportion of individuals who are not equipped to participate in online surveys is constantly decreasing, many surveys face an under-representation of individuals who do not feel IT literate enough to participate. Using experimental data from a probability-based online panel, we study which recruitment survey mode strategy performs best in recruiting less IT-literate persons for an online panel. The sampled individuals received postal invitations to conduct the recruitment survey in a self-completion mode. We experimentally vary four recruitment survey mode strategies: one online mode strategy, two sequential mixed-mode strategies, and one concurrent mode strategy. We find the recruitment survey mode strategies to have a major effect on the sample composition of the recruitment survey, but the differences between the strategies vanish once respondents are asked to proceed with the panel online.
{"title":"What about the Less IT Literate? A Comparison of Different Postal Recruitment Strategies to an Online Panel of the General Population","authors":"Barbara Felderer, J. Herzing","doi":"10.1177/1525822X221132940","DOIUrl":"https://doi.org/10.1177/1525822X221132940","url":null,"abstract":"Even though the proportion of individuals who are not equipped to participate in online surveys is constantly decreasing, many surveys face an under-representation of individuals who do not feel IT literate enough to participate. Using experimental data from a probability-based online panel, we study which recruitment survey mode strategy performs best in recruiting less IT-literate persons for an online panel. The sampled individuals received postal invitations to conduct the recruitment survey in a self-completion mode. We experimentally vary four recruitment survey mode strategies: one online mode strategy, two sequential mixed-mode strategies, and one concurrent mode strategy. We find the recruitment survey mode strategies to have a major effect on the sample composition of the recruitment survey, but the differences between the strategies vanish once respondents are asked to proceed with the panel online.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"35 1","pages":"219 - 235"},"PeriodicalIF":1.7,"publicationDate":"2022-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49627250","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-18DOI: 10.1177/1525822x221134756
Nestor Hernandez, Kristen Olson, Jolene D Smyth
Questionnaire designers are encouraged to write questions as complete sentences. In self-administered surveys, incomplete question stems may reduce visual clutter but may also increase burden when respondents need to scan the response options to fully complete the question. We experimentally examine the effects of three categories of incomplete question stems (incomplete conversational, incomplete ordinal, and incomplete nominal questions) versus complete question stems on 53 items in a probability web-mail survey. We examine item nonresponse, response time, selection of the first and last response options, and response distributions. We find that incomplete question stems take slightly longer to answer and slightly reduce the selection of the last response option but have no effect on item nonresponse rates or selection of the first response option. We conclude that questionnaire designers should follow current best practices to write complete questions, but deviations from complete questions will likely have limited effects.
{"title":"“Are You …”: An Examination of Incomplete Question Stems in Self-administered Surveys","authors":"Nestor Hernandez, Kristen Olson, Jolene D Smyth","doi":"10.1177/1525822x221134756","DOIUrl":"https://doi.org/10.1177/1525822x221134756","url":null,"abstract":"Questionnaire designers are encouraged to write questions as complete sentences. In self-administered surveys, incomplete question stems may reduce visual clutter but may also increase burden when respondents need to scan the response options to fully complete the question. We experimentally examine the effects of three categories of incomplete question stems (incomplete conversational, incomplete ordinal, and incomplete nominal questions) versus complete question stems on 53 items in a probability web-mail survey. We examine item nonresponse, response time, selection of the first and last response options, and response distributions. We find that incomplete question stems take slightly longer to answer and slightly reduce the selection of the last response option but have no effect on item nonresponse rates or selection of the first response option. We conclude that questionnaire designers should follow current best practices to write complete questions, but deviations from complete questions will likely have limited effects.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":" ","pages":""},"PeriodicalIF":1.7,"publicationDate":"2022-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43988017","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-14DOI: 10.1177/1525822x221124469
Rachel Stenger, Kristen Olson, Jolene D Smyth
Questionnaire designers use readability measures to ensure that questions can be understood by the target population. The most common measure is the Flesch-Kincaid Grade level, but other formulas exist. This article compares six different readability measures across 150 questions in a self-administered questionnaire, finding notable variation in calculated readability across measures. Some question formats, including those that are part of a battery, require important decisions that have large effects on the estimated readability of survey items. Other question evaluation tools, such as the Question Understanding Aid (QUAID) and the Survey Quality Predictor (SQP), may identify similar problems in questions, making readability measures less useful. We find little overlap between QUAID, SQP, and the readability measures, and little differentiation in the tools’ prediction of item nonresponse rates. Questionnaire designers are encouraged to use multiple question evaluation tools and develop readability measures specifically for survey questions.
{"title":"Comparing Readability Measures and Computer‐assisted Question Evaluation Tools for Self‐administered Survey Questions","authors":"Rachel Stenger, Kristen Olson, Jolene D Smyth","doi":"10.1177/1525822x221124469","DOIUrl":"https://doi.org/10.1177/1525822x221124469","url":null,"abstract":"Questionnaire designers use readability measures to ensure that questions can be understood by the target population. The most common measure is the Flesch-Kincaid Grade level, but other formulas exist. This article compares six different readability measures across 150 questions in a self-administered questionnaire, finding notable variation in calculated readability across measures. Some question formats, including those that are part of a battery, require important decisions that have large effects on the estimated readability of survey items. Other question evaluation tools, such as the Question Understanding Aid (QUAID) and the Survey Quality Predictor (SQP), may identify similar problems in questions, making readability measures less useful. We find little overlap between QUAID, SQP, and the readability measures, and little differentiation in the tools’ prediction of item nonresponse rates. Questionnaire designers are encouraged to use multiple question evaluation tools and develop readability measures specifically for survey questions.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":" ","pages":""},"PeriodicalIF":1.7,"publicationDate":"2022-10-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42385232","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1177/1525822X221115844
Michael Daniel, Alexey Koshevoy, I. Schurov, N. Dobrushina
In this article, we address the issue of reliability of quantitative data on multilingualism of the past obtained as recall data. More specifically, we investigate whether the interviewees’ assessments of the language repertoires of their late relatives (indirect data) provide results that are quantitatively similar to those obtained from the people of the same age range themselves (direct data). The empirical data we use come from an ongoing field study of traditional multilingualism in Daghestan (Russia). We trained machine learning models to see whether they can detect differences in indirect and direct data. We conclude that our indirect quantitative data on L2 other than Russian are essentially similar to direct data, while there may be a small but systematic underestimation when reporting others’ knowledge of Russian.
{"title":"Can Recall Data Be Trusted? Evaluating Reliability of Interview Data on Traditional Multilingualism in Highland Daghestan","authors":"Michael Daniel, Alexey Koshevoy, I. Schurov, N. Dobrushina","doi":"10.1177/1525822X221115844","DOIUrl":"https://doi.org/10.1177/1525822X221115844","url":null,"abstract":"In this article, we address the issue of reliability of quantitative data on multilingualism of the past obtained as recall data. More specifically, we investigate whether the interviewees’ assessments of the language repertoires of their late relatives (indirect data) provide results that are quantitatively similar to those obtained from the people of the same age range themselves (direct data). The empirical data we use come from an ongoing field study of traditional multilingualism in Daghestan (Russia). We trained machine learning models to see whether they can detect differences in indirect and direct data. We conclude that our indirect quantitative data on L2 other than Russian are essentially similar to direct data, while there may be a small but systematic underestimation when reporting others’ knowledge of Russian.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"34 1","pages":"288 - 302"},"PeriodicalIF":1.7,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48799728","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-09-30DOI: 10.1177/1525822X221115830
Henning Silber, Joss Roßmann, Tobias Gummer
Attention checks detect inattentiveness by instructing respondents to perform a specific task. However, while respondents may correctly process the task, they may choose to not comply with the instructions. We investigated the issue of noncompliance in attention checks in two web surveys. In Study 1, we measured respondents’ attitudes toward attention checks and their self-reported compliance. In Study 2, we experimentally varied the reasons given to respondents for conducting the attention check. Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.
{"title":"The Issue of Noncompliance in Attention Check Questions: False Positives in Instructed Response Items","authors":"Henning Silber, Joss Roßmann, Tobias Gummer","doi":"10.1177/1525822X221115830","DOIUrl":"https://doi.org/10.1177/1525822X221115830","url":null,"abstract":"Attention checks detect inattentiveness by instructing respondents to perform a specific task. However, while respondents may correctly process the task, they may choose to not comply with the instructions. We investigated the issue of noncompliance in attention checks in two web surveys. In Study 1, we measured respondents’ attitudes toward attention checks and their self-reported compliance. In Study 2, we experimentally varied the reasons given to respondents for conducting the attention check. Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"34 1","pages":"346 - 360"},"PeriodicalIF":1.7,"publicationDate":"2022-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47430637","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-09-29DOI: 10.1177/1525822X221115831
Tanja Kunz, Katharina Meitinger
Although list-style open-ended questions generally help us gain deeper insights into respondents’ thoughts, opinions, and behaviors, the quality of responses is often compromised. We tested a dynamic and a follow-up design to motivate respondents to give higher quality responses than with a static design, but without overburdening them. Our results showed that a follow-up design achieved longer responses with more themes and theme areas than a static design. In contrast, the dynamic design produced the shortest answers with the fewest themes and theme areas. No differences in item nonresponse and only minor differences in additional response burden were found among the three list-style designs. Our study shows that design features and timing are crucial to clarify the desired response format and motivate respondents to give high-quality answers to list-style open-ended questions.
{"title":"A Comparison of Three Designs for List-style Open-ended Questions in Web Surveys","authors":"Tanja Kunz, Katharina Meitinger","doi":"10.1177/1525822X221115831","DOIUrl":"https://doi.org/10.1177/1525822X221115831","url":null,"abstract":"Although list-style open-ended questions generally help us gain deeper insights into respondents’ thoughts, opinions, and behaviors, the quality of responses is often compromised. We tested a dynamic and a follow-up design to motivate respondents to give higher quality responses than with a static design, but without overburdening them. Our results showed that a follow-up design achieved longer responses with more themes and theme areas than a static design. In contrast, the dynamic design produced the shortest answers with the fewest themes and theme areas. No differences in item nonresponse and only minor differences in additional response burden were found among the three list-style designs. Our study shows that design features and timing are crucial to clarify the desired response format and motivate respondents to give high-quality answers to list-style open-ended questions.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"34 1","pages":"303 - 317"},"PeriodicalIF":1.7,"publicationDate":"2022-09-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41497024","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-09-29DOI: 10.1177/1525822X221115829
Rainer Schnell, Sarah Redlich, A. Göritz
Frequency of behaviors or amounts of variables of interest are essential topics in many surveys. The use of heuristics might cause rounded answers, resulting in the increased occurrence of end-digits (called heaping or digit-preference). For web surveys (or CASI), we propose using a conditional prompt as input validation if digits indicating heaping are entered. We report an experiment, where respondents in an online access panel (n = 2,590) were randomly assigned to one of three groups: (1) no input validation; (2) conditional input validation if rounding was presumed; and (3) input validation every time a numerical value was entered. Conditional input validation reduces heaping for variables with high proportions of heaped values. Unconditional input validation seems to be less effective.
{"title":"Conditional Pop-up Reminders Reduce Incidence of Rounding in Web Surveys","authors":"Rainer Schnell, Sarah Redlich, A. Göritz","doi":"10.1177/1525822X221115829","DOIUrl":"https://doi.org/10.1177/1525822X221115829","url":null,"abstract":"Frequency of behaviors or amounts of variables of interest are essential topics in many surveys. The use of heuristics might cause rounded answers, resulting in the increased occurrence of end-digits (called heaping or digit-preference). For web surveys (or CASI), we propose using a conditional prompt as input validation if digits indicating heaping are entered. We report an experiment, where respondents in an online access panel (n = 2,590) were randomly assigned to one of three groups: (1) no input validation; (2) conditional input validation if rounding was presumed; and (3) input validation every time a numerical value was entered. Conditional input validation reduces heaping for variables with high proportions of heaped values. Unconditional input validation seems to be less effective.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"34 1","pages":"334 - 345"},"PeriodicalIF":1.7,"publicationDate":"2022-09-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48387016","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-09-27DOI: 10.1177/1525822X221114226
Yali Liu, L. Buckingham
To date, research on elite interviews has primarily focused on political or business settings in European and Anglo-American contexts. In this study, we examine the procedures involved in conducting elite interviews in academic settings, drawing on fieldwork with 53 senior scholars at 10 universities across five regions of northern China. We provide a detailed, critically reflective account of strategies to gain access, develop trust, and manage the power imbalance. Our account reveals the importance of the researcher’s professional identity in gaining participants’ trust and determining adequate forms of reciprocity.
{"title":"A Critical Approach to Interviewing Academic Elites: Access, Trust, and Power","authors":"Yali Liu, L. Buckingham","doi":"10.1177/1525822X221114226","DOIUrl":"https://doi.org/10.1177/1525822X221114226","url":null,"abstract":"To date, research on elite interviews has primarily focused on political or business settings in European and Anglo-American contexts. In this study, we examine the procedures involved in conducting elite interviews in academic settings, drawing on fieldwork with 53 senior scholars at 10 universities across five regions of northern China. We provide a detailed, critically reflective account of strategies to gain access, develop trust, and manage the power imbalance. Our account reveals the importance of the researcher’s professional identity in gaining participants’ trust and determining adequate forms of reciprocity.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"35 1","pages":"236 - 252"},"PeriodicalIF":1.7,"publicationDate":"2022-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43521189","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-09-16DOI: 10.1177/1525822X221115506
E. Aizpurua, Gianmaria Bottoni, R. Fitzgerald
Despite the widespread use of examples in survey questions, very few studies have examined their impact on survey responses, and the evidence is mainly based on data collected in the United States using questionnaires in English. This study builds on previous research by examining the effects of providing examples using data from a cross-national probability-based web panel implemented in Estonia (n = 730), Great Britain (n = 685), and Slovenia (n = 529) during Round 8 of the European Social Survey (2017/18). Respondents were randomly assigned a survey question measuring confidence in social media using Facebook and Twitter as examples, or another condition in which no examples were offered. The results show that confidence in social media was significantly lower in the example condition, although the effect size was small. Confidence in social media varied across countries, and the effect of providing examples was heterogeneous across countries and education levels. The implications of these findings are discussed.
{"title":"The Devil Is in the Details: A Randomized Experiment Assessing the Effect of Providing Examples in a Survey Question across Countries","authors":"E. Aizpurua, Gianmaria Bottoni, R. Fitzgerald","doi":"10.1177/1525822X221115506","DOIUrl":"https://doi.org/10.1177/1525822X221115506","url":null,"abstract":"Despite the widespread use of examples in survey questions, very few studies have examined their impact on survey responses, and the evidence is mainly based on data collected in the United States using questionnaires in English. This study builds on previous research by examining the effects of providing examples using data from a cross-national probability-based web panel implemented in Estonia (n = 730), Great Britain (n = 685), and Slovenia (n = 529) during Round 8 of the European Social Survey (2017/18). Respondents were randomly assigned a survey question measuring confidence in social media using Facebook and Twitter as examples, or another condition in which no examples were offered. The results show that confidence in social media was significantly lower in the example condition, although the effect size was small. Confidence in social media varied across countries, and the effect of providing examples was heterogeneous across countries and education levels. The implications of these findings are discussed.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"35 1","pages":"198 - 218"},"PeriodicalIF":1.7,"publicationDate":"2022-09-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46631737","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-09-15DOI: 10.1177/1525822X221113181
Matthew Jannetti, A. Carroll-Scott, Erikka Gilliam, Irene E. Headen, Maggie Beverly, F. Lê-Scherban
Place-based initiatives often use resident surveys to inform and evaluate interventions. Sampling based on well-defined sampling frames is important but challenging for initiatives that target subpopulations. Databases that enumerate total population counts can produce overinclusive sampling frames, resulting in costly outreach to ineligible participants. Quantifying eligibility before sampling using machine learning algorithms can improve efficiency and reduce costs. We developed a model to improve sampling for the West Philly Promise Neighborhood’s biennial population-representative survey of households with children within a geographic footprint. This study proposes a method to estimate probability of study eligibility by building a well-calibrated predictive model using existing administrative data sources. Six machine-learning models were evaluated; logistic regression provided the best balance of accuracy and understandable probabilities. This approach can be a blueprint for other population-based studies whose sampling frames cannot be well defined using traditional sources.
{"title":"Improving Sampling Probability Definitions with Predictive Algorithms","authors":"Matthew Jannetti, A. Carroll-Scott, Erikka Gilliam, Irene E. Headen, Maggie Beverly, F. Lê-Scherban","doi":"10.1177/1525822X221113181","DOIUrl":"https://doi.org/10.1177/1525822X221113181","url":null,"abstract":"Place-based initiatives often use resident surveys to inform and evaluate interventions. Sampling based on well-defined sampling frames is important but challenging for initiatives that target subpopulations. Databases that enumerate total population counts can produce overinclusive sampling frames, resulting in costly outreach to ineligible participants. Quantifying eligibility before sampling using machine learning algorithms can improve efficiency and reduce costs. We developed a model to improve sampling for the West Philly Promise Neighborhood’s biennial population-representative survey of households with children within a geographic footprint. This study proposes a method to estimate probability of study eligibility by building a well-calibrated predictive model using existing administrative data sources. Six machine-learning models were evaluated; logistic regression provided the best balance of accuracy and understandable probabilities. This approach can be a blueprint for other population-based studies whose sampling frames cannot be well defined using traditional sources.","PeriodicalId":48060,"journal":{"name":"Field Methods","volume":"35 1","pages":"137 - 152"},"PeriodicalIF":1.7,"publicationDate":"2022-09-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47100891","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：