5 Survey design, item dev.

This section will cover specific ‘survey’ related content, as well as some content that overlaps with experiments and trials.^M[ore technical and involved discussion of overlapping content will often be deferred to the ‘experiments’ related sections, with placeholders and links.]

5.1 Some key resources and things to incorporte

Katja Grace, ‘things I believe after making some surveys’
Link or incorporate William Elsey’s work HERE,
and Reinstein’s sloppy notes here focusing on probability sampling and sampling rare populations; see embeds below

:::

5.2 (Willem Sleegers, to incorporate)

item development

wayback link

Code

knitr::include_url("https://willemsleegers.github.io/how-to-science/content/methodology/survey-design/item-development.html")

Reinstein misc notes to incorporate

Code

knitr::include_url("https://daaronr.github.io/metrics_discussion/surveys.html")

5.3 Suggested sections…

‘Qualitative’ issues (hopefully backed by evidence)

How to ask good survey questions
Avoiding pitfalls (framing, agreeability bias, comprehension, attention, etc.)
Validating the measure is capturing ‘the thing you intend’

‘Quantitative issues’

Constructing reliable indices and scales
Sampling issues and representativeness
- Elsey on “Mr P”
- Reinstein on ‘sampling’ and ‘measuring a rare population’ with reference to the EA survey

5.4 Discussions to integrate

Google discussion doc

Framing effects Disputing claim that “One doesn’t need to worry about framing effects” because of Morewedge et al, 2015; Khan et al, 2006; Baumer et al, 2015 private slack conversation link

Ordering attitude measures In asking a series of attitude measures that have a natural order, present them in that order or randomize (e.g., here https://docs.google.com/document/d/16lmVZu9sjoHLceh8Zdgs8A_n4sxmZudahn7825ctRW4/edit#heading=h.gwnnifgo8qd) … I found it interesting, I am enthusiastic, I would definitely sign up, etc.)

Discussion of ordering attitude measures

DR: I see the case for increasing order is ‘it makes it more clear for them to follow/more coherent to respond. On the other hand, it could lead to less careful consideration of each response and a mechanical ’I better increase it’

DM: If we want to test how many people will say they would be likely to do 1/2/3/4/5/6 I think we want to minimise the influence of order (or anything extraneous) on their likelihood of selecting 1/2/3/4/5/6. Ordering them by perceived commitment seems like it can only serve to nudge them in untoward ways to be more or less likely to select some.

Also I don’t think we have good grounds to predict what the order of increasing commitment is or to assume this is invariant across respondents.

DR: OK, I see your point, if we actually want to know, and see these as measures of how many people will do/think each of these things. As a ‘measure of commitment and affinity’ the ordering makes some sense to me.

Quiz questions, ‘information search’ (googling) and honesty

We often want to measure knowledge or recall but in online surveys people can look things up. Will they do so, and how does it depend on the incentive? Can this be deterred?

Discussion of honesty and Hugh-Jones work

Re whether people cheat on unincentivized survey responses to quiz questions online when googling is possible, I consulted my ‘experiment gang’. David Hugh-Jones had a paper published in 2016 in which (he says) most US and UK people were honest and didn’t look things up[ even when there was an incentive. I’m trying to dig out those precise numbers but to me…

This is possibly a lower bound on the level of honesty without compensation if we asked, say quiz questions that indicate familiarity with 80,000 hours. (of course it’s not airtight, you could make a case that people might be more honest in a case where there is compensation because they feel like it’s more of a moral issue or that there will be more curiosity in this case, or that lookup was more work in his case… but still)

Detecting and Deterring Information Search in Online Surveys Matthew H. Graham*

From Twitter:

Looking up the answers to knowledge questions in online surveys is far too prominent.

A simple promise not to cheat can cut down a lot of it.

So can implementing a supplied script to detect when the participant leaves the screen.

# Survey design, item dev. {#surveydesign} This section will cover specific 'survey' related content, as well as some content that overlaps with experiments and trials.^M[ore technical and involved discussion of overlapping content will often be deferred to the 'experiments' related sections, with placeholders and links.] ## Some key resources and things to incorporte - [Katja Grace, 'things I believe after making some surveys'](https://www.lesswrong.com/posts/oyKzz7bvcZMEPaDs6/survey-advice) - Link or incorporate William Elsey's work [HERE](https://willemsleegers.github.io/how-to-science/content/methodology/survey-design/), - and [Reinstein's sloppy notes here](https://daaronr.github.io/metrics_discussion/surveys.html) focusing on probability sampling and sampling rare populations; see embeds below ::: ## (Willem Sleegers, to incorporate) [item development](https://willemsleegers.github.io/how-to-science/content/methodology/survey-design/item-development.html) [wayback link](https://web.archive.org/web/20220630084700/https://willemsleegers.github.io/how-to-science/content/methodology/survey-design/item-development.html) ```{r } knitr::include_url("https://willemsleegers.github.io/how-to-science/content/methodology/survey-design/item-development.html") ``` ## Reinstein misc notes to incorporate {-} ```{r } knitr::include_url("https://daaronr.github.io/metrics_discussion/surveys.html") ``` ## Suggested sections... **'Qualitative' issues** (hopefully backed by evidence) - How to ask good survey questions - Avoiding pitfalls (framing, agreeability bias, comprehension, attention, etc.) - Validating the measure is capturing 'the thing you intend' \ **'Quantitative issues'** - Constructing reliable indices and scales - Sampling issues and representativeness - Elsey on "Mr P" - Reinstein on ['sampling'](https://daaronr.github.io/metrics_discussion/surveys.html#survey-samplingintake) and ['measuring a rare population'](https://daaronr.github.io/metrics_discussion/surveys.html#jazz-case) with reference to the EA survey ## Discussions to integrate [Google discussion doc](https://docs.google.com/document/d/1FpsUM0N100AesImLIP80X0-q-R4lf7c4j8k9R-e9coQ/edit) **Framing effects** Disputing claim that "One doesn't need to worry about framing effects" because of Morewedge et al, 2015; Khan et al, 2006; Baumer et al, 2015 [private slack conversation link](https://rethinkpriorities.slack.com/archives/G01962YABHB/p1655407493604669) **Ordering attitude measures** In asking a series of attitude measures that have a natural order, present them in that order or randomize (e.g., [here](https://docs.google.com/document/d/16lmVZu9sjoHLceh8Zdgs8A_n4sxmZudahn7825ctRW4/edit#heading=h.gwnnifgo8qd) https://docs.google.com/document/d/16lmVZu9sjoHLceh8Zdgs8A_n4sxmZudahn7825ctRW4/edit#heading=h.gwnnifgo8qd) ... I found it interesting, I am enthusiastic, I would definitely sign up, etc.) ::: {.callout-note collapse="true"} ## Discussion of ordering attitude measures > DR: I see the case for increasing order is 'it makes it more clear for them to follow/more coherent to respond. On the other hand, it could lead to less careful consideration of each response and a mechanical 'I better increase it' > DM: If we want to test how many people will say they would be likely to do 1/2/3/4/5/6 I think we want to minimise the influence of order (or anything extraneous) on their likelihood of selecting 1/2/3/4/5/6. Ordering them by perceived commitment seems like it can only serve to nudge them in untoward ways to be more or less likely to select some. > Also I don't think we have good grounds to predict what the order of increasing commitment is or to assume this is invariant across respondents. > DR: OK, I see your point, if we actually want to know, and see these as measures of how many people will do/think each of these things. As a 'measure of commitment and affinity' the ordering makes some sense to me. ::: **Quiz questions, 'information search' (googling) and honesty** We often want to measure knowledge or recall but in online surveys people can look things up. Will they do so, and how does it depend on the incentive? Can this be deterred? ::: {.callout-note collapse="true"} ## Discussion of honesty and Hugh-Jones work Re whether people cheat on unincentivized survey responses to quiz questions online when googling is possible, I consulted my ‘experiment gang’. David Hugh-Jones had a paper published in 2016 in which (he says) most US and UK people were honest and didn’t look things up[ even when there was an incentive. I’m trying to dig out those precise numbers but to me... This is possibly a lower bound on the level of honesty without compensation if we asked, say quiz questions that indicate familiarity with 80,000 hours. (of course it’s not airtight, you could make a case that people might be more honest in a case where there is compensation because they feel like it’s more of a moral issue or that there will be more curiosity in this case, or that lookup was more work in his case… but still) ::: ::: {.callout-note collapse="true"} ## Detecting and Deterring Information Search in Online Surveys Matthew H. Graham* From Twitter: > Looking up the answers to knowledge questions in online surveys is far too prominent. > A simple promise not to cheat can cut down a lot of it. > So can implementing a supplied script to detect when the participant leaves the screen. :::