2  RP Surveys+: Ex. projects/q’s/data

In this section we aim to outline a set of projects and methodologically-relevant questions that typify the work of Rethink Priority’s Survey and Movement Building Team1, as well as the work of adjacent projects like the EA Market Testing Team.

We will also extract, reference, and outline some example data, to use in the chapters below.

9 Dec 2022: So far, the chapters below draw from the EA Survey data. However, as that data itself cannot be in the public domain, this is less useful for people who don’t have access to our Github repo, and less than ideal in general.

Some sources of data and examples could include:

  1. Data from the EA Market Testing Team’s public work, see the data treatment here; simplifying this data might be helpful for our examples

  2. Other Reinstein projects, including data from the ICRC chapter of the ‘impact of impact information’ project, or JustGiving data e.g., see codebook here

  3. Simulated data like (a simplified version of) the EA Survey data

  4. Any RP or adjacent survey data that has been put in the public domain2

  5. Simulated data resembling that of other confidential EAMT and RP projects

2.1 Some common/typical cases and problems

Community Survey omnibus

Typical example: EA Survey3

Research questions:

  • Current community composition (e.g., ‘what share are non-male’)
  • Actions taken and level of involvement
  • Retrospective ‘how did you get involved?’
  • Relative preferences and attitudes (e.g., longtermist or non-longtermist)
  • Changes in all of this over time
  • Actions vs intentions

Issue-oriented survey of the population

Typical examples:

  • EA Pulse,
  • surveys of awareness of EA;
  • attitudes towards long-termism;
  • polling on animal welfare legislation, etc.

Research questions:

  • Awareness and knowledge of a subject
  • Attitude to an issue
  • Potential voting choices and other behavior
  • Differences across personality, demographic traits, etc.
  • Clustering, dimension reduction, factor analysis: How do responses ‘cohere’
  • Differences over time, particularly in response to campaigns (before-after)

Data may include:

  • Ordered Likert responses and indices
  • Open-response
  • Hypothetical choices among a set of optinos
  • Binary, continuous, other

Naturally occuring behavioral data (signups, donations, etc)

Typical examples:

  1. All fundraisers for effective charities started on JustGiving during a certain period, with donation amounts

This data is shared in the current github repo, and read in with code below 4

read in Just Giving data
library(here)
library(readr)

jg_eff_sample <- readRDS(here::here("sample_data", "fdd_fd_sample_eff.rds"))

  1. Meat and animal product consumption data5

  2. A time series6 of signups for 80,000 hours newsletter, particularly considering the difference before and after a report in the Guardian newspaper

We share

Marketing trials/experiments and surveys

This is distinct from the ‘naturally occuring’ data just mentioned, because we are intervening to ‘do or test something’ and measure the response.

Typical examples:

  • EA Market Testing comparison of GWWC Facebook ads to encourage email signup
  • A Prolific survey to helping an EA filmmaker choose a movie title and blurb7
  • An effective charity wants to know how to present their impact information in mail appeals, if at all. They may test this with an actual set of appeals (field experiment, aka ‘split test’).

Research questions and goals:

  • Baseline response rates and cost per outcome
  • Find ‘best approach within a large space’ (reinforcement learning)
  • Test different categories of approaches in a limited set (‘which works better’)
  • Find most promising audiences (profiling)
  • Interaction (tailored messaging, etc.)

Data could include

  • Attitudinal (Likert) response data
  • Suggestive outcome data (e.g., ‘clicks on the web site’)
  • Actual ‘A/B testing’ data for outcomes and responses of interest

  1. DR: I have suggested we should rebrand as something involving ‘quantitative social science’↩︎

  2. Are there any?↩︎

  3. flesh this out↩︎

  4. Not evaluated here atm↩︎

  5. This may be more relevant for the Farmed Animal Welfare team↩︎

  6. time series analysis will tend to use distinct approaches↩︎

  7. A ‘survey experiment’ might present a different title to each participant (‘between-subject variation’). An alternative approach might present a range of titles to each participant and compare ‘within subject’.↩︎