6  Experiments: Qualitative design, implementation

• See Reinstein notes on ‘why run an experiment’ (somewhat more relevant to ‘lab experiments’)

• See informal ‘practitioners’ discussion below, from ‘EA market testing’

6.1 Other relevant resources

John List ‘Some Tips for “Start-Ups” to do Better Field Experiments and Getting Your Work Published’

6.2 Proposed structure of section

• Basic design choices and terminology

• Types of experiments: ‘lab and field’

• Formulating hypotheses and ‘learning and adaptation goals’

• Hypothesis testing versus ‘reinforcement learning’ goals

• Design concerns and pitfalls

  • Confounded designs

  • Attrition and failed randomization

  • ‘Demand effects’

  • Naturalness versus cleanliness

• Practical and implementation issues

  • Survey and experiment platforms (see ‘surveys’) chapter

  • Field experiments, A/B and lift tests, and marketing trials

  • Designing, coding, and implementing experiments: IT issues

  • Failure and success modes

  • Capturing outcome data

  • Pre-registration and pre-analysis plans

  • … See Reinstein discussion notes on the benefits and costs here