Unit 5: Evaluation and Measuring Change

Because we’re generating evidence for others, communicating the result of pilots is incredibly important. We need to be clear and direct, avoid jargon, and follow the reverse pyramid-style used in journalism: with the most important conclusion at the top and the evidence coming after. This shouldn’t be a 20 page paper with citations, but rather a summary of the results of the pilot, then how we ran the pilot, then the pressure it was based on and the evidence we have for that pressure.

By using a consistent pilot outcome statement, it makes the outcomes between different pilots easy to  compare and decisions easier to make.

• Confidence – how confident are you that the intervention will change behavior? Not confident, somewhat, or very confident. This is just a plain language version of p value.
• Intervention – what did you do? Without too much detail, be clear about the proposed intervention to be scaled.
• Direction –this is straight from your behavioral statement; are you trying to get people to do something more or less?
• Behavior – you already know this, just copy/paste from your behavioral statement.
• Data – and again!
• Effort – the effort it takes to scale the intervention. This could be money ($5 per user), time (three months), or however the organization thinks about cost (3 months to setup, then $5 per user after that).
• Impact – the result of the behavior change. This could be money ($5 per user), time (three months), or however the organization thinks about benefit ($5 per user in savings and three months less in conversion time).