Linking from the IDinsight blog.
By Siobhan McDonough, heather lanthorn, Kondwani-yobe Mumba, and Kevin Kelsey
This post explores how IDinsight addressed the tendency of program staff to want their most successful participants surveyed.
Introduction
It can be a huge challenge identifying survey respondents in sparsely populated regions with limited information about where people live. This is especially the case when participants are recruited outside of the home: for example at a clinic, school, or as we’ll cover in this post, an agricultural extension training. For a process evaluation in Zambia, we needed to find hundreds of randomly sampled farmers — but all we had was a list of people who attended a training. Many of them were impossible to track down without someone from the program we were evaluating supporting the effort to find where they lived. But once we started working with program staff, they were interested in taking us to their strongest participants. This post shares how we overcame the tendency of program staff to only want their most successful trainers or farmers to be surveyed — or to otherwise positively influence the findings of the survey.
Our two takeaways:
- Sometimes it is better to be slow when sampling if it means avoiding bias.
- It is always important to account for implementers’ conflicting incentives, especially when it may be in their best interest for an evaluation to show their program’s success.