With the development of policies specifically relating to rural health and health care provision, it seems apposite to consider evaluating their outcomes. Although little has apparently been done that specifically studies policy or processes, much of rural health research implicitly 'measures' policy effects; for example, study of the effects of rural medical education. Given what is known about the policy-making process, rural health researchers should beware of thinking that policy outcome evaluation might be straightforward or that evidence produced from evaluation will seamlessly influence future or evolving policy. Nonetheless, as rural health research and policy mature, it is worth adopting some of the complex approaches to health policy outcome measurement and applying them to understand our field--to find out the extent that policy, and indeed our role and research, have effects on rural health and care provision. In this paper, we identify some of the quirks of policy and policy evaluation and provide examples.