How Do Different Groups Judge the Quality of Research Questions Which Inform Evidence-Based Policymaking?

Abstract

Magda Osman and Nick Cosstick

Science-policy co-production depends on successfully coordinating exchanges between different researchers and policymakers— acknowledging that they may vary in their interpretation of the problem and the questions that need addressing. In the UK, ‘Areas of Research Interest’ (ARI) are questions generated by government departments, agencies, and public bodies to invite responses from external experts, such as researchers. There are two broad aims, to communicate the information needs of government departments and to initiate a co-productive process. But are such questions assessed in the same way by policymakers and researchers? The present study examines the properties of questions to understand whether there is agreement across different groups (i.e. public N = 383, academia N = 182, public administration N = 211) regarding the types of questions which are judged to be better than others. The study presented participants with seven types of questions (Instrumental/Procedural, Causal Analytic, Verification/Qualification, Explanation/Example, Explaining/Asserting Value Judgments, Comparisons, and Forecasting) on the same topic (i.e. climate change) that varied in length (i.e. long vs. short), and that presented as either posed by policy professionals or researchers. Participants were required to assess questions based on quality of communication, neutrality, and overall goodness. The findings show that assessments were unaffected by proposer, sample, and demographics (e.g. age, gender, level of education). Of the seven types of questions investigated, Instrumental/Procedural type questions were rated the best. The implications of these findings are considered with respect to co-production between academia and policy.

PDF