Principal investigator: Viviane Pires Ribeiro
Paper Title: When evidence is not taken for granted: the use and perception of “evidence” in the Czech Republic ministries
Authors: Arnošt Veselý, František Ochrana, Martin Nekola
Location of the Intervention: República REPUBLIC
Sample Size: 1351 questionnaires
Main theme: Economic Policy and Governance
Main Variable of Interest: Evidence
Type of InterventionAnalysis of the role of evidence in policymaking.
Methodology: Research with large N and detailed interviews.
The role of evidence in policymaking is one of the most researched topics in public policy and public administration. However, surprisingly, little research has been done on how public officials actually use evidence in everyday practice. Veselý, Ochrana, and Nekola (2018) address this gap. Using a large-N survey of Czech ministerial officials and in-depth interviews with them, the authors explore what is understood by the term "evidence," what type of evidence is used and preferred by public officials, and why.
Evaluation Context
The rise of the discourse of “evidence-based politics” (EBP) is one of the most visible trends in public administration and public policy in the last decade. Although EBP is a well-recognized movement, its definitions vary from a rather narrow view (a particular methodology for producing a specific form of evidence) to a broader and more comprehensive view of what it represents. While many governments and practitioners have been very enthusiastic, EBP has also been criticized and challenged for various reasons, for example, for assuming that research evidence can provide objective answers – and ultimately, resolve inherently political issues.
Despite all these criticisms, many initiatives and projects on "bridging the gap between policy and research" have been carried out, especially in Anglo-Saxon countries. In these countries, the discourse of evidence-based practice (EBP) is deeply rooted in public administration practices. Consequently, in countries that have been strongly influenced by this discourse, it seems difficult to challenge the basic assertion that evidence should be used in policymaking. The use of evidence – whatever the meaning of this term – is taken for granted.
The Czech language has the word "evidence," but with a different meaning than in English. In Czech, evidence is derived from the verb "evidovat," which means "to record." Thus, the Czech word "evidence" is generally associated with other types of activities, such as Evidence obyvatel (population register) or elektronická Evidence tržeb (electronic sales register). Therefore, the word evidence is generally understood as a "record," that is, centrally collected and stored information.
Few people in public administration and even in academia are familiar with the concept of EBP. Occasionally, however, the concept is mentioned. Given that evidence in the Czech language is associated with “recording” (which is quite different from its original meaning in English), EBP has often been translated as politika založená na důkazech. This literally means “evidence-based policy.” This is, of course, a rather restrictive interpretation of evidence, as “proof” includes only unquestionable evidence that gives definitive answers. To avoid this terminological confusion, some authors use the term “poznatky” instead of “evidence,” as is the case with Veselý, Ochrana, Nekola (2018). This concept is broader and the most neutral of all related concepts. It can be translated as “knowledge,” or more precisely as knowledge created through the process of cognition. The concept of poznatsky has a slight connotation with research (research knowledge), but it is not necessarily associated with research.
Intervention Details
The study conducted by Veselý, Ochrana, and Nekola (2018) explores the understanding and use of evidence in Czech ministries as central institutions of public administration. This country represents an interesting case study in the use of evidence in policymaking. The production and use of policy-relevant information has a long tradition in the Czech Republic, and Czech social science has always had a strong practical orientation. Furthermore, the demand for scientifically based policy knowledge led to the establishment of different types of research institutions that directly serve ministries. Although both their number and impact have decreased since 1989, the emphasis on relevant and practical social science has prevailed. At the same time, however, the idea and discourse of EBP (Evidence-Based Practice) have never seriously entered the debate in public administration and policymaking. Few people in public administration are familiar with the concept of EBP. There have been no initiatives or projects to incorporate more evidence into policymaking and decision-making in public administration, and the use of research evidence in policymaking appears to be quite limited. Furthermore, the basic concept of "proof" is hardly translatable into the Czech language.
Thus, Veselý, Ochrana, and Nekola (2018) raise some traditional questions about the use of evidence in policy in the context of Czech Republic ministries. Specifically, they asked the following questions: What is considered evidence by public officials? What type of evidence is used? What type of evidence is considered important? When and how is evidence used in the policymaking process? In doing so, the authors draw on four currents of theories that focus on different aspects of evidence use. They discuss the extent to which these theories, mostly developed in a context with a strong discourse on evidence-based practice (EBP), can be applied in a context where EBP is almost unknown.
Methodology Details
To answer the research questions, Veselý, Ochrana, and Nekola (2018) used both quantitative and qualitative data. Regarding the quantitative data, the team conducted a large-N survey of policy bureaucrats in Czech ministries between April and July 2013. Step by step, 11 ministries agreed to participate in the survey. In seven ministries, data was collected through face-to-face interviews: interviewers met with respondents, asked a series of pre-defined standard questions, and recorded their responses on a paper form (CAPI) or a computer application (PAPI). For two ministries that preferred to participate without interviewer involvement, data was collected by the administration in the form of online questionnaires (CAWI). In one ministry, a combination of CAPI and CAWI was implemented. Respondents were randomly selected from each ministerial sample frame. After several waves of random sampling, all individuals from each sample frame were invited to participate. Thus, what was intended to be a random sampling ended up being a census. A total of 1351 completed questionnaires were obtained, and the response rate was 29,4%.
The qualitative data comes from detailed interviews conducted by the authors that focused specifically on the use of evidence. These interviews were conducted from April 2016 to May 2017. In total, 23 interviewees from different Czech ministries participated. Four researchers, members of the research team, conducted the interviews following a standard interview guide. The interview topics concerned the interviewee's description of their strategic work, their use of scientific or other knowledge, their experience with the use and status of evidence in the broader context of their ministry and state administration. The semi-structured interviews with experts lasted approximately one and a half to two hours, were recorded, and field notes were compiled. The recordings were transcribed in full, and the interviews were coded and analyzed using thematic analysis focusing on identified patterns in the use of knowledge across various contexts and phases of the policy process.
Results
The use of evidence in the Czech Republic differs in many respects from what is reported in countries with a strong discourse on evidence-based practice (EBP). Public officials do not report a commitment to using evidence in the sense of "what works" (although they do not oppose it). More precisely: they do not demonstrate a commitment to scientific evidence, but to evidence from "administrative data." In general, they are not interested in "proof," but in "evidence" that can be used to legitimize political objectives and support negotiated consent on policy.
Veselý, Ochrana, and Nekola (2018) argue that it is impossible to determine with precision to what extent these differences are caused by different governance structures in the Czech Republic and to what extent they are caused by the low impact of the EBP movement in the country. It is likely that these two are linked. All interviewees refer to the relative political instability and frequent changes in the overall public policy landscape, which has two consequences for their work. First, they perceive time pressure, i.e., it is necessary to act very quickly. Second, they report the (repeated) experience of policies and strategies not being implemented because of personal changes in ministry leadership. Time pressure leads civil servants to find and prepare easily understandable documents. Interviewees who worked on a policy document that was left unfinished, abandoned, or even withdrawn from the government's decision-making process because of a change in the minister's or deputy minister's position report less motivation for any complex work and data analysis. This is then combined with low internal pressure (within a given ministry) to use evidence in the policy-making process, only slowly changing the overall culture of the ministries.
Many commonalities were found with the results from other countries. Most notably, public officials use different types of evidence and apply their own criteria to what counts as evidence and what is “useful evidence.” That is, public officials use very diverse sources of evidence, but evidence produced by other officials (or for officials) is used more intensively. In general, all four theoretical streams appear to be useful in generating hypotheses about the use of evidence also in the context without EBP (Evidence-Based Practice).
Lessons in Public Policy
Although civil servants and academics do not differ so profoundly in their training, they do differ in their discourse and vocabulary, and also in terms of the standards by which they judge the merit of evidence. Thus, the study conducted by Veselý, Ochrana, and Nekola (2018) suggests an important role for experts or “knowledge intermediaries” who translate the world of science into the world of practice. However, more detailed research is needed to explore exactly who these knowledge intermediaries are and how they function.
Furthermore, the analysis also suggests that the recent movement toward adding context to the analysis of evidence use is quite promising. The use and understanding of evidence are strongly influenced by the everyday practices of public officials, as well as their immediate social environment. The use of evidence cannot be separated from the entire political process in which it is embedded. Public officials want research that “helps them in their work.” And what should help them is influenced by the stage of the policy and their expectations. Combined with the recent emphasis on the type of knowledge and evidence, this could be a very promising avenue for future research.
References
VESELÝ, Arnošt; OCHRANA, František; NEKOLA, Martin. When evidence is not taken for granted: the use and perception of “evidence” in the Czech Republic ministries. Network of Institutes and Schools of Public Administration in Central and Eastern Europe. The NISPAcee Journal of Public Administration and Policy, v. 11, no. 2, p. 219-234, 2018.
