Introduction

The development of biomedical research in the last 50 years has broadened the professional horizons of a large number of researchers and constitutes an economic activity that generates substantial sums of money: as an example the macroeconomic impact of biotechnology in Spain, measured directly and indirectly, included a figure of 44,333 jobs and a turnover amounting to 0.6% of the state GDP in 2005 [1]. In Spain in 2008, 1.2% of GDP was dedicated to Scientific Research and Development (R+D), while R+D in health was little more than 0.2% of GDP [2]. In the United Kingdom the annual return rate of biomedical health research varies between 24% [3], and 28% [4].

In Spain during 2017 were 133.213.188 jobs on research and the state GDP expenditure on R+D was 14.063,444 millions of euros, of which 5.471,159 millions was the Spanish Government budget. Only the 779.302 millions (5,54%) were destined to health research [5].

In the United States, according to the data of the report published in May 2016 (United for Medical Research Report), the National Institute of Health of the United States of America (NIH) with its research funds generated more than 350,000 jobs and an economic activity of 60,171 billion dollars in 2015. The People’s Republic of China tripled its investment in biomedical research from 2.6 billion dollars in 2004 to 9.7 billion in 2012 [6]. Worldwide it is estimated that investment in this area was a quarter of a trillion dollars in 2010 [7].

A course under the auspices of Instituto Carlos III was held in July 2015 in the Menéndez Pelayo University, on the socioeconomic impact of biomedical research, where the importance of this sector in Spain was apparent, as was the need for an ongoing evaluation of its scientific and social impact. Many of the analyses and proposals generated are still waiting for dissemination and implementation.

An analysis of publications on ethical issues in biomedical research reveals how a vast percentage of scientific production revolves around the issues of consent and conditions of experiment subjects [8-11]. The problem of funding biomedical research and its ethical implications is reduced to a relatively low number of publications. On the other hand, it is logical that when analysing a particular project, the methodological and design aspects of the protocol require analysis from an ethical- legal perspective of enforceable standards; however, in the establishment of a priority research line, ethical analysis must focus primarily on the objectives and expected social impact.

Private entities, i.e. the pharmaceutical industry, can invest with economic efficiency criteria and business performance in mind, as is their right as long as it is within the limits of our legal system. A very interesting topic would be an ethical analysis of objectives and results of projects financed through pharmacy-industry funds [12], (but, as Rudyard Kipling would say, that’s another story).

However, public funds must, at least in theory, be used with criteria whereby the common good and public interest prevail over economic interest. Quoting Artells Herrero (2000) [13]: “Research financed and carried out with public funds participates in the public good, where the Public Sector, rather than the market, intervenes as an allocation mechanism to produce a fundamental service of non-exclusive social use and unrivalled collective consumption.”

Public institutions should ensure a proper use of their funds for greater social performance, but terms such as general interest or social performance are not ideologically neutral, and in their formulation a series of values intervene, which, according to prioritisation criteria, may modify development criteria. It is important to remember that the funds obtained from public calls for proposals are considered as a public good, and therefore their management and the establishment of their final destination require the application of criteria additional to those of exclusive economic performance [14].

There is no doubt that an honest and rational application of prioritisation criteria in the allocation of resources has many advantages and has led to important progress in the design of research policy. The problems arise from the practical difficulty of carrying out a critical analysis of the results and criteria of the priority lines and evaluating the efficiency and effectiveness of the research [15-18]. And perhaps the most controversial and difficult part is the selection of experts and their relationship with political powers, since they are ultimately responsible for defining the priority lines [19-21].

The evaluation and prioritisation of research activity is essential for the development of any science and technology system, and is a very effective tool for optimising the development of research and the establishment of evidence-based health policies [22,23].

In an ideal world (obviously not ours), dialogue between experts and politicians demands that each one fulfils their role, without trying to the other’s function [24]. The ethical responsibility of both sides depends not only on the criteria used, but also on scientific consistency and clarity in the relation of reasons for a specific decision.

So-called consensus methods, such as the nominal group or the Delphi method, intend to define agreement levels amongst experts on controversial issues [21]. They have been used in recent years to identify and prioritise lines of research in biomedicine [25-27]. However, they present certain limitations, such as those derived from the selection of experts who contribute subjective opinions according to their greater or lesser knowledge on the subject and their ideological positions, as well as the possible existence of conflicts of interest [27]. Valera (1991) [28] says: Since these are group dynamic techniques, based on subjective opinions, with intermediate imperfect processes of analysis and synthesis, with manipulations aimed at fomenting convergences, their validity and reliability are not well established.

We know that it is not an easy task to establish universal values that are accepted by all, and to avoid falling into the temptation of relativism we must at least try to reach criteria based on consensual arguments that, although not universal, can freely be accepted by the greatest number of people, trying to minimize any ideological and/or religious bias that could generate division or controversy [21,29].

There are ideological and differences in belief that are logical and legitimate, as long as they are expressed in the discussion and no attempt is made to mask a belief with a layer of science to disguise it; and the same applies to the interests of the different actors involved in the decision taking. Whatever the case, post-evaluation of the results will allow us to refine the process and objectify the errors committed in the decision-making system.

A critical analysis was carried out from the Spanish national reality, which does not differ too much from the rest of the countries of the European Union, although there may be different nuances with other national contexts with a greater weight of private patronage, (USA, United Kingdom), ethical problems in the selection of priority lines financed through public funds maintain a relative similarity, which allows to introduce elements in the discussion of this generalizable problem to most developed countries.

Objective

The aim of this paper is to discuss the different criteria of evaluating the priority of lines of research and how political decisions affect in the development of scientific researches, emphasizing the ethical implications that they entail.

How to Evaluate Priority Lines of Research?

It is not only necessary to assess quality and the scientific-technical relevance of the research (which as we shall see, is not difficult), but also the impact on society and possibility of improving on the prior situation of the problem meant to be solved [30,31]. Social impact has been defined by the United Kingdom MRC (Medical Research Council) as “Increasing the effectiveness of services and public policies. Improving quality of life, health and creative performance” [32].

There is a growing interest in extending the objectives of evaluating the impact of research to aspects other than the strictly academic to assist in the design of appropriate policies [33-35].

Raftery, et al. (2016) [36] published an excellent monograph on different existing models to apply them routinely in order to guarantee and help establish cost/benefit balances. But an important issue that we shall visit later is the need to publish the results so that they are not restricted to the scientific community or experts, so that they can be debated, criticised and questioned by the population, in order to establish a system of continuous feedback that helps us think of not just the technical-scientific dimension, but also a wider ethical-social dimension [21,37,38].In the United Kingdom, evaluation of research impact was institutionalised in 2014 through the REF (Research Excellence Framework) [39], the results of which are used to distribute around £1.6 billion annually among university research centres [40]. Models exist for measuring the indirect social impact of biomedical research, such as that proposed by Jones and Hanney (2016) [41].

Saarni, et al. (2008) [42], established a series of domains that must be evaluated in the implementation of new health technologies: Health problem in question and current use of technologies

a) Description and technical characteristics of the technology to be applied

b) Their safety

c) Clinical efficacy

d) Cost and economic evaluation

e) Ethical analysis

f) Organisational aspects

g) Social implications

h) Legal aspects

Including all or most of the factors in any evaluation programme will offer a fairly complete and complementary analysis of the different elements that must converge in an analysis.

There is a research program, ASPIRE (Action to Support Practices Implement Research Evidence) directed at the development of intervention models for the assessment of clinical research, which is structured into five successive different stages [43]:

1. The selection of NICE guidelines and associated quality standards to obtain a set of “High Impact” indicators based on the burden of disease, the significant potential benefit for the patient derived from the improvement of clinical practice, the probability of saving costs without damage to the patient and the feasibility of measuring change using routine data collections.

2. Transversal analysis of patient data to identify high impact recommendations with a wider margin for improvement (low adherence) and explore variations in adherence.

3. Interviews with primary care professionals to explore the barriers related with the selected high impact recommendations, and to make changed behavioural techniques coincide with identified barriers and capacitors to develop an adaptable intervention package (based on audit and the indications obtained after feedback).

4. Evaluation of the effectiveness and profitability of the intervention package adapted to the implementation of high impact recommendations.

5. Carrying out an evaluation of the parallel process to examine the intervention process, mechanisms of action and unwanted consequences.

Until now we have been moving in the domain of effectiveness indicators and research impact; however, sophisticated approaches to intervention development, diffusion actions and transfer efforts are not always effective, and it is the link between intervention results, researcher diffusion activity and a variety of contextual factors after the research that ultimately determined whether a study had an impact on health policy and practice. Given the complex interaction between the various factors, there seems to be no simple formula for determining which intervention studies should be funded to achieve optimal impacts on health policy and clinical practice [44].

An area where evaluation is essential is in the development of health technologies. Health Technology Assessment (HTA) is the multidisciplinary study of the implications of the development, diffusion and use of health technologies. It provides a common pool of knowledge for decision makers and provides the basis for health policy decisions [42,45]. To be more relevant about policy, the HTA extends its reach beyond effectiveness and costs to consider social, organisational and ethical implications of technologies. However, a normally accepted method of analysing the ethical aspects of health technologies is lacking [42].

Saarni, et al. (2008) [42] developed an ethical analysis model of health technology that has the necessary capabilities to be able to be used in different institutional settings and in different cultures. The model is part of the EUnetHTA project, which is centred on the transferability of the impact of Health Technology Assessments (HTAs) across countries. This ethical model of EUnetHTA is based on the idea that the HTA process is driven by values [42]. It is not enough to simply analyse the ethical consequences of technology, but it is also necessary to take into account the ethical issues associated with the whole process of the impact of health technology assessments [42]. The selection of evaluation themes, methods and results is essentially a decision driven by values. Health technologies can test values and moral or cultural beliefs, and their application can also have significant impact on people other than the patient, and these considerations are essential to health policies. This ethical model has been articulated from key ethical questions rather than philosophical theories, so that it can be applied to different cultures and used by non-philosophers. For these authors, the integration of ethical considerations into HTAs can improve the relevance of technology assessments for health care and health policies in both developed and developing countries [42].

The objective of the 2010 SESPAS report (Sociedad Española de Salud Pública y Administración Sanitaria) is to contribute to advancing in the incorporation of the Health´s goal in all public health policies in Spain [46]. In the chapter that closes the report [47], ethical and economic arguments are presented to answer those who questioned certain public health policies for their possible invasion of individual freedom. The authors review the limitations of individual freedom from the point of view of efficiency, equity and social justice, concluding that the adoption of public health measures, as long as they meet certain ethical and technical requirements, do not limit, but protect and expand individual freedom.

It is evident that the systematic evaluation of project research impact developed for the different priority lines of the calls for public funds and their public diffusion outside specialized circles, is not frequent in our environment, and the evaluations that are carried out are usually only done by analysing the publications that are generated, the impact factor and citations [48]. In the work published by Martín-Moreno, Toharia, and Fuentes (2008) [22], on researchers’ opinions concerning project evaluation systems, although they all agree on the need for an efficient integrated evaluation system, there are doubts about the criteria used, particularly among clinical researchers. They also refer to the fact that there is some consensus that quality, applicability (transfer), social, clinical and industrial interest (patents), thematic area and type of research (basic) should be taken into consideration when establishing prioritisation criteria.

How Political Decisions Affect in Scientific Researches

It is important to reflect on the role of experts in their function of advising public authorities. Quoting Cubides and Durán (2002) [49]: “However, on many occasions, this is not the adopted perspective, since it is based on an over dimensioning of specialised knowledge, the role of the expert, and the possibilities of methods of scientific disciplines, which are apparently more rigorous. On the other hand, it is a matter of understanding that for some time now knowledge has ceased to be the exclusive domain of intellectuals and their heirs (Researchers, “Social Engineers” or “Symbolic Analysts”) and has become a common thing and an important device through which societies organise themselves, change and adapt to new historical circumstances”.

Without whole heartedly agreeing with this statement, it serves as an introductory element to analyse some of the problems that arise in the selection and performance of experts. In human relations the principle of trust is a basic tool and from it, communication and influence flows are established and constructed. Politicians choose the people who will advise them in their decisions, providing a technical-scientific vision that will provide consistency to their decisions. The role of scientific advisers is essential in all areas, but it is especially fundamental in directing research policy [19,50] point out that the participation of health professionals in health policy decision-making is essential to guarantee the implementation of measures based on evidence, with a high degree of professional support, thus maintaining the quality of the services provided.

The human condition is a constant, just like the force of gravity or the number “Pi”, and this must always be borne in mind, in order to establish surveillance and modulation systems to correct possible “Undesirable” effects. If we add to this the strong competitiveness that exists among some investigation groups, the desire for limelight, science globalisation and increasingly frequent economic crises, we define an ecosystem where it is very necessary to introduce correction systems and develop habits of ethical analysis for the decisions we make. A simple analysis of the conflicts of interest that exist in the case of some scientific advisers is sufficient to confirm this [19,51,52].

We are speaking about a phenomenon that affects all countries indiscriminately - the human condition knows no borders, languages or ideologies - but corrective measures have not achieved the same success (or even been instituted) in all countries of the world. In the particular case of Spain, solutions must undoubtedly be sought to improve the situation. The relationship between science and politics has hardly ever been peaceful; independence involves a cost in most cases, and, unfortunately, in a clash of interests intelligence and common sense do not always triumph.

Strassheim and Kettunen (2014) [53] identify the different ways in which politicians can interfere in the design of evidence-orientated decision making:

a) Asymmetry in knowledge of resources and the tendency to protect “self-knowledge” against evidence that questions it.

b) The black box phenomenon, whereby the complexity of statistical models prevents knowledge and understanding by those actors without sufficient knowledge.

c) Using evidence to displace and avoid political responsibilities and transfer it to other actors.

d) Oversimplification, based on large-scale planning schemes and management techniques based on limited rational principles.

An associated problem is the difference in time between research evidence and political decisions – different rhythms that do not always coincide [18,54].

The SUPPORT project (SUPporting POlicy Relevant Reviews and Trials) is an international project funded by the 6th Framework Programme of the European Commission and its objective is to develop a series of tools to help decision makers in health policy and health programmes based on the best scientific evidence available.

Gómez, et al. (2006) [55] mentioned the following determining factors in the relationship between scientists and politicians for the latter to take decisions:

• Main theme of policies and studies.

• Personal characteristics of researchers and politicians.

• Way in which researchers and policy makers engage in these processes.

• Context in which research and decision-makers interact.

• Potential impact of studies.

Other useful tools are the GRADE EtD frameworks that helps decision makers to use scientific information in a structured and transparent way, to inform on the formulation of clinical recommendations and other types of decisions [56,57].

A basic problem is the concept of social value. It is not easy to establish unanimous criteria to establish what it means, although that does not prevent it from being used as a term to determine priorities. For Emanuel (1999) [58] “Social value comparisons are an integral part of the determination of funding priorities. But when considering if a certain clinical research protocol is ethical (not whether it should be funded), evaluation focuses appropriately on that if this “should focus on whether it has some social value”.

Conclusion about Priority of Biomedical Research Lines

To claim absolute neutrality in the definition of a priority line is impossible, since ranking implies prioritising values, and the priority of values necessarily implies an ideological component. This is something common in political decisions; and neither are scientific criteria neutral, but it is evident that technical assessment allows more objective criteria to be used. The ability to explain some criteria and their justification from elementary ethical premises is an indicator of their pertinence. However, from the scientist’s point of view, rigour, solidity and consistency of criteria are necessary elements, but not always sufficient. It is necessary to make explicit the values and interests of the advisory scientists, since the information will complement the rest of factors and allows an integral perspective of the reality being analysed.

A very frequently criterion used is the necessity based on the objective conditions of the problem [14]. This criterion includes urgency, transcendence and magnitude as well as other elements that would make the list very long. Quantification tools that can introduce an objective element are always necessary. But analysis requires the reference framework to be broadened to include possible alternatives among other elements. It is evident that we are moving in a far from peaceful context and the interests of the actors will always be present, so that making conflicting interests explicit and visible becomes even more necessary.

An example of the difficulties that may arise using traditional criteria is the evaluation of the social impact of research on “Rare Diseases” (a disease is considered rare, infrequent or of low prevalence, when it affects less than one in two thousand cases in the European Union, or less than two hundred thousand in the United States). In this case, due to its low prevalence, the choice of financing and the prioritisation of one research line over another process requires a totally different model in which subjectivity and timeliness can condition the results. Low prevalence and the infrequency of rare diseases compared to the total set of pathologies makes them the Cinderella of quantitative systems of assessment [59,60]. The Instituto de Investigación de Enfermedades Raras (Institute of Research into Rare Disease) (IIER) of ISCIII (Spanish Ministerial Decree SCO/3158/2003 of 7 November) was created and the E-RARE of the European Research Area Network (ERA-NET) was launched in 2006. Among the objectives of the IIER are to ensure that adequate health care is provided to rare disease patients, in which respect, in 2006, its ethics committee drew up recommendations on the ethical aspects of population screening programmes for rare diseases [61].

What can be done to minimise the risks? Answers include defining processes that are as transparent as possible, an evaluation system that includes scientific impact, socioeconomic impact and ethical social aspects, using objective criteria, establishing full independence in the evaluation process and choosing the evaluators according to their competence, monitoring possible conflicts of interest, and making the results public to allow a feedback process and debate of the results. Because only by identifying the errors can we correct them. Oscar Wilde said that experience is the name we give to our mistakes - we cannot and must not repeat mistakes, but, at least let us be original in our mistakes.

Prioritising and defining the ethical values involved in a research line is not an easy task, and it is not always possible to reach an agreement to everybody’s satisfaction. But we can all agree that it is not difficult to present and explain them, which is what is lacking in the current situation. Politicians often hide behind scientist’s criteria and reduce explanations for their decisions to an exclusively technical area, ignoring explanations in ethical and social areas. Each member of the equation must assume his or her responsibilities clearly, without overlapping in their roles and without confusing their objectives.

In moments like these, where budgetary conditions force us to be very demanding in terms of efficiency [62], as understood in a broad sense and with a well understood social dimension, it is necessary to articulate a critical evaluation system of who decides, and with what criteria, the priority lines of research and what to do with the results of previous lines that are still in force.

It is not an easy or comfortable task, but it is necessary, but it is our duty to cooperate in generating a positive inertia in this area.

The identification of problems and search for solutions is our responsibility as professionals, but also to promote their implementation by those responsible. As in other fields, we are hopeful pessimists, but it should be possible to introduce elements of improvement in some decisions. On our perseverance and efforts will depend the final outcome.

References

1. García-Romero A (2008) La evaluación del impacto de la investigación biomé Situación actual y perspectivas de futuro. Medicina Clínica131: 1-5.
2. Adam P, Permanyer-Miralda G, Pons JMV (2010) El impacto social de la investigación biomédica en Cataluñ Nota d’Economia: 127-153.
3. Glover M, Buxton M, Guthrie S, Hanney S, Pollitt A, Grant J (2014) Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes. BMC medicine 12: 1.
4. Sussex J, Feng Y, Mestre-Ferrandiz J, Pistollato M, Hafner M, et al. (2016) Quantifying the economic impact of government and charity funding of medical research on private research and development funding in the United Kingdom. BMC medicine14: 1.
5. The Organisation for Economic Co-operation and Development (OECD). Gross domestic expenditure on R&D by sector of performance and Socio-Economic Objective (SEO). OCDE - Stats. 2017.
6. Moses H, Matheson DH, Cairns-Smith S, George BP, Palisch C, et al. (2015) The anatomy of medical research: US and international comparisons. Jama313: 174-189.
7. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, et al. (2014) How to increase value and reduce waste when research priorities are set. The Lancet383: 156-165.
8. del Río Sánchez C (2013) El consentimiento informado en menores y adolescentes. Contexto ético-legal y algunas cuestiones problemá Informació Psicológica : 60-67.
9. Giménez N, Pedrazas D, Redondo S, Quintana S (2016) Ensayos clínicos y consentimiento informado: visión de investigadores, pacientes y médicos de familia. Atención Primaria 48: 518-526.
10. Meléndez MF, Martínez LY, Sánchez MC, Jiménez EO, Hurtado RB, et al. (2013) Conocimiento sobre consentimiento informado para atención médica en unidades de primer y segundo nivel de atenció Revista CONAMED: 53-58.
11. Rodríguez Yunta E (2004) Comités de evaluación ética y científica para la investigación en seres humanos y las pautas CIOMS 2002. Acta bioethica 10: 37-48.
12. Martínez C, Edgar E (2003) Patents in the pharmaceutical industry: between ethics and property rights. Journal of Public Health 5: 18-23.
13. Herrero JA (2000) Estrategia y asignación de recursos en la investigación bioméGaceta Sanitaria 14: 391-397.
14. Carrasco Mallén M (2004) Políticas y gestión de la investigació Educación Médica 7: 66-80.
15. Escobar-Díaz FA, Agudelo CA (2009) Evaluación del proceso de priorización en salud en Colombia: perspectiva de grupos de investigación ubicados en Bogotá. Rev Salud Publica (Bogota) 11: 212-224.
16. ARMAS R, TORRES A, ARRIAGADA J, MUÑOZ F, SALINAS R, et al. (2010) Proceso para priorizar las líneas de investigación esencial de interés para el Ministerio de Salud de Chile. Revista médica de Chile 138: 401-405.
17. Elliott H, Popay J (2000) How are policy makers using evidence? Models of research utilisation and local NHS policy making. Journal of Epidemiology & Community Health 54: 461-468.
18. Camí J (2000) Priorización de la investigación biomédica: implicaciones para la investigación en salud pública y servicios sanitarios. Gaceta Sanitaria 14: 327-329.
19. Guston DH (2009) Between politics and science: Assuring the integrity and productivity of reseach. Cambridge University Press.
20. Delbecq AL, Andrew H, Gustafson DH, Villegas García C (1989) Técnicas grupales para la planeació 1989.
21. Fink A, Kosecoff J, Chassin M, Brook RH (1984) Consensus methods: characteristics and guidelines for use. American journal of public health 74: 979-983.
22. Martín-Moreno JM, Toharia JJ, Fuentes JAG (2008) Evaluación y priorización de la investigación científica en Españ El punto de vista de los investigadores. Medicina Clínica131: 12-19.
23. Peiró S, Herrero JA (2001) La gestión de la investigación en los centros sanitarios. Una exploración mediante la técnica de grupo nominal. Gaceta Sanitaria 15: 245-250.
24. Harding S (1992) After the Neutrality Ideal: Science, Politics, and" Strong Objectivity". Social Research: 567-587.
25. Bernal-Delgado E, Peiró S, Sotoca R (2006) Prioridades de investigación en servicios sanitarios en el Sistema Nacional de Salud. Una aproximación por consenso de expertos. Gaceta Sanitaria 20: 287-294.
26. Carbonell C, Gascón E, Nolasco A, Alvarez-Dardet C (1991) Estudio de opinión sobre prioridades en salud en la comunidad valenciana. Gaceta Sanitaria 5: 135-138.
27. Peiró S, Artells JJ, Meneu R (2011) Identificación y priorización de actuaciones de mejora de la eficiencia en el Sistema Nacional de Salud. Gaceta Sanitaria 25: 95-105.
28. Varela J (1991) Los métodos de consenso en el sector sanitá Gaceta Sanitária 5: 114-116.
29. Moreno SP, Argelaguet EP (1993) No todo es acuerdo en el consenso: limitaciones de los métodos de consenso en los servicios de salud. Gaceta Sanitaria 7: 294-301.
30. Fernández Polcuch E (2000) La medición del impacto social de la ciencia y la tecnologí 2000.
31. Baker JL (2000) Evaluación del impacto de los proyectos de desarrollo en la pobreza: Manual para profesionales. L-241-Bake_Judy-2000-399.
32. MRC (2015) Research funding policy and delivery. Guidance for applicants. Medical Research Council. Accessed January 3, 2017
33. Banzi R, Moja L, Pistotti V, Facchini A, Liberati A (2011) Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Research Policy and Systems9: 1.
34. Bornmann L (2013) What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology64: 217-233.
35. Milat AJ, Bauman AE, Redman S (2015) A narrative review of research impact assessment models and methods. Health Research Policy and Systems13: 1.
36. Raftery J, Hanney S, Greenhalgh T, Glover M, Blatch-Jones AJ (2016) Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme. Health Technol Assess 20: 1-254.
37. Solís DP, Fernández LR (2005) El acceso abierto a la información cientí Bol Pediatr 45: 61-64.
38. Rodríguez López J (2005) Ciencia y comunicación científica: edición digital y otros fundamentos del libre acceso al conocimiento. El profesional de la información 14.
39. http://www.ref.ac.uk/ (accessed Enero 2017).
40. Pollitt A, Potoglou D, Patil S, Burge P, Guthrie S, et al. (2016) Understanding the relative valuation of research impact: a best–worst scaling experiment of the general public and biomedical and health researchers. BMJ open6:  
41. Jones TH, Hanney S (2016) Tracing the indirect societal impacts of biomedical research: development and piloting of a technique based on citations. Scientometrics107: 975-1003.
42. Saarni SI, Hofmann B, Lampe K, Lühmann D, Mäkelä M, et al. (2008) Ethical analysis to improve decision-making on health technologies. Bulletin of the World Health Organization86: 617-623.
43. Willis TA, Hartley S, Glidewell L, Farrin AJ, Lawton R, et al. (2016) Action to Support Practices Implement Research Evidence (ASPIRE): protocol for a cluster-randomised evaluation of adaptable implementation packages targeting ‘high impact’ clinical practice recommendations in general practice. Implementation Science11: 11-15.
44. Newson R, King L, Rychetnik L, Bauman AE, Redman S, et al. (2015) A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: perceptions of Australian researchers. BMJ open5: e008153.
45. Orden SCO/2765/2003 de 19 de septiembre por la que se convocan ayudas económicas del programa de Promoción y Fomento de la Investigación Biomédica y en Ciencias de la Salud del Instituto de Salud Carlos III, con el fin de realizar estudios e investigaciones sobre evaluación de tecnologías sanitarias (BOE de 8/10/2003).
46. Artazcoz L, Oliva J, Escribà-Agüir V, Zurriaga Ó (2010) La salud en todas las políticas, un reto para la salud pública en Españ Informe SESPAS 2010. Gaceta Sanitaria 24: 1-6.
47. Aviles MA, Lobo F (2010) La justificación de las políticas de salud pública desde la ética y la eficiencia econó Informe SESPAS 2010. Gaceta Sanitaria 24: 120-127.
48. Segura Benedicto A (2000) La influencia de la investigación en las políticas de salud y en la práctica sanitaria. Revista Española de salud pública 74.
49. Cubides HJ, Durán A (2002) Epistemología, ética y política de la relación entre investigación y transformación social. Nómadas: 10-24.
50. Heras-Mosteiro J, Otero-García L, Sanz-Barbero B, Aranaz-Andrés JM (2016) Percepciones de médicas y médicos de atención primaria de Madrid sobre las medidas de ajuste en el sistema público de salud. Gaceta Sanitaria 30: 184-190.
51. Guston DH (1999) Stabilizing the boundary between US politics and science: The role of the Office of Technology Transfer as a boundary organization. Social studies of science 29: 87-111.
52. Guston DH (2001) "Boundary organizations in environmental policy and science: an introduction." 26: 399-408.
53. Strassheim H, Kettunen P (2014) When does evidence-based policy turn into policy-based evidence? Configurations, contexts and mechanisms. Evidence & Policy: A Journal of Research, Debate and Practice, 10: 259-277.
54. Buse K, Mays N, Walt G (2012) Making health policy. McGraw-Hill Education (UK).
55. Gómez A, Darío R, Orozco G, Alejandra D, Rodríguez O, et al. (2006) Políticas públicas y salud: relación entre investigación y decisióRevista Facultad Nacional de Salud Pública 24: 105-118.
56. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, et al. (2016) GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. bmj353: 2016.
57. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, et al. (2016) GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. bmj353: i2089.
58. Emanuel E (1999) Qué hace que la investigación clínica sea ética? Siete requisitos éPellegrino Filho A, Macklin R Investigación en sujetos humanos: experiencia internacional. Santiago de Chile: Programa Regional de Bioética OPS/OMS, 39.
59. Darbà J, Kaskens L (2013) Consideraciones económicas para la evaluación de los medicamentos huérfanos en las decisiones de financiación en Españ Pharmacoeconomics Spanish Research Articles 10: 141-146.
60. Hernández-Vásquez A, Rubilar-González J, Pisfil-Benites N (2014) Financiamiento de enfermedades raras o huérfanas: una tarea pendiente. Revista Peruana de Medicina Experimental y Salud Pública 31: 393.
61. Arribas MC (2006) Comité de Ética del Instituto de Investigación de Enfermedades Raras. Recomendaciones acerca de los aspectos éticos de los programas de cribado de población para enfermedades raras. Gaceta Sanitaria 20: 27-32.
62. Cortès-Franch I, López-Valcárcel BG (2014) Crisis económico-financiera y salud en Españ Evidencia y perspectivas. Informe SESPAS 2014. Gaceta Sanitaria 28: 1-6.