Abstract

Background : Systematic reviews (SRs) are important synthesis mechanisms for aiding the epidemiology background in decision-making process. Since 2004, the Brazilian Ministry of Health (MoH) have been used the international reviews for compiling health technology assessment (HTA) reports and rapid reviews. SRs provide information of greater consistency for decision-making relating to the healthcare system, since they increase the magnitude and precision of the results evaluated.

Objective: To describe the use of SRs HTA reports and in the MoH’s decisions.

Methods: This was a retrospective descriptive study conducted by reviewing the MoH’s documents and official registers and by using personal experience

Results: Between 2004 and 2011, the MoH requested 65SRs. Of these, 27 (41.5%) were sent to the MoH’s permanent consultative committees as support material. Eight were used as information in making decisions, and seven were cited in eight HTA reports of the MoH. Considering all the systematic reviews (Cochrane Library/Plus and Brazilian Cochrane), it was observed that there were 83HTA reports with citations in total among 262 reports produced between 2006 and 2011.

Conclusion and implications for practice and for research: SRs are sources of information for decision-making and for compiling HTA reports for the MoH. The implication of this study for practice was that a standardized method for registering SRs was developed. The implications for new research consisted of identifying barriers against and factors favoring the use of SRs, along with measuring the economic impact of systematic reviews on efficiency in the healthcare system.

Keywords

Health technology assessment; Use of health research; Systematic review implementation

Introduction

The rapid growth of health technologies and the range of therapeutic options have given rise to a need for decisions on healthcare service coverage. For effectiveness and safety to be attained in relation to therapeutic alternatives that are chosen, systematic searches and critical analysis of the literature are fundamental.

Systematic reviews (SRs) form the first step in health technology assessment and coverage decision processes [1]. They are used to achieve better understanding of a topic or problem; to define priorities for assessments and new research; to support decisions relating to reimbursement for medications; to form the basis for economic assessment models; and to reduce the barriers against translation of knowledge [1-10].

Through a literature review and interviews with decision-makers in Canada and the United Kingdom, authors such as Lavis et al [10] identified characteristics that favored the use of SRs by decision-makers. From experience of the Cochrane Collaboration network in Canada, Grimshaw [11] reported on the contributions that these SRs made towards the activities of translating knowledge. Perrier [12] reviewed published studies on interventions that had the aim of increasing the search for and evaluation and application of evidence coming from SRs, and concluded that little empirical data had been published.

Few studies have described practical experiences relating to using SRs, especially in developing countries. Such studies are important for obtaining evidence about what SRs are used for, through seeking ways of measuring these results.

Given that in implementing the field of health technology assessment (HTA) in the MoH, the partnership with the Brazilian Cochrane Center was the first step towards producing SRs on topics that were prioritized by the national administrator, it is important to describe how these SRs have been used as a source of evidence in the process of assessing and incorporating health technologies at national level within the Brazilian National Health System (SUS).

The aim of this paper was to describe how SRs from the Cochrane were used in composing HTA reports in the MoH’s and as information for decision-making by the permanent consultative committees of the MoH between 2004 and 2011.

Materials and Methods

A descriptive analytical study was conducted using retrospective data covering the period 2004 to 2011, before the National Committee for Health Technology incorporation (CONITEC - Lay 12401/2011).

The markers observed in order to measure the use of SRs are described in the following. Marker A refers to the presence of a bibliographic citation of a Cochrane SR published in the Cochrane Library/Plus or produced by the Brazilian Cochrane Center, which was included in the reference list of the HTA report. Marker B refers to registration of a SR abstract in the information system of the Brazilian HTA Network (REBRATS). Marker C refers to SRs sent as support material to the permanent consultative committees of the MoH. Marker D refers to SRs used as a source of information for decision-making by the permanent consultative committees of the MoH.

The MoH’s permanent consultative committees taken into consideration in this study were the Technology Incorporation Committee (CITEC - after 2011 became CONITEC) and the Working Group for Clinical Protocols and Therapeutic Guidelines (PCDT). Over the period evaluated, the HTA Unit of the Department of Science and Technology (DECIT) was in charge of these committees and was responsible for producing HTA reports to provide support for the decision-making process. Use of SRs in these permanent committees’ work was ascertained through analyzing the records of DECIT and CITEC (Chart 1), and through the first author’s experience as a full member of the committees.

To correlate the markers observed in this study with the concept of use of research (in these case SRs) the conceptual model of Hanney et al was used [2]. According to these authors, the stock or reserve of knowledge can interact with the political, professional, industrial and society environments.

Starting from this concept, these authors proposed several stages for feeding the stock of knowledge and supplying information for decisionmaking processes. The stages and interfaces of how the products and research results can be made use of by healthcare policymakers are as follows: i) identification of research questions; ii) selection and specification of the research project; iii) inputs for the research; iv) research development process; v) production of primary products from the research; vi) dissemination of these results; vii) backing for policy formulation; viii) adoption by professionals and the public; and ix) return of the final results to the store of knowledge.

According to these authors, SRs are used in policy formulation and in their adoption by professionals or political engagement. SRs are products from research into support for policies, and also, have the function of contextualizing this knowledge within the social environment.

For the purposes of this study, the stages of use of the conceptual model of Hanney [2] were taken to be the stage of dissemination of the results from the research and the phase characterized as backing for policy formulation.

Marker B, i.e. registration of SRs in the information system of the Brazilian HTA network (REBRATS) [13], was allocated as being in the stage of dissemination of the results from SRs.

The markers A, C and D were situated as indicators of results from SRs that provided backing for policy formulation.

For the present study, HTA reports were taken to be technical notes from rapid reviews (TNRRs) and scientific technical reports (STRs). TNRRs are synthesis documents that are produced by the internal team at DECIT within 20 days, and they place value on SRs, technology assessments from international agencies and randomized controlled trials (RCTs). STRs are also synthesis documents, produced within four months either by the DECIT team or ordered from institutions within REBRATS. These place value on all types of quality study, contain structures searches and a table showing a critical evaluation on the evidence.

The markers observed were identified through reading and analyzing official documents and records from the MoH, as listed in Chart 1 (Chart 1: Description of the data sources as additional files)

Types of Record in the Ministry of Health (MoH)	Definition of the record	Body responsible for registration	Year of registration	Format
F1 - Management reports from the technology assessment unit of the Department of Science and Technology (DECIT)	Description of the actions performed and studies ordered during the year under analysis	DECIT	2005	Word
F2 - Table named “All the CNPq HTA announcements (4)”	Listing containing the projects selected in announcements from DECIT in conjunction with CNPq in the field of health technology assessment, in the years 2005 to 2009	DECIT	2007	Excel spreadsheet
F 3 - Electronic files from the HTA unit of DECIT.	HTA reports contained in DECIT’s administrative records	DECIT	2004	File directory
F4 - DECIT’s history of participation in meetings of the MoH’s Technology Incorporation Committee	Register of the main decisions from CITEC and the requests for studies made to DECIT.	DECIT	2006	Excel spreadsheet
F 5 - Situation of reports compiled by DECIT and CITEC’s conclusions (2007 to 2010)	Description of studies produced or ordered by DECIT and their results	DECIT	2008	Excel spreadsheet
F6 - Technology incorporation 24082011	Register of processes submitted to CITEC	CITEC	2011	Excel spreadsheet
F7 - Minutes of CITEC meetings	Record of the main decisions and positions of the members	CITEC	2008	Word
F8 - Register of decisions made	Register of the decisions made with the members’ signatures	CITEC	2008	Word
F 9 - Official memoranda and e-mails	Document formalizing requests for and sending of studies from/by DECIT	Various sectors	2006	Word
F10 - Information system of the Brazilian HTA Network (REBRATS)	Database containing studies under development or abstracts and complete texts of concluded studies.	DECIT	Continuous	Database on internet
F11 - CITEC decisions 09_02_2012	Charts citing technologies incorporated and not incorporated by CITEC	CITEC	2006	PDF on Internet

Chart 1: Description of the data sources (As additional files)

The eligible analysis was systematic reviews ordered from the Brazilian Cochrane Center and all the SRs (Cochrane Library/Plus and Brazilian Cochrane Center) cited in the HTA reports.

The method for documenting the use of SRs followed the protocol described in Chart 2. Presence of one or more markers was taken to be a record of use for each marker defined. (Chart 2: Correlation between the markers and information sources of the retrospective study).

Marker A: Bibliographic citation of all SRs (Brazilian and general) included in the composition of HTA reports, as described in the methods section

a.  Survey of the list of SRs ordered, in the management reports (F1) and register of projects selected in announcements (F2).

b. Survey of the list of HTA reports (F3).

c.  Complete reading of the HTA reports (F3), with identification of citations of Cochrane reviews.

d. Analysis on the titles of the Cochrane reviews cited and stratification according to whether the review was obtained from the Cochrane Library/ Plus or from the Brazilian Cochrane Center (BCC).

Marker B: Systematic reviews registered in the information system of the Brazilian HTA Network (REBRATS).

a.  Search for and reading of abstracts of BCC reviews registered in the REBRATS information system (F10).

Marker C: SRs sent as support material to the permanent consultative committees of the Ministry of Health.

b. Reading of the minutes of CITEC meetings (F7).

c.  Search for the name of the technology and studies compiled by DECIT in the spreadsheet (F7) and CITEC’s conclusions registered in the spreadsheet (F6).

d. Checking of the spreadsheet (F4) and second checking of the spreadsheet (F6).

e.  Analysis on the records of CITEC’s decision-making (F8)

f.  Search for studies in DECIT’s files (F3)

g. Search for CITEC’s decisions (F11)

a.  Reading of official memoranda and e-mails (F9) to investigate sending of reviews to aid the working group for clinical protocols and therapeutic guidelines.

Marker D: Brazilian SRs used as information for decision-making by the permanent consultative committees of the ministry of Health

a.  Search for the name of the technology and studies compiled by DECIT in the spreadsheet (F7) and CITEC’s conclusions registered in the spreadsheet (F6).

b. Analysis on the records of CITEC’s decision-making (F8)

c.  Search for studies in DECIT’s files (F3)

d. Search for CITEC’s decisions (F11)

b. The first author’s experience as a full member of CITEC during the study period.

Chart 2: Correlation between the markers and information sources of the retrospective study.

Results

Between 2006 and 2011, 77 SRs were ordered from the Brazilian Cochrane Center by the MoH, and 65 were concluded. There are different forms of use of SRs within the MoH’s scope of the MoH according to the markers and definitions used.

In relation to marker A, i.e. bibliographic citations of SRs included in the composition of HTA reports (Table 1), a total of 83 reports with citations of all SRs were observed in 262 reports, thus corresponding to 31% (83/262) of the reports produced between 2006 and 2011. No records covering the period before 2006 were found for analysis.

Year	Number of reports produced (STRs and TNRRs)	Total number of reports with citations	Reports with citations of SRs in the Cochrane Library/Plus	Reports with citations of SRs from the Brazilian Cochrane Center
2006	15	1	1	0
2007	29	4	4	0
2008	69	24	18	6
2009	58	19	18	1
2010	41	21	20	1
2011	50	14	14	0
Total	262	83	75	8

Table 1: Citations of all Cochrane reviews in HTA reports produced by the Brazilian Ministry of Health, from 2006 to 2011.
HTA: Health Technology Assessment; STR: Scientific Technical Report; TNRR: Technical Note from Rapid Review; SR: Systematic Review

SRs obtained via the Cochrane Library/Plus were cited in 75 HTA reports between 2006 and 2011. SRs ordered from the Brazilian Cochrane Center (BCC) were cited in eight reports over the same period.

In relation to marker C, 27 Brazilian SRs (BSR) reached the definition of material for support to the consultative committees (Scorer C in the Graph 1). Eight were also used as information for decision-making in the technology incorporation committee (Scorer D) and seven were cited as references in HTA reports (Scorer A). These three markers were present in the following topics: gastric band implantation surgery for treating severe obesity; efalizumab for treating psoriasis; imatinib for treating chronic myeloid leukemia; and coated stents versus conventional stents for treating arterial insufficiency.

Graph 1: Situation of systematic reviews from the Brazilian Cochrane Center funded Ministry of Health, between 2004 and 2011, according to the marker (scorer) selected.
Source: Compiled by the authors.
BSR: Systematic Review from the Brazilian Cochrane Center; SISREBRATS: Information System of the Brazilian Health Technology Assessment Network; HTA: Health Technology Assessment.

As shown in Graph 1, 38 Brazilian SRs (BSR) were available in full texts from the information system of REBRATS (SISREBRATS – scorer B).Since 2008, abstracts or SRs have been included in the database of the REBRATS. These BSRs can be accessed by individuals responsible for decision-making in the MoH and by the general public.

Most of the SRs were directed towards evaluating medications. Thirty-two of the SRs assessed related to medications, followed by nineteen relating to devices and equipment. All of the SRs evaluated efficacy and safety. (See additional file - Chart 3 Brazilian systematic reviews produced by the Cochrane Center on request from Ministry of Health, according to the categories selected, from 2005 to 2011).

Sl No.	Concluded systematic reviews produced by the Brazilian Cochrane Center	Year of conclusion	Marker
1	Radio therapeutic approaches and palliative treatment for prostate cancer	2011	B
2	Acupuncture for treating primary headache	2005	B
3	Acupuncture for treating lateral epicondylitis	2005	B
4	Acupuncture for treating lumbalgia	2005	B
5	Acupuncture for treating carpal tunnel syndrome	2005	B
6	Adalimumab for treating rheumatoid arthritis	2006	C, D
7	Alemtuzumab for treating chronic lymphoid leukemia	2009	B
8	Drotrecogin alfa for treating severe sepsis	2005	B
9	Peginterferon alfa-2α for treating chronic hepatitis B	2009	C
10	Peginterferon alfa-2α and peginterferon alfa-2β for treating genotype 1 of chronic hepatitis C	2009	C
11	Ferrara ring for treating keratoconus	2005	B
12	Monoclonal antibodies for treating breast cancer	2011	B
13	Atypical antipsychotics for treating schizophrenia that is refractory to typical antipsychotics	2009	C
14	Update of systematic review: coated stents for cardiovascular disease	2008	B
15	Evaluation of percutaneous intervention with micro-springs (coils) for arteriovenous malformation	2009	B
16	Evaluation of tenofovir for treating chronic hepatitis B and pegylated interferon versus conventional interferon for treating chronic hepatitis B and delta	2011	B
17	Intragastric balloon for treating obesity	2005	C
18	Mason bands and gastroplasty for treating obesity	2005	A, C, D
19	Bevacizumab (Avastin) in ophthalmology	2008	A, C
20	Blood glucose control (insulin) in patients with diabetes mellitus types l and ll	2009	C
21	Roux-en-Y gastric bypass for treating obesity	2006	C
22	Efalizumab for treating psoriasis	2005	A, C, D
23	Efficacy and effectiveness of pegvisomant for treating acromegaly	2009	C
24	Efficacy and effectiveness of piribedil for treating Parkinson disease	2007	C
25	Embolization for treating uterine myoma	2005	A, C
26	Deep cerebral stimulation using electrodes for treating Parkinson disease	2005	B
27	Etanercept for psoriasis in psoriatic arthritis and plaque	2009	C
28	Endovascular intervention in comparison with open surgery for treating abdominal or thoracic aortic aneurysm	2009	B
29	Interventions for reducing maternal-child mortality	2009	B
30	Idursulfase for treating mucopolysaccharidosis type II	2009	C
31	Imatinib for treating gastrointestinal tumors	2005	C, D
32	Imatinib for treating chronic myeloid leukemia	2005	A, C, D
33	Immunosuppressants for preventing kidney transplant rejection	2009	B
34	Infliximab for treating psoriasis in moderate to severe plaque	2009	C
35	Enzyme activity inhibitors for advanced/metastatic renal carcinoma refractory to initial treatment	2011	B
36	Catechol-O-methyltransferase inhibitors for treating Parkinson disease	2009	C
37	Prolactin inhibitors for treating hyperprolactinemia	2010	B
38	Interventions among schoolchildren aiming towards lifestyle changes: interventions for treating obesity among children	2011	B
39	Digital mammography in comparison with conventional mammography	2009	B
40	Multisite pacemaker for cardiac resynchronization therapy	2005	B
41	Skin replacement material for treating burns	2005	B
42	Adjuvant diagnostic methods for coronary or atherosclerotic diseases: efficacy and safety of intracoronary ultrasound in coronary insufficiency cases	2010	B
43	Nucleoplasty for treating disc hernia	2005	B
44	Oxcarbazepine for refractory epilepsy	2008	A, C
45	Prostheses for male urinary incontinence (artificial sphincters)	2011	C, D
46	Diabetes screening in the initial diagnosing of tuberculosis	2011	B
47	Reuse of balloon catheters for angioplasty	2005	B
48	Reuse of electrophysiological catheters for angioplasty	2005	B
49	Stents coated with rapamycin or paclitaxel versus conventional stents for treating arterial insufficiency	2005	A, C, D
50	Weekly iron supplementation among children for preventing iron deficiency anemia	2010	B
51	Suburethral suspension as a technique for surgical treatment of urinary incontinence among women	2005	B
52	Mechanical suturing in colorectal anastomosis surgery	2005	B
53	Duodenal and Scopinaro switch for treating obesity	2005	B
54	Surgical techniques for treating epilepsy	2005	B
55	Intersomatic fusion techniques of single or double intervertebral level for treating degenerative cervical disc disease	2005	B
56	Teriparatide for treating osteoporosis in postmenopausal women	2005	C
57	Treatment of rheumatic carditis	2009	B
58	Treatment of chronic obstructive pulmonary disease	2011	C
59	Treatment of ankylosing spondylitis	2009	C
60	Treatment of pulmonary arterial hypertension	2009	C, D
61	Treatment of wounds and burns: dressings with carboxymethyl cellulose for treating burns	2011	B
62	Treatment of systemic lupus erythematosus	2009	B
63	Photodynamic therapy with verteporfin (Visudyne®) for treating age-related macular degeneration	2005	A, C
64	Pediatric use of corticosteroids for respiratory disorders	2011	B
65	Validation of diagnostic procedures involving rapid screening tests for hepatitis B and C	2011	B

Chart 3: Concluded systematic reviews produced by the Brazilian Cochrane Center on request from DECIT, Ministry of Health, according to the categories selected, from 2005 to 2011. (As additional files)

The eight SRs were used in decision-making by technology incorporation committees are listed in Chart 4. The concordance analysis was conducted, comparing the recommendations in the SR and the decisions made by the committee.

From Chart 4, it can be seen that the majority of the decisions had a high degree of concordance with the conclusions in the SR, except in relation to the urinary sphincter, for which there was discordance, and in relation to the treatments recommended for pulmonary arterial hypertension, for which the two medications under analysis were indicated. It should be borne in mind that in addition to the SR, the committee had an arsenal of information relating to the relevance of the technology for the healthcare services, in the light of the costs and reports from the sectors responsible for care policies.

Sl No.	Systematic reviews from Brazilian Cochrane Center that were used as information	Year of conclusion	Year of decision	Degree of concordance
1	Adalimumab for treating rheumatoid arthritis	2006	2007	Concordance with incorporation
2	Mason bands and gastroplasty for treating obesity	2006	2010	Concordance. Authors judged that there was insufficient evidence. Committee decided not to incorporate it.
3	Efalizumab for treating psoriasis	2005	2006	Concordance. Authors judged that it was too early to adopt its use. Committee decided to withdraw it from the agenda.
4	Imatinib for treating gastrointestinal tumors	2005	2011	Concordance. Authors reported that adoption should be monitored because they judged the evidence to be insufficient. Committee decided not to incorporate it.
5	Imatinib for treating chronic myeloid leukemia	2005	2006	Concordance with incorporation
6	Prostheses for male urinary incontinence (artificial sphincters)	2011	2011	Discordance. Authors recommended adoption for severe cases of post-prostatectomy incontinence. Committee decided to ask DECIT for cost-benefit analysis.
7	Stents coated with rapamycin or paclitaxel versus conventional stents for treating arterial insufficiency	2005, updated in 2008	2007	Concordance. Authors judged that there was no advantage in several outcomes. Committee decided not to incorporate based also on other economic information.
8	Treatment of pulmonary arterial hypertension	2009	2010	Inconclusive. Authors demonstrated that sildenafil and bosentan produced effects. Committee chose sildenafil.

Chart 4: Systematic reviews from the Brazilian Cochrane Center that were used as decision-making information (marker D), from 2005 to 2011

The seven BSRs that were cited in eight HTA reports according to the type of report, title, content and conclusion of the report. The data showed that all the SRs cited were ordered by the MoH. The conclusions of five HTA reports followed the results of the BSR. Among these, the case of macular degeneration in elderly individuals can be highlighted. Two reviews were cited because the content of the introduction was used, referring to the characteristics of the disease - psoriasis and rheumatoid arthritis (Chart 5).

Systematic reviews from Brazilian Cochrane Center used in HTA reports from DECIT	Year ordered	Year of conclusion	Title of HTA report	Type of report	Year of HTA report	Target public	Content used	Conclusion of HTA report
Bevacizumab (Avastin) in ophthalmology	2008	2008	NT – Information on the medication bevacizumab (Avastin®) for treating subretinal neovascularization	TNRR	2008	CITEC	Results	Recommended incorporation; concordance with the SR
			NT – Scientific evidence available on efficacy and safety of bevacizumab (Avastin®) for treating neovascular ocular diseases	TNRR	2009	PAO in Cascavel	Results	Recommended incorporation; concordance with the SR
Oxcarbazepine for treating refractory epilepsy	2008	2008	NT on supplying the medication oxcarbazepine	TNRR	2008	FGLO	Results	Recommended non-incorporation; concordance with the SR
Verteporfin (Visudyne®) for treating age-related macular degeneration	2004	2005	NT – Information on inclusion of photodynamic therapy with verteporfin (Visudyne®) in the list of procedures of the National Health System (SUS) for individuals with age-related macular degeneration	TNRR	2008	PAO in Sao Paulo	Results	Because of the introduction of anti-neoangiogenic drugs, it concluded that verteporfin should not be incorporated
Stents coated with rapamycin or paclitaxel versus conventional stents (*)	2005 e 2008	2005 e 2008	NT – Request for incorporation of the medication CYPHER® (sirolimus-eluting coronary stent)	TNRR	2009	SBHCI	Results	Recommended non-incorporation; concordance with the SR
Embolization for treating uterine myoma	2004	2005	NT – Request for inclusion of uterine embolization procedure in the SUS table	TNRR	2010	Senate	Introduction	Cited in characterization of the disease
Adalimumab for treating rheumatoid arthritis	2004	2006	STR on use of rituximab for treating rheumatoid arthritis	STR	2008	CITEC	Introduction	Cited in characterization of the disease
Efalizumab for treating psoriasis	2004	2005	PTC on use of efalizumab for treating psoriasis	STR	2008	CITEC	Results	Mentioned that there was an effect, but recommended new clinical trials, in concordance with the SR

Chart 5: Systematic reviews (SRs) from the Brazilian Cochrane Center cited in HTA reports from DECIT (marker A), between 2005 and 2011
Source: Compiled by the authors. Legend: TNRR: Technical Note From Rapid Review; STR: Scientific Technical Report; CITEC: Technology Incorporation Committee of the Ministry of Health; SBHCI: Brazilian Society for Hemodynamics and Intervention Cardiology; PAO: Public Attorneys’ Office; FGLO: Federal General Legal Office.

Discussion

Summary of findings

The findings from the MoH’s experience reveal that SRs were used, but the small number of citations may have been due to many situations. For example, the topic requested might not have been a subject covered by the SRs; the SRs have not concluded; or the SRs results are inconclusive.

All the topics were prioritized by the MoH, but the decision-making moment within the MoH’s permanent consultative committees could not always be reached. For example there was a lack between the time of the SR conclusion and the time of the meeting of decision-making by the committee.

Validity of results

The limitations of the present study comprised difficulties organized the records of the HTA reports and SRs produced, along with documents used in incorporation decisions made by the consultative committees over the period studied. Since this was a descriptive study using retrospective data, there was the possibility of memory bias and measurement bias, since the registers were not planned for this purpose.

Implications of the findings

Since 2004, the MoH has had a partnership with the REBRATS institutions to support decisions made by the MoH’s Technology Incorporation Committee (CITEC), and for MoH to draw up HTA reports. The perspective adopted was that SRs supplied information of greater consistency for decision-making relating to the healthcare system, since they increased the magnitude and precision of the results relating to the outcomes assessed.

For seven years, the Brazilian MoH has been funding the production of SRs on topics prioritized by healthcare administrators for decisionmaking, through directly ordering them or through public calls with the National Council for Scientific and Technological Development [14]. The partnership the REBRATS institutions included provision of master’s degree courses as professional training and courses on evidence-based medicine for technical specialists at the MoH [15,16].

However, contextual factors involve the degree of use, as the permanence of barriers as lack of prioritization process from the decision-makers and informal research body to compile HTA reports.

In this paper, we have suggested that markers should be used to prospectively monitor the use of SRs funded by the MoH. Analysis on the impact and use of SRs or HTA reports can be fundamental for avoiding duplication and expenditure on the decision-making process.

Comparison with the literature

Empirical studies on the use of research and on decisions informed by the results from research started to be published internationally in 1999. From a conceptual point of view, a variety of models for analyzing the use of research in policy formulation have been reported, with the conclusion that the process is diffuse and difficult to measure [17,18].

The majority of studies on models and levels of use of research in formulating policies have been conducted in developed countries, covering all types of research [19,20].

The impact of the ten years of existence of the National Coordinating Centre for Health Technology Assessment (NCCHTA) was evaluated by a research team [21], through applying interviews and analyzing documents. It was concluded that HTAs produced for the NICE had greater impact, in terms of application to decision-making in the British system, but that few studies had estimated returns in monetary values, in terms of costbenefit analysis or estimates of cost reduction. The review studies cost 40,000 pounds on average and clinical trials started at one million pounds. A total of 133 projects were analyzed. The impact indicators applied were publication in peer-reviewed journals, qualification of personnel, funding for new projects, presentation of results to different audiences (both academic and in practice). Qualitative analysis indicated that the professional body at national level and the agencies were the biggest beneficiaries. Overall, the times taken to present results, the quality of the results and the closeness of links with decision-makers were impact factors.

The impact of 21 HTA reports produced by a Canadian committee [22] was evaluated through estimates of cost reductions applied to the results and recommendations. Document analysis, interviews, questionnaires and databases were used. The results demonstrated that only three reports influenced policies and that cost minimization studies promoted savings of between US$ 16 and 27 million per year. It was concluded that precise estimates of the impact are rarely possible, but that systematic analysis of documentation on the effects was viable.

Through empirical studies some authors demonstrated the factors that influenced the use of healthcare research and highlighted the following as the most frequent factors: i) interactions between researchers and policy formulators through creation of formal advisory networks and committees or informal relationships; ii) research that brought together beliefs, values and interests or policy objectives; and iii) social pressure [22,23].

The factors that influence the use of research evidence by public policymakers are funding, human resources, time, evaluators’ skills and access to databases [24].

The implication of this study for practice is that SRs are advantageous, since they cost 30,000 reais, which is around 10% of the cost of a review paid for in England and 1% of the cost of an RCT. In this case, the cost of a decision should also be taken into account, given that there is a series of stages in producing an assessment that will inform the decision-making process [25].

Meta-assessments on the assessment process itself have been conducted by consultancy companies such as Charles River Associates [26]. In analyzing 15 HTA organizations as case studies representing countries that included France, United Kingdom, Germany, Canada, Australia, South Korea and Brazil, they questioned the quality, duplication and costs incurred in the assessment processes used in the regulatory processes of health technology incorporation.

The mixed method study demonstrated a total of 1,502 new and updated reviews were produced by the 20 National Institute for Health Research (NIHR) -funded Cochrane Review Groups between 2007 and 2011.These Cochrane reviews have an impact on clinical guidance development and may have influenced the conduct of primary research. Limited evidence was found to demonstrate that Cochrane reviews had had a direct effect on clinical practice, but they may have an indirect impact on health care through their role in clinical guidance [27].

Measuring the economic impact of the results from SRs used in healthcare decision-making will need investigation to aim promote efficiency within the healthcare system [28].

Conclusion

The present descriptive study demonstrated that SRs were used in HTA reports and decision-making process. These practices can be increased. Many barriers were found, as the topic requested might not have been a subject covered by the SRs; the SRs have not concluded; or the SRs results are inconclusive.

However, the notion of using evidence for decision-making is legitimated through the Brazilian federal constitution. Brazilian legislation has created rules based on scientific evidence for incorporation decision making process (Law 12401/2011). S0065 lection of technologies for assessment followed three basic criteria: those under development or at the pre-registration phase; those already incorporated in the healthcare system but with a need for economic evaluation or assessment of their effectiveness for new indications; and those already registered but not yet incorporated.

The element that generated use of SRs for supporting the MoH’s decisions was the prioritization process, which took into consideration the three criteria described above and the demands from the National Committee for Health Technology Incorporation (CONITEC, reformulated through Law 12401/2011).

Use of evidence is fundamentally favored by communication mechanisms directed towards different target audiences. Decentralization of use is another aspect highlighted. For this, the interactions with state and municipal administrators need to be strengthened from the perspective of managing knowledge and identifying factors for stimulating greater use of SRs.

The implications for research cover periodic updating of SRs and the need to estimate impacts through expanding the theoretical approach from the point of view of translation of knowledge. There is also a need to measure the economic impact of systematic reviews for promoting the efficiency of the healthcare system, which will be the subject of a future study.

Competing Interests

The authors were responsible for the partnership between the Ministry of Health and the Brazilian Cochrane Center.

Authors’ Contributions

FTS and ANA conceived the question for the study. FTS created the design, analyzed and interpreted the data and wrote the paper. ANA made interpretations and critical reviews on the paper

Acknowledgements

We thank Prof. Reinaldo Guimarães for the initiative of formalizing the partnership with the Brazilian Cochrane Center during the period when he was the Secretary of Science, Technology and Strategic Supplies at the Ministry of Health; and Prof. Carlos Augusto Gabrois Gadelha (the present Secretary) and Prof. Jailson de Barros Correia (the present Director of the Department of Science and Technology), for maintaining the partnership and supporting the development of this study. We also thank the technical specialists of DECIT’s technology assessment unit for the institutional work developed.

References

1. Milne R, Clegg A, Stevens A (2003) HTA responses and the classic HTA report. J Public Health Med 25: 102-106. [Ref.]
2. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M (2003) The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 1: 2. [Ref.]
3. World Health Organization (2012) The Alliance for Health Policy and Systems Research. [Ref.]
4. The Cochrane Collaboration (2001) Informing Judgment: Case Studies of Health Policy and Research in Six Countries September 2001. Milbank Memorial Fund, New York, USA. [Ref.]
5. Drummond MF, Iglesias CP, Cooper NJ (2008) Systematic reviews and economic evaluations conducted for the National Institute for Health and Clinical Excellence in the United Kingdom: a game of two halves. Int J Technol Assess Health Care 24: 146-150. [Ref.]
6. Sun X, Orlewska E, Yang H, Li Y (2005) Assessing comparative costeffectiveness for selection of essential drugs in China: an evidencebased approach. XIII Cochrane Colloquium, Melbourne, Australia 22-26.
7. Atallah AN, Castro AA (1998) Revisão sistemática e metanálises. In: Evidências para melhores decisões clínicas. São Paulo. Lemos Editorial.
8. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE (2012) Knowledge translation of research findings. Implement Sci 7: 50. [Ref.]
9. Garrido MV, Kristensen FB, Nielsen CP, Busse R (2008) Health Technology Assessment and health policy making in Europe. Current status, challenges and potential. Eunehta, European Observatory, Europe 1-197. [Ref.]
10. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, et al. (2005) Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 1: 35-48. [Ref.]
11. Grimshaw JM, Santesso N, Cumpston M, Mayhew A, McGowan J (2006) Knowledge for knowledge translation: the role of the Cochrane Collaboration. J Contin Educ Health Prof 26: 55-62. [Ref.]
12. Perrier L, Mrklas K, Lavis JN, Straus SE (2011) Interventions encouraging the use of systematic reviews by health policymakers and managers: a systematic review. Implement Sci 6: 43. [Ref.]
13. Pacheco Santos LM, Moura EC, Barradas Barata Rde C, Serruya SJ, da Motta ML et al. (2011) Fulfillment of the Brazilian Agenda of Priorities in Health Research. Health Res Policy Syst 9: 35. [Ref.]
14. Vanni T, Stein At, Souza Km, Freitas Mg, Patterson I, et al. (2015) Evaluative intelligence network: building consensus on Technology Assessment in Health. Revista Eletrônica de Comunicação, Informação & Inovação em Saúde 9: 1. [Ref.]
15. Novaes HMD, Elias FTS (2013) Using the evaluation of health technologies in analysis processes for incorporating technology in the Unified Health System in the Ministry of Health. Cad Saúde Pública 29: S7-S16. [Ref.]
16. Weiss CH (1979) The many meanings of research utilization. Public Admin Rev 39: 426-431. [Ref.]
17. Elias FTS, Patroclo MAA (2005) Use of research: How to Build theoretical models for evaluation? Science & Public Health 10: 215-227.
18. Hanney S, Buxton M, Green C, Coulson D, Raftery J (2007) An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess 11: 1-180. [Ref.]
19. Innvaer S, Vist G, Trommald M, Oxman A (2002) Health policymakers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy 7: 239-244. [Ref.]
20. Jacob R, McGregor M (1997) Assessing the impact of health technology assessment. Int J Technol Assess Health Care 13: 68-80. [Ref.]
21. Ellen ME, Lavis JN, Ouimet M, Grimshaw J, Bédard PO (2011) Determining research knowledge infrastructure for healthcare systems: a qualitative study. Implement Sci 6: 60. [Ref.]
22. Trostle J, Bronfmen M, Langer A (1999) How does research influence decision-makers? Case studies of Mexican policies. Health Policy Plan 14: 103-114. [Ref.]
23. Lavis J, Hammill AC, Gildiner A, RJ McDonagh, MG Wilson, et al. (2005) A systematic review of the factors that influence the use of research evidence by public policymakers: Report submitted to the Canadian population health Initiative. Hamilton, Canada.
24. Glasziou P, Djulbegovic B, Burls A (2006) Are systematic reviews more cost-effective than randomised trials? Lancet 367: 2057-2058. [Ref.]
25. Wilsdon T, Serota A (2011) A comparative analysis of the role and impact of Health Technology Assessment. Charles River Associates, London. [Ref.]
26. Bunn F, Trivedi D, Alderson P, Hamilton L, Martin A, et al. (2015) The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research. Health Technol Assess 19: 1-99. [Ref.]
27. Elias FT, Silva EN, Belfort R Jr, Silva MT, Atallah ÁN (2015) Treatment Options for Age-Related Macular Degeneration: A Budget Impact Analysis from the Perspective of the Brazilian Public Health System. PLoS One 10: e0139556. [Ref.]

Download Provisional PDF Here

 

Article Information

Article Type: Research Article

Citation: Elias FTS, Atallah AN (2016) The Role of Systematic Reviews as Part of DecisionMaking Process: Descriptive Study of the Brazilian Experience. J Epidemiol Public Health Rev 1(5): doi http://dx.doi.org/10.16966/2471-8211.127

Copyright: © 2016 Elias FTS, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Publication history: 

Received date: 26 May 2016

Accepted date: 31 May 2016

Published date: 03 Jun 2016