Penny Brohn Cancer Care - Quality of information

What do evaluation instruments tell us about the quality of complementary medicine information on the Internet?

Introduction

Up to 55% of Internet users access health-information 1. Several studies have highlighted problems with the quality of information. Searching for information on complementary therapies may be particularly challenging with reports that adherence to advice from internet sources has led to serious consequences ².

Several strategies have been designed to help information seekers obtain good quality information. These include ‘Codes of Conduct’, which give a seal of approval to sites adhering to a list of guidelines. ‘Gateways’ are collections of pre-approved sites which have been deemed as being of good quality by a governing organisation. In addition to these, many instruments have been designed to enable an internet user to rate the quality of health information websites. These range from simple checklists to detailed documents offering instruction on how to rate information.

Aims

To review available evaluation instruments and assess their performance when used by a researcher to evaluate websites containing information on complementary medicine and breast cancer. In particular, we wanted to see if instruments used the same criteria, agreed on the ranking of web sites, were easy to use by a researcher, and if use of one tool was sufficient to assess website quality.

Methods

Literature search for assessment instruments
Medline, AMED, BNI, CINAHL, Embase and PsychInfo were searched using the terms: ‘[evaluat* OR assess* OR rating OR rat* OR ranking OR rank* OR quality OR criteria] AND [website* OR world wide web OR Internet]’.

Search for 12 complementary medicine websites
To search for web sites to be assessed, a search for ‘Complementary (Medicine OR therapies) AND Breast Cancer’ was performed using the Google search engine in February 2007. The first 6 results and a purposive six from different origins (charities, sponsored, commercial) were chosen.

Assessment of websites

The twelve websites were each evaluated using each of 12 assessment instruments. Each site was given a mark using the individual scoring system for each instrument, which was then converted to a percentage score.

Comparison of assessment instruments
The range of criteria contained in each instrument was compared to nine main criteria identified by a previous study.
Agreement between instruments was assessed by a correlation matrix using Spearman rank correlation on the instruments’ rankings of the 12 websites.

Illustrative comparison of best and worst sites
Statements found on the site ranked best by the sum of the 12 instruments were compared to those on the worst site.

Table 1
	Method of scoring
WEB FEET HEALTH Collection Criteria for Site Selection	24 statements to be agreed or disagreed with: Score Range 0-24
HONcode principles	8 desirable properties of sites containing health information: Score Range 0-8
Emory University School of Public Health - Health-Related Web Site Evaluation Form	36 questions about the site which gain differing amounts of points depending on whether they are disagreed with (+1), agreed with (+2) or N/A (0): Score Range 0-72 although total varies if questions are not relevant to site
University of Michigan – Website Evaluation Checklist	43 questions, each weighted and has a selection of answers which give a variety of positive and negative scores: Score Range: -80 - +80
Kellogg Library Criteria (University of Dalhousie)	31 questions and statements for the user to agree or disagree with: Score Range: 0-31
DISCERN - Quality Criteria for Consumer Health Information	16 questions which each have a 5-point visual analogue scale covering answers from ‘No’ to ‘Yes’. Score Range:16-80
NCCAM Basics	10 questions followed by an explanation of how to answer each one. Score Range: 0-10
U.S. Pharmacist Tool	15 questions with Yes/No answers. Score Range: 0-15
Minervalidation tool	A semi-automated tool which requires the URL of the site being assessed. Drop down menus to answer questions of content and usability, a rating is automatically calculated. Score Range: 0-100%
Nicoll, L.H. (Authors’ Guidelines)	A mnemonic (PLEASED) which represents Yes/No questions followed by author justification of the importance of each.. Score Range: 0-9
Silberg et al (Authors’ Guidelines)	4 items in checklist of authors’ ‘Core Standard’ which sites containing health information should meet. Score Range: 0-4
Sandvik score	7 questions with 3 options which achieve scores of 0-2. Score Range: 0-14

Results

Assessment instruments available

39 instruments were located, 12 of which met the inclusion criteria (Table 1).

Cancer web sites sampled

4 charity sites were rated by the instruments, 2 commercial sites, 3 sites offering cancer treatment, 1 funded by advertising, 1 funded by sponsorship and 1 run by a network of health professionals.

Agreement between instruments
There was high correlation between the majority of instruments i.e. using either would give a similar result. Honcode and Webfeet seemed to assess different characteristics to the other instruments.

Comprehensiveness

There was no general consensus of quality markers for web sites between the instruments.

Comparison of statements found on the ‘best’ and ‘worst’ sites

Table 2 compares statements found on the highest and lowest ranked sites.

Table 2
Best site (Imaginis)	Worst site (Alternative Cancer)
‘While anecdotal evidence reveals that many alternative or complementary medicines may be beneficial to patients, extensive research is still needed to determine whether non-traditional medicines are truly effective.’	‘Proven Therapies. Includes a list of successful, long-standing alternative treatments from around the world going unused by the conventional medical system. There is one reason they are the oldest - in the hands of experienced practitioner they work! For example: the very successful nutritional based Gerson therapy. It has been used by untold thousands of people worldwide for over 50 years.’
‘Chinese herbs have been shown to lessen the side effects of chemotherapy and acupuncture has been shown to reduce nausea (a possible side effect of chemotherapy and other drug therapies).’ ‘Not all alternative or complementary medicines are safe.’	‘Every day worldwide, quietly behind the scenes, there are over 100 proven alternative therapies used successfully against cancer. (Get a list of the 78 most popular below) The problem is, nobody bothers to tell the public. Plus, conventional cancer doctors (MD Oncologists) are not taught anything about them in medical schools. This must change!’
‘Not all alternative or complementary medicines are safe.’	‘The one true secret to success: There are six basic types of proven alternative cancer treatments, and you must use them all together.’
‘In a recent study published in the Journal of the National Cancer Institute, researchers found that advanced breast cancer patients with high stress levels were less likely to live as long as patients who coped well with stress.’	‘Anvirzel® : A new weapon against cancer and AIDS from Ozelle Pharmaceuticals - a herbal extract which is nontoxic and causes no adverse side effects. Closed clinical trials are showing that the drug is especially effective against prostate and breast cancer. The materials of the company promoting Anvirzel. say that Dr Ozel treated 494 cancer patients with the extract, resulting in a high rate of success. The company has organized phase I and II trials in , and states that the trials confirmed the efficacy of the extract in cancer. They say the patients were improved in their quality of life as well as regression of cancer, while reporting no notable side effects. Best results were said to be in prostate, lung and brain cancers. Sarcomas showed stabilization.
‘Some preliminary studies have shown that vitamins may help reduce risk of breast cancer or treat the disease.’	‘Artemisinin: A Chinese herb, sweet wormwood (qinghao in Chinese). In test tube studies, breast cancer cell research resulted in a 28% reduction of breast cancer cells treated only with artemisinin, and an amazing 98% decrease in breast cancer cells within 16 hours that were treated with artemisinin and an iron-enhancing molecule, transferrin. These treatments had no significant effect on normal human breast cells. This research pointed to the involvement of free iron in the toxic effect of artemisinin toward cancer cells, while basically sparing healthy cells. ("Selective toxicity of dihydroartemisinin and holotransferrin toward human breast cancer cells," Life Sciences 70 {2001) 49-56.’

Conclusion

Although the instruments did not agree on what attributes of websites indicated good quality information, there was agreement between most as to which sites were best and worst. The comparison of the statements made by the websites suggest that the instruments have face validity. It is unclear from examining the criteria contained in the HONcode and Web Feet instruments why these ranked websites differently to the others.

Some instruments contained more than 30 questions and these were extremely time-consuming to apply to websites.

Instruments identified by several previous studies suggest that many instruments have a limited ‘shelf-life’ and often become unavailable. The most frequently cited instruments were felt the least likely to become obsolete. The HONcode, DISCERN and NCCAM instruments were cited most frequently and of these, HONcode did not agree with the other instruments and NCCAM was felt to be easier to use than DISCERN. It is not known to what extent these instruments are used by consumers but they could be useful for developers of health information websites and gateway providers.

References

Satterlund MJ, McCaul KD, Sandgren AK. Information gathering over time by breast cancer patients. J Med Internet Res 2003; 5(3):e15
Hainer MI, Tsai N, Komura ST, Chiu CL. Fatal hepatorenal failure associated with hydrazine sulfate. Ann Intern Med 2000; 133(11):877-80

For the complete publication visit http://www.jmir.org/2008/1/e3/HTML