Criteria for Assessing the Quality of Health Information on the Internet

APHA Action Board Implementation Plan
Resolution #200031 Submitted September 19, 2002

Summary
The Action Board "Health Information on the Internet Work Group" has prepared an implementation plan, described below, for Resolution #200031: Criteria for Assessing the Quality of Health Information on the Internet (Attachment 1). The resolution was rated among the top five resolutions in 2000 by the APHA Governing Council. The implementation plan is built on the premise that APHA should take a leadership role in promoting quality standards for health information on the Internet. Not satisfied with merely creating the plan, the Work Group took steps to implement key parts of it. To educate APHA members on this important issue, member Karyn Pomerantz submitted a session abstract which was accepted for the 2002 meeting entitled "Providing Health Information to the Public". The session (#3145.0) will occur on Monday at 12:30pm in the Technology Center. The Work Group wrote an article entitled "Evaluating Health Information Websites" (Attachment 2) intended for placement on the APHA website and wide distribution to APHA partners (see Stakeholder list below). The article identifies several Web-based tutorial programs in English and Spanish that should be helpful to seekers of health information. The editor of the Nation's Health has agreed to publish an article about the topic and the Work Group's report in an upcoming issue, hopefully the November issue. The Work Group also encourages APHA to join with other stakeholders to advocate quality standards for health information provided on the Internet.

    Background Information
  1. Resolution - pp.38-39
  2. Review existing criteria and other information on the following websites:
    www.hon.ch and http://hitiweb.mitretek.org/hswg/;
    Needs Identified
  1. To develop and widely disseminate brief and understandable evaluation criteria to seekers of health information on the Internet (general audiences and specific populations such as older adults, people with medical conditions, etc.).
  2. To investigate technological options for communicating criteria to the public and those that provide health information.
  3. To identify partners who will join APHA in promoting quality standards for health information on the Internet.
  4. To identify topics/issues where quality health information is not available and work with partners to fill these gaps so that people searching for quality health information have something to find.
    Stakeholders
  • Community Technology Centers (funded by the Commerce Department and HUD)
  • US Agricultural Extension Service and 4H Centers
  • Library organizations, such as the Medical Library Association, American Library Association
  • AARP
  • Medical and nursing societies
  • State and local departments of Health
  • Primary Care Associations (organized by state)
  • Neighborhood/Community Health Centers
  • Consumer Advocacy groups (both national and geographically/demographically specific)
  • School Librarians, Health Educators, Nurses, and Counselors (much Internet access is available through schools)
  • Webmasters of large directory sites, search engines and health sites
  • Internet service providers or home computer resellers
  • Health institutions
  • Health resource centers
    Deliverables
  1. APHA session "Providing Health Information to the Public" (Karyn Pomerantz)
  2. Selected "exemplary links." See Attachment A.
  3. Newsletter article for wide dissemination and education
  4. An electronic curriculum on evaluating health information on the Internet.
  5. A take-away (4x6 laminated card, wallet size card, clip out listing) for APHA members attending the conference - funding needed.
    Dissemination Plan
  • APHA members at annual meeting
  • APHA partner organizations' communication channels
  • The Nation's Health
  • The APHA website
  • Other stakeholder organizations
  • Internet service providers
  • Health newsletters
  • Health information centers

Resources

Eysenbach, G., & Diepgen, T. (1998b). Towards quality management of medical information on the internet: Evaluation, labelling, and filtering of information. British Medical Journal, 317 (7171), 1496-500.

Jadad, A. R., & Gagliardi, A. (1998). Rating health information on the Internet. Journal of the American Medical Association, 279 (8), 611-614.

Kim, P., Eng, T. R., Deering, M. J., & Maxfield, A. (1999) Published criteria for evaluating health related web sites: review. British Medical Journal, 318 (7184), 647-649.

http://mlanet.org/resources/medspeak/topten.html - General criteria for the selection of these sites are listed. Criteria include: "credibility, sponsorship/authorship, content, audience, currency, disclosure, purpose, links, design, interactivity, and disclaimers." Also, top ten health resources on the Internet.

www.cancerlinks.org - Has a tutorial in English and Spanish about how to use the Internet.

www.lib.berkeley.edu/TeachingLib/Guides/Internet/FindInfo.html - The Berkeley Library has a Web tutorial and a evaluating Web sites section and how to cite Internet and electronic sources.

Workgroup Participants (see Attachment 3 for complete contact information)

Elizabeth Howze, Chair
Louise Villejo, Co-Chair
Kristine Alpi
Michael Barnes
Cynthia Bauer
Jill Becker
Kelly Brumbaugh
Maia Dock
Joan Dzenowagis
Karen Fowler
Kathye Gorosh
Sallie Craig Huber
Karyn Pomerantz
Karen Richard
Michael Slater
Virginia Sublet

Attachment 1

Resolution 200031: Criteria for Assessing the Quality of Health Information on the Internet

The American Public Health Association,

  • noting that in 1998 more than 22 million Americans went to the Internet for health-related answers, and expecting this number to grow to 33.5 million in 2000;1 and
  • Finding that nearly 70 percent of those searching for health care information on the Internet did so before visiting a doctor's office;2 and
  • Realizing that the Internet can be a valuable resource for users seeking health information and presents a powerful mechanism for helping users improve their health care decision-making by providing easy and rapid access, exchange, and dissemination of enormous amounts of health information; and
  • Acknowledging that the quality of health information is critically important, as it could potentially affect health outcomes for millions; and
  • Understanding that the quality of health information on the Internet is extremely variable and difficult to assess; and
  • Recognizing that health information is proliferating on the Internet and that there is a growing need for objective, reproducible, widely accepted criteria that can be used to evaluate the quality of the information; and
  • Building on APHA's long-standing commitment to accurate health education (Resolution 7320 and Position Paper 7742), especially on controversial topics (Resolution 85248) on APHA's long-standing belief in preventing health fraud through provision of accurate health information (Resolution 88139), and on APHA's long-standing concern with the role of the electronic media health education (Resolutions 520210 and 762211); and
  • Aware that as of June 24, 1999, the United States Federal Trade Commission had found 800 Internet sites containing inaccurate health claims, and undertook settlements with four businesses accused of deceptively marketing health products on the Internet; and
  • Noting that incidents relating to health-related websites demonstrated the blurring of lines between objective information, advertising, pro-motional content, and proper disclosure;3 and
  • Highlighting the time-sensitive nature of this issues; and
  • Understanding that users must be made aware of the potential for misinformation and recognize the critical need to assess the quality of the information provided; and
  • Discerning that the choice of appropriate evaluation criteria for the information is crucial and that no uniform guidelines for quality assessment of Internet-based health information existed until the Health Summit Working Group produced the policy paper,4 Criteria for Assessing the Quality of Health Information on the Internet;5 and
  • Recognizing that the Health Summit Working Group selected, defined, ranked, and evaluated seven major criteria for assessing the quality of Internet health information: credibility, content, disclosure, links, design, interactivity, and caveats (advisories); and
  • Observing that content providers must be encouraged to develop and post high-quality information, and policymakers and health care professionals must be educated on this important health issue;

Therefore,

  1. Urges individuals and organizations to be-come involved in promoting the application of appropriate criteria for assessing health information on the Internet, whether it be
    • Educating consumers and health care professionals on how to evaluate Internet health information,
    • Using criteria to develop their own Internet health site, or
    • Promoting the use of educational tools to help assess quality;
  2. Supports continued FTC monitoring and enforcement of fraudulent health claims and deceptive health marketing on the Internet;
  3. Endorses the development, publication, and dissemination of criteria for evaluating Internet health information that address such items as those defined by the Health Summit Working Group:
    • Credibility: includes the source, currency, relevance/utility, editorial review process for the information, and financial disclosure,
    • Content: must be accurate and complete, and appropriate disclaimer provided,
    • Disclosure: includes informing the user of the purpose of the site, as well as any profiling or collection of information associated with using the site,
    • Links: evaluated according to selection, architecture, content, and back linkages,
    • Design: encompasses accessibility, logical organization (navigability), and internal search capability,
    • Interactivity: includes feedback mechanisms and means for exchange of information among users,
    • Caveats: clarification of whether the site's primary function is to market products and services or to serve as a primary information content provider.

References

1. Davis R. Miller L. Net Empowering Patients, USA Today. July 14, 1999.

2. Brown MS: Healthcare information seekers aren't typical Internet users. Medicine on the Net. February 1998; 4(2):7-18.

3. Noble HB: Hailed as Surgeon General, Koop Criticized on Web Ethics, New York Times. September 4, 1999.

4. Members of the Health Summit Working Group include: Helga Rippen, MD, PhD, MPH, Health Information Technology Institute, Mitretek Systems; Roger Guard, University of Cincinnati and Association of Academic Health Sciences Libraries; Marshall Kragen, JD; Patricia Byrns, BSN, MD, University of Colorado Health Sciences Center; William Silberg, Journal of the American Medical Association; Denise Silber, MBA; et al.

5. Criteria for Assessing the Quality of Health Information on the Internet http://hitiweb.mitretek .org/docs/olicy.html, last visited November 3, 1999.

6. Res 7320: Increased Efforts in Health Education.

7. PP 7742: Toward a Policy on Health Education Public Health.

8. APHA Resolution 8524: Support for Accurate Public Information on Abortion.

9. APHA Resolution 8813: Nutrition Fraud and AIDs.

10. APHA Resolution 5202: Use of Television in Health Education.

11. APHA Resolution 7622: Television and Health.

Attachment 2

Evaluating Health Information Web Sites*
Health information is one of the most popular topics on the Internet. People search for answers to many health questions, including "Why don't I feel well?" "What are these symptoms?" "Which treatment is the best and safest for me?" "How can I stay healthy and eat right?" "How do I lose this weight?"

Reading different points of view of the Internet can help us form questions to ask our health care providers and lead us to new solutions. However, we all need to make sure the information we use to make decisions about our health-and our family's health-is reliable. For example, we need to make sure the information does not advise us to try harmful medicines or follow unhealthy diets. The problem is, anyone can post health information on the Internet. There are ways to protect yourself and increase your chances of finding reliable information. You can look for organizations you trust that sponsor Web sites. You can look at more than one Web site and read different points of view.

Here are some easy, basic questions to ask yourself. Use them to help you decide if the Web site is one you can rely on for accurate and useful information.

    Who
  • Does the Web site sponsor provide or write the information? The sponsor should be someone you consider a reliable source.
  • Does the site have names and contact information for the sponsors or authors?
  • Does the sponsor or author have a health background? Look for their credentials (degrees, work experience, books they've written). Are they telling a personal story, or is it hard to tell?
  • Does the sponsor or author seem to be trying to sell you something?
  • Has someone you trust recommended this site? Have you talked with a health professional about this site or about information you've obtained from this site?

*This article is intended to be a drop-in for newsletters, fliers, Web sites, or other communication materials. It can also be the basis for developing more extensive guidance for Internet information seekers. The article was developed by the APHA Action Board Work Group on "Criteria for Assessing the Quality of Health Information on the Internet." The listing of Web sites does not imply endorsement of any particular Web site by the American Public Health Association.

    What
  • Does the Web site get its information from reliable sources? Does it make outrageous claims, or is it hard to tell?
  • Does it provide references for the information? Is it someone's opinion, or is it hard to tell?
  • Does it include a broad range of information (both the pros and cons)?
  • Is there information about different ways to improve your health? You may need to look at more than one Web site to support the information you've found, or to learn about different ways to improve your health.
    How
  • Is the information well organized and presented?
  • Is it easy to navigate and easy to read? Does the Web site use words you understand? Will you be able to ask someone to explain the information to you if you don't understand it?
  • Does it protect your privacy? Can you tell how? Can you search for information without giving personal information?
  • Can you follow the links? Do they work?
  • Are there useful definitions or pictures to help make the information clear?
    When
  • Can you tell when the information was written and the site updated? Health information frequently changes.
  • Is the information current and appropriate?

A Quick Checklist: What To Look For

  • A clear purpose
    ***If you cannot check off the majority of these items, you may want to think twice about using this Web site and look for another source.***
  • Names of the sponsors and authors of the information
  • Useful, easy to understand information
  • Specific answers to your questions
  • Links that work
  • Dates to make sure the information is current
  • Privacy protections

Health Information Links

Following is a select list of Web sites that contain links to health information on a wide variety of health topics:

The Centers for Disease Control and Prevention (www.cdc.gov) provides health topics and travel information for the public. English and Spanish.

The National Cancer Institute (http://cis.nci.nih.gov/fact/2_10.htm#1) provides a style sheet on evaluating information online.

The Environmental Working Group (www.ewg.org) publishes reports on the environment that government sites might not include.

Family Doctor (http://familydoctor.org) is produced by the American Academy of Family Physicians and offers flow charts to help you make decisions about your health.

Healthfinder (http://healthfinder.gov) is produced by the U.S. Department of Health and Human Services and links to carefully selected information and Web sites from academic, government, and nonprofit health-related organizations. English and Spanish.

MEDLINEplus (http://medlineplus.gov) is produced by the National Library of Medicine and links to information on health topics, drugs, dictionaries, and resources. English and Spanish.

The New York Online Access for Health site (known as NOAH; www.noah-health.org) is hosted by the New York Academy of Medicine for a partnership of New York libraries working with librarians nationwide. It links to a broad range of health topics and resources. English and Spanish.

http://mlanet.org/resources/medspeak/topten.html - General criteria for the selection of these sites are listed. Criteria include: "credibility, sponsorship/authorship, content, audience, currency, disclosure, purpose, links, design, interactivity, and disclaimers." Also, top ten health resources on the Internet.

www.cancerlinks.org - Has a tutorial in English and Spanish about how to use the Internet.

www.lib.berkeley.edu/TeachingLib/Guides/Internet/FindInfo.html - The Berkeley Library has a Web tutorial and a section on evaluating Web sites.

Attachment 3

Criteria for Assessing the Quality of Health Information on the Internet Work Group Contact Information

Elizabeth (Libby) Howze, ScD, CHES (Co-chair)
Director, Division of Health Education and Promotion
Agency for Toxic Substances and Disease Registry
1600 Clifton Road, MS-E33
Atlanta, GA 30333
404-498-0101 (phone)
404-498-0062 (fax)
eah6@cdc.gov

Louise Villejo (Co-chair)
Director, Patient Education Office
M. D. Anderson Cancer Center
1515 Holcombe - 21
Houston, TX 77030
(713) 792-7128
(713) 794-5379 FAX
LVillejo@mdanderson.org

Kristine Alpi, MLS
Information Services Librarian
Samuel J. Wood Library
Weill Medical College of Cornell University
1300 York Ave., New York, NY 10021
(212) 746-6086 voice
(212) 746-8364 fax
kma2002@med.cornell.edu

Michael Barnes, PhD, CHES
Associate Professor of Community Health
Department of Health Sciences
213 Richards Building, Box 22080
Brigham Young University
Provo, UT 84602
Phone: 801.378.4428 or 801.378.3327
Fax: 801.378.4388
e-mail:
michael_barnes@byu.edu

Cynthia Baur, Ph.D.
Senior Policy Advisor, ehealth
Office of Disease Prevention and Health Promotion
U.S. Department of Health and Human Services
200 Independence Ave, S.W. Rm 738G
Washington, D.C. 20201
202-205-2311 (t)
202-205-0463 (f)
cbaur@osophs.dhhs.gov

Julie Becker
jbecker@astro.ocis.temple.edu

Kelly Brumbaugh
Division of Reproductive Health
National Center for Chronic Disease Prevention and Health Promotion
Centers for Disease Control and Prevention
4770 Buford Highway, NE, MS K-21
Atlanta, GA 30341
KBrumbaugh1@cdc.gov
770-488-6344 (phone)

Maia Dock
maiamatija@hotmail.com

Dr Joan Dzenowagis
Project Manager, Health InterNetwork
World Health Organization
20 Ave Appia
CH-1211 Geneva 27
Switzerland
tel +41 22 791 2504
fax +41 22 791 4806
eMail dzenowagisj@who.int

Karen Fowler, MPH
National Institutes of Health
NIDCR
Office of Health Communication and Education
45 Center Drive
MSC 6400
Bethesda, MD 20892
Phone: 301-594-5388
Fax: 301-496-9988

Kathye Gorosh, Executive Director
The CORE Foundation
2020 West Harrison
Chicago, IL 60612
Phone: 312-572-4765
Fax 312-572-4771
kgorosh@corecenter.org

Sallie Craig Huber
Principal Program Associate
Management Sciences for Health
165 Allandale Road
Boston, MA 02130
Phone: (617) 524-7799
Fax: (617) 524-2825
shuber@msh.org

Karyn L. Pomerantz, MLS, MPH
Distance Education Program
GW School of Public Health & Hlth Svcs.
2300 I St., NW, #202; Washington, DC 20037
202-994-2976 (phone), 202-994-9867 (fax)

Karen Richard-Lee, MPA
Public Health Advisor
Division of Cancer Prevention and Control
National Center for Chronic Disease Prevention and Health Promotion
Centers for Disease Control and Prevention
4770 Buford Highway, NE, MS K-64
Atlanta, GA 30341
krichard@cdc.gov
770-488-4737 (phone), 770-488-4760 (fax)

Michael D. Slater, Professor
Journalism and Technical Communication
Psychology (joint appt.)
C229 Clark Building
Colorado State University
Fort Collins, CO 80523
email:
michael.slater@colostate.edu
http://lamar.colostate.edu/~slaterjt
phone: 970-491-5485
fax: 970-491-2908

Virginia Sublet, Ph.D.
Senior Toxicologist
Oak Ridge Institute for Science and Education
8436 Woburn Court
Windermere, Florida 34786
(407) 909-4742
sublet@ix.netcom.com