Assessing Infertility Information on the Internet: Challenges and Possible Solutions

  by Yakov M. Epstein, Ph.D.[a] [b]  and Helane S. Rosenberg [c]

Capsule:

Patients visiting consumer-related internet sites often have no clear way to evaluate content within these sites.  This article reviews the problem and offers suggestions to remedy the situation.

Abstract

Although patients frequently visit consumer-related internet sites to help them make informed decisions about treatment, there is no clear way for these consumers to evaluate content within these sites.  This article reviews the problem and offers suggestions to remedy the situation.

            Over the past decade, there has been rapid growth in the use of the internet by American adults. Around 1997,  the proportion of the U.S. adult population, (22.1%) accessing the internet became substantial. Since 1997 the proportion of adult internet users in the U.S. has increased markedly:  22% in 1997, 40% in 2000, 59% in 2002, and 78% in 2003. (1-2).

Around 1997, when internet usage began to approach one quarter of the US adult population, concern about the quality of health information available on the internet  was voiced and became the subject of numerous publications. In 1997 the editor of JAMA and colleagues published an editorial raising these concerns and advocating the creation of procedures that would enable users to ascertain the quality of online information in a manner comparable to ascertaining the quality of print information (3).  This call for action was followed by the publication of AMA guidelines in 2000 (4) .  These guidelines offer principles in 4 areas: content, advertising/sponsorship, privacy/confidentiality, and e-commerce. 

In this paper, we want to concentrate on the first of these areas – content.  In our view, as internet access has become more widespread and more frequent, consumers have become much more sophisticated in their understanding of potential dangers associated with loss of privacy, unethical use of advertising or unlawful e-commerce schemes – the other 3 areas addressed by the AMA guidelines.  So though the AMA concern about addressing these matters is laudable, it is not nearly as timely as it was when the guidelines were established.  Concerns about content, on the other hand, are more important than ever as more and more people turn to the internet to find this information.

The Huang group (5) studied private practice websites which they compared with hospital/university affiliated sites.  Interestingly, most consumers do not go to either of these types of venues when looking for infertility information or support.  A Google search on the term “infertility” conducted in mid September, 2004, reveals that of the top ten sites located only 1 was to a hospital related site and 1 was to private practice website.(6) The rest were to organizations such as the International Council on Infertility Information Dissemination (INCIID), RESOLVE, the American Infertility Association (AIA) and similar consumer-related sites.

The AMA guidelines for content highlight 8 areas of concern. (4)  Curiously, only one of these 8, “quality of editorial content,” goes to the heart of the issue of information which might confuse or mislead a consumer. The other 7 (site ownership, site viewing, view access payment and privacy, funding and sponsorship, linking, intersite navigation, downloading files, and navigation of content) are issues which most consumers can handle on their own without having to rely on the AMA’s assistance to ensure that they have a high quality experience.  When it comes to awarding value to an online health resource, content is truly king.

The problem, however, is that the AMA guidelines offer no practical way for consumers to know which content to cherish and which to discard. For example, the AMA recommends and we agree that the posted content should have been reviewed. But how is a consumer to know whether it has been reviewed and if it has been reviewed how credible is the review panel?

Huang et al used the AMA criteria to evaluate private and hospital-affiliated infertility websites and reported that “the overall quality fails to meet the American Medical Association (AMA) Internet health information guidelines.” (5)  To ascertain the ideal of how a website should be constructed and how quality content should be made available, we visited the AMA web site (http://www.ama-assn.org/).  We navigated to the patient section and then to the MEDEM patient information resource where we visited the “Life Stages” section and then, more specifically, the “women’s health” section and finally the “pregnancy and fertility issues” section.  We then applied Huang et al’s criteria to evaluate this site. (5) We invite readers to try this themselves, perhaps using one of the other patient areas of MEDEM.

Ownership and affiliations could be found by clicking on “About Medem (1 point). Copyright information was present (1 point.)  Restrictions on access to content were available by clicking “terms of service” (1 point). There was information about “secure pay” (1 point).  Privacy notices were easy to find (1 point). Funding information is ambiguous.  The FAQ states “Revenue for Medem comes from traditional sponsorships, e-commerce, licensing subscriptions and transaction fees” but it does not indicate whether the content that appeared had any particular sponsor. As to the final criteria, ability to distinguish content from advertising, Huang’s guidelines did not permit us to enter a score of 1 or 0.  There were several ads on the content page such as “click here to order Family Health Supplies.”  We had no way to know whether a consumer would believe this was part of or separate from the content.  Overall, however, the site earned a good score for this area (5 or 6 out of 7).

A second area that Huang’s group evaluated was navigation. (5)  Navigating the site was easy.  There was no information about the optimal browser or platform. The internal links all worked (1 point) as did the external links (1 point.). Medem permitted us to return to previously viewed sites (1 point) and did not redirect us to unintended sites (1 point.)   There was no information about PDF software but neither were any of the documents in PDF format.  Do we award a point or not? The Huang group offered no guidance on this matter. There was a site map (1 point) and an FAQ page (1 point). There was a “contact us” link to provide feedback (1 point).  There was no specific customer service mechanism (but we might possibly award a point because we could contact the site so it was not clear whether 1 or 2 points should be awarded.) There was a search engine (1 point).  All in all navigation would earn a good score (at least 8, although it would be hard to enter this score into a statistical analysis since Huang’s criteria for evaluation described in his paper were too ambiguous to award a distinct score.)

The real difficulty for consumers, however, involved an evaluation of content. The site had relatively little information and none of it dealt with fertility issues.  There was no information about peer review for the posted content. Language was not complex (1 point). There was no description of the editorial process.  There were no posted names of staff members responsible for content. There was no date for the posting of this content. There were no author bylines or organizational affiliations indicated. No financial disclosure information was posted.  Finally, no references were cited. Using the Huang group’s criteria, the AMA’s own consumer information site earned a poor score, a score of  1 out of 8 for editorial content.

Of greater concern, however, was the fact that nowhere on the page of this organization which published guidelines, was there any information about how a consumer could evaluate the quality of the content she found there.  Near the bottom of the page, in typically small print was a heading called “medical disclaimer.”  When we clicked on it we saw the following warning: Rely at Your Own Risk: RELIANCE ON ANY INFORMATION ON THIS WEB SITE IS SOLELY AT YOUR OWN RISK. Hardly a guideline to help consumers assess the quality of the content on this site!

Thus using the Huang group’s criteria, the AMA earns good scores for ownership and affiliation and for ease of navigation but a failing grade for evaluating the content posted. This is not at all surprising nor does it reflect a failing on the part of the AMA.  It is easy to evaluate navigation and information about privacy and ownership.  It is a much more difficult and complex problem to evaluate content.  But evaluation of content and not ease of navigation is how these sites should be judged.

Even more to the point, the AMA guidelines don’t deal with an even more problematic issue of online access: discussion groups.  In 1998, we conducted a survey of internet consumers looking for information and support on the internet.(7)  The 589 respondents we studied spent approximately 1½  hours/day searching for information and support in various interactive internet forums.  Some of these venues were moderated by physicians but others involved only lay participation. Some of the information imparted by physicians was most likely accurate but it is difficult to know how accurate the information posted by lay participants was.  Nonetheless, these consumers considered the information they received extremely valuable. Approximately  45% believed they got valuable medical information and 40% reported that this information helped them deal better with their physicians. Nearly 1 in 4 said their participation influenced them to switch from an OB/GYN to an RE.  Approximately 1 in 5 stated that they decided to ask for a test or procedure they had never had before because of the views of participants in these forums. We have no way of ascertaining how good the information was that influenced these decisions or whether the decisions were appropriate or helpful in each case. On the other hand, we do know of a patient whom we counsel at the Fertility and Gynecology Center where we consult who insisted on pursuing a course of action contrary to what the physicians told her based solely on poor advice she received from members of a discussion group.  Following their advice she had a poor cycle outcome and later on lamented her decision to follow the forum advice rather than the counsel of her doctors.

There probably is no way to develop a system to evaluate these discussion forums. One could state that a forum moderated by a physician is superior to a lay run group.  Discussion group guidelines similar to those which the AMA created would be welcomed and probably even more important than they are for static content.  We should know whether a group is moderated by a physician, the physician’s background and training, experience with various procedures, information about privacy and how to contact the physician to share information the patient does not want to share with the public, whether the physician has any financial relationship with the organization sponsoring the discussion group or any links to a pharmaceutical group that might influence the suggestion of a particular medication.

One possible mechanism for enhancing interactive information would be to have ASRM sponsorship of discussion groups operated from the ASRM website using a rotating group of moderators.  Physicians could volunteer to moderate these sites and their qualifications could be ensured by an ASRM group who would evaluate their credentials.

As to the evaluation of static content, numerous articles have attested to the difficulty of creating any reliable evaluation tool that could be used by a consumer to evaluate the content of a site.(8,9,10,11)  In the UK, the NHS has funded the DISCERN project which developed an instrument designed to enable consumers to judge the quality of written health information.(12) Extensive testing of this instrument revealed that it could be used reliably by health professionals but consumers were unable to use it effectively. It was suggested that perhaps if consumers received special training they could use this instrument but we believe such an approach is impractical.  

Evaluating static content can be done through filtering.  Two approaches are possible: downstream filtering and upstream filtering. (8)  Downstream filtering is the process by which consumers use criteria suggested by experts to evaluate the content themselves.  As the DISCERN project has demonstrated, this approach is unlikely to work. Upstream filtering, on the other hand has experts reviewing the content and evaluating it. The result of such an approach which has been suggested by several people could be the creation of a “trustmark” mechanism.(13) This mechanism would amount to a seal indicating that the material meets certain standards of accuracy and appropriateness that have been established by this ASRM group of professionals.  Rather than placing the onus on a layperson to evaluate the content of a site, the consumer could simply look to see whether the content has earned an ASRM trustmark.  An ASRM assembled group could be charged with assigning trustmarks to content. Persons wishing to have the ASRM trustmark associated with some item of content would submit the material to this group for evaluation and the award of the trustmark. 

While trustmarking could help, it would still be an imperfect solution.  It is difficult to know, for example, in what context the user encounters information.  Some information, if read out of context can be misleading. (8)  Ultimately, there is no substitute for consumer education provided by physicians. The time is ripe for the ASRM, perhaps in partnership with consumer organizations such as RESOLVE, INCIID, and AIA to launch a task force to study the problem of managing infertility information on the internet and proposing useful solutions.

References

1.         Newburger E. Computer Use in the United States. U.S. Census Bureau 1997;PPL-114:1-11.

2.         US-Department-of-Commerce. A Nation Online: How Americans Are Expanding Their Use of the Internet, 2002 Available at www.ntia.doc.gov/ntiahome/dri/index.htm

3.         Silberg WM, Lundberg GD, Musacchio RA. Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet: Caveant Lector et Viewor-Let the Reader and Viewer Beware. JAMA 1997;277:1244-1245.

4.         Winkler MA, Flanagin A, Chi-Lum B, et al. Guidelines for medical and Health Information Sites on the Internet: Principles Governing AMA Web Sites. JAMA 2000;283:1600-1606.

5.         Huang JYJ, Discepola F, Al-Fozan H, Tulandi T. Quality of fertility clinic web sites. Fertility and Sterility 2004.

6.         Google. available at www.google.com  Accessed on September 12, 2004

7.         Epstein Y, Rosenberg H, Grant T, Hemenway N. Use of the internet as the only outlet for talking about infertility. Fertility and Sterility 2002;78:507-514.

8.         Eysenbach G, Diepgen TL. Towards quality management of medical information on the internet: evaluation, labeling, and filtering of information. BMJ: British Medical Journal 1998;317:1496-502.

9.         Eysenbach G, Powell J, Kuss O, Ra E-R. Empirical studies assessing the quality of health information for consumers on the Word Wide Web. JAMA 2002;287:2691-2715.

10.       Gagliardi A, Jadad AR. Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination. BMJ: British Medical Journal 2002;324:569-573.

11.       Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ: British Medical Journal 1999;318:647-649.

12.       Charnock D, Shepperd S, Needhan G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology & Community Health 1999;53:105-111.

13.       Sheldon T. Trust mark launched as a guarantee of safety in the Netherlands. BMJ: British Medical Journal 2002;324:567.

---

[a]  Center for Mathematics, Science, and Computer Education, Rutgers – The State University of New Jersey, Piscataway, NJ

[b]  Correspondence: Center for Mathematics, Science, and Computer Education, Rutgers – The State University, 118 Frelinghuysen Rd. Piscataway, NJ 08854-8019  email: yepstein@rci.rutgers.edu

[c]  IVF New Jersey Fertility and Gynecology Center, Somerset, NJ