Health Information and Publications Network (HIPNET) Home Page

Appendix 1

Map of Information Products and Services Indicators to USAID Results Framework Indicators

No.	Information Products and Services Indicator	IR 1.0: Global leadership demonstrated	IR 2.0: Knowledge generated, organized & communicated	IR 3.0: Support provided to the field
	REACH
	*Area 1: Primary Distribution (Push)*
1	Number of copies of a product initially distributed to existing lists		IR 2.3
2	Number of copies of a product distributed by a publisher through additional distribution		IR 2.3
	*Area 2: Secondary Distribution (Pull)*
3	Numbers of products distributed in response to orders		IR 2.3
4	Number of file downloads in a time period		IR 2.3
5	Number of times a product is reprinted by recipients		IR 2.3
6	Number of people reached by media coverage of the material or generated by it		IR 2.3
	*Area 3: Referrals*
7	Number of instances that products are indexed or archived in bibliographic databases		IR 2.3
8	Number of postings of products by other Web sites or links to products from other Web sites		IR 2.3
9	Number of instances that products are selected for inclusion in a library		IR 2.3
10	Percentage of users who share their copies or transmit information verbally to colleagues		IR 2.3
	USEFULNESS
	*Area 1: User Satisfaction*
11	Percentage of those receiving a product or service who have read or browsed it		IR 2.3
12	Percentage of users who are satisfied with a product or service		IR 2.3
13	Percentage of users who rate the format or presentation of a product or service as usable		IR 2.3
14	Percentage of users who rate the content of a product or service as useful		IR 2.3
15	Number/percentage of users who report knowledge gained from a product or service		IR 2.3
16	Number/percentage of users who report that a product or service changed their views		IR 2.3
	*Area 2: Product or Service Quality*
17	Number and quality assessment of reviews of a product in periodicals		IR 2.1 & IR 2.3
18	Number and significance of awards given to a product or service		IR 2.1 & IR 2.3
19	Number of citations of a journal article or other information product	IR 1.1	IR 2.1 & IR 2.3	IR 3.1
20	Journal impact factor		IR 2.1 & IR 2.3
21	Number/percentage of users who pay for a product or service		IR 2.3
22	Number/percentage of information products or services guided by theories of behavior change and communication		IR 2.3
	USE
23	Number/percentage of users intending to use an information product or service	IR 1.1	IR 2.1	IR 3.1
24	Number/percentage of users adapting information products or services	IR 1.1	IR 2.1	IR 3.1
25	Number/percentage of users using an information product or service to inform policy and advocacy or to enhance programs, training, education, or research	IR 1.1	IR 2.1	IR 3.1
26	Number/percentage of users using an information product or service to improve their own practice or performance	IR 1.1	IR 2.1	IR 3.1
	COLLABORATION & CAPACITY BUILDING
	*Area 1: Collaboration*
27	Number of instances of products or services developed or disseminated with partners	IR 1.2 & IR 1.3		IR 3.3
28	Number of instances of South-to-South or South-to-North information sharing		IR 2.3	IR 3.1 & IR 3.3
	*Area 2: Capacity Building*
29	Number and type of capacity-building efforts			IR 3.3

USAID Office of Population and Reproductive Health Results Framework

IR 1.0: Global leadership demonstrated in FP/RH policy, advocacy, and services	IR 2.0: Knowledge generated, organized and communicated in response to field needs	IR 3.0: Support provided to the field to implement effective and sustainable FP/RH programs
1.1 Tools, protocols, procedures, systems, methodologies, guides, curricula, indices and/or key actionable findings incorporated into the work of other organizations	2.1 Tools, protocols, procedures, systems, methodologies, guides, curricula or indices with demonstrated programmatic value validated, scaled up, and/or replicated in contexts other than where they were originally developed.	3.1 Contraceptive methods, tools, protocols, procedures, systems, methodologies, guides, curricula, indices, and/or key actionable findings incorporated into mission or country programs (incorporation may be core or FS-funded, bilateral, HC gov’t or other donor funded) or adopted/applied by other CAs
1.2 Resources leveraged globally for FP/RH activities from non-USAID sources by core or FS funds	2.2 Key actionable findings and experiences identified, generated, pooled, or summarized and their lessons extracted	3.2 Percentage of total contraceptive shipments that are shipped on time (within one month from the desired ship date)
1.3 Number of partnerships with organizations that do not traditionally focus on FP/RH	2.3 Target audiences reached with tools, protocols, procedures, systems, methodologies, guides, curricula, indices, key actionable findings (i.e., the products reported in 2.1 and/or 2.2)	3.3 Organizational capacity to undertake activity as measured on a continuum from: - implementing w/significant TA - implementing/replicating with limited TA - implementing/replicating independently - serving as a resource for others/leveraging resources
	2.4 Number of new and current contraceptive leads/methods under development or evaluation; advancing to the next stage; and/or approved by FDA	3.4 Number of missions using PRH-designed or PRH-managed mechanisms for program implementation (includes non-PRH projects that PRH co-manages)
		3.5 Ratio of field support to core funding within centrally funded projects designed to support the field, disaggregated by Pop FS/Pop core and all FS/all core
		3.6 TA visits by PRH staff to support field programs
		3.7 Percent of respondents based in missions that report being satisfied with TA provided by USAID/W and/or GH CAs

Appendix 2

Types of Documents and Publications and Their Characteristics

Document or Publication Type (Format, Length, Medium)	Purposes and Objectives	Illustrative Audiences	Examples
Project and technical reports (unpublished)	Fulfill reporting requirements (periodic report, trip report, progress report)	Donors	Quarterly, semi-annual, annual reports to USAID
Evaluation, evaluation report, or assessment study	Inform client organizations about successes and improvements needed, fulfill reporting requirements	Client organizations, CAs, donors	Population Council/ FRONTIERS: Final reports
Case studies	Serve as teaching or training materials, illustrate lessons learned	Health professionals, trainers, CAs	AED, Speak for the Child: Community Care for Orphans and AIDS-Affected Children, Case Study, Kenya
Technical update/note/brief, policy brief, research summary	Communicate technical or policy experience or research findings in short format for wide audience, sometimes for a specific country or region	Health professionals and managers worldwide or in- country, policymakers, CAs, donors	Population Council/ FRONTIERS: OR summaries; INFO Project: Global Health Technical Briefs; AGI: Facts in Brief, Issues in Brief, Research in Brief
Information, Education, and Communication (IEC) materials	Provide information to the public; health education, health communication and promotion	Clients, patients, communities	Poster or flier for local use; radio program, brochures on personal health and healthy behaviors
Success story or project final report	Illustrate partnerships and share successes with implementing organizations and donors	Donors, CAs, client organizations, health managers	FHI: Addressing Health Worker Shortages
Occasional paper, white paper, working/issue/ concept/position paper	Provide wide, low-cost access to discussions of timely topics	Professional/technical audiences	Population Council: Policy Research Division Working Papers
User's guide with CD-ROM or for software	Provide tools to implement specific public health or management functions and improvements; guide people in using software	Health workers, program or facility managers at multiple levels	MSH: Financial Management Assessment Tool (FIMAT); JSI: ProQ User's Manual
Interactive CD-ROM	Serve as an interactive learning/training tool and/or means for sharing large volume of information in a widely accessible format	Various audiences (dependent on subject)	URC: Health Journalism CD-ROM
Technical poster	Summarize data or state-of-the-art information for quick reference	Professional/technical audiences	Population Reference Bureau: World Population Data Sheet
Monograph, booklet (medium length, on a narrowly defined topic)	Communicate experience, success, lessons learned	Donors, CAs, senior level in client organizations; some academics, libraries, professionals	AED: Community Health Worker Incentives and Disincentives
Manual, booklet, handbook, guidelines, textbook	Offer practical information on clinical or public health management topics; “how to”; clinical protocols; checklists; provide comprehensive, up-to-date information and teaching and reference materials on public health topic	Health care providers and managers at multiple levels, trainers, CAs	EngenderHealth: COPE® Handbook: A Process for Improving Quality in Health Services; MSH: Managers Who Lead Population Council: HIV/AIDS OR handbook
Audiovisual materials	Various	Various	Audiotapes, videotapes, PowerPoint presentations, DVDs; URC: Everybody’s Business: Media, Youth, and Drugs
Training guide or curriculum	Provide training modules or curricula for on-site training or virtual learning	Trainers, CAs, client institutions	FHI: HIV/AIDS Care and Treatment: A Clinical Course for People Caring for Persons Living with HIV/AIDS
Compilation or anthology (CD-ROM or book)	Collect a large body of separately authored materials on one topic	Health professionals and senior managers worldwide, CAs, donors, libraries	Population Council/ FRONTIERS: Findings from OR in 30 Countries, 1998-2005 (CD-ROM)
Serial technical newsletter	Disseminate state-of-the-art information widely, provide continuing education materials	Health managers at multiple levels, trainers	PATH: Directions in Global Health; Futures Group: Maternal and Neonatal Program Effort Index; INFO Project: Population Reports; MSH: The Manager
Conference paper, poster	Communicate technical experience widely; establish CA as an authority on certain topics	Academics (including students), health professionals, CAs	Presentation at Global Health Council conference
Book, including conference proceedings	Synthesize experience or research on clinical topics or public health management and health areas or countries; offer a long view	Senior-level policymakers and managers, professionals, academics, libraries, CAs, and others	MSH: Community-Based Health Care
Journal, journal article	Communicate technical experience or research findings widely through professional (peer-reviewed) journals; establish publisher or author as an authority on certain topics	Academics (including students), researchers, libraries, health professionals and senior managers worldwide, CAs, donors	Population Council: Studies in Family Planning
Web site	Various, ranging from interactive learning to communicating technical experience widely. A site may serve as the medium to distribute virtually all of the materials listed above or contain original content available in no other medium.	All audiences	MSH: Provider’s Guide to Cultural Competence

Notes: With the exception of CD-ROMs, all of these may be online, printed, or both. Promotional materials, such as brochures, have been omitted. “Unpublished” means not offered for sale and not widely distributed. (Online materials are considered published.)

Compiled by Management Sciences for Health, Dec. 1, 2005.

Success Stories:

Using Selected Data Collection Methodologies

Maintaining a Client Database and Mailing List for Initial Distributions and Requests

In 1997 the Center for Communication Programs (CCP) at the Johns Hopkins Bloomberg School of Public Health transformed its large mailing list—built up since 1973—into a client database. The database now includes more than 115,000 contacts. CCP has collected information to better serve its customers, including interest categories, job responsibilities, and organization types. This has enabled CCP to match clients’ needs with resources. CCP maintains the accuracy of its records by employing advanced international and domestic address hygiene software and by directly contacting its clients for address updates. Electronic and printed order forms offer clients the opportunity to update their information.

The database also connects to requestors’ order histories, which contain fields such as publication name, quantity requested, and request notes. Taken together, these records enable CCP to gauge the popularity of a particular product or topic. Staff also can analyze the data to see which materials do well in specific countries or regions, and how well various materials meet the needs of specific segments of the audience—for example, health care providers, program managers, policymakers, or trainers. In addition to content, the database keeps records of format requested—print, electronic (PDF or HTML), and CD-ROM. CCP has also established partnerships with various organizations listed in the database. As a result even more people are being reached with the information and products they need at the right time and in the right format.

CCP's database also is used to formulate mailing lists for other organizations, tailored to specific regional or interest criteria. This approach also has been used to co-disseminate flyers or other materials to a portion of the database, thus saving mailing costs. Other collaborations arising from the client database have led to in-country redistribution through Southern organizations.

As can be seen above, there are advantages to keeping a client database/mailing list:

1. Requests can be analyzed to assess the popularity of topics, which can contribute to decision-making on future publication topics.

2. Histories of requests can be used to target promotions to the clients most likely to be interested.

3. Mailing list databases are needed to verify subscription information, change of address, and requests for more materials.

4. Databases can be used to record detailed information about readers' interests, for promotion, and also for surveying and needs assessments, or for selecting potential participants in focus group discussions.

5. As more people come online, databases can be linked to Web sites and electronic ordering information such as shopping carts.

Disadvantages of client databases include the labor-intensive work, high maintenance costs (depending on number of records), and difficulties keeping records updated.

Establishing and Maintaining a System to Track Unsolicited Feedback about Information Products

Management Sciences for Health created an archive of “testimonials” in 2002 to track readers’ unsolicited feedback about publications. The archive was developed in Microsoft Outlook in a public folder on the organization’s computer network. Testimonials are entered monthly as new postings. (This system uses a customized form in Outlook Notes; reports can be generated using a simple program created in Access. Creating the form and program for Access took about half a day.) These records can be searched by the name, organization, or country of the author of the testimonial, by the name of the product, by date, by language, and by keywords in the testimonial.

As of September 2006, there were 530 records in this database. Each record is set up so that uses can be classified as follows:

Change Systems or Procedures

Testimonials come from a wide range of people—nurses, doctors, clinic workers, health program directors; of countries, both developed and developing; and of institutions—ministries of health, NGOs, libraries, and health facilities, among others.

Advantages of maintaining a testimonials system include:

1. The system makes it possible for projects to report qualitative information about the usefulness of specific information products.

2. The system provides a convenient repository of unsolicited feedback about information products of all types throughout the “lives” of the products.

3. The information from the system can be used in publicizing products (for example, selected testimonials can be quoted in publications catalogs and fliers or on Web sites).

4. Some of the information collected illustrates outcomes.

5. Can be made easy to access and search.

Some disadvantages are:

Tracking and Analyzing Visits to Web-Based Information Products

Management Sciences for Health does occasional research on the performance of selected electronic products, using a free Web-hosted tool called Google Analytics (https://www.google.com/analytics/home). This tool for analyzing Web usage can provide information about, for example, the number of unique views, number of page views, average time spent at a particular part of a Web site, and the percentage of users who exited the site from that point. The analysis can be carried out for a defined period such as a specific month. It is also possible to some degree to segment users by country (see Appendix 4). The program does not use cookies and does not collect any personal information about visitors.

For instance, eight months after the launch of MSH’s first Occasional Paper, on expediting the financing of HIV/AIDS programs (which was announced via listservs and publicized at several conferences), analysis made it possible to report the following: “There have been 673 visits (816 page views) to the Occasional Papers part of MSH’s Web site since July 2005. Of those visits, one-third originated in the US (232, or 34%), and the remaining visits were made from South Africa (8%), the UK (5%), Kenya (3%), India (2%), France (2%), Brazil (2%), the Netherlands (2%), Uganda (2%), and other countries (40%).”

Advantages of using Google Analytics to track visits include:

1. They provide quantitative information about the reach of an online information product.

2. Research can be done quickly.

3. Information about where visitors come from help with developing future online products. The visitors’ countries help with designing future online products.

4. The amount of time users spend looking at a page suggests the value of the content (i.e., it might be inferred that the more time they spend, the more interesting or useful they found the content to be).

5. The performance of different products can be compared.

Some disadvantages are:

1. A Webmaster with HTML training is needed to tag Web pages using a special script.

See also Appendix 4, p. 60.

Using “Bounceback” Questionnaires to Routinely Collect Client Feedback

To obtain quick feedback on a limited number of questions, the Population Reference Bureau (PRB) includes a “bounceback” questionnaire with new materials it disseminates through its mailing list. Questions are posed in both multiple choice and open-ended formats. Typical questions ask recipients to rate the usefulness of the material, describe how they might use it, and estimate how many others will see or use it. The questionnaire doubles as an order form for additional copies. The results tell how information is used, its potential impact, and what topics and formats are useful.

In 2004 The Wealth Gap in Health data sheet in English was disseminated to 11,000 recipients worldwide using PRB’s international mailing list. Some 222 recipients returned completed questionnaires, for a response rate of just over two percent (response rates usually range from one to five percent). Ninety-one percent of the respondents rated the material “very useful,” while nine percent rated it “somewhat useful.” When asked how they would use the publication, respondents listed a wide variety of uses, including writing reports and speeches (64 percent); research (60 percent); and in trainings and workshops (58 percent). Based on respondents’ responses, it was projected that nearly 27,000 others would see the material and more than 6,000 would use it in their work.

Some advantages of bounceback questionnaires:

Disadvantages of bounceback questionnaires:

While the multiple choice questions tend to yield predictable responses, comments and examples provide a wealth of qualitative information at low cost. Key informant interviews, requiring more time and resources, have not produced as much specific information. Bounceback responses often describe potential use—for example:

“As researchers and program consultants for the country’s health care programs, this information is vital to our work. The information can help us in future research works and programs directed to solve our country’s health inequities.”

Director, Institute of Clinical Epidemiology, National Institutes of Health, University of the Philippines, Manila

“I plan to use Wealth Gap figures to write an Africawoman article about women's health especially to do with pregnancy and antenatal care. I will also use the information in the rural newspaper (Bukedde) I write for.”

Features Editor, New Vision Publications, Kampala, Uganda
Using Online Survey Tools to Collect Client Feedback on Products and Services

Over the years, the Center for Communication Programs (CCP) has used various types of information technology (IT) to reach public health professionals around the world in addition to print publications. CCP IT products and services include: online and CD-ROM versions of print publications, Web portals, online databases, virtual collaboration tools, and listservs and online forums. To gauge how the users view print and IT products, CCP has been using online surveys.

An online survey, or Web-based questionnaire, can identify the level of satisfaction regarding the usefulness and relevance of a publication, determine topics that clients would like covered in the future, and solicit suggestions for further improvement and enhancement of products and services. Because many electronic survey tools are affordable and easy to use, online surveys enable CCP to collect and analyze information from a large number of clients quickly and efficiently.

Electronic survey tools offer a variety of methods to invite people to respond to a Web-based questionnaire. A URL link can be emailed to clients and/or posted on a Web page, or a pop-up window can appear when a client visits a certain Web page. A combination of such approaches can be used to increase the number of responses. Using these methods, CCP has achieved up to a 25% response rate in voluntary online surveys.

The advantages of electronic survey tools are numerous and include:

1. Low cost: no printing or postage costs, no need for interviewers or data entry personnel.

2. Short data collection time: no need to wait for the mail. Clients have immediate access to a survey when it is launched, and the publisher has immediate access to their responses, regardless of their location.

3. Data accuracy: Web-based questionnaires can have automatic validation capabilities by structuring questions in a certain way (e.g., set the number of desired answers or type of answer and set skip patterns).

4. Efficient data processing and analysis: Web-based questionnaires can be programmed to move responses automatically into data analysis software, such as SPSS or Excel.

5. Anonymity of survey respondents: online surveys allow respondents to answer questions anonymously, which can be especially useful when dealing with sensitive topics.

Web Usage Statistics

The ability to monitor and evaluate how information is accessed, exchanged, and used improves every year. This is true across formats and access opportunities. There are more options today to track the use of print publications than there were ten years ago. With each option, evaluators must be careful to understand the factors that affect their data on use of print publications and the conclusions that they draw about that usage.

The same is true for Internet-mediated information access and exchange, but the situation is further complicated by the very dynamic nature of Internet use.

The indicators used to evaluate Internet-mediated information access and exchanges have evolved both in their focus and accuracy as use of the Internet has evolved. For instance, in the early nineties, it was common to track “hits to a page” over a period of time. In the ensuing years, this was largely replaced by the more informative “unique visitors to a page.” This, too, has its limitations and grey areas of interpretation and will, no doubt, either be replaced in the coming years by a more meaningful indicator or be tracked by a better method.

Because the Internet and the indicators of its use are newer than print publications and their indicators, the issues around the meaningfulness and clarity of specific data can sometimes take on disproportionate importance in people’s minds relative to print.

There are several approaches taken to deal with these concerns. One is simply not to track any data where provenance or accuracy is in any doubt. For instance, Web log software tracks the origin—usually the country—of each Web site visitor’s Internet service provider (ISP). This may in fact be where the visitor is located, but perhaps not. For example, a user in one city may log onto the Internet via an ISP whose server is located in another city, state, or even country. For example, all America On Line (AOL) users, regardless of where they actually are, appear to be in Vienna, Virginia, where AOL’s servers are located.

This complication has led some organizations to stop tracking some or all geographical data on Web site use. Other organizations take the approach of tracking data at a higher level or aggregating the data to a level where they have confidence that the conclusions of user origin are accurate. Examples of this are data gathered at a city level but reported out at a national or regional level.

Other organizations continue to track and report geographical data in tandem with an ongoing analysis of the data’s usefulness. For instance, many organizations consider it still useful and reasonable to track Internet use at a national level in the developing world because currently there are not enough supranational ISPs such as AOL to seriously skew data on country of origin. This becomes less true as an organization’s evaluation lens narrows from a global level to a regional, national, or even city level. An organization working in southern Africa may forgo country-level data collection if, for instance, it learns that many users from countries bordering South Africa are accessing the Internet via an ISP in South Africa.

That said, there are some indicators that are commonly tracked with varying levels of understanding of the data’s accuracy. These indicators are usually tracked via software that analyzes log data maintained on a Web server of visits to a site. Examples are:

Web site usage also can be tracked via other methods, including analysis of others’ links to a Web site, as well as targeted user surveys and interviews.

Each organization needs to make its own judgment about the data it collects in light of its understanding of their dataset, their ability to evaluate those data, objectives in collecting the data, ability to act on the data, donor/stakeholder expectations, and standard practices for indicator tracking and reporting. These same considerations, which are receiving increasing attention in the evaluation of online services and products, deserve attention in the evaluation of other media as well, such as print and radio.

For Writers: Questions to Ask of an Outline

INFO developed this tool to assist writers in developing targeted information products, guided by communication theory. Evaluators may refer to this tool to assess how well these guidelines have been followed in developing an information product or service.

1. Who is the information product for?

2. What do you expect the audience to do as a result of reading it?

To make the answers to those first two questions more actionable when formulating the outline, researching, and writing consider:

3. Are you addressing the issues that most concern the intended audience(s)?

4. What practice, behavior, or policy do you want to encourage and enable through the presentation and analysis of evidence and experience?

a. For improving access?

b. For improving quality?

5. Are you emphasizing the facts that support or call for this new behavior?

6. Are the actions/practice/policy that the facts suggest sufficiently highlighted and easy to find? Will a browsing reader spot them?

7. Are facts linked with actions that the intended audience can take? (For example, when we discuss side effects of contraceptive methods, do we also discuss how to counsel about them and/or how to manage them? And when we discuss the gap between perfect effectiveness of a method and effectiveness as commonly used, do we discuss how that gap can be reduced?)

8. Are you presenting material—and presenting it in a way—that makes use of the five characteristics of readily adopted behavior?

· Relative advantage

9. Have you sacrificed for focus? That is, have you held to a minimum, or else presented in the most efficient fashion, information that is supplemental or already well known, so as to focus readers’ attention on what is essential to improving practice and services? (For example, can you use tables, with little or no explanatory text, to present standard information or unremarkable survey findings. Have you used graphics where graphics convey a point better than text?)

Source: Developed by Ward Rinehart for the editorial staff of the INFO Project, 2006.

Illustrative Readership Survey

We greatly value your feedback on our information products and services. Please take the time to fill out this short survey. We will use your answers to guide future product development in order to better meet your information needs.

	*Question*	Indicator[1]
	REACH
1	Do you usually receive this publication? � Yes � No Comments:	11
2	Other than you, how many people normally read at least some part of this publication? � More than 10 people � 6-10 people � 1-5 people � No other people � Other ____________________	10
	USEFULNESS
3	Do you usually read this publication? � Yes, I read it cover to cover. � Yes, I read the parts that interest me. � No, I do not usually read it. Comments:	11
4	How useful is this publication in your daily work? (Check one) � Highly useful � Somewhat useful � Not useful Comments:	12
5	How would you rate the length of this publication? (Check one) � Too short � Just right � Too long Comments:	13
6	Please choose the answer that best describes the readability of this publication (Check one): � Easy to read � Somewhat easy to read � Not easy to read Comments:	13
7	Please rate your satisfaction with the following elements of this publication: *Relevance of program examples* ¦ Satisfied ¦ Somewhat satisfied ¦ Not satisfied *Ease of understanding key points* ¦ Satisfied ¦ Somewhat satisfied ¦ Not satisfied *Ease of finding specific information* ¦ Satisfied ¦ Somewhat satisfied ¦ Not satisfied	13
8	How would you rate the coverage of topics in this publication? (Check one) � Too little � Just right � Too much Comments:	14
9	What suggestions do you have for making the *content* of this publication more useful and relevant to your work?	14
10	What other topics would you like to see covered in this publication?	14
11	Did you learn anything new from this publication? � Yes � No Please explain.	15
12	Did the information contained in this publication change your mind about a specific issue? � Yes � No Please explain.	16
	USE
13	Have you or do you intend to adapt this publication for a specific use? � Yes, I have adapted this publication. � Yes, I intend to adapt this publication. � No Please explain.	23,24
14	How often have you used this publication for the following purposes? (Check in all rows that apply)	25, 26
		Frequently	Sometimes	Never
	To develop policy	¦	¦	¦
	To ensure accurate media coverage	¦	¦	¦
	To increase public awareness	¦	¦	¦
	To design projects or programs	¦	¦	¦
	To develop training programs or workshops	¦	¦	¦
	To design educational materials	¦	¦	¦
	To improve service quality	¦	¦	¦
	To guide research agendas or methods	¦	¦	¦
	To write funding proposals	¦	¦	¦
	To develop presentations	¦	¦	¦
	To write reports or articles	¦	¦	¦
	For reference in daily work	¦	¦	¦
	For professional or academic interest or development	¦	¦	¦

15	Has the information in this publication led to changes in policies or procedures or influenced the provision of health services? � Yes � No Please explain.	25
16	Please give specific examples of how you (or your colleagues) have used this publication in your work and explain the results of that use (if known).	26

Do you usually receive this publication?

Other than you, how many people normally read at least some part of this publication?

� More than 10 people

� Other ____________________

Do you usually read this publication?

� Yes, I read it cover to cover.

� Yes, I read the parts that interest me.

� No, I do not usually read it.

How useful is this publication in your daily work? (Check one)

How would you rate the length of this publication? (Check one)

Please choose the answer that best describes the readability of this publication (Check one):

� Somewhat easy to read

� Not easy to read

Please rate your satisfaction with the following elements of this publication:

Relevance of program examples

¦ Satisfied ¦ Somewhat satisfied ¦ Not satisfied

Ease of understanding key points

¦ Satisfied ¦ Somewhat satisfied ¦ Not satisfied

Ease of finding specific information

¦ Satisfied ¦ Somewhat satisfied ¦ Not satisfied

How would you rate the coverage of topics in this publication? (Check one)

What suggestions do you have for making the content of this publication more useful and relevant to your work?

What other topics would you like to see covered in this publication?

Did you learn anything new from this publication?

Did the information contained in this publication change your mind about a specific issue?

Have you or do you intend to adapt this publication for a specific use?

� Yes, I have adapted this publication.

� Yes, I intend to adapt this publication.

How often have you used this publication for the following purposes? (Check in all rows that apply)

To ensure accurate media coverage

To increase public awareness

To design projects or programs

To develop training programs or workshops

To design educational materials

To improve service quality

To guide research agendas or methods

To write funding proposals

To develop presentations

To write reports or articles

For reference in daily work

For professional or academic interest or development

Has the information in this publication led to changes in policies or procedures or influenced the provision of health services?

Please give specific examples of how you (or your colleagues) have used this publication in your work and explain the results of that use (if known).

BACKGROUND INFORMATION

In which country do you work?

Please select the category that best describes your organization type (Check one):

� Academic institution

� Private sector (for profit)

� Government or ministry

� Medical or health organization

� NGO or PVO (local or international)

� Research institution

� Religious/Faith-based organization

Please choose the category that best describes the focus of your work (Check one):

� Health communication

� Health or medical service delivery

� Program development or management

� Research or evaluation

� Teaching or training

[1] This column provides the indicator number that corresponds to the question. It would not be included in a survey.

Source: This illustrative survey draws heavily from the Population Reports Reader Survey developed by INFO in 2006.

Indicators for M&E of Provision of Information Products and Services:

Results of Ranking at August 9, 2006, HIPNET Meeting

HIPNET members ranked these indicators according to their relevance to their organizations work on a scale from 1 (highly relevant) to 5 (not very relevant). Some of the indicators, marked as not applicable (N/A), were not included in the list of indicators reviewed at the August 9, 2006, meeting and so do not have a rank.

No.	Indicator	Average Rank
	REACH
	*Area 1: Primary Distribution (Push)*
1	Number of copies of a product initially distributed to existing lists	1.3
2	Number of copies of a product distributed by a publisher through additional distribution	2.3
	*Area 2: Secondary Distribution (Pull)*
3	Numbers of products distributed in response to orders	1.4
4	Number of file downloads in a time period	1.4
5	Number of times a product is reprinted by recipients	2.0
6	Number of people reached by media coverage of the material or generated by it	2.3
	*Area 3: Referrals*
7	Number of instances that products are indexed or archived in bibliographic databases	2.6
8	Number of postings of products by other Web sites or links to products from other Web sites	2.0
9	Number of instances that products are selected for inclusion in a library	2.3
10	Percentage of users who share their copy or transmit the information verbally to colleagues	2.2
	USEFULNESS
	*Area 1: User Satisfaction*
11	Percentage of those receiving a product or service who have read or browsed it	N/A
12	Percentage of users who are satisfied with a product or service	1.3
13	Percentage of users who rate the format or presentation of a product or service as usable	1.7
14	Percentage of users who rate the content of a product or service as useful	1.3
15	Number/percentage of users who report knowledge gained from a product or service	1.5
16	Number/percentage of users who report that a product or service changed their views	N/A
	*Area 2: Product or Service Quality*
17	Number and quality assessment of reviews of a product in periodicals	2.7
18	Number and significance of awards given to a product or service	3.1
19	Number of citations of a journal article or other information product	2.6
20	Journal impact factor	3.0
21	Number/percentage of users who pay for a product or service	3.2
22	Number/percentage of information products or services guided theories of behavior change and communication	N/A
	USE
23	Number/percentage of users intending to use an information product or service	N/A
24	Number/percentage of users adapting information products or services	1.4
25	Number/percentage of users using an information product or service to inform policy and advocacy or to enhance programs, training, education, or research	1.3
26	Number/percentage of users using an information product to improve their own practice or performance	1.6
	COLLABORATION & CAPACITY BUILDING
	*Area 1: Collaboration*
27	Number of instances of products or services developed or disseminated with partners	1.8
28	Number of instances of South-to-South or South-to-North information sharing	2.0
	*Area 2: Capacity Building*
29	Number and type of capacity-building efforts	N/A

Area 1: Primary Distribution (Push)

Number of copies of a product initially distributed to existing lists

Number of copies of a product distributed by a publisher through additional distribution

Area 2: Secondary Distribution (Pull)

Numbers of products distributed in response to orders

Number of file downloads in a time period

Number of times a product is reprinted by recipients

Number of people reached by media coverage of the material or generated by it

Area 3: Referrals

Number of instances that products are indexed or archived in bibliographic databases

Number of postings of products by other Web sites or links to products from other Web sites

Number of instances that products are selected for inclusion in a library

Percentage of users who share their copy or transmit the information verbally to colleagues

Area 1: User Satisfaction

Percentage of those receiving a product or service who have read or browsed it

Percentage of users who are satisfied with a product or service

Percentage of users who rate the format or presentation of a product or service as usable

Percentage of users who rate the content of a product or service as useful

Number/percentage of users who report knowledge gained from a product or service

Number/percentage of users who report that a product or service changed their views

Area 2: Product or Service Quality

Number and quality assessment of reviews of a product in periodicals

Number and significance of awards given to a product or service

Number of citations of a journal article or other information product

Journal impact factor

Number/percentage of users who pay for a product or service

Number/percentage of information products or services guided theories of behavior change and communication

Number/percentage of users intending to use an information product or service

Number/percentage of users adapting information products or services

Number/percentage of users using an information product or service to inform policy and advocacy or to enhance programs, training, education, or research

Number/percentage of users using an information product to improve their own practice or performance

COLLABORATION & CAPACITY BUILDING

Area 1: Collaboration

Number of instances of products or services developed or disseminated with partners

Number of instances of South-to-South or South-to-North information sharing

Area 2: Capacity Building

Number and type of capacity-building efforts

[1] This column provides the indicator number that corresponds to the question. It would not be included in a survey.

· Relative advantage	· Observability
· Compatibility	· Trialability
· Simplification

HIPNET - Health Information and Publications Network

Appendices