The HRSA/HAB/SPNS Cooperative Agreement 1994 – 1999: What We Learned

The HRSA/HAB/SPNS Cooperative Agreement 1994 – 1999: What We Learned, How We Learned It, and [with Hindsight] What Else We Might Have Learned with Alternate Methodologies

Handout to Accompany a Presentation at the HRSA/HAB/SPNS Border Health Initiative Grantee Meeting
in San Francisco on January 11, 2001

George J. Huba, Ph.D.
The Measurement Group
5811A Uplander Way
Culver City, California 90230
310.216.1051
ghuba@TheMeasurementGroup.com

This summary lists major types of information available about the Cooperative Agreement Projects funded as Special Projects of National Significance by the HIV/AIDS Bureau of HRSA between 1994 and 1999. The information cited is available on the Internet at the web site of The Measurement Group [http://www.TheMeasurementGroup.com] with specific URLs on the site given below to enable the user to directly access certain types of information. HRSA funded 27 projects in 1994, with 23 of these funded for five years. The Measurement Group was the national [cross-cutting] evaluator for the initiative and was primarily responsible for the Evaluation and Dissemination Center. Key personnel at The Measurement Group involved in the evaluation included G. J. Huba, Ph.D., Lisa A. Melchior, Ph.D., Abigail T. Panter, Ph.D., and Diana E. Brief, Ph.D. The web site was developed under the direction of Dr. Huba. A listing of the Project Directors from the 27 Cooperative Agreement Projects and Grantee Project Staff as well as HRSA SPNS Staff is given online at http://www.themeasurementgroup.com/KnowBase/Acknowledge.htm.

What We Learned

a. Empirical Studies. Major empirical results from the five-year evaluation studies are presented online in a series of more than 500 Knowledge Items arrayed in a Knowledge Base. Access this Knowledge Base at http://www.themeasurementgroup.com/KnowBase/CoopAgree/knowbaseca.htm. This section of the web site contains the primary cross-cutting evaluation results from the 27 projects. The projects produced a number of desirable outcomes.

b. Empirical Studies [Patient Outcomes Summary]. Special sections of the Knowledge Base summarize patient-client outcomes. See http://www.themeasurementgroup.com/KnowBase/CoopAgree/Index_Client/SummaryOutcomes.htm. The page lists links to about 50 key Knowledge Items that summarize patient outcomes. Overall, health improved, quality of life improved, the impact of symptoms decreased, barriers to services decreased, psychological distress decreased, and patients were retained in services.

c. Empirical Studies [Training Outcomes Summary]. Major outcome results from healthcare provider training efforts are summarized online at http://www.themeasurementgroup.com/KnowBase/CoopAgree/Index_Training/PatientCare.htm, http://www.themeasurementgroup.com/KnowBase/CoopAgree/Index_Training/SystemChange.htm, and http://www.themeasurementgroup.com/KnowBase/CoopAgree/Index_Training/ProvPractices.htm. Training was used by most participants to improve their clinical practice and/or to improve the systems by which services were provided at their institutions.

d. Summary Report on Results in the Online Knowledge Base. A summary report is online at http://www.themeasurementgroup.com/KnowBase/CoopAgree/SummaryReport.htm on major results in the Knowledge Base.

e. Process and Achievements Reports. The process for managing the initiative is described in first-year and second-year reports. These reports also show the major accomplishments of each grantee at the end of the first and second years of the projects. The process used was rated by the grantees, at the end of five years, to have been a successful one.

f. Lessons Learned. A short summary of the major lessons learned and accomplishments of each project – as defined by that project – was developed at the end of the grant cycle.

g. Fact Sheets. Short summaries of projects were distributed by through the SPNS/FAX Electronic Report Program weekly between 1996 and 1998. Fact Sheets were also prepared by The Measurement Group to summarize major data studies.

h. Project Descriptions. A Project Index consisting of individual project descriptions and achievements is available on the web. The project pages include both project-developed and Evaluation-Center-developed materials.

i. Initiative Poster Sessions for HRSA Staff. Results from the cross-cutting evaluation and individual projects have been presented at four grantee-sponsored poster sessions in the District of Columbia [in 1995, 1996, 1997, and one at HRSA in September 1999 at the completion of the projects]. Posters from the final 1999 session and an agenda for the all-day symposium sessions held concurrently with the poster session are available. The session at HRSA at the end of the funding cycle was an effective way to make agency staff aware of the major findings; ideally this might also be done after two or three years.

j. Publications. A number of papers have been published from these projects. A master list is available at http://www.themeasurementgroup.com/publications/papers.htm and changes on a regular basis as new papers are accepted or as in-press publications appear. This list also includes presentations made to various professional audiences.

k. Process and Management Reports. A number of reports are available online including reports from individual work groups, reports from individual projects, and specialized analyses.

l. Special Issue of Drugs and Society on Individual Comprehensive Care and Community Based Organization Projects. A [double] Special Issue of the journal was developed and also published as a book. The Table of Contents and Abstracts for the papers are available online.

m. Special Issue of Home Health Care Services Quarterly on High Need and Vulnerable Populations. A [double] Special Issue of the journal was developed and also published as a book. The Table of Contents and Abstracts for papers are available online.

n. Book on End of Life AIDS Services under a Capitated Funding Model. A Special Issue of Home Health Care Services Quarterly was done by one project on their local evaluation results and also published as a book, AIDS Capitation [Cherin and Huba, editors]. The Table of Contents and Abstracts for the papers are available online.

How We Learned It

a. Evaluation Designs. Evaluation designs for five work groups encompassing the individual projects.

b. Evaluation Modules. More than 100 evaluation modules are contained online and available for use.

c. Interview Forms. A series of structured and semi-structured interviews were developed for the projects.

d. Lessons Learned. Summary forms for coding project successes and lessons learned.

e. Data Technical Issues. Technical issues in coding certain kinds of data for outcomes are online and may be accessed at http://www.themeasurementgroup.com/KnowBase/CoopAgree/Index_Special/TechnicalIssues.htm.

f. Decision Making Process for Grantee Initiative. We ran a very participatory group process for five years with 3 or 4 meetings per year. The decision making model can be viewed online.

g. Publication and Dissemination Models. Publication and dissemination models are described online at http://www.themeasurementgroup.com/KnowBase/CoopAgree/designkb/3/KnowledgeItem.htm.

h. CyberTours of Programs. A very good way of presenting information about innovative programs is the CyberTour, a method developed by TMG for presenting an online tour of programs along with evaluation results. Some representative CyberTours of programs are available. Only one of the three programs shown was from the Cooperative Agreement; The Measurement Group was the local evaluator for all of these projects and probably such CyberTours should be considered as the responsibility of local evaluators.

i. Focus Groups. A limited amount of work was done with focus groups. Some reports online are those on protease inhibitor use by youth and unmet advocacy needs.

j. Possible Evaluation Questions. The Measurement Group posed 61 possible evaluation questions for innovative HIV/AIDS services questions. Most of these questions are addressed in one form or another in the more than 500 Knowledge Items in the Online Knowledge Base. While these questions were developed independently of the first version of the HRSA/HAB Evaluation Questions, most of these map directly into one of the larger HRSA areas, and later versions of the HRSA/HAB questions appear to have been adapted to some degree from these questions.

k. Classification Trees for Presenting Major Findings. In addition to using traditional inferential statistical tests, a number of evaluation issues were examined using methods for developing classification trees using the CHAID [Chi-square Automatic Interaction Detector] method. The technique is useful for presenting complex data for non-statisticians that can be interpreted in an intuitive manner. A list of major findings generated using this method is given at http://www.themeasurementgroup.com/KnowBase/CoopAgree/Chaid_Analyses.htm. Some specific technical issues and comparisons with alternate methods are given online at http://www.themeasurementgroup.com/KnowBase/CoopAgree/TechnicalIssues/5/KnowledgeItem.htm.

l. We Dealt with a Lot of Missing Data. Some discussions about how to deal with missing data are given at http://www.themeasurementgroup.com/KnowBase/CoopAgree/TechnicalIssues/1/KnowledgeItem.htm, http://www.themeasurementgroup.com/KnowBase/CoopAgree/TechnicalIssues/2/KnowledgeItem.htm, and http://www.themeasurementgroup.com/KnowBase/CoopAgree/TechnicalIssues/7/KnowledgeItem.htm. The entire Knowledge Base and its component analyses were explicitly designed to deal with the fact that some projects did not collect certain predictive indicators; models are developed which do, and do not, actually include "missing" as a data category.

m. Evaluation Conference. An Evaluation Conference was held in 1995 with Speakers from the EDC, the Cooperative Agreement Projects, HRSA, and outside experts. Selected conference materials are available online, and most of those papers are still as timely in 2001 as they were in 1995.

n. Other. Numerous other materials are available on the Evaluation and Dissemination Center index of the web site.

o. Contrast to a Second Evaluation and Dissemination Center. The Measurement Group was also responsible for an earlier Evaluation and Dissemination Center for 10 HRSA/HAB/SPNS projects on services for adolescent and youth, with or at high-risk for, HIV/AIDS. Materials from that earlier EDC including instruments, a Knowledge Base, presentations, program descriptions, and reports can be accessed.

What Else We Might Have Learned with Alternate Methodologies

a. Organizational-Infrastructure Analysis. We might have done more qualitative work both in cross-cutting evaluation studies and as local evaluations. One good example of this work was done with three Comprehensive Healthcare projects and a similar report was done with eight infrastructure change projects. Another good example was a follow-up interview study of healthcare professionals trained by various project and a summary report can be accessed online. More work of this kind would have been beneficial.

b. Lessons Learned and Achievements. At the end of the project we collected project bullets about major lessons learned and achievements. We probably should have done this several times during the process, perhaps coordinating such statements with the annual reports that each project makes to HRSA. A more formal way of coding and presenting this information should be developed and the information probably should be collected annually. This type of analysis needs to be done on a regular basis.

c. Focus Groups. It would have been desirable to run focus groups on outcomes at each of the projects. This was not done.

d. Retrospective Outcome Interviews. Outcome interview studies should have been retrofit into the cross-cutting designs and conducted retrospectively near program completion or after completion. One project did such interviews with a relatively large degree of success [see the PROTOTYPES local evaluation]. The structured interview used is also posted and may be accessed.

e. Medical Chart Abstraction. Grantees were unwilling to adopt the detailed chart review specifications that were initially proposed for the Cooperative Agreement Projects. The way that medical outcomes were coded is given online at http://www.themeasurementgroup.com/KnowBase/CoopAgree/TechnicalIssues/3/KnowledgeItem.htm and required large amounts of data equating by the Evaluation Center from project-specific databases, many which were "downloads" from hospital or clinic management information systems. Representative findings after we re-recoded the data are given at http://www.themeasurementgroup.com/KnowBase/CoopAgree/Index_Client/MedicalOutcomes.htm. An updated methodology, reflecting recent advances in the state of the art for HIV/AIDS treatment was developed for another evaluation by The Measurement Group in Broward County, Florida. Our work in this area is ongoing and will be posted at http://www.TheMeasurementGroup.com as it is completed. We strongly recommend that later projects use chart abstraction as a key method for obtaining medical outcomes data.

f. Local Evaluations. Although not intended in the beginning of the projects, many projects limited their local evaluations to the cross-cutting data and did project-specific analyses only on their portion of the cross-cutting data. Much more should have been done in individual projects to develop instruments and indicators that were specific to special programmatic elements of those projects. Among the projects with more evolved and unique local evaluations were those at PROTOTYPES, the Visiting Nurse Association, the University of Nevada School of Medicine, the Center for Women Policy Studies/Metro DC Collaborative, the University of Texas at San Antonio Medical School, the University of Washington, and the Cook County/Hektoen Institute Comprehensive HIV/AIDS Care Center. Materials from those projects are posted on this website.

g. Project Logs of Key Implementation Events. Each project could have kept a log of the key events in implementing and maintaining its program. Such logs would have helped other "replication" programs deal with key milestones in implementing similar programs. The closest thing to this was a study of the implementation issues in three university-based comprehensive care clinics. We strongly urge each project to keep a timeline log of all key events in the project and to present this information graphically with short summaries of the importance of the events and how key issues were resolved.

h. CyberTours. It would have been desirable to have a CyberTour of every program, both as that program appeared early in its implementation and much later after it was in its final form. A CyberTour was developed for one Cooperative Agreement Program on which The Measurement Group was also the local evaluator.

Related Information:

Evaluation and Dissemination Center: Innovative Models of HIV/AIDS Care 1994-2001