61 Evaluation Questions for HIV/AIDS Services Projects

This list of evaluation questions was proposed by George J. Huba, Ph.D., Lisa A. Melchior, Ph.D., Abigail T. Panter, Ph.D., and the staff of The Measurement Group as applicable to the 27 HRSA SPNS Cooperative Agreement Projects. The list of questions was developed between March and May 1998. The questions are also more generally appropriate for other projects studying the issues in implementing HIV/AIDS services.

The questions were approved as a set of working hypotheses for the 27 Cooperative Agreement Projects by the Steering Committee on May 7, 1998. The questions form the basis for many of the analyses of the Knowledge Base derived from the cross-cutting evaluation of the projects.

The questions are given in three sets: a) those applicable to projects that focus on HIV services for individual patients or clients; b) those applicable to projects that focus on training individuals who provide HIV services to patients or clients; and c) those applicable to projects that are seeking to change the infrastructure for HIV services.

Client-Level Project Evaluation Questions

1. What are the best ways to empirically describe the characteristics of the persons who receive direct clinical services from the cooperative agreement projects?
2. Are there distinct types of persons who receive direct clinical services from the cooperative agreement projects?
3. What are the best ways to empirically describe the continuum of psychosocial services provided by he cooperative agreement projects?
4. What are the best ways to empirically describe the continuum of medical (and psychosocial support) services provided by the cooperative agreement projects?
5. Are there distinct types of psychosocial service utilization patterns for clients (patients) of the cooperative agreement projects?
6. Are there distinct types of service utilization patterns for patients (clients) served by the cooperative agreement projects that include medical as well as psychosocial support service components?
7. How are behaviors and other client (patient) characteristics and program factors related to patterns and rates of psychosocial service utilization in the cooperative agreement projects?
8. How are behaviors and other patient (client) characteristics and program factors related to patterns and rates of service utilization in cooperative agreement projects that include medical as well as ancillary psychosocial support service components?
9. How are psychosocial service utilization rates (and their determinants) related to client (patient) medical and psychosocial outcomes in the cooperative agreement projects?
10. How are medical service utilization rates (and their determinants) related to patient (client) medical and psychosocial outcomes in the cooperative agreement projects?
11. How do psychosocial service utilization rates affect medical service utilization rates and subsequent medical and psychosocial outcomes in the cooperative agreement projects?
12. How long are clients (patients) retained in care by the cooperative agreement projects, and how is retention related to characteristics of the clients (patients) and the service models?
13. How is temporal retention in care by the cooperative agreement projects related to client (patient) outcomes such as health status, quality of life, and psychosocial functioning?
14. How are client (patient) characteristics related to patterns of service sequencing in the cooperative agreement projects?
15. How does the use of case management by clients (patients) of the cooperative agreement projects impact on the overall patterns of service utilization?
16. What client (patient) characteristics are related to the utilization of specific types of case management services across the cooperative agreement projects?
17. How does the use of psychosocial support services by clients (patients) of the cooperative agreement projects impact on the overall patterns of service utilization?
18. What client (patient) characteristics are related to the utilization of specific types of psychosocial support services in the cooperative agreement projects?
19. How does the use of ancillary support services by clients (patients) of the cooperative agreement projects impact on the overall patterns of medical and psychosocial service utilization?
20. What patient (client) characteristics are related to the utilization of specific types of medical services in the cooperative agreement projects?
21. How does the use of substance abuse services by clients (patients) of the cooperative agreement projects impact on the overall patterns of service utilization?
22. What client (patient) characteristics are related to the utilization of specific types of substance abuse services in the cooperative agreement projects?
23. How does the use of mental health services by program clients (patients) of the cooperative agreement projects impact on the overall patterns of service utilization?
24. What client (patient) characteristics are related to the utilization of specific types of mental health services in the cooperative agreement projects?
25. Do individuals seen by the cooperative agreement projects with multiple diagnoses (e.g., HIV, substance abuse, and/or mental illness) have different patterns and rates of service utilization and other outcomes as compared to clients (patients) who are not multiply-diagnosed?
26. How are the characteristics of clients (patients) seen by the cooperative agreement projects related to barriers they may have faced when seeking HIV services?
27. Do clients (patients) of the cooperative agreement projects experience a reduction of barriers to HIV care as a function of their involvement with these programs?
28. How are the characteristics of clients (patients) of the cooperative agreement projects related to their self-perceived health, psychosocial, and ancillary service needs?
29. Do self-perceived service needs change among clients (patients) of the cooperative agreement projects as a function of their involvement with these programs?
30. Do indicators of health-related quality of life change among clients (patients) of the cooperative agreement projects as a function of program involvement and client (patient) characteristics?
31. Do indicators of patient (client) health status and functioning change among patients (clients) of the cooperative agreement projects as a function of program involvement and patient (client) characteristics?
32. What are the levels of client (patient) satisfaction with care from the cooperative agreement projects as a function of program involvement and client (patient) characteristics?
33. Do levels of psychosocial functioning among clients (patients) of the cooperative agreement projects change as a function of program involvement and client (patient) characteristics?
34. Do levels of self-efficacy among clients (patients) of the cooperative agreement projects change as a function of program involvement and client (patient) characteristics?
35. Do levels of general well-being among clients (patients) of the cooperative agreement projects change as a function of program involvement and client (patient) characteristics?
36. Do patterns and rates of hospitalization and emergency room utilization change among patients (clients) of the cooperative agreement projects as a function of program involvement and patient (client) characteristics?
37. What are the models of cost-effectiveness associated with the idealized models of psychosocial and medical HIV care for the cooperative agreement projects?
38. Across the cooperative agreement projects, do persons living with HIV who have co-occurring problems of substance abuse and/or mental disorders have different temporal retention rates than other clients (patients)?
39. Across the cooperative agreement projects, do persons living with HIV who have co-occurring problems of substance abuse and/or mental disorders have different medical outcomes than individuals who are not multiply-diagnosed?
40. Across the cooperative agreement projects, do persons living with HIV who have co-occurring problems of substance abuse and/or mental disorders have different psychosocial outcomes than individuals who are not multiply-diagnosed?
41. Is there a different mix of psychosocial and medical services that is optimal for producing desired outcomes for individuals with HIV who have co-occurring problems of substance abuse and/or mental disorders?
42. Across the cooperative agreement projects, are there differential costs and levels of cost-effectiveness associated with providing services to persons living with HIV with co-occurring problems of substance abuse and/or mental disorders?
43. Across the cooperative agreement projects, how is staff composition related to client (patient) outcomes?

Training Project Evaluation Questions

44. What are the best ways to empirically describe the characteristics of the training activities provided by the cooperative agreement projects?
45. What are the best ways to empirically describe the kinds of impact of the training activities provided by the cooperative agreement projects?
46. How are the major characteristics of trainings provided by the cooperative agreement projects related to the various impacts of those activities?
47. How are the characteristics of training activities provided by the cooperative agreement projects related to participant satisfaction with the trainings?
48. How are the characteristics of training activities provided by the cooperative agreement projects related to changes in clinical practice related to HIV?
49. How are the characteristics of training activities provided by the cooperative agreement projects related to changes in behavior other than that related to clinical practice among those individuals trained? How are the training characteristics related to changes in attitudes or knowledge among participants?
50. What are the best ways of describing the characteristics of the people who provide training for the cooperative agreement projects?
51. What are the best ways of describing the characteristics of the people who receive training from the cooperative agreement projects?
52. How are training outcomes related to the interaction of the characteristics of training activities, persons trained, and trainers across the cooperative agreement projects?

Infrastructure-Advocacy Project Evaluation Questions

53. What are the best general ways of describing the characteristics of the service infrastructure change activities of the cooperative agreement projects?
54. What are the best ways of measuring immediate "success" for the cooperative agreement projects that attempted to affect change in the way services are provided for persons living with HIV/AIDS?
55. What are the best ways of measuring long-term "success" for the cooperative agreement projects that attempted to affect change in the way services are provided for persons living with HIV/AIDS?
56. How are the major characteristics or types of infrastructure change activities of the cooperative agreement projects related to the impact of those activities?
57. How are the characteristics of the infrastructure change activities of the cooperative agreement projects related to changes in clinical practice related to HIV among the target individuals for those activities? That is, do different types of systems change activities produce different levels, patterns, or types of change among practitioners?
58. How are the characteristics of infrastructure change activities provided by the cooperative agreement projects related to changes in behavior other than that related to clinical practice among the target individuals for those activities?
59. What are the best ways of describing the characteristics of the people involved as agents of change for the cooperative agreement projects that focused on infrastructure change, and how are such characteristics related to project outcomes?
60. What are the best ways of describing the characteristics of the people participate in infrastructure change activities across the cooperative agreement projects, and how are these characteristics related to project outcomes?
61. What is the estimated effectiveness and cost-effectiveness of a process of 27 projects working in the cooperative agreement structure utilized by this group of grantees, as opposed to working as 27 individual grantees?

© Copyright 1998-2005 by The Measurement Group LLC. All rights reserved.