The PROMIS Initiative and Mid-Course Review

The Patient-Reported Outcomes Measurement Information System (PROMIS) initiative, one of the NIH Roadmap efforts designed to re-engineer the clinical research enterprise, establishes a collaborative relationship between NIH and individual research teams through a cooperative agreement (UO I) mechanism.

PROMlS is developing new ways to measure patient-reported outcomes (PRO) such as pain, fatigue, emotional distress, physical functioning, and social role participation, which have a major impact on quality of life across a wide variety of chronic diseases. The focus and assessments of the PRO items, thus far, has been on the following clinical populations: cancer, chronic obstructive pulmonary disease (COPD), heart disease, osteoarthritis, psychiatric conditions, rheumatoid arthritis, and spinal cord injury?

The broad objectives of the PROMIS Network (the Network) are to:

• develop and test a large Item Library measuring PRO;
• create a computerized adaptive testing (CAT) system that allows for efficient, psychometrically robust assessment of PRO in clinical research for a wide range of chronic diseases; and
• create a publicly available system which will allow clinical researchers to access a common item repository and CAT.

Each item is a question with an associated set of response options on a survey. The PROMIS Item Banks (Item Banks) represent a collection of items that measure the same construct, such as pain.

The Mid-Course Review was conducted from May 2007 to September 2007 and assessed the progress of the Network over the first two-and-a-halfyears offunding with a focus on whether the objectives and milestones of the program were being met. The Mid-Course Review helped assess short-and long-term needs of PROMIS in order to bring the project to fruition.

The Roadmap Implementation Coordination Committee (RICC) approved the following general questions to guide the Mid-Course Review:

• Is the project being conducted as planned?
• Have the mid-point goals been achieved?
• Have the goals been modified since the beginning of the project? Ifso, what is the progress in these modified goals?
• Was the rationale appropriate for process changes since the project's inception?
• Does the project continue to be relevant and significant in relation to re-engineering the clinical research enterprise?

Executive Summary

The PROMIS Request for Applications (RF A) objectives guided the implementation ofthe Network and provided the context for the Mid-Course Review. The Review Panel organized them into four categories to focus the review: organizational, operational, research, and dissemination.

Conclusions

The Review Panel concluded the PROMIS Principal Investigators are successfully collaborating to implement this very important and innovative project and the Network has exceeded expectations with its first two-and-a-half years of accomplishments.

The Review Panel found the PROMIS Principal Investigators have developed an organized, functional, and productive Network that conforms to the RFA objectives. The Network has not modified the program goals since the beginning of the project.

The Network operates as a collection ofcollaborative committees and working groups and has the intellectual resources to achieve its goals. The PROMlS Principal Investigators are leaders in their respective fields and have demonstrated their collective commitment to the success of the project.

The domain framework, Item Bank selection, and data collection methods, which included focus groups, cognitive interviews, general population and specific patient population sampling, provide a solid foundation on which the PROM IS Principal Investigators can build a system that has the possibility to transform how clinical researchers collect PRO information.

The Network activities are ahead of the timetable described in the RF A, enabling the Network to plan a second wave of clinical validation studies for four chronic disease areas during the fourth and fifth years of the funding period.

PROMIS represents a paradigm shift in how assessments for PRO are created and used in clinical research. In particular, the dynamic functionality of the CAT system is an exciting new feature. The Item Banks and CAT system represent much improvement over the use of assessments which focus on a single disease and are likely to be outdated and/or are culturally irrelevant. Thus, the project is highly relevant and in a position to transform clinical research. It will be critical to have the project's tools and content freely and openly available to clinical researchers.

The Review Panel concludes that the PROMIS goals for the first three years have been met within the first two-and-a-half years and that the project is relevant and timely to reengineering the clinical research enterprise. However, the ultimate success of the project will lie in the validation studies, the implementation of the CAT system, and the adoption of the PROMIS products by the clinical research community.

Executive Summary

III.A. Conclusions

In response to the general questions that guided the Mid-Course Review, the Network is conducting the project as planned. The goals have not been modified. The mid-point goals have not only been achieved, but have been surpassed. Wave I data collection ended ahead of schedule and data analysis is nearing completion. The project is relevant and timely in terms of re-engineering the clinical research enterprise. The Review Panel recommendations in the next section are designed to support the initiative's continued relevance.

III.B. Recommendations

The Review Panel commends the Network on the progress and products to date. Once the project is complete, management of the PROMIS products must be accompanied by continuing research to ensure their dynamic viability.

III.B.l. Organization ofthe Network

Network organizational activities included the creation of a unifying structure for the individual grantees and the formulation of rules and processes for communication, collaboration, and decision-making. The Review Panel raised the following potential issues and offers recommendations for the Network to consider as the project progresses:

• Sustainability - If one of the PROMIS grantees leaves the Network, the success of the initiative could be compromised.
  • Recommendation: The Network must ensure that a solid, dynamic infrastructure is in place as the PROMIS project continues development and operation.
• Contractor Advantage - Contractors or subcontractors currently involved with the initiative may appear to have an unfair advantage or conflict of interest in competing for future contracts.
  • Recommendation: The SC should consider whether those currently working on the project should be allowed to compete for subsequent contracts.

III.B.2. Operations

Operational activities have included identification of the domains, items, and assessments, as well as methods for assessing their relevance and utility in a diverse population. Potential issues and recommendations for these areas are:

• Clinical Relevance - For the PROMIS Item Banks and items to retain clinical relevance, they must be dynamic and reflect changes in the environment that may render items obsolete or irrelevant.
  • Recommendation: The Network should formulate a plan or process for ensuring the items remain up-to-date and clinically relevant.
    Final Report
• Sample Diversity - The lack of ethnic, minority, and educational diversity in the focus groups and cognitive interviews may affect widespread use of the Item Banks.
  • Recommendation: The Network should develop a concrete plan for future testing which ensures greater ethnic, minority, and educational representation so the PROMIS products are generalizable.

IILB.3. Research

The Review Panel commends the Network on the detailed approach to validation of the Item Banks and offers the following recommendations.

• Item Validity - The Review Panel recognizes it is often difficult to prove that items are measuring what they are intended to capture and the respondents' understanding of an item is the same as the researcher's. This concept, content validity, must be assessed as the Item Banks are tested in additional chronic disease populations.
  • Recommendation: The Network should continue testing the items and Item Banks in different chronic disease areas and ethnic groups to demonstrate their content validity. Moreover, the Network should conduct studies to help end-users interpret what the CATgenerated scores and changes in scores represent.
    
    Responsiveness to change must be demonstrated for the items and instruments to be considered useful.

III.B.4. Dissemination

Distribution of the Item Banks and CAT system strategies for dissemination, and ongoing development and maintenance are necessary steps to ensure long-term sustainability.

• Public Information Strategies - Dissemination of information about the project to potential users, and other groups, such as patient advocacy groups, may help to obtain funding and general support for the PROMIS products.
  • Recommendation: The Network should consider its multiple constituencies in developing a comprehensive dissemination strategy for PROMIS. This will assist in long-term funding and maintenance of the Item Banks and CAT system.
• Presentation of Project Information - Dissemination of the Network methods and findings are key to its sustainability. However, it is important that the project is presented in the context of both its capabilities and limitations.
  • Recommendation: The Network should have a process for ensuring presentations and publications do not overstate results, and they illuminate both strengths and weaknesses of the project.

Final Report

• Item Banks and Computer Software - The Network's research efforts and the PROMIS products are supported by public funds and public use. Thus, the items and software algorithms should be freely available to all potential users.
  • Recommendation: The Review Panel urges the Network to put the system software code, including item selection algorithms, into the public domain to facilitate widespread use by clinical researchers, institutions, and commercial organizations.
• Short Forms - If short forms constructed from the Item Banks contain items with high potential for variability across similar populations, the DIF impact could compromise their validity.
  • Recommendation: The Network should carefully consider what short forms will be constructed and how these will be used. Short form items should be selected to minimize the potential for DIF impact.
• Long-Term Support - The Review Panel raised concern over the long-term support of the project, including continued research and maintenance of the CAT system and Item Banks.
  • Recommendation: The Network should explore public-private partnerships to manage the long-term costs of maintaining the project. All potential partners should agree that the products of PROMIS should be free ly available.
• Industry Acceptance - The Review Panel believes that the Network needs to explore strategies that will induce the clinical research industry, including pharmaceutical companies, to utilize PROMIS.
  • Recommendation: To gain the interest and acceptance of the biopharmaceutical industry, the Network must demonstrate that the PROMIS products can be used for multiple chronic disease populations, to capture adverse events, and to distinguish the effects of therapeutic response.