Guidelines for Program and Institutional Quality Reports
The following guidelines for program reviews and for institutional quality reviews were originally adopted by the Rhode Island Board of Governors on June 6, 1986, as described in the paper Quality in Higher Education, Program and Institutional Review Processes. The guidelines were modified to include information on trigger mechanisms (July, 1987), and new indicators of quality were added (December, 1988). The institutional review process was slightly refined to streamline the process (January, 1990).
On the basis of additional experience, the institutional review process has been further redesigned to be less time-consuming to prepare and to review and more useful to the institutions and the Office of Higher Education. Minor editorial changes have been made in this edition of the guidelines (December 1994).
Guidelines for program reviews
Each institution has a distinct mission, academic expertise, and understanding of its own programs. Therefore, each institution’s program review process should be incorporated into the foundation of the system-wide process. In addition, those departments and/or programs that regularly undergo national accreditation assessments can use these accreditation reports in place of reviews conducted by the institution.
The following factors should guide an institution in the preparation of program review reports for the Board of Governors:
1. Scope of program reviews
Reviews usually will encompass an entire academic department or the equivalent; both the undergraduate and graduate program units will be considered. Institutions may, however, choose to conduct special program reviews on some other basis — see Point 5.
All program review teams should include at least one external evaluator. Persons who are external to the department and/or the institution provide an additional perspective that enhances the program review.
2. Programs that are nationally accredited
Institutions will provide the Office of Higher Education with a listing of all programs that undergo national accreditation. For each program listed, the name of the accreditation agency and the schedule for the accreditation process will be included.
All programs accredited in a particular year will become part of the program review report for that year. The information in Point 4 will be provided on the programs reviewed.
Each institution should provide the Office of Higher Education with a synopsis of the findings of regional and national accreditation reports. The reports should contain information from both the self-study and the accrediting team’s report. In addition, each institution should submit on an annual basis a listing of scheduled accreditation visits for the coming year.
3. Programs that are not nationally accredited
Those programs that are not reviewed by national accreditation agencies will undergo a program review initiated by the institution. Within a cyclical timeframe of three to seven years, each institution will determine a specific schedule for program reviews. The information listed in Point 4 will be provided on the programs reviewed.
4. Reporting of program reviews
Program review synopses will focus on the findings and the recommendations for all program reviews completed for a given period.
Each report should contain the following information:
- name of the program reviewed
- general description of the program
- trends, past and projected, in enrollment and degrees granted
- characteristics of program
- description of the review criteria
- major strengths and weaknesses of the program
- review findings
- institutional action based on the review
- timetable for future reviews
- names and titles of the program review team and accreditation agency, if appropriate
5. Special program reviews
Under unusual circumstances, an institution may initiate a review, or the Office of Higher Education may request that a program be reviewed. The following considerations might trigger a special review:
- number of degrees granted is below the prescribed number — see Point 6
- enrollment in the program is below a certain number
- trend in enrollment is downward
- an imbalance in faculty/student ratio exists
- cost-effectiveness of the program is questionable
- jobs for graduates are scarce
- accreditation report is negative
- program exists at another Rhode Island public institution
6. Triggered reviews
If the number of degrees granted by a department/program falls below the specified number for two consecutive years, the question as to whether the department/program should be reviewed out of cycle will be triggered. The trigger mechanism for degrees awarded is:
- 10 associate’s degrees
- 5 bachelor’s degrees
- 3 master’s degrees
- 2 first professional and doctor’s degrees
Using data contained in the annual Integrated Postsecondary Education Data System (IPEDS) Completions Report, the Office of Higher Education will identify small programs as determined by the number of degrees granted. Any programs which fall below the prescribed indicators for two consecutive years will be identified. If a department/program is triggered for an out-of-cycle review, additional information and/or justification for the program may be required.
The Office of Higher Education will maintain an inventory of small programs, but institutions should also annually review small programs.
7. Follow-up reviews
In some instances, members of the Planning and Program Committee may request a follow-up report of a program review. These reports will be prepared for the following year and should comment of the progress made by the department/program to address identified problems.
8. Program review synopsis form
A program review synopsis form has been developed as a guideline for the preparation of quality review reports. See appendix A for a copy of the form.
9. Schedule
A seven-year proposed schedule for both external and internal reviews will be maintained by the Office of Higher Education.
Guidelines for institutional quality reviews
1. Regional accreditation reviews
Institutions are reviewed at least every ten years by the New England Association of Schools and Colleges (NEASC). These regular reviews follow cycles established by the institution and NEASC. In addition, five-year updates, other special reviews, interim reports and/or special reports are sometimes required.
After any regular regional accreditation review or special reviews or reports are completed, the institution should submit the entire report or a synopsis to the Planning and Program Committee and to the Office of Higher Education. The following information should be included in the synopsis, as appropriate:
- description of the review process and names and titles of the accreditation team
- description of the review criteria, review findings, and recommendations
- institutional responses and actions based on review recommendation
- timetable for future reviews
2. Reports on institutional quality
In conjunction with the Office of Higher Education, the institutions will be prepared to report to the Planning and Program Committee on the quality indicators (see next section). All three institutions will report on the same indicator at the same time, so that, in addition to developing a system-wide perspective, the institutions can share information.
In general, the reports will include the following types of information, as appropriate for the reporting institution, on the indicator selected for examination
- summary data on the indicator
- significant changes that have occurred in this area since the last report
- any changes in this area which are under consideration
- any major impediments to effecting change in this area
- description of the review process
Selection of quality indicators
The joint report prepared by the academic vice presidents includes a caution that “observations not be mistakenly interpreted as judgments about quality.” However, as the term quality indicator is used here, that is to imply the presence of quality in the absence of more direct, in-depth measures, even observational measures are preferable to no measures at all. As stronger indicators are developed, less reliance on simple observations will be necessary. It is suggested, therefore, that observational information be provided as background or contextual information for the discussion of quality, while at the same time, in-depth analysis procedures continue to be developed.
It also should be noted that the institutions will be asked to supply an interpretation of the data provided. These interpretations would stress the relationship between the indicator and efforts being made by the institution to enhance quality.
In the pages that follow, the suggested quality indicators are grouped into similar categories. For each indicator, suggestions are made concerning the information sought and the appropriateness of the indicator for particular institutions. These matches between indicators and institutions are based, in part, on the responses by the institutions.
A. Background information on students
The institutions point out that, although they provide direction to elementary, junior high and high school students, evidence of student learning prior to matriculation does not reflect the quality of instruction at the institution of higher education. It does, however, influence the impression the public has about the quality of the institution. Concern about the characteristics of entering students is one of the reasons the institutions collect and analyze information on the students’ backgrounds and is also why the information should continue to be provided to the Board of Governors. This information should be examined in terms of trends to determine whether there is any perceptible change in the make-up of the incoming student body.
Pre-matriculated measures
Scope: Describe any placement tests or other pre-matriculation measures in use or planned. The results of basic skills tests used to place students in courses or into remedial programs should be reported in terms of the most recent data and comparison with previous results.
Institutions: URI, RIC, CCRI
Admissions
Scope: The annual admissions profile, which includes data on applications, acceptance rates, yields, enrollment rates and aptitude indicators (including SAT scores, class rank, merit-based scholarships, etc.), should be continued. This profile should be compared with the information for the past five years. This quality indicator report will be made annually.
Institutions: URI, RIC, CCRI (where appropriate)
B. Resource measures
The analysis of the availability and use of resources is frequently seen as one of the best available indicators of quality in an academic institution. The following resource indicators were selected:
Support for libraries/learning resource centers
Scope: Using the standards available for library collections, describe the degree to which the library/learning resource center at the institution compares with the standards for institutions of similar mission and size.
Institutions: URI, RIC, CCRI
Financial aid support
Scope: Financial aid is necessary to attract and retain students who might not be able to attend college without this assistance. The difference between the needs of the students for financial support and the availability of financial aid resources should be analyzed and reported.
Because of the positive correlation between students’ academic performance and work experiences on-campus as opposed to off-campus, the availability of support for on-campus student help should be included as part of the financial aid analysis.
Institutions: URI, RIC, CCRI
Incentives for promoting quality
Scope: If the institution has or plans to initiate any financial incentive programs intended to promote quality, the report should include a description of these programs, including participation rates and estimated costs. Incentive programs might include such items as faculty recognition awards, research fund awards, student achievement awards, etc.
Institutions: URI, RIC, CCRI
C. Faculty
As was indicated by the institutions, the “quality of any academic institution is a direct consequence of the quality of its faculty and professional staff.” Accordingly, the institutions have developed programs, policies, and standards that are intended to attract, retain and reward faculty members who are judged as contributors to quality.
Part-time faculty
Scope: Institutions use part-time faculty both as means of retaining flexibility and to conserve resources. When the number of part-time faculty becomes excessive, the quality of the institution may be affected because part-time people normally have a different type of commitment to the institutions than do full-time faculty.
The trend in the use of part-time faculty by departments should be analyzed and reported. Significant changes in the use of part-time faculty should also be reported.
Institutions: URI, RIC, CCRI
Support for faculty professional development
Scope: Describe any institutional programs designed to improve the quality of teaching, research and general development of the faculty. Indicate the degree of participation in these programs.
Include in this report a description of the institutional support provided for faculty development activities such as sabbatical leaves, research time, attendance at professional meetings, and institutional programs.
Institutions: URI, RIC, CCRI
Research and other scholarly/creative activities
Scope: Present data on the following faculty research/scholarly/creative activities:
- source and amount of external funding for research projects
- publications in refereed journals
- critical reviews of research by external review panels
- formal citations and awards for research
- appropriate measures for the creative and performing arts
Institutions: URI, RIC, CCRI (as appropriate)
Service
Scope: Service is recognized as one of the three components of faculty responsibility. Increasing service to the community, to the state and/or to the country is viewed as a contribution that higher education should make to improve the quality of life.
Provide information on the efforts being made by the institution to promote public service by providing incentives and recognition and by giving significant weight to this factor in decisions on faculty promotion and tenure.
Institutions: URI, RIC, CCRI
D. Special programs
To a large extent, the information on specific academic programs will be provided through the related program review process. However, there are some general programs which either cut across traditional departmental lines or which are run separately from the departmental structure. Information pertinent to these programs should be included in the institutional quality report.
Remedial programs
Scope: Provide a description of any remedial programs in place. Include in this description the institutional policy for granting credit for remedial courses.
Provide information on the preparation of entering students and the extent to which students require remediation.
Provide data on the extent of student utilization of these programs and measures of their effectiveness.
Institutions: URI, RIC, CCRI
General education
Scope: Provide information on the current requirements for the general education program.
Provide information on the methods used to assess achievement of general education objectives by undergraduate students.
Institutions: URI, RIC, CCRI
Guidance and advisement
Scope: Provide information on current guidance and advisement procedures
Provide data on the participation rate of students in these programs
Institutions: URI, RIC, CCRI
E. Outcome measures
The interest in using outcome information as a measure of academic quality has increased as a result of the questions raised about the validity of input measures to judge academic quality. Outcome measures suffer from some of the same problems as other measures in that they are related to student background factors which are not under the direct control of the institutions. Consequently, it has been argued that “criteria-based outcome measures provide a much clearer picture of college impact on student growth” than do other outcome measures of student performance. For example, assessing what students have learned using both pre- and post- tests that are based on the goals of the program are much more useful measures than just using final examination results. The development and implementation of criteria-based measures is a long-term, expensive process and may not be an assessment technique suitable for all institutions. In the interim, measures that will give some indication of student outcomes should be used.
Retention and completion
Scope: Provide current data on retention rates and compare with the past trends for the institution.
Provide information on existing retention procedures and any changes that have been made or are anticipated to be made to improve retention. The relationship between the retention procedures and the philosophy of the institution should be addressed.
Institutions: URI, RIC, CCRI
Outcome assessment and evaluation
Scope: Provide information on any outcome assessments and evaluations including any special entry, retention and/or exit requirements by each major field of study where such requirements exist.
Given the current interest and concern regarding teacher education programs, details on these programs should be addressed. In addition to information on entrance, retention and exit standards, provide information on any requirements of the teacher education programs that are external to the field of education (e.g., requiring a major in a subject matter field). Information on how students in teacher education programs compare with students in other fields on measures such as SAT scores and grade-point averages should be included.
Provide data on any assessment techniques that are in place or that are being planned. Describe the program and provide a proposed schedule for data reporting.
Provide information on the extent to which periodic reports are made to high schools on the performance of each high school’s graduates in basic skills tests, progress toward a degree and other possible indicators of academic success.
Institutions: Any of the three institutions that have specific evaluation criteria for programs. [Note: This information could be included in the program review reports as these are prepared.]
Follow-up on graduates and other former students
Scope: Provide a description of the procedures used to collect information on graduates and the types of information collected. Present data on future academic study, graduate or professional schools attended, GRE scores, professional examination scores, and such other follow-up data that are relevant to quality assessment.
Provide a description of the procedures used to collect information on former students who did not graduate, and on the types of information collected. Present data on the reasons students decided not to continue at the institution, and other information that might be relevant to quality assessment.
Institutions: URI, RIC, CCRI (transfer rates to senior institutions)
Student/alumni satisfaction
Scope: Provide a description of any measures of student/alumni satisfaction currently in use.
Report the results of the most recent assessment of student/alumni satisfaction programs and of data on career status of student/alumni.
Institutions: URI, RIC, CCRI
F. Other changes in academic policy
In addition to the quality indicators listed above, the institutions can be anticipated to make changes in academic policy for the specific purpose of improving quality. Any major changes in academic policy should be brought to the attention of the Planning and Program Committee. These changes might include implementation of standards for non-traditional courses (e.g., television instruction), modifications in the grading policies, newly instituted incentives to improve quality, or any other changes that are made to enhance the program.
APPENDIX A
Program Review Synopsis
Institution:
Date:
Department/Program and Degree Level(s):
Reason for Review:
Accreditation Internal Review
Special Review Triggered Review
Follow-up Review Other Review
Accreditation Agency (if applicable):
Date of Current Review:
Date of Last Review:
General Description of Program:
Enrollments (Five year trend and projected):
Degrees Granted (Five year trend and projected):
Class Size (Are class sizes atypical? If so, explain.):
Faculty Characteristics (Number of Full-time and Part-time, FTE):
Average Teaching Assignment for Full-time Faculty:
Number of Graduate Assistants:
Research Dollars:
Summary of External Activities:
Description of Review Criteria:
Strengths of Department/Program:
Concerns with Department/Program:
Major Conclusion of Review:
Institutional Responses to Conclusions of Review:
Names and Titles of the Review Team:
