Notes Toward Improving Program Reviews

by Edward Schiappa
Associate Dean
The Graduate School

January 2000 Draft

Executive Summary

Across the nation, academic program reviews in general and graduate program reviews in particular are being rethought and revised. The primary problem is that there is often a gap between such reviews and the everyday institutional structures concerned with program planning and resources. Reviews imposed "from above" are too often treated as a nuisance rather than as an opportunity. To improve this situation at the University of Minnesota, the following recommendations are forwarded in this report:

Introduction

This report was drafted in response to a request by Vice President for Research and Graduate School Dean Christine M. Maziar to evaluate the current practices of program review and to identify possible improvements. The opportunity for an evaluation was created when Dean Maziar suspended most department and program review activities while the University made the transition to semesters.

The report is informed by research and interviews conducted by the author with a number of stakeholders.

• Appropriate present and former deans and directors of the Graduate School.
• Deans from the University's Schools and Colleges.
• Various faculty, students, and staff from programs recently reviewed.
• The Council of Graduate Students.
• Various University data managers and planning analysts.

Materials from other institutions and organizations interested in graduate program review were solicited and reviewed including:

• The American Association of Higher Education project on program review.
• The Council of Graduate Schools.
• The Association of American Universities.
• Selected graduate schools identified as having model review programs.

My research was supplemented in valuable ways by a draft report funded by the Pew Colloquium on Quality Assurance titled Evaluation of Academic Departments: A Strategy Paper published in 1999 by Jon F. Wergin and Judi Swingen (hereafter cited as Wergin & Swingen) and by Strategies for Change and Improvement: The Report of the Task Force on Accreditation of Health Professions Education, also published in 1999.

Why evaluate the program review process now? For the most part, the pause in program review afforded by semester transition is simply a good opportunity to see if the process could be improved. Also, there is a fairly widespread belief that the University is not always getting its money's worth from the current program review process. Most departments appear to view program review as something to be endured and that rarely produces tangible results. The relative benefit of the program review process, for most participants, does not seem to outweigh the considerable time and effort that goes into reviews.

Some of the core recommendations in this document can be found in the 1992 "Report of the Committee to Review the Graduate School," better known as the Johnson Committee Report. That report called for regular data collection "relevant to the assessment of program performance" (p. 11). Such an initiative is consistent with President Mark Yudof's "Workplan" for 1999-2000 in which he calls for "a more data-driven environment and management-oriented system for the collection, analysis, and dissemination of institutional data" and for continued implementation of the compact planning agreements (p. 3). Accordingly, this is an opportunity to improve a process that could be of considerable benefit to the institution.

A caveat is in order at this point: This report does not pretend to be a detailed or exhaustive analysis of all aspects of the current review process. It is unavoidable in a report such as this that the descriptions are oversimplified and that there are notable exceptions to most of the generalizations made here. Nonetheless, I believe that it is accurate to say that "most" Deans, chairs, and programs share the perceptions summarized within this report.

Attitudes Toward the Current Review Process

The goal of program review is to assess quality and to aid planning. According to the Protocol for Program Reviews under the aegis of the Graduate School:

The goals of a review and evaluation include:

1. assessment of the health and vitality of a program,
2. determination of its strengths and weaknesses,
3. analysis of its present effectiveness in terms of stated objectives,
4. critical examination of program objectives in view of present or anticipated circumstances, and
5. recommendations that develop a course of action to maintain and foster strengths, and to remedy weaknesses determined in the review process; or, on occasion, to close programs. (p. 1)

Reviews are an opportunity to assess the strengths and weaknesses of individual programs and departments and to help plan for the future. Such assessments are important not only for the department but also for the Graduate School, the respective school or college, and the Provost's Office. The problem is that there is a widespread belief that our current program review procedures provides less "added value" to the quality assurance process than they should.

I believe the problem with the current review process is partially structural, partially cultural. In general, program review works best when the individual college and program buys into the review process and sees it as an opportunity rather than as a nuisance. For a variety of historical reasons, the review process is currently viewed by most programs and collegiate deans as more of a compliance issue than as a productive opportunity to evaluate and improve programs. In my research, every single review that was referred to as a success was the result of fortunate timing where the program was going through a transition and wanted outside guidance. Otherwise, most reviews were seen as having little value. I believe this is true for several reasons:

First, the review process is run from the top-down. Though individual departments and programs can request a review, the standard comprehensive review is typically initiated by the Graduate School according to a random calendar cycle. Because most reviews are imposed rather than invited, a compliance mentality is quite common. As noted by Wergin & Swingen, the impacts of program reviews are, as a result, "generally modest" (p. 4).

Second, most reviews take place only about once every ten years. As a result, such reviews are not integrated into the culture of the departments and colleges as part of an on-going process of quality assurance and self-assessment. Program reviews are seen as "one-shot affairs, not well integrated into the life of the institution" (Wergin & Swingen, p. 4). At the current time at the University of Minnesota, for example, there is no explicit or obvious institutional connection between program review and strategic/budgetary planning process, whether in annual compact discussions or in formulating significant new budget initiatives for the legislature.

Third, the self-studies departments are required to do are felt by many to be a very time-consuming exercise in futility. Though some departments indicate the process of self-study was a valuable exercise in reconsidering the department's identity and future, most view the self-study as a nuisance. With only one exception, collegiate deans and members of external review committees commented that the self-studies generated are often "useless" because they are too long and do not include what deans and reviewers feel is valuable information. There is a widespread belief that "no one reads the self-studies" after they are completed.

Fourth, the most important agents of change within a department or program are the faculty, and the vast majority of faculty feel negatively or indifferently about program review. Such attitudes are not surprising. Faculty are reviewed annually for their accomplishments as individual teachers and researchers. There is rarely reward and thus motivation to invest in collective action to improve the department. As Wergin & Swingen note, "Usually faculty work alone, and especially in larger institutions they are rewarded according to standards of quality dictated by their disciplines, not by standards specific to their institutions or departments" (p. 2).

Fifth, in most (though not all) cases collegiate deans do not believe external reviews produce much value. The strongest form of this attitude was expressed by one dean in roughly the following fashion: "The self-studies are self-serving and defensive, departments pick friends as their outside reviewers, and the whole thing is approached as an occasion to press for more resources." Several deans claimed that they had never learned anything from an external review team that they did not already know about a department.

There were other criticisms of the review process that came up that will be discussed below within the context of specific recommendations.

While there are widely held negative feelings about program reviews, no one ever urged that we simply abolish them. Almost invariably someone would bring up an example of some insight that was gained or benefit derived from a well-timed review. Thus, the challenge is to improve the process so that the goal of quality assurance is better met and all concerned feel that reviews are worth their time and effort. I should note also that there is a difference between a "nuisance" and a "major problem." No one I spoke to felt that there is a crisis or major problem being caused by the current review process. Furthermore, deans who already run their own program reviews-such as in the Medical School, Law School, and School of Dentistry-are quite content with their current procedure. But concerning those reviews administered by the Graduate School, there is a good deal of sentiment that we could be getting much more for the current investment of time and money.

The remainder of this report will consist of a series of recommendations and supporting rationale. Some recommendations overlap while others are quite independent. Collectively the recommended changes constitute a major departure from the current review process. They invite a significant institution-wide change in the way program quality is assessed.

Summary of Recommendations

1. Augment the current review approach with annual "departmental profiles" collected through a web-based instrument and included as a supplement to departments' annual compact documents.
2. Increase the range of review process options, including the use of "focussed," internal, and "mini" reviews.
3. Modify the calendar cycle approach to increase the ease of programs to invite a review out of sequence, and increase the ease of programs to obtain an exemption from a review.
4. Increase the role of the budgetary college in organizing and conducting the reviews. The Graduate School should limit its role to those portions of the review pertaining to the graduate programs of the department.
5. Increase the role of the Provost's Office in organizing and conducting the reviews. Specifically, reviews should be conducted officially under the auspices of the Provost's Office in conjunction with the Graduate School and the budgetary college.
6. There should be an additional consultation involving the Provost's Office, the collegiate dean, and a Graduate School representative before and while reviews are being organized.
7. Except in the case of special requests, the current suspension of reviews should continue for an additional year.

Recommendations and Rationale

1. Augment the current review approach with annual "departmental profiles" collected through a web-based instrument and included as a supplement to departments' annual compact documents.

   Rationale: Program quality cannot be accurately ascertained and assured with a review only once per decade. Rather, those metrics that are felt to be important indicators of program effectiveness and efficiency can and should be identified and utilized on an annual basis as part of a process of continuous quality assessment. In particular, schools and colleges (including the Graduate School) should collect appropriate data for each program that is, in turn, routed to appropriate administrators involved in on-going policy and budget planning.

   To some extent current budgetary and compact planning already involves departments and programs regularly assessing strengths, weaknesses, directions for the future, and resource needs. The challenge is to include in such annual conversation the issues and data appropriate for quality assurance and improvement.

   This initiative is consistent with a recommendation made by the Johnson Committee in 1992 to conduct "annual audits" of graduate program data "such as GPA and GRE scores of incoming students, time to completion of degree recipients, and the progress of cohorts through a program" (p. 13).

   Some colleges already routinely require collection of departmental data. CLA, for example, submits a list of questions to all departments each year prior to budget meetings. CLA's questions do not include a call for quality-assessment indicators, but it would be fairly simple to add such questions. By contrast, CBS maintains a "Data Book" with various quality indicators that normally are collected only through once-per-decade reviews. Currently the Graduate School requires various performance indicators be reported by programs through a web-based survey that could serve as a model for other administrative units. Almost any data that an administrator would want is being collected by someone-the point is for that data to be processed and presented on a more regular basis. Accordingly, the suggested change here involves logistics more than policy.

   The point here is to begin to change the culture. Given the relative ineffectiveness of once-per-decade reviews for monitoring and assuring quality, the institution must more fully embrace the idea that quality assurance and improvement are continuous processes. Some departments produce Annual Reports that utilize various indicators of excellence to demonstrate program effectiveness, while others will only produce or consider such data if prodded. The goal here is to identify those performance indicators we feel are important and to be sure that they are collected and distributed to appropriate university offices on an annual basis. As Wergin & Swingen note, "the less the evaluation of departments is controlled by central administration, the less it is couched in terms of a product (that is, a report or series of recommendations) rather than as a process that contributes to continuous improvement, the more credible, and thus more effective, the evaluation is likely to be" (p. 11).

   Such an initiative is also consistent with the Task Force on Accreditation of Health Professions Education's recommendations for improving the accreditation process. In their report, Strategies for Change and Improvement, they suggest that "Commitment to continuous improvement should be part of the culture of health professions education, and processes related to accreditation should be integrated into everyday activities rather than conducted periodically as burdensome, externally imposed mandates" (p. 18).

   Concretely, what would on-going quality assessment entail at the University of Minnesota? The substance of this suggestion is to take the sort of program data now collected once every ten years (or so) and make sure it is reported annually to the relevant administrators and in annual program "compact" policy and budget planning documents. This information includes such items as enrollment figures, faculty teaching efficiency and effectiveness, research and grant productivity, etc. There is not so much need for new data (though the need for new categories may emerge) as it is the need to collect and distribute data pertaining to quality on a more regular basis. This is an important point because if this recommendation is perceived to be a large new burden on colleges and schools there will be resistance.

   For the Graduate School, this recommendation means initially to continue to develop the web-based survey as an instrument to collect performance indicators and to utilize such data in on-going policy and budgetary planning. At the University of Iowa, traditional program reviews are supplemented with a statistical "Profile" created by the Graduate College for use "as the basis for consideration of possible enhancements, reductions or elimination of programs" (p. iii). We should examine Iowa's approach carefully and consider adding to our survey components found in Iowa's profile that would enhance our quality assessment.

   For the provost's office and for some collegiate deans, this recommendation requires the creation of a list of certain baseline indicators. Attached is a draft document that describes the performance indicators that could become part of an annual supplement to departments' compact planning documents. I suggest that the Provost's Office and the Graduate School, in consultation with the collegiate deans, create a web-based instrument similar to what we have in the Graduate School that would allow individual programs and departments to create an annual "profile" similar to what one might produce in an annual report. Such profiles should be relatively simple and straightforward, designed to identify particularly outstanding achievements by programs as well as to report routine data concerning teaching, research, and service.

   I should note that the deans with whom I shared this idea were supportive of it. Some deans felt that they already has such profiles in place, while others acknowledged the need to improve their data collection process.

2. Increase the range of review process options, including the use of "focussed," internal, and "mini" reviews.

   Rationale: The Graduate School should not try to make one size review fit all programs. That is, the Graduate School should encourage flexibility in terms of using different sorts of reviews, including a greater use of internal reviews where faculty from other University of Minnesota programs are brought in to review a program, or where a team of one or two outside reviewers are combined with one or two home faculty to conduct a more focussed, "mini" review of only those aspects of the program that the dean and faculty of the program feel needs the most attention.

   Reviews generally should be focussed rather than comprehensive. The chairs, faculty, external reviewers, and deans I interviewed all agreed that reviews are almost never thorough and comprehensive because of the time pressures involved. Typically, in a Graduate School-initiated review, the External Review Committee is given an extensive list of questions that tend to be comprehensive in scope. These questions are rarely all answered, or if they are many of the answers are very brief. Rather that set up the review to fail to meet its stated goals, or run the risk that the ERC is spread too thin and does not devote enough time to the most pressing concerns of the program, reviews should be focussed on the key issues that the relevant deans and faculty agree deserve most attention.

   A number of the graduate programs associated with health and professional schools undergo periodic accreditation reviews. While some deans expressed a preference to keep program review and accreditation processes separate, others, including Dentistry, Public Health, and Nursing were strongly in favor of smaller, focussed reviews that could be coordinated with accreditation. Such a review would focus just on those parts of the graduate program that are not covered by accreditation. For example, in the coming year the School of Nursing will undergo accreditation review of its M.S. program but not its Ph.D. degree program. Dean Edwardson suggested that it would be easier on the program to have its Ph.D. program reviewed now when they have gathered a good deal of data about the School of Nursing, as opposed to a review timed as a result of a random calendar cycle. In short, an added advantage of focussed reviews is that they are much easier to "piggyback" to accreditation reviews than a comprehensive review would be.

   The 1992 Johnson Committee report's recommendation #10 includes such suggestions as more focussed reviews and periodic internal reviews (pp. 12-13).

   Focussed reviews should involve the ERC members in the creation of a schedule for their on-campus visits, since the current template for such visits is not a good fit for a focussed approach and tend to spread ERC members' time too thin.

   As a logistical improvement, the University of Minnesota should consider purchasing laptop computers for ERC use while on campus that has the focussed questions already loaded. Such laptops could also contain digital copies of the program's self-study and other data that may be useful to have convenient for ERC members.

3. Modify the calendar cycle approach to increase the ease of programs to invite a review out of sequence, and increase the ease of programs to obtain an exemption from a review.

   When specific program reviews were praised, it was always due to fortuitous timing. Programs are most interested and willing to "buy into" a review process when it recognizes that there is a significant problem or faces a transition of some sort. Deans praised those reviews that happened at a time when there was a felt-need for outside consultation. On the other hand, programs who felt that the timing of a review was inopportune engaged in various strategies to resist the review including, in some cases, refusing to write a departmental response after the review was over.

   There is no particular merit to a random calendar approach except that it allows the relevant administrators to avoid responsibility for the timing of a review. Furthermore, once-per-decade reviews are too infrequent to assurance quality programs.

   Accordingly, Deans and program chairs should be encouraged to invite reviews when there is a felt-need for them. This is already an option, but it is an option that deserves to be reemphasized. Additionally, when a program comes up in the random calendar cycle system, the program should be allowed to be "exempted" from review if the relevant collegiate Dean, the Graduate School Dean, and the Provost's Office all agree.

4. Increase the role of the budgetary college in organizing and conducting the reviews. The Graduate School should limit its role to those portions of the review pertaining to the graduate programs of the department.

   The Board of Regents' Policy makes it clear that all college constitutions should include a provision that makes the collegiate dean responsible "for the development and periodic review of departmental programs." Indeed, the one college constitution I researched- CLA-specifies that the Dean's responsibilities "include continuous review of departments, programs, and schools within the college." The Graduate School's own constitution similarly gives the Dean executive responsibility for the review of graduate degree programs and their faculties (Article II-4-k).

   Accordingly, the most appropriate scope of reviews organized and initiated by the Graduate School would be limited to graduate educational efforts. The tradition of having the Graduate School "staff" most departmental reviews has evolved to the point that the Graduate School is responsible for initiating, coordinating, and funding such reviews. Unfortunately, such a tradition has led to the problematic perception that the reviews are "owned" by the Graduate School.

   Despite the oft-cited caveat that the Graduate School merely staffs program reviews that are supposed to be comprehensive in scope, there is a widespread perception that these are "Graduate School reviews" that focus primarily on graduate programs. This perception is reinforced by the fact that during site visits the Graduate School's presence is rather significant compared, for example, to the Provost's Office. During the opening and closing meetings there tends to be a heavy emphasis on graduate-related issues. Sometimes, of course, the graduate program should be the main focus. The problem is that there is a mistaken impression that we "own" the reviews and that the graduate program is our primary point of interest.

   There are several deans who are under such an impression, and it is safe to assume that many chairs and faculty believe similarly. It is also a mistaken impression held by many external reviewers unless and until they are told otherwise. Such a perception is symptomatic of a mentality that sees reviews as merely a matter of compliance rather than an opportunity for reflection and change.

   The best reason for locating the primary responsibility for organizing and conducting reviews in the budgetary college is to enhance the interface between program reviews and the most appropriate institutional structures involved with planning and resources. A review is pointless if those best positioned to effect change are unwilling or uninterested in doing so. The current practice leads to rather undesirable though unintended problems. In one case a collegiate dean told me that she felt her most pressing concerns about two departments were not addressed in a recent review and so a valuable opportunity for significant change was lost. In other cases departments or programs are skeptical of the review process because they believe a dean will simply ignore ERC suggestions if they do not fit with that dean's plans. In any case, it is reasonable to infer that deans are less invested in the conduct and results of program reviews than they would be if they had more intimately involved in financing and organizing the review.

   What sort of structures would this recommendation require? In some units, this recommendation is already the status quo. No changes would be necessary for the Law School, Medical School, Dentistry School, or the College of Agriculture, Food, and Environmental Sciences, for example. Probably the most significant change would be for the College of Liberal Arts, but this is also the College where the review process is most often seen as a nuisance. Reviews within CLA and all colleges should be limited to those that the relevant deans are genuinely interested in being involved with and committed to following through on the review's outcomes.

   To be sure, the Graduate School can and should play various sorts of supporting roles, including the training of relevant associate deans to conduct their own college's reviews and to participate in those portions of the reviews that pertain to the department's graduate programs. For example, if it is decided to piggyback a focussed review of a graduate program with an accreditation review of an undergraduate or graduate professional school, it makes sense for the Graduate School to take responsibility for organizing and funding the graduate program review in coordination with the relevant dean's office. Also, the responsibility for coordinating reviews of interdisciplinary graduate programs may rest with the Graduate School. However, the primary locus of control for more comprehensive departmental review ought to be in the hands of the relevant collegiate dean.

   While I believe this suggestion may receive the most opposition of all the recom-mendations, I am convinced that if we want program reviews to improve dramatically in quality and effectiveness, this is the single best step to take to accomplish such a goal.

5. Increase the role of the Provost's Office in organizing and conducting the reviews. Specifically, reviews should be conducted officially under the auspices of the Provost's Office in conjunction with the Graduate School and the budgetary college.

   Rationale: The rationale for this suggestion is similar to that of the previous point. As noted above, we in the Graduate School understand that our function is to staff and organize comprehensive reviews of programs. But most people outside of the Graduate School do not understand this and believe that these are "Graduate School reviews" that are primarily focussed on the graduate program. The result is problematic in part because the Provost's Office is not involved in the review process as thoroughly as it should be. Also, the perception that the focus is on the graduate program sometimes results in a disconnect between the department's felt-need to focus on resources and faculty lines and the external review committee's felt-need to assess the graduate program quality. More than one dean also expressed the feeling that more emphasis needs to be given to the undergraduate program of departments under review.

   By conducting the reviews officially in conjunction with the Provost's Office and the budgetary college we can reinforce for all concerned the fact that these reviews are not limited to the graduate programs involved. Furthermore, we can enhance the connection between the improvements needed that are identified in the review process and those university offices (collegiate deans and provosts) who are in the best budgetary and policy position to facilitate such changes.

   Such a change would not require additional resources though, depending on how many external reviews occur in a given year, the number of meetings that require a provost's participation could increase.

6. There should be an additional consultation involving the Provost's Office, the collegiate dean, and a Graduate School representative before and while reviews are being organized.

   Rationale: This recommendation simply sums up the process implications of the previous two recommendations. Currently a review might be initiated at the request of a dean, a department chair, or the Graduate School. The first organizational meetings typically involve only representatives of the department and the associate dean of the Graduate School who administers reviews. Prior to such a meeting, there should be consultation with the appropriate collegiate dean and the Provost's Office to facilitate the process of focussing the review on issues of particular concern. Such consultation will enhance the connection between such reviews and subsequent policy and budgetary planning at the college and university level.

7. Except in the case of special requests, the current suspension of reviews should continue for an additional year.

   Rationale: Several deans specifically requested that reviews be suspended for an additional year. Starting up the process for fall 2000 means that programs will have to work on self-studies and lining up external reviews in the coming spring semester. Some deans felt that this was too soon since many programs have implemented significant curricular change starting this fall and it will be too soon to assess the effect of such changes. Furthermore, the fatigue factor was mentioned since we would be starting up the review process in the midst of our first year in semesters.

Comments or Questions? Contact Edward Schiappa at 626-0338 or schia001@umn.edu

¹I have met with the following: Vice President and Graduate School Dean Christine M. Maziar; Associate Vice President Robert B. Kvavik; Assistant Vice President and Graduate School Associate Dean Victor A, Bloomfield; Dean Mary Heltsley, College of Human Ecology; Dean Steven Rosenstone, Associate Deans Ann Waltner, Michael Hancher, & Barbara Reid, CLA; Dean Alfred Michael & Associate Dean Robert Howe, Medical School; Dean Robert Elde, CBS; Dean Thomas Fisher, College of Architecture and Landscape Architecture; Dean John Brandl, the Humphrey Institute of Public Affairs; Associate Dean Pat Kumar, IT; Dean Marilyn Speedie, College of Pharmacy; Dean Steven Yussen & Associate Dean Robert Serfass, College of Education and Human Development; Dean Jeffrey Klausner & Associate Dean Larry Schook, College of Veterinary Medicine; Dean Edith Leyasmeyer & Associate Dean John Finnegan, School of Public Health; Dean Sandra Edwardson, School of Nursing; Dean Michael Hill, School of Dentistry; Dean Charles Muscoplat & Interim Dean Philip Larsen, College of Agriculture, Food, and Environmental Sciences; Dean David Kidwell, the Carlson School of Management; Dean Al Sullivan & Associate Dean Steven Laursen, the College of Natural Resources.

²IncludingDarwin D. Hendel, Susan M. Grotevant, Lincoln Kallsen, Bradley E. Bostrom, and Peter M. Radcliffe.