Protocol for Program Reviews Under the Aegis of the Graduate School

I. Purpose

The purpose of systematic, periodic review and evaluation is to strengthen education throughout the University. Under Graduate School aegis, all units with graduate responsibilities will normally engage in the review process. These reviews should assist the faculty in the field to examine and evaluate their program (including its graduate and undergraduate components), aided by colleagues in related fields and by external visitors from other institutions and from business and government, if appropriate.

The goals of a review and evaluation include:

• assessment of the health and vitality of a program,
• determination of its strengths and weaknesses,
• analysis of its present effectiveness in terms of stated objectives,
• critical examination of program objectives in view of present or anticipated circumstances, and
• recommendations that develop a course of action to maintain and foster strengths, and to remedy weaknesses determined in the review process; or, on occasion, to close programs.

Both qualitative and quantitative data and assessments should form the bases for specific recommendations.

As mandated by the Board of Regents, all graduate programs will be reviewed on a cyclical basis. The Graduate School will support the review process in ways outlined in Section VIII below.

II. Documents Produced in the Review Process

There are three important documents produced in the course of the review process.

• Self Study. This report is developed by the faculty and administration of the program. Details of the Self Study are outlined in IV below.
• External Review Committee Report. As outlined in V below, an External Review Committee is appointed by the Graduate School Dean. Their task is to prepare a written report based on review of the Self Study and a site visit to the program.
• Program Response. The third major document is the program's response to the External Review Committee Report, As outlined in VI below.

III. Steps in the Review Process

Appropriate procedures may differ with field and circumstances. The procedures specified below are thus suggestive, not constraining.

The time required for the complete review and evaluation of a field (or group of fields) will usually be approximately one academic year. The Graduate School, in consultation with the relevant budgetary college dean(s), makes the determination that a given program will be reviewed. Once this determination is made, the following order of events usually will occur:

• Programs to be reviewed each year are selected by consultation between the Graduate School and the relevant budgetary college dean(s). Normally, programs slated for review in a given academic year will be so informed in Spring of the prior academic year. The process begins with an orientation meeting between relevant program faculty (chair, DGS, DUS and possible other administrative personnel), the budgetary college dean(s) and the Graduate School dean. When feasible, this meeting is held early in the Fall term.
• The faculty in the program being reviewed recommends to the Graduate School dean a pool of names of possible external reviewers. The Graduate School dean, aided by the list and in consultation with the budgetary college dean(s), identifies the External Review Committee.
• Each program under review, using its own data plus information from the Graduate School and the budgetary college, prepares the Self Study (See IV below for details). This preparation typically begins in the Fall and is completed in time for the External Committee site visit, usually the following Spring. When the self study document has been prepared by the program faculty, the Graduate School forwards copies of that document to the External Review Committee members, along with any other pertinent materials.
• The Graduate School and budgetary college dean(s), relying upon their general knowledge of the program and upon the program's self-study materials, prepare a list of questions to be considered by the External Review Committee.
• The External Review Committee holds a site visit, coordinated by the Graduate School, to review the program, typically in the Spring. The committee is requested to provide a written report within one month of the site visit. (For details see "A Note on the Site Visit" at the end of this section.)
• The External Review Committee Report is submitted for formal comment to faculty in the program under review.
• The faculty in the program under review prepares a written response to the External Review Committee Report, addressing questions and suggestions raised and, where relevant, plans for implementing those suggestions deemed appropriate and executable. This response is submitted to the Graduate School within two to three months after receipt of the External Review Committee Report.
• After receipt of the program response, the Graduate School schedules an implementation meeting with the program leadership, Graduate School and collegiate deans, and appropriate provost(s) to deal with recommendations arising from the Self Study, External Review Committee Report, and Program Response Document.

IV. Self Study

Below is a list of elements to be included in the Self Study for use by the External Review Committee. The list is suggestive rather than exhaustive and does not imply that subjects need be addressed in the order presented. The final contents of the Self Study will be determined by the program under review. Generation of the information included will be done primarily by the program under review, supplemented by data from the Graduate School and the budgetary college.

Where possible, tables and graphs should be used rather than discursive presentation. In many instances it is helpful to demonstrate how relevant indicators have changed in recent years, particularly since the last periodic program review.

General

• A concise history of the department, indicating pertinent points of its activities in graduate and undergraduate study
• A considered statement of the objectives of graduate and undergraduate study in the area
• A statement of the plan being followed to achieve program objectives
• Degrees offered (including date of establishment)
• Description of administrative, budgeting, and student affairs structure
• Departmental and collegiate organization (including lines of responsibility for daily decision-making)
• Promotions and appointments process for faculty
• Student admissions processes at the program and Graduate School levels
• Student financial aid processes at the program and Graduate School levels
• Graduate and undergraduate student advising processes (including description of grievance procedures)

Faculty

The following information will prove useful in constructing a schematic faculty profile: distribution by age, academic work, sub-fields of interest, scholarly activity, and participation in the graduate and undergraduate programs.

• Aggregate statistical information on the number and level of faculty, on student-faculty ratios, and on faculty examining committee assignments within and outside the program (if possible, statistical data from comparable institutions should be included).
• Curriculum vitae of faculty including: academic rank, degrees with dates and awarding institutions; years on the graduate faculty; number of students advised to completion of masters and doctoral degrees; current number of graduate student advisees; service on student examination committees; current research projects and support; publications; all-University and relevant community service; professional honors received, etc. Curriculum vitae should be non-discursive in form with information easily readable and extractable for an overall profile of the program's graduate faculty.

Students

1. Profile of graduate students admitted to study in the program area.
   • Undergraduate institutions which admitted students attended and, where applicable, institutions of prior graduate work.
   • If available, mean undergraduate GPA, mean prior graduate GPA, and mean GRE and TOEFL scores.
   • The number of annual inquiries (if available) and completed applications, including the fraction admitted and rejected, and the percentage of admitted students who actually register.
   • Statistical data on minority and women applicants
2. Enrollment and related information, both graduate and undergraduate, where appropriate.
   • Course enrollments for the past five years.
   • Degrees awarded during the past five years.
   • Length of time students are in the program, credits accumulated, time taken for students to complete stages toward degree goals; e.g., course work, preliminary examination or equivalent, thesis, etc.
   • Attrition rate; the number of students who voluntarily terminate program (including the reasons offered)
3. Financial support
   • The number of teaching and research assistantships
   • Service time expected for graduate assistantship appointments
   • Stipend levels
   • Fellowships
   • Maximum length of time a student may hold service and non-service appointments
4. Student placement
   • Description of the process by which the program assists students in finding employment
   • Information on the place of initial employment
   • Present employment opportunities for students who complete the program

Graduate program (curriculum and degree-related)

1. Identification of curricular offerings in the program (including arrangement by sub-field, frequency of offering information, and compilation of curricular and program changes in past five years)
2. Description of the approximate length of various degree programs in terms of number of credits and time required to reach degree
3. Description of typical programs (including major, minor, supporting fields, etc.)
4. Where applicable, a description of special program requirements or options for Plan B master's degrees
5. Nature of special program requirements or options for candidacy for Ph.D.; e.g., preliminary written examinations, papers required, or other written requirements
6. Description of student involvement in research and scholarship (including how initiated, how supervised, and how graded)
7. Course grade distributions; results of written and oral preliminary examinations, and final examinations
8. List of students' theses for the past five years
9. List of student honors, awards, participation in professional meetings and publications.
10. Description of advising and thesis advising process (including scope of adviser's responsibility vis-a-vis graduate studies committees, number of students "counseled out" of the program and on what basis--course work, research, or examination performance)
11. Description of the process, if any, by which a student is guided to prepare for examinations and research

Facilities and Resources

1. Number and location of private and double offices for faculty
2. Amount and location of space available for graduate assistants
3. General description of library holdings, services, and resources as they relate to the program (e.g., specialized collections, departmental libraries, etc.)
4. Number and size of teaching and research laboratories (including a description of equipment and instrumentation)
5. Description of computer and technical support services
6. Geographical relationship of the program to other programs with which it normally interacts

Internal Ratings or Measures of Quality

Information should be provided about current graduate students' perceptions of the program (e.g., with respect to effectiveness and general climate). Where appropriate, assessments of relevant program aspects should also be provided by the program's civil service staff and post-doctoral associates.

External Ratings or Measures of Quality

Where available, information should be provided about the evaluation of the program by groups and individuals outside the field and not directly connected with the University. This may include:

1. Assessments by recent graduates (within the last five years) of their experience and the adequacy of the program as preparation for subsequent professional work
2. Evaluations by other University programs with which the program interacts
3. Evaluation by professional or other external accrediting or reviewing agencies where these exist
4. National Research Council rankings, or other external evaluations if appropriate

Interdisciplinary involvements of the program, including:

1. Faculty participation in other programs via, e.g., joint or adjunct appointments, cross-listed courses, graduate committees, research and writing.
2. Graduate student participation via, e.g., minors or interdisciplinary programs.

Academic plans

Future plans for the program as developed by the faculty in the field should be described. Program changes in view of modified or new objectives, anticipated enrollment, personnel, facilities, and resource requirements should be described. If long-range plans require the commitment of additional resources, an estimate of probable costs should be included.

V. External Review

The suggested format of the visit for the review and evaluation of the program by the External Review Committee is as follows:

First Day

1. Meeting with Graduate School deans and collegiate deans
2. Interviews with program administrators; interview with representatives of related units
3. Lunch with deans and program leadershipInterviews with program faculty; examination of facilities
4. Evening meeting of External Review Committee only

Second Day

1. Continuation of visits with program faculty
2. Interviews with graduate and undergraduate students
3. Working luncheon for External Review Committee only to prepare final oral report
4. Final oral report of External Review Committee presented to graduate and collegiate deans and appropriate provost(s) to report tentative findings and recommendations

Modifications in this format are required when longer site visits are needed.

It is intended that the External Review Committee will assess the program based on the Self Study and information gained during the site visit. Since the examples cited above for content of the Self Study are only suggestive, it is assumed that the committee will widen (or contract) the scope of the evaluation in keeping with the unique characteristics of the program being reviewed.

Many of the evaluative questions applicable to the components of the Self Study will be readily apparent to members of the review committee. However, a few illustrative questions for selected components are offered below:

• IV A.2,3. How well are program objectives formulated? Are the objectives appropriate for developing and maintaining a high quality program?
• IV A.5. Does the advising process permit students adequate access to advisers and systematic counseling at all stages of their career?
• IV B How effective are the faculty and the general program climate as viewed by the students currently enrolled in the program?
  Are graduate advising loads and examining committee assignments evenly distributed among the graduate faculty?
  Does graduate faculty associate or full status adequately reflect each faculty member's research activity and advising capabilities?
  Is the quality of students admitted to the program adequate? Should there be any procedural changes to improve the admissions process?
• IV C.2. Is the average time taken by students to obtain advanced degrees reasonable or excessive? If excessive, what procedural changes can be recommended to expedite the process while maintaining appropriate academic standards?
• IV C.3. Are opportunities for TA and RA appointments adequate in the opinion of the graduate students? In the opinion of the External Review Committee?
• IV D. Does the program sufficiently take into account individual student needs and aspirations? Is the program providing offerings that meet the changing nature of the field and social needs? Conversely, has the program over-reacted to transitory internal and external elements?
• IV D.3,4,5 Do the program requirements permit overspecialization which might hamper the student's placement opportunities and future development in the field?
  Do new developments in the field, the interests of the faculty, and interaction of curricular offerings with other disciplines suggest the feasibility of developing new cross-disciplinary graduate programs?
• IV E. Is student access to information sources adequate (e.g., libraries, special collections, e-mail)?

The above are examples of the sort of questions that could be posed with respect to all categories of information and data regarding the program.

VI. PROGRAM RESPONSE

The External Review Committee Reports will typically contain critiques, recommendations for improvement, and suggestions about optimal future directions for the program. In some instances, critiques and suggestions may be deemed by the program faculty to be either inappropriate or unworkable. The purpose of the Program Response is to provide the program personnel opportunity to revise plans where appropriate, initiate changes where needed, and provide justification for not following recommendations where it is believed that those recommendations are either based on misunderstandings or are ill-advised in light of existing constraints or alternative opportunities. This does not mean that the Program Response is simply an appologia for making no changes, but rather a reasoned consideration of the validity of points raised and the steps to be taken to maximize benefits derived from the External Review Committee Report. As noted above, assuming that the External Review Committee Report has been received within one month of the site visit, the Program Response should be prepared and forwarded to the Graduate School dean within the academic term following the site visit.

VII. IMPLEMENTATION MEETING.

After receipt of the Program Response in the Graduate School, the implementation meeting will be scheduled. Participants in that meeting will include relevant program staff (chair, DGS, DUS, ...), the budgetary college dean(s), the Graduate School dean(s) and the appropriate provost(s). The intent of the meeting is to reach final agreement on issues raised during the entire review process.

VIII. SUPPORT PROVIDED BY THE GRADUATE SCHOOL

The following types of support are offered by the Graduate School to assist the program:

• Data on enrollments, courses, quality of applicants, etc. These data are available from the Graduate School data base. When a unit is scheduled for review, the Graduate School dean will alert data management personnel in the Graduate School to prepare and forward the data to the program staff.
• Contacting and arranging for travel and lodging of the External Review Committee.
• In coordination with program personnel, arranging the schedule of the External Review Committee members during the site visit.
• Paying the costs of
  • the site visit by the External Review Committee
  • duplication of the Self Study for distribution

   

Download this Document