Academic Program Review Process & Guidelines

Three Stages of Academic Program ReviewIRO DATA Packet – to be sent to Program ChairTarget Timeline for Academic Program Review | Useful Resources

 

The AUA Academic Program Review Process Guidelines

The Academic Program Review Process comprises the Self-Study, Self-Study Review and Wrap-Up Meeting. It is an opportunity to evaluate and improve our degree programs based on data, stakeholder input, collaborative analysis, student learning assessment outcomes, and feedback from external reviewers.   Although we continuously strive to improve our academic programs, the Academic Program Review process and specifically the Self-Study calls on faculty to periodically engage in evidence-based introspection about learning at the university.  Led by the Program Chair and overseen by the Dean, the Self-Study focuses the faculty on a data- and evidence-based investigation of the program’s activities resulting in formulation of recommendations for program improvement.  The Self-Study Review reviews the Self-Study for accuracy and completeness and provides an external perspective on related issues and recommendations.

This document outlines the required components of the Academic Program Review.The questions and evidence suggestions are meant as a guide to writing the Self-Study and Review, which should be presented in narrative form.

In general, Self-Study Reports are 15-20 pages, excluding appendices.

Three Stages of Academic Program Review

  1. Self-Study
  2. External Review Report
    1. Review Report prepared by Review Committee, composed of intramural and external-to-the-university reviewers from outside the program
    2. Factual Clarification Meeting/Conference Call (between Program Chair and Review Committee)
    3. Corrected Review Report submitted by External Review Committee to the Office of the Provost.
  3. Wrap-Up Meeting with Administration
    1. Wrap-up Meeting among Program Chair, External Review Committee Chair, President and Provost resulting in prioritized recommendations
    2. Program prepares and submits the jointly approved Prioritized Recommendations and Plans, which serve as the baseline for the next Self-Study

 

Self-Study

The Self-Study is an opportunity for program faculty to examine and analyze the educational effectiveness of the program through data and evidence.  The Self-Study should identify strengths and limitations and make recommendations to address identified problems, challenges and additional resources.

Model Table of Contents for the Self-Study Report

  1. Introduction/Context
    1. Internal and External Context
    2. History of the Program (with an emphasis on recent history)
    3. Mission Statement
  2. Disposition of Last Self-Study Prioritized Recommendations and Plans
  3. Evidence-Based Analysis of Program Quality
    1. Student Profile
    2. Curriculum and Learning Environment
    3. Student Learning, Assessment and Success
    4. Faculty
  4. Evidence-Based Analysis of Program Viability
    1. Demand for the Program
    2. Allocation of Resources and need for additional resources
  5. Summary & Plans
    1. Summary of Analysis
    2. Plans including multi-year student learning assessment plan
Types of evidence and questions to consider:
II.   Prioritized Recommendations from the previous APR should be itemized and discussed.

III.a.  Student data: age, gender, citizenship, test scores, employment status. 

III.b. Are the curriculum and program goals and learning outcomes up to date?  Is the curriculum aligned with learning outcomes? Is the curriculum sufficiently broad and varied?  Is the curriculum properly sequenced?  Is the curriculum map accurate? Is it updated? Does the curriculum reflect input from stakeholders (incl. external stakeholders)?  How does the curriculum benchmark against best practices?   Is the assessment process inclusive?  How?

III.c.  Disposition of annual SLA.  Samples of student work (i.e. assignments, portfolios, capstones), Student retention and graduation rates. Are students achieving learning outcomes, on time and at appropriate levels?  Are students prepared for work or further study? Are Capstones reflective of expected learning outcomes? How is student learning assessed?

III.d.  What are faculty qualifications? Do the qualifications fit the program mission and goals?  What are faculty background, expertise, teaching and research records, scholarship, professional service?  Have faculty participated in professional development/training?

IV. a. Demographic admissions trends (program v. university as well as program v. other national standards); application/acceptance ratios. Market trends for graduates. Competitor analysis.  What is the program’s recruitment plan? What is the history of demand for your program?

IV. b.  Faculty: Is there enough faculty and other resources to support the program including Student Support:  Academic and career advising programs and resources, tutoring, training of TA’s, remediation support, financial support, support for research, Information and Technology Resources, Facilities. Staff? Financial Budget: Including trends for 3-5 year period.

V. a.  SWOT:  Alignment of curriculum practices, processes and resources with goals of program? Program aligned with goals of students and faculty? Program aligned with University goals? Bottom-line: Are student learning outcomes being achieved?

V.b. What are your goals for the next few years?  How do you intend to achieve those goals?  Address identified weaknesses, build on strengths, improvement within existing resources, reallocation of resources, improvements requiring additional resources (with estimates), possible collaboration/synergies to improve quality

 

Required Appendices

  1. Curriculum Map
  2. List of Faculty Research and Scholarship
  3. Program Strategic Plan Milestones Update
  4. Syllabi for previous two years
  5. Links to on-line archive–where appropriate–for Faculty Meeting Minutes; Annual Student Learning Assessment; Course Descriptions; Degree Requirements; Student Handbook, Application, Admission, Retention Statistics; Entrance Score and Grade Trends; Student Demographics; Student Exit, Alumni and Employer Survey Results; Capstones or other Student Work Portfolios; Research Center Reports

Upon completion, the Program Chair submits the Self-Study Report with Transmittal Memo (see templates online) to the Office of the Provost.

 

External Review

The External Review  reviews the Self-Study and all of the relevant documentation and prepares a report.  The goal of the External Review is to ensure objectivity of the program review process, determine how the program compares to other similarly situated programs, and identify commendations as well as make recommendations for improvement.

  1. Goals
    1. Review the Self-Study for accuracy and completeness
    2. Note any additional issues, recommendations, taking into account their special expertise (e.g., disciplinary, student learning assessment)
  2. Process
    1. Program Chair submits completed Self-Study to the Office of the Provost, who reviews it for structural completeness (not substantive adequacy, just verifying that all sections are filled in and all appendices are included). If there are omissions, the report goes back to the Program for completion and resubmission within 14 days.
    2. The Review Committee  should include at least two internal and at least two external members.  The Review Committee Chair is generally a university faculty member experienced in AUA review processes, appointed by the Provost. .  The Dean, Program Chair and program faculty compile an annotated list of suggestions of external-to-the-university Reviewers with discipline-specific expertise.  The Provost selects from the list and invites the external-to-the-university reviewers.  (See External-to-the-University Instructions below.)  An annotated list should include a brief explanation of qualification for participation and any past affiliations with AUA.  If none of the names suggested are acceptable to the Provost, s/he provides a brief explanation why the candidates are not acceptable and tasks the Dean, Program Chair, and program faculty with expanding the list until a mutually acceptable external-to-the-university reviewers are selected.  The Provost may chose an external reviewer not from the list if a mutually acceptable name is not identified in a timely manner.  The university faculty members of the Review Committee are appointed by the Provost and typically one faculty member experienced in program review.  The qualifications for participation on the Review Committee include:
      1. Senior academic leadership and/or highest degree in the relevant discipline (for external-to-the-university reviewers),
      2. Experience in conducting academic program reviews (for external-to-the-university reviewers),
      3. No conflict of interest (This is not intended to be an exhaustive list of possible conflicts of interest but provides guidance in this regard)
        1. No direct reporting to the program chair or dean of the program being reviewed
        2. No recent (within the period being reviewed) connection to the unit under review (i.e. former students, instructors, chair, dean, researchers) or the executive leadership or board of the university
        3. No close personal (i.e. close friend, family member,)or professional (i.e. co-authorship of research, co-applicant for a grant proposal or other funding, either way direct reporting, either way dissertation supervision) ties with any of the faculty in the program being reviewed, executive leadership or board of the university
        4. When impossible to avoid a conflict, the program in review should disclose any potential conflicts of interest of suggested reviewers when submitting names
      4. Whenever possible, external-to-the-university reviewers should be from similarly situated institutions (i.e. size, experience)
    1. The Provost forwards the Self-Study to the Review Committee members who review the contents and identify any questions.
    2. Review Committee holds meeting/conference calls with IRO to review Guidelines, Self-Study, outline any questions related to the tasks, outline time frame in line with the timeline below (see page 9 of these Guidelines)
    3. Review committee meets with stakeholders (students, faculty, alumni, employers, program leadership) as needed to clarify issues that arose in the “document review.” External reviewers should have opportunities to meet separately with internal AUA constituencies (students, faculty, alumni, employers, etc.)
    4. The review committee maintains a record of all meetings related to the review.
    5. All discussions related to the program review must be conducted during meetings scheduled and documented in the review report.
    6. Review committee prepares Review Report (see outline below). The Review Report should reflect the opinions of all reviewers.
    7. Review Committee Chair delivers draft review report to the Program Chair, Dean, and Provost who review for completeness
      1. If there are omissions, the document is sent  back to the Review Committee (with copy to Dean, Program Chair, and Provost) for completion and resubmission within 14 days
      2. Program Chair circulates draft Review Report among current core faculty for comments and corrections and sends factual corrections within 14 days
      3. A meeting/Conference call may be scheduled by the Program Chair and Review Committee, if necessary, to review findings/factual clarifications  (Review Committee makes corrections at its discretion)
    1. Once complete, the Review Committee Chair submits the Final Review Report with Transmittal Letter to the Program Chair, Dean, and Provost.
    2. The provost  schedules the Wrap-Up Meeting (see Wrap Up Meeting description below)
    3. Review Progress Check: The Review Committee Chair is tasked with notifying the Provost if the timeline for completion is unexpectedly altered
    4. Review Process Check: At any point in the Review, the Program Chair, Dean, Review Committee Chair or Provost may request a process check to address questions or concerns about the review process.   In rare circumstances of material irregularities, the process may be suspended until the irregularities have been addressed.
  1. Outline of Review Report – The Review report is an audit of the Self-Study. It generally follows and reflects on the adequacy of the data and analysis presented in the Self-Study and is coupled with data and information gathered during the review process. The report is prepared by the Review Committee with the participation of all members including the external-to-the-university reviewer.   The external-to-the-university member may submit a separate report to be included in the record if s/he is unable to take part in the report drafting or has additional comments.    Recommendations, where appropriate, should be made at the end of each section.
    1. Introduction/Context
      Is it complete and accurate?  Has the mission statement been reviewed and updated, if necessary.
    2. Disposition of Last Self-Study Prioritized Recommendations and Plans
      Have all of the recommendations from the last Self-Study been thoroughly addressed?
    3. Evidence-Based Analysis of the Program’s Quality
      Is this section complete and accurate? Was appropriate data and analysis used?  Did the program draw appropriate conclusions from student profile data? Were student learning assessments properly and timely completed? Did the program close the loop on student learning assessment recommendations? Did stakeholder meetings confirm the Self-Study’s analysis?
    4. Evidence-Based Analysis of Program Viability
      Is this section complete and accurate?   Has the program developed an appropriate recruitment plan based on trends, demographics, competitor analysis?   Are resources allocated effectively?
    5. Summary & Plans
      Are strengths, weaknesses and opportunities appropriately and thoroughly identified with supporting evidence?  Did the program make appropriate recommendations for addressing weaknesses, building on strengths, responding to opportunities, and (re)allocating resources based on the evidence and analysis presented?  Does the Self-Study include at least an outline for a multi-year (minimum 2 years) student learning assessment plan?
    6. Review of Self-Study Appendices:   Are these complete and accurate?
    7. Appendices to Review Report:  External-to-the-university reviewer CV; list of meetings (stakeholders, dates) conducted by the review committee during the review.

Additional Instructions for External-to-the-University Reviewer

  1. Review the Academic Program Self-Study Report (15-20 page document), paying special attention to the discipline-specific parts, i.e., curriculum, instruction, faculty qualifications, and plans (especially Sections III and V), as well as appropriate international trends and best practices.
  1. External-to-the-university reviewers should participate in the drafting of the Review Committee Report.   In the event they are unable to participate or  have additional comments,  external-to-the-university members submit a separate report to be included in the record. In the case of a site visit, the external-to-the-university draft report should be submitted before departure from Armenia.
  2. Provide CV.

Wrap-Up Meeting with Administration – Discuss and Prioritize Recommendations

Participants:   President, Provost, Dean, Program Chair, Review Committee Chair.

Goal:  Prioritized List of Recommendations and Plans, jointly approved by Administration and Program.

Preparation:   Program Chair may draft a list of Prioritized Recommendations based on the Self-Study and Review Reports.  Participants should review the Self-Study and Review Reports and list of Prioritized Recommendations (if available) ahead of time and be prepared to discuss and agree upon recommendations on priorities and plans for the program for the coming 3-5 years.

Outcome:  Prioritized Recommendations and Multi-year Plan, including budgetary recommendations and Wrap-Up Memo, written by the Program Chair based on Wrap-Up Meeting, which is reviewed and finalized by the Dean and becomes the baseline for the next self-study.

  

IRO DATA Packet – to be sent to Program Chair

  1. Student Exit Survey (program results) For period of review
  2. Employer Survey (program results) For period of review
  3. Alumni Survey (program results) For period of review
  4. CVs for faculty (for most recent full academic year) For period of review
  5. Admissions Data (Application, Acceptance, Enrollment) for program (for period covered in Self-Study) for program with a year to year summary comparison w/ university totals – disaggregated by gender For period covered in the Review
  6. Admissions Data (Test Scores for Accepted Students) for program (for period covered in Self-Study) for program with a year to year summary comparison w/ university totals – disaggregated by gender For period covered in the Review
  7. Retention and Graduation Rates for program (for as many full cohorts as available)
  8. Summary statistics of course evaluations for the program and the university (With online course evaluations, reports are accessible through the Dean)
  9. Average grades for courses administered by program in the last two years

Additional Resources to be cited and used in Self-Study Preparation

  • Previous Academic Program Review (Self Study, Review Report, Wrap Up Memo, Prioritized Recommendations)
  • Program Student Handbook
  • Faculty Meeting Minutes
  • Latest academic program Strategic Plan
  • University’s Strategic Plan
  • Annual Student Learning Assessment Reports – http://iro.aua.am/student-learning-committee/
  • Program Mission Statement
  • Program Goals and Student Learning Outcomes
  • Curriculum Map
  • Course Descriptions
  • List of Faculty Research and Scholarship
  • Exit Interviews (if applicable)
  • AUA Factbook – http://iro.aua.am/factbook/
  • AUA Student Exit Surveys; Alumni Surveys; Employer Surveys –http://iro.aua.am
  • Research Center Reports

 

Target Timeline for Academic Program Review

Normally the entire Academic Program Review process, including any unexpected delays, will be completed by May 15th.

  • August 15: IRO Data Packet sent to Program Chair. Copy to Dean.
  • September: Initial consultative meeting with Assessment Team to discuss process and finalize timeline.
  • End of September: Initial faculty meeting to review data and determine time table for analysis (this could include appointing subcommittees).
  • October: Dean, Program Chair, provide Provost with list of potential external-to-the-university reviewers.  Stakeholder meetings (student, faculty, employer, alumni, as needed).
  • Early November: Follow up meeting with Assessment Team which confirms timeline for Self-Study completion.
  • November – Review committee Appointed. Committee establishes its timeline and schedule for the review process with a target of completing the review report by March 30.
  • January: Submit Self-Study to the Office of the Provost.
  • November – January: Drafting report;
  • February 1:  The Office of the Provost sends Self Study to review committee.
  • February: Review Committee members meet and finalize timeline
  • February – March 30: Review is finished.
  • March: Wrap-up session including Prioritized Recommendations and Multi-Year Plan.

 

 

Useful Resources

  1. Self-Study Transmittal Template
  2. Review Transmittal Template
  3. Checklist for Self-Study Chair
  4. Checklist for Review Chair
  5. External Stakeholder Input Guidelines (How to hold and document a focus group)
  6. Academic Program Review Recommended List of External to the University Reviewers Template
  7. Prioritized Recommendations and Timeline Template
  8. Tentative Self-Study Schedule (2012-2021)