Annual Student Learning Assessment Summaries

4.1 Produce and deliver written and oral presentations, and communicate with specialists and nonspecialists using appropriate media and technology

· Interviews with students

Hasmik Mirabyan, CSE operations and communications coordinator

• There is confusion among students about the concept of reproducible research.
• Students need additional training on reproducibility and ethics in research.

·         Contribute to the work of the placement institution in a way that is valued by co-workers (PG230)

·         Adapt to professional work environment by combining guided learning with self-learning (PG230)

·         Recognize main concepts and theories of political science; be able to relate them to reality (PG231)

·         Apply appropriate theories, methodologies, and/or evidence to their research topic (PG230 and PG231)

·         Develop papers, reports, or presentations that describe their findings and conclusions on their research topics for professional or academic audiences (PG230 and PG231)

Review of Capstones: 35 completed capstone projects were reviewed, focusing on the overall topics chosen, the structure of the paper, the types of methodologies used, the formatting and references mechanics.

Discussion with Capstone Supervisors: several discussions took place during regular faculty meetings, focusing on capstone process and troubleshooting. A wrap-up focus group discussion with capstone supervisors was also organized on June 12.

33 out of 38 students completed the capstone projects successfully and on time for graduation. Several capstones were of very high quality, two resulted in student-supervisor co-authorship papers currently in the process of being published, one was submitted to an international conference. The average grade was 3.34 (B+). Three capstones were recommended by supervisors to be featured on the BAPG website. Two students failed their capstones due to inappropriate use of AI. Most projects demonstrate a consistent engagement with contemporary political and social issues, with recurring themes centered on identity, governance, public administration, security, and economic development. Majority of the projects fall within qualitative approaches, using case studies, document analysis, discourse analysis, interviews, and comparative methods. Methodologies are often lacking explanation for the choice of methods or sources. Technical issues persist in several submissions indicating a need for better formatting checks and adherence to submission guidelines. During the discussions, faculty repeatedly noted that students struggled with literature review and “connecting the dots” between the literature, the methodology and the findings. They often started with unrealistically broad topics. The capstone process started too late, procrastination was a problem. Despite setbacks, it was an overall positive first experience for the program. Through the capstone process students started to understand academic practices, showed awareness of how academic writing and research works. Many felt ownership and were excited about their projects.

·         Update Internship Guidelines with more details and helpful tips for students

·         Update Capstone Guidelines, possibly introduce intermediate deadlines

·         Begin capstone planning in Fall semester; organize a September workshop for capstone launch

·         Showcase best Capstones on BAPG website. [Done].

·         Pilot creative project options with clear guidelines (in Fall 2025 or next year?)

·         Explore a possibility of introducing a third-year research paper course (during the first Program Review next year?)

2. Understand the applicability of various research methods on specific topics.

Once the questions were finalized the evaluation team divided them into two sets of 10 questions each and identified a classroom that has the largest number of second year MPSIA student enrollment. In order to maximize the sample size, the evaluation team waited until the beginning of the semester on August 25 and managed to collect answers from 26 second year MPSIA students.

The assessment of student comprehension was evaluated on a scale used by the program to evaluate students’ knowledge. The scale uses the following categories:

Not yet competent (NYC) Developing (D) Proficient (P) Advanced (A)Added to these categories are also intermediate scales making the overall rubric consist of seven categories: NYC; NYC/D; D; D/P; P; P/A; A

– Just above 25% of the respondents have above proficient (this includes D/P, P and P/A) knowledge in research methods. Since this is their second to last semester before they start working on their MA capstones, this is a bit of a challenge.

– Almost 45% have developing or less than developing knowledge about politics and political processes in Armenia.

– Over 80% have developing or less than developing understanding of IR concept and global issues.

– The “best” results or the domains where the students are more knowledgeable are in Armenian politics (~55% at Developing/Proficient level or above), regional politics (~49), and comparative politics (~38%).

– What is not expressed in the statistics is the fact that students are fairly good at making connection of concepts IF they know what the concepts are.

– The students need focus on fact analyses rather than provide logical and conceptual argumentations.

– While preparing for their capstones, focus more on research methods.

– Emphasize (political) theory so as to help students become “big” thinkers and readily able to apply concepts and theories to practical issues.

– Theory-heavy for purposes of creating problem solvers/critical thinkers.

– Try to synchronize course planning/syllabus organization (when possible) so that the faculty can overlap on main themes and theories in order to increase students’ retention of material.

– Provide the students with a suggested reading list (either throughout the semester or at the end of it) for them to develop their wider knowledge of ideas and concepts.

– Encourage students to read more regional and even local news and critically reflect on them.

– Commit to becoming systems thinkers as a department so as to help our students adopt a wider picture of politics and where they fit in the body of ideas.

– In order to achieve the goals above, it is recommended that the PSIA program appoints a task force to evaluate first year MA students as early as Fall 2021 and start working with them (and the subsequent cohorts) to establish and reach benchmarks during their first as well as second year of studies.

3.3: Apply theoretical frameworks for literary criticism, linguistic analysis, and communication theory

– Evaluation of student papers according to the rubric by two full-time professors

– Interviews with faculty members who teach first-year students and advise them for capstone projects

– Survey of 2020 & 2021 EC graduates

The results of the assessment seem to indicate that the main program goals are being accomplished when it comes to writing instruction. When it comes to basic writing skills, as articulated in Goal 1.1, there seems to be a marked improvement over the four years, and by the time they write their capstone papers, most students are able to write with an extremely high level of clarity.  However, when it comes to more advanced academic writing skills, like the ability to integrate theoretical frameworks into their writing or the ability to compose an original research paper, some students are still struggling. These results are reported by the students and graduates, and supported by the scores given by the evaluators reviewing the selected student papers.

The interviews with the instructors reveal some key observations about student writing. A certain challenge was reported about the students with little background in writing instruction in the Armenian school context, who had considerable struggles in certain modes of writing, as well as in coming up with their own original arguments, expressing a writer’s voice (as compared by summarizing other people’s ideas)

– MS CIS student focus group meeting, spring 2020.

– Faculty Survey, spring 2020.  The survey was taken by faculty members who are employed in IT in Armenia and are actively involved in a field covering the three courses.

– Review and Evaluation of student projects produced in CS326, CS312 and CS323 to evaluate if their leads SLO2.

The Academic Program Review Committee suggested that students interested in data science should complete at least one database course to understand where data is coming from, how it is stored, organized and used. The details and suggestions of the student focus group did not reveal any general issues regarding CS326.  However, students reported that CS323 did not consider deepening the necessary conceptual knowledge, while CS312 had repetitive content with CS323.  They also reported that the course duration for both courses could be shortened.

The survey to the faculty and IT experts abroad showed that the content of CS326 course is up to date with the industry and equips students with the necessary knowledge.  However, the topics included in CS312 and CS323 seemed to be outdated and overlapping with the undergraduate courses. The review of student projects overseen by instructors of the courses CS326, CS312 and CS323 revealed the following.  Projects for CS326 meet SLO2 in content and the use of state-of-the-art technologies.  The content of projects undertaken in courses CS312 and CS323 meet SLO2 in content and design solutions.  However, these courses did not seem to be up-to-date technology-wise.

1.2 To evaluate the post-graduation salary profiles

1.3 To evaluate the use of learning outcome at the workplace

– Instructor Interviews: Semi-structured interviews exploring integration, assessment, and relevance of the SLO .

– Student Survey: Descriptive analysis of Likert items and open responses; visualized with pie/bar charts; n=19 .

– Data Analysis: Triangulation (QUAL+QUAL+QUAN), member checking, peer debriefing, reflexivity, narrative descriptions .

• Students report average-to-high confidence in policy analysis/application; awareness is high.

• Practical, hands-on policy tasks are limited; assessment of policy learning is mostly informal (discussions/reflections).

• Faculty suggest the SLO wording may be overly ambitious (“develop policies”) and recommend emphasis on analysis/awareness.

• Quantitative survey (n=19) supports increasing structured opportunities (e.g., workshops).

• Integrate policy-focused assignments into core courses (e.g., Curriculum Design, Research Methods).

• Provide workshops/guest speakers to strengthen practical engagement.

Overall to assess the degree of achievement of the SLO 1.2 two major directions were taken. First, we have analyzed the syllabi of the courses to identify course level learning outcomes that would match SLO 1.2. Then, together with course instructors we have tried to match the assignments that would be targeted at assessing practical applications of knowledge obtained. Formally this was implemented through a joint syllabus review conducted by the program chair and relevant course instructor. We have also considered major assessment items, e.g. midterm and final exam contents, project assignments. Second source of information is the student feedback and their perceptions. To collect this piece of evidence we have developed and conducted a survey among graduated and capstone course taker MSSM students in the Summer.

– Assessment questions both for projects and exams should be changing over time to make sure that various sections of syllabus are periodically assessed.

-Course level student learning outcomes for some of the courses analyzed should be revised to make them more suitable for subsequent assessment purposes. SLOs that are defined in extremely broad terms are difficult to be measured or evaluated. In general, the instructors should be paying a bit more attention to make sure that SLOs are coherent with curriculum map and that they are aligned with course assignments.

– Feedback received from the faculty teaching the courses highlights the fact that different background of students often times impedes proper coverage of graduate level material. Still the recommendation is to make sure that syllabi of graduate level courses have a rigor required for MS programs.

– Evaluation of students’ perceptions about how “applied” are the courses analyzed here generally gives positive outlook. In this term Capstone and Business analytics stand out, but that was also expected given highly applied nature of both courses. –     – Students were asked to respond to a set of MCQs that were checking “remaining” knowledge after the courses in 6 months or 1 year period. The questions were suggested by the instructors themselves to make sure that they are relevant and properly formulated. The rate of correct responses is around 50%, but there are variations both across courses and across questions within the courses. While this exercise is far from being representative, it can serve as a food for thought for the instructors in terms of what is working and what is not working that well in the course.

 3.4 Oral Presentation and Advocacy Skills. The objective of the review is to understand how well oral presentation and advocacy skills have been incorporated into the course design from perspectives of both instruction methodology and syllabi structure to meet the students’ needs in preparing them for successful professional legal career.

The assessment was conducted using both direct and indirect evidence to come to a conclusion about the effectiveness of overall course design and instruction methods with regard to oral presentation and advocacy skills as a learning objective among LLM program students. In this regards, the syllabi of 6 courses with 3.4 skills were examined, as well as interviews had been conducted with students, alumni and faculty.

The study reveals that the overall course design, structure as well as teaching methods, in general, duly incorporate oral presentation and advocacy skills mechanisms to help students develop both theoretical and practical knowledge and relevant skills.  The survey results display the positive feedback on the courses which lead to advance students’ oral presentation and advocacy skills in their daily work and legal practice. Both the alumni and students appeared to be more confident in their argumentations and judgments at the end of the course, since verbal skills are crucial part in their professional career. The survey reveals that lecturers and instructors’ put special emphasis on oral presentation and advocacy skills advancing techniques during their courses, and practice that regularly based on the situation and the need of the class audience regardless of the fact whether or not SLO 3.4 objectives are spell out in the syllabi.

The syllabi review as well as interviews with alumni and students reveal that there are six courses which practice SLO 3.4 objectives oral presentation and advocacy skills enhancement exercises during the classes. Furthermore, there are several courses in the Program that do not state SLO 3.4 objectives in their syllabi, however they practice so in the classroom. Therefore, it is recommended to remind faculty to revise their syllabi and incorporate SLO 3.4 objectives in their course as they practice.

Integrating Experience Project papers written by the MPH students in 2019 and written products of the course PH 350: Project Development and Evaluation were used for the direct assessment of the SLO 1.  The papers were graded using the pre-existing Grading Rubric developed based on the MPH program competencies, corresponding to this SLO.  The criteria were assessed on a numeric scale ranging from 1 (performance criteria not met or missing) to 4 (excellent). The average scores for each criterion and each student were developed and presented in the assessment report.The indirect assessment included the analysis of the results of the End of Program Evaluation conducted for the MPH students at the end of studies in 2019. Students’ responses to the closed-ended question used for indirect assessment of the SLOs of interest were translated into summary scores and compared to the results from the previous years.