Assessing and Evaluating Learning Outcomes of the Information Systems Program

Program assessment is an essential procedure of academic programs accreditation review. As such, it is a cornerstone of quality, enhanced education. At Ajman University, the process of assessing and evaluating courses is done at the departmental level. This paper describes a model that the Department of Information Systems uses to assess the achievement of its program learning outcomes. This model enables the measurement of the level of achievement of each learning outcome to identify areas for improvement in students’ performance and suggest remedial actions in consultation with faculty concerned. The results of program assessment are used to suggest changes to curricula and courses structure and content to be implemented in the following reaccreditation cycles when these changes are considered substantial.

second form collects students' feedback related to their academic advisors during the semester.
Information gathered by these two forms were used to identify and solve persistent problems occurred in courses and/or with instructors and used in the annual evaluation of faculty members.
Today, academic programs assessment is considered as an imperative process to ensure quality education. At AU, recognition of the importance of program assessment started back in 2005, when AU established the Quality Assurance and Institutional Research Unit (QAIRU). The role of this unit is to provide colleges with the necessary assistance to define their procedures and develop tools to measure their Program Learning Outcomes. In fact, QAIRU provides instructions and guidelines on all aspects of program and course learning outcomes.
The assessment and evaluation of the Information Systems program started in 2006, where a model was defined and used to measure the achievement of its Program Learning Outcomes (PLOs), based on direct and indirect measurements. Courses included in the program were mapped to PLOs by examining individual learning outcomes of each course. The first step was to examine the achievement of learning outcomes of each course. Then, each PLO was analysed individually based on data collected from courses' exam results, faculty, students, alumni, internship, and employers to measure its level of achievement (Mehdi & Abou Naaj, 2013). The results produced by this model were not accurate as the same weight were given to all direct and indirect assessment tools contributing in the assessment of the same PLO.
This paper describes an assessment model based on performance indicators which were defined to measure the attainment levels of each of the IS PLOs. It was applied to the IS program as part of its re-accreditation process in 2016. The model adopts the definition of Program Educational Objectives and Program Learning Outcomes provided by ABET, which are in-line with the assessment criterion specified in CAA (2011); they are stated as follows (ABET, 2010): "Program Educational Objectives are broad statements that describe what graduates are expected to attain within few years of graduation. Students (Program) learning outcomes describe what students are expected to know and be able to do by the time of graduation. This relates to the knowledge, skills and behaviours that students acquire as they progress through the program".
The result of this study will provide colleges and other concerned individuals with data regarding the effectiveness of their academic programs. It will assist colleges in developing strategies that extend the quality assurance framework to support sustainable quality education, which will contribute to produce creative and dynamic graduates who will be able to find adequate job opportunities and ensure satisfactory well-being. In 2017, Slade the author defined quality education as the one that "provides the outcomes needed for individuals, communities and societies to prosper".

Literature Review
Assessment of PLOs became an important process to ensure effective, sustainable and improved education that is increasingly recognized and required by accrediting bodies (Buzzetto-More & Alade, www.scholink.org/ojs/index.php/wjer World Journal of Educational Research Vol. 4, No. 4, 2017 526 Published by SCHOLINK INC. 2006). Assessment is a process of identifying, collecting, and analyzing student achievements data to measure the attainment of each learning outcome. Effective assessment uses quantitative, qualitative, direct and/or indirect measures as appropriate to the outcome being measured (ABET, 2010).
Moreover, assessment is an integral part of certifying that an educational institution meets the standards and has the necessary resources to provide quality education (Love & Cooper, 2004). Most accreditation bodies require programs to: i. Specify the skills and knowledge that they expect students to achieve by the time of their graduation (identify a set of program learning outcomes), ii. Set up assessment processes to determine the extent to which the program is successful in supporting students to achieve these learning outcomes, and iii. Implement a continuous improvement process commonly referred to as closing the loop. This process is used to improve the teaching and learning experiences at course and program levels (Alzubaidi, 2017).
Learning outcomes focus on the rational, interactive and collaborative development of students as they cooperate to succeed in a learning activity. They are what students are expected to demonstrate in terms of knowledge, skills and attitudes upon completion of a learning experience (Asheim, Gowan, & Reichgelt, 2017).
Learning outcomes have direct implications on curriculum design as well as on quality assurance. They represent a transformation from the traditional teacher-centred viewpoint to the adoption of the student-centred approach, which produces a focus on the teaching-learning-assessment relationship and the fundamental links between the design, delivery and measurement of learning (Adam, 2004). They have created the most anxiety among faculty dealing with the accreditation process (Jones & Price, 2002). However, adopting learning outcomes based approach has proven beneficial at the program level (Clarke & Reichgelt, 2003), individual courses level (Rigby & Dark, 2006) as well as at library level that allows students to achieve the specified skills (Gowan, MacDonald, & Reichgelt, 2006).
To implement a learning outcomes approach, we should first start by formulating the program educational objectives that address the institution as well as the program's mission statement. Then, the PLOs are expressed to reflect the knowledge, skills and behaviour the program's graduates will have.
Different types of assessment are being used to measure the attainment of program learning outcomes.
In 2003, Sanders and McCartney describes a set of twelve assessment tools used in their program accreditation process. These tools include senior exit survey, alumni survey, written and oral exit exams, portfolio and external advisory panel, which have a set of limitations, as they are all considered as indirect assessment tools insofar not course-based.
Another type of assessment focuses on course assessment tools (Blanford & Hwang, 2003) Schwartzman suggested to establish a departmental steering committee of senior faculty members to consolidate all course learning outcomes into a final set of PLOs. Blanford and Hwang (2003) recommended an assessment day as an effective way for faculty to meet, evaluate assessment results and provide improvement recommendations.
In 2017, Alzubaidi suggested an assessment approach for direct measurement of how well students achieve the course learning outcomes and the PLOs by defining a set of measurable performance indicators in strong relationship with courses being taught. These performance indicators are measurable attributes identifying the performance required to meet a program's outcomes (Rogers, 2003).

Purpose of Study
The purpose of this study is to:

Program Learning Outcomes
The Information Systems program offered at AU offers two concentrations: 1) IS2. Apply knowledge of core concepts, techniques and practices to IS applications.
2) IS3. Use analytical and critical thinking skills to solve IS problems.
3) IS4. Address information requirements and provide solutions that reflect current business needs and changes.
4) IS5. Select and adopt emerging technologies for computerized business information systems.
5) IS6. Manage information systems components to maintain business sustainability.

Performance Indicators and Rubrics
Performance Indicators (PI) are measurable performance benchmarks that students must meet as an indication of achievement (ABET, 2010). They indicate what tangible actions students should be able to perform after their participation in the program.
For each PLO listed above related to the IS program, the knowledge, skills and expected students' behavior required to achieve that outcome were listed in order to define various PIs for the IS program.
A rubric is associated to each PI and related to a specific assessment tool of a particular course. A rubric may consist of one or more dimensions specified to evaluate PI. A dimension in general relates to a more specific area of the corresponding PI. Moreover, scoring scales were defined to evaluate each dimension as well as specific target (attainment threshold) to measure the achievement of that dimension ( Figure 1). The measurement of the attainment level of each PLO is carried out by using one or more PIs with corresponding scoring scales defined for each PI. A sample of seven performance indicators (A to G) with their rubrics designed for a specific PLO is described in Figure 1.

Program Learning Outcome #5 (IS5)
Select and adopt emerging technologies for computerized business information systems.

Performance Indicators
A. Producing technical resources, processes and services in organization.

Program Learning Outcomes Attainment
In order to measure the attainment levels of PLOs, each learning outcome is assigned number of PIs from two to seven. One or more dimensions define each rubric associated to a PI. Each dimension divides students into four categories as follows: The level of attainment for each PLO is computed as follows: a. The percentage of students in each of the above score categories for a particular PI is calculated, as the average percentage scored over the dimensions of that PI.
b. The percentage of students in each category for a particular PLO is the average score for that category over all PIs for that PLO.

c.
A PLO is considered achieved if the combined percentage of students in the "meet expectations" and "exceed expectations" categories are within 50% score. This is roughly equivalent to 50% of the students scoring grade C (70%) and above.
A PLO is considered achieved if it scores a total value greater than or equal to 70%; an unachieved PLO is defined with a score less than 50%. PLOs with a level of achievement between 50% and 60% are considered as marginally attained, and those with a total score of achievement between 60% and 70% need improvements (further enhancements in some knowledge and skill areas are required where the percentage of students achieving "meet or exceed" levels of expectations is not satisfactory).

Sample
The sample of 144 students used in the study was chosen from the pool of undergraduate students

Results
As sample, Figure 2 gives a summary of the level of attainment of the PI-E of the PLO#5. Dimension 3 indicates that the skills of students in writing JavaScript code are extremely below expectation (6.42%) and need substantial improvements.
Using the same way shown in the previous section, dimensions related to all PIs of the IS program were evaluated and consequently, all PIs were assessed. Figure 3 gives a summary of the level of attainment of each of the eleven IS PLOs. It shows that Learning outcomes IS5 and IS6 are marginally attained according to the 50% criterion.

Program Learning Outcomes Evaluation-Closing the Loop
Based on the results obtained in the previous section, IS PLOs were divided into three categories as follows: 1) Learning outcomes (PLO#2, PLO#4, PLO#7, PLO#8, PLO#9, PLO#10 and PLO#11) are considered as achieved with a total score greater than 70%.  Vol. 4, No. 4, 2017 2) The Learning outcomes PLO#5 and PLO#6 are marginally attained with a total score less than 60%.
3) The two learning outcomes with scope for improvements are PLO#1 and PLO#3 with a total score less than 70%.
To improve the level of attainment of PLOs, the IS department has developed a set of possible remedial actions. One or more remedial action(s) may be applied to courses involved in those PIs. Actions can be one or more of the following: a.
Engaging students with more assignments.

Conclusion
This paper describes a model that the IS Department within the College of Information Technology at AU has used to implement a complete outcome-based assessment and evaluation process for the re-accreditation of its program.
This model does not depend on assessment of individual courses, but rather on PIs and rubrics related to knowledge, skills and behaviours that students are required to acquire at graduation. These PIs and rubrics were capable of showing the degree to which each PLO has been achieved. However, the model proposed assumes that PIs used to measure a PLO have an equal weight on that learning outcome. We can enhance this model by assigning different weights to performance indictors in order to reflect the contribution of each performance indicator to a particular learning outcome. Moreover, this model would be enhanced by including alumni and employer survey forms as well as instructors and students evaluations as indirect assessment instruments in addition to the internship survey forms.
Implementing a model to measure the achievement of PLOs for any academic program helps institutions to identify problematic areas and take appropriate remedial actions. The model described in this paper is generic: it can be applied to any academic program with measurable learning outcomes.
In this study, data collection and analysis were carried out manually. These two tasks require a considerable amount of time from faculty members to be achieved. As a future work, we could consider to computerize these tasks, which could facilitate the whole process of assessing and evaluating PLOs of any academic major.