Study in Using Integrated Rubrics in Assessment

This case study takes a look at an integrated approach to assessment. The paper walks through the process of alignment of college mission and learning outcomes with departmental and course level outcomes. The department developed integrated assignments that are implemented across a four-year program to ensure learning outcomes. In order to assess those outcomes, the department developed an integrated rubric that is applied at multiple stages of the program levels as well as in various courses. This allows for feedback that is both summative and formative for future changes. The feedback loop has helped provide insight into course level and department level changes, but it has also enhanced departmental climate and work processes.

Published by SCHOLINK INC. mission and focus (Note 8, Note 9, Note 10). While it may seem counterintuitive to end with institutional goals instead of starting there, the last step was set up as a check to ensure consistency.
The original assessment learning outcomes were aligned with the mission of the college so congruency should have been maintained throughout the process (Note 11). However, if a mission alignment exercise had never taken place, a department would want to start with first assessing the institutional mission and then moving into learning outcome development.

Business Department Themes:
Critical Thinking

Multiple Framing
Reflective Exploration of Meaning Practical Reasoning

Figure 2. Business Department Themes
Next, the department mapped their curriculum. Each member of the department utilized a form to look at the learning outcomes addressed in each course they taught. They identified which learning outcome(s) each course addressed, how they measured that outcome (test, paper, exercise), the measurement tool (grade, score on a rubric) and if it was an introduction, a reinforcement or a demonstration of the concept. By doing this each faculty member could see where the outcomes were occurring, if the department was using a layered approach and the total impact of the learning outcomes.
The additional benefit was that this process lent itself to the identification of gaps and overlaps in the curriculum (See Appendix A: Curriculum Map).
Once the curriculum map was created, faculty shared what skills and knowledge base they needed for their students to have when they entered each course. This helped each faculty member identify what learning outcomes were important from a course sequencing perspective.
With outcomes aligned and mapping complete, the faculty had an opportunity to ensure the assignments in different courses within a sequence were building in a manner that allowed for the layered approach to learning that was being sought. It also opened up communication between faculty about course outcomes and ways to further integrate courses. In addition, it allowed for the elimination of some of the concepts being taught in upper-level classes that were already being taught and achieved in lower level classes. The heightened understanding of each course outcome and skill/knowledge needs in upper level courses allowed for a utilization of assignments that built on knowledge gained in earlier sequenced courses.
Opening up the communication amongst the faculty members was, thus, extremely helpful in making the program more integrated and enhancing the outcomes of students. This allowed for a more highly Published by SCHOLINK INC. syllabi, cohort advising, etc.) and informally (through discussion of assignments, foreshadowing learning needed in courses later in the sequenced curriculum, etc.). The coordinated approach to outcomes allowed the department to shift to four key learning outcomes instead of the previous 35 outcomes identified. Department Learning Objectives: 1. Critical Thinking -Students will develop a comprehensive understanding of how to access business; and to critically analyze complex material in order to evaluate evidence, construct reasoned arguments and communicate both inferences and conclusions.
2. Multiple Framingthe ability to analyze individuals, systems, businesses and societies through multiple frames of reference.
3. Reflective Exploration of Meaningthe ability to see things beyond utility and to wrestle with the self-reflective aspects of learning, to raise questions, recognize the human side of business and approach things in an ethical manner.
4. Practical Reasoningarticulate how key theoretical principles can be used to explain individual, organizational, and system processes to inform business policy, decision making and practical approaches to human and business issues across multiple contexts

Figure 3. Department Learning Objectives
This process provided a feedback loop at multiple levelsassessment committee, department administrator, and individual instructors. For the assessment committee, the department report provided a much more manageable number of results to review, due to the decreased number of inputs (See Appendix B -Assessment Report). This allowed the committee to see if the major(s) were functioning as claimed. In addition, it provided a much more integrated report instead of a collection of several sets of courses data. Focusing on a smaller number of outcomes that were built from the 100 (Foundation) to the 400 (Integration) level courses allowed for a macro look at the overall effectiveness of the program. From an institutional perspective, this built cohesion within the program and its individual majors.
For the department administrator, the results provided a guide of where to focus attention. When certain areas were falling short in a given year or trends were identified over several years, the administrator could narrow in on the source of the concern and address it more effectively. This outcome of curriculum mapping provided the opportunity for a more focused analysis of problem identification and problem-solving in the program.
With the ability to narrow in on specific courses and compare courses across the curriculum, individual faculty could see how the outcomes in their courses were contributing to overall program outcomes and compare their results with colleagues. This allows for a better understanding of how to develop Published by SCHOLINK INC. learning activities in their own individual courses (Schutte, 2018). In addition, it encourages coordination between faculty across courses to ensure learning objectives are being fully addressed but also creates engaging student activities by integrating projects across courses.

Development of Rubric and link to Direct and Indirect Evidence
The development of an integrated rubric started with collecting all rubrics used by faculty in their courses. From those rubrics and the college wide learning outcomes a collective rubric was created that addressed the college wide learning objectives as well as departmental learning objectives. All departmental learning objectives were tied to the college learning objectives through this initial rubric.
As a department we wanted to create a standard rubric used for assessment for many reasons: we wanted to be able to do an apples to apples analysis of outcomes, we wanted to be able to better understand our outcomes and we wanted to be able to track and analyze student progress and potential areas of improvement.
Once the initial standardized rubric was created that rubric was then tied to the standardized rubrics created by AAC&U (Rhodes, Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics, 2010) as well as rubrics used by faculty in their individual courses. The AAC&U rubrics were used here as a benchmark for the same reason we focused on the AAC&U earlier in the process. We then used the two rubricsthe department rubric addressing college and department learning objectives and the AAC&U VALUE rubrics -to create a master rubric. The intent of this rubric was to create a working rubric that would be used for assessment, as well as course rubrics (Note 12).
Traditionally, faculty created and used their own rubrics for each course. Assessment was not tied to any one rubric in a course, but to that faculty members' assessment of the coursework completed. This standardized rubric would allow for the assessment to be more focused and allow for all parties to better understand what was being evaluated, how it was being evaluated, and the outcomes of the evaluation.
The standardized rubric would provide a starting point for all faculty in their courses. Faculty were given the two options regarding use of the rubric. They were required to use the standard rubric for assessment purposes. However, they were not required to use the standardized rubric for class assignments but could grade the assignment a second time using the standard rubric. This second grading would not impact a students' grade. Or they could integrate the rubric fully into the course rubrics.
www.scholink.org/ojs/index.php/jecs Journal of Education and Culture Studies Vol. 4, No. 3, 2020   Significant work has been done to obtain buy-in from all members of the department. A department retreat was planned in order to work through the rubric and gain approval and acceptance from all department faculty. Prior to the departmental retreat the sample rubric, tying college learning objectives to departmental learning outcomes and AAC&U best practices, was created and circulated to the department, allowing faculty to review the rubric in advance and come to the retreat prepared for discussion.

Assignments Included and Alignment with LO -Department and College -
Over the summer the department met for a retreat. We started with a discussion of the rubric: how it was created, the goals we were attempting to achieve, and the purpose of the rubric. Once all questions regarding the use of the rubric were addressed, we reviewed our learning goals for the department.
Using those learning goals, we discussed gaps in the integrated rubric both in rubric content and course coverage. Changes were made to the rubric to simplify wording and clarify measurement.
Significant concern was raised regarding use of the rubric. Faculty did not want to be forced to use the rubric for direct student grading. Faculty were reassured that they were not required to use the rubric for direct grading. Additional concern was raised about collecting the data. Our current Learning Management System (Moodle) does not easily collect and maintain data from student assignments where no grade was given. Faculty agreed to continue looking at options but to move forward with a manual collection process for now.
Once the department felt comfortable with the standardized rubric we moved to course mapping. In this process we had several goals: to align assignments to learning objectives, to ensure we were measuring learning objectives at different levels of course work, to make sure that no one faculty member carried too much of the load for assessment, and to use existing assignments as much as possible (Note 13, Note 14). A table was used to list all courses offered. The department then worked through the list using syllabi and faculty input to determine which classes would best measure each learning outcome (See Appendix D: Integrated Rubric). Next faculty identified which assignments would give the best direct evidence of knowledge in that learning objective (See Appendix C: Assignments and Rubrics).
Like any change process, alignment of the integrated rubric has had challenges. Our Learning Management System (Moodle) does not currently allow for the use of a rubric with no grade connected to it. Because of this, the integration of the rubric has been primarily manual. In addition, most faculty have chosen to not use the integrated rubric for grading assignments in their courses. This has required the double grading of assignments and the manual collection of data. However, each iteration of data collection has improved and efforts are being made to upgrade Moodle to aid in our data collection efforts.
The collection of this data has led to some changes at different levels in the curriculum. Our curriculum is set up in different levels. The first level is foundations that focuses on obtaining a first level of understanding of business including the context (economics) and the language (accounting) of the field.
The second level focuses on decision making around critical thinking and abstract thinking. This is normally taken in their second and third years and provides the ability to practice the field of business while obtaining additional knowledge. These courses utilize case studies, debates, projects with real client interaction and continuation of development of their own business ideas. The third level is integration where we focus on transferring the knowledge and skills learned to different contexts. This level forces students to work in contexts that are not contrived but allow for ambiguity so they have to apply what they have learned in new ways. This learning includes projects, internships and a full business plan of their business idea development.

Figure 5. Business Department Curriculum
By looking at multiple assignments at different levels we were able to incorporate changes based on the data provided. In the first level we have added increased computer and writing proficiency assignments to help facilitate future projects and client interaction. We have also added teaching in the LEAN process to facilitate ideation and running ideas through a quick process to move them forward or let them fail fast. In the second level we have brought the ideation from the first phase into classes to let them take what they have learned from LEAN and put more substance around their ideas. This forces them to apply what they are learning to something they are creating and developing on their own. We have found this has increased their engagement with the idea. In the final level we have the students take what they have learned throughout their program and put it into a final business plan. It brings together all their learning and forces them to make some decisions about the idea they have worked on in previous courses. This plan is then presented to outside judges. This helps them learn to interpret their idea for others by putting themselves in the judge's shoes. This culmination of the learning in a public forum ups the stakes and provides an opportunity to showcase and align learning.
The development of the integrated rubric allows us to focus on outcomes across the levels of the program. Thus, providing us with data to break the process of growth into stages. By seeing where students are progressing or stagnating allows us to drill deeper into pedagogical methods and assignment methodology to enhance the learning outcomes in our students. The utilization of different assignments (including papers, cases, presentations) gives us insights into not only the way students may be developing but provides an opportunity to see how a variety of learning methods may impact different learning styles (Note 15, Note 16). This is something we have just scratched the service on but hope to explore in more depth when we have a good collection of data.
The integrated approach to assessment has had a significant impact on the department. This includes vision, teamwork and student engagement. The first area that we have seen growth in is vision for the department and how it fits within the overall college mission. Our alignment with college learning outcomes, strategic vision and implementation of goals has allowed departmental members to see the congruency between what our department does and what the college is doing. It gives us a sense of all rowing in the same direction. In addition, the opportunity to discuss outcomes, identify gaps and ideate around better methods helps us to further understand and operationalize our departmental vision.
In addition, we have worked together as a team to build a strong program instead of just a collection of courses. By understanding what is needed from one another and what each of us are doing in our courses we can see how it all fits together at a new level. This increases our interdependence and allows us to share ideas and resources more readily. This knowledge has provided formal conversations through retreats and departmental meetings as well as informal hallway and "I have an idea" conversations. Thus, the collision of ideas has led to some unique classroom projects, resource sharing and a deeper understanding of the collective approach to ensuring our students are learning at a high level.
Finally, this has developed a deeper understanding within our students of the goals of our program.
They are able to see the learning outcomes emphasized in each course and how those outcomes are tied to assignments throughout the program. This allows students to see the "why" behind the assignment and coherency between courses. We are able to signal to them that they will need a given concept in a later class or remind them that they learned about this idea in an earlier class. This provides them with a better understanding of the interconnectedness of the courses and the overall learning outcomes they will walk away with. I have heard students talk about this in one of their final courses in the program.
They share how their views of business have shifted over their four years and they often bring up the learning outcomes without knowing that is what they are doing. These outcomes are embedded in their understanding of what they have learned and what they are taking with them. Thus, their sense of the coherency of the program is very refined around the goals we have outlined.

Opportunity for Improvement
There are three areas we are focusing on for improvement: continuous re-evaluation and updating, modification of learning management system, and integration of the rubric into courses. Because this is a new process it will be key to continue to re-evaluate the integrated rubric to ensure that we are still measuring key learning objectives in the most effective manner possible. We will need to make sure that we are continuing to map courses to learning objectives to ensure we have the proper layering as well as to ensure we are collecting data from the most appropriate assignments. If courses change we will need to be diligent in ensuring that those changes are addressed in our map and that we are not losing a touch point for evaluating a learning objective. In addition, it will be essential to make sure that we continue to ensure that our department learning objectives continue to align with the college learning objectives.
As mentioned earlier, one of the issues with Moodle is the inability, currently, to create a zero grade rubric. We are working to get the upgrade implemented that would simplify this process. However, in the meantime we are using manual processes to collect the data. While this can be time consuming it is currently a sufficient fix for the department. However, should this modification be installed, it would greatly improve our data collection and our ability to analyze the results we collect.
Finally, as we move through this process we continue to encourage faculty to integrate these rubrics into their assignments. As more faculty integrate the rubric into their assignments data collection will improve and time to collect data will improve as well. An additional benefit of tying the integrated rubric into assignments will be an additional opportunity to reinforce to students the connection between the assignment and the learning objectives for the department and college. If we are able to upgrade Moodle, this may be the encouragement faculty need to make the change as they will be able to easily add the rubric elements to their Moodle assignments without impacting grades.

Notes
Note 1. " [T]here are many different types of questions currently being asked by labor market analysts, including:  For students: Which programs, majors, credentials, or institutions provide the best likelihood of accessing jobs that match their interests and provide solid earnings and employment opportunities?
 For institutions: Which programs are well aligned to the actual needs of employers and available opportunities for graduates, and which programs may need to be improved, redesigned, expanded, or eliminated?
 For policymakers and system leaders: How well are the state's institutions meeting the needs of regional or state employers, driving economic growth, and advancing the social well-being of the state's citizens? How should institutions be funded to meet these needs?" (The Aspen Institute, 2015).
Note 2. "The #1 reason why Americans value higher education is to get a good job. And although 96% of chief academic officers of colleges and universities have confidence they are preparing students well for the workplace, no one else believes them; 13% of US adults, 11% of C-level business executives and 6% of college trustees have the same levels of confidence in how well college grads are prepared for work" (Busteed, 2019).
Note 3. "Alignment is the connection between learning objectives, learning activities and assessment.
An aligned course means that your learning objectives, activities and assessments match up so students learn what you intend and you accurately assess what students are learning" (University of Colorado -Denver, 2007).
Note 4. "When employers are asked what they want to see most in college grads, the top things they cite are work-related experiences and industry-relevant skills. And the strongest predictor of alumni saying their education was worth the cost and high quality is the degree to which they feel their education is relevant to their current work." (Busteed, 2019).
Note 5. "Over the last 30 years, the need to verify this assumption through the measurement of student learning outcomes has risen, largely due to the evolving and increasing pressure from stakeholders such as state and federal governments, accreditation agencies, students, parents, and society at large (Ewell, 2009)" (Russel & Markle, 2017).
Note 6. "If we haven't specified our objectives or our learning outcomes, how should we do it?
You might work through the following questions: o What would the ideal graduate of our program look like (knowledge, skills, beliefs and values)?
o What experiences (assignments, papers, productions, internships, etc.) do students carry out through our program that would provide evidence of their achievements?
o What standards would we expect our students to achieve for those experiences?
o Can we express those experiences and standards in ways that would both guide our students in determining whether they have achieved what we want and provide us clear criteria for our assessments?" (Skidmore College, 2020).
Note 7. "The VALUE rubrics are providing a means by which campuses and their faculty can create common standards for evaluating the quality of performance expected for attainment of specified degree levels, e.g., Associate or Baccalaureate degrees. In essence, what has emerged is a framework of quality standards without standardization" (Rhodes, Emerging Evidence on Using Rubrics, 2012).
Note 8. "Learning outcome statements are most useful when they are crafted to inform effective educational policies and practices, not to meet compliance demands by external groups. When they are concrete and clear about the proficiencies students are to achieve, such statements provide reference points for student performance, not just for individual courses but the cumulative effects of a program of study. … describing desired outcomes also make it possible for faculty to align curriculum and pedagogy with intended proficiencies, which is essential to ensuring that a program is, indeed, achieving its purposes. Such statements also make it easier for students to understand and appreciate institutional and program expectations for their performance and how their learning will equip them to handle what they encounter after college" (University of Illinois and Indiana University, 2016).
Note 9. "5 Steps Prescribed by National Institute of Learning Outcomes Assessment: 1. Develop specific, actionable learning outcomes statements.
2. Connect learning goals with actual student assignments and work.
3. Collaborate with the relevant stakeholders, beginning with the faculty.
4. Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve student and institutional performance.

rubrics were developed as part of AAC&U's VALUE (Valid Assessment of Learning in Undergraduate
Education) project-part of AAC&U's Liberal Education and America's Promise (LEAP) initiative.
The VALUE rubrics were developed as an alternative to the snapshot standardized tests used by many to assess student learning" (Rhodes, Emerging Evidence on Using Rubrics, 2012).
Note 11. "The assessment data and information provided at the institutional level should convey the alignment between the learning objectives and the assessment.  Determine whether you are already assessing student work in ways that could be employed for assessment purposes. For example, do all your students have to take a capstone course or perform field research or undergo an internship experience? Can you systematically evaluate those for assessment purposes? Are student performances evaluated by groups of faculty or professionals in the field? Can those be adapted for assessment purposes? Do the faculty as a group or a sub-group of the faculty evaluate student progress in the earlier stages of your program? Can those evaluations be employed as one of your assessment methods?
 The assessment process should involve the entire faculty. All faculty should review the results and discuss whether any improvements need to be made. Don't ask one faculty member to carry out any one assessment. Individual faculty give grades in their courses and can gather the examples of student work that you need, but groups of faculty review the evidence for assessment purposes" (Skidmore College, 2020).
Note 14. "After writing learning objectives, you should to think about how to assess the students' achievement of the different learning objectives.
 Different learning objectives define the scope and methods of assessment.
 Relating different levels of objectives with assessment instruments and all items that are graded will make sure you achieve your specific objectives.

BUS 301
To enhance the ability to collect and analyze data in a way that enhances decision making in the workplace Transfer -Apply the knowledge they have gained throughout their education to create value for an outside organization.

BUS 406
Students will integrate and apply the knowledge and skills of all functional areas of business from all of the courses in the major as well as novel and evolving observations of the external environment in order to meaningfully analyze real business problems and real businesses.