A Systematic Review of Cognitive Diagnostic Assessment and Modeling through Concept Mapping

Traditional assessments are typically constructed on logical taxonomies and content specifications but lack explicit cognitive models of the processes and problem-solving strategies that underlie student performance. Cognitive Diagnostic Assessment (CDA) fills this gap by combining cognitive science and psychometrics. CDA is in its infancy, but over 1,000 relevant studies have been conducted in this area during the last 20 years. Facing these complicated studies, many beginners struggle to understand the whole picture of CDA. This paper systematically reviews the literature on CDA and relevant cognitive diagnosis models (CDMs) with the application of a concept mapping technology. Concept mapping is graphical representation of concepts and their relationships. Its use in this study allows researchers and students to gain in-depth knowledge about CDA and CDM and identify areas of future research.


Introduction
The history of assessment can be traced back to the Chinese civil service in 2200 B.C. (Bowman, 1989), ancient Greeks' individual difference testing (Doyle, 1974), and the European governments' test systems of the 19th century. Testing received a great boost in the 20th century due to rising interest in individual differences (Kline, 2005). In the United States especially, spurred by the A Nation at Risk report in 1983 and the No Child Left Behind Act in 2001, the role of Large-Scale Assessment (LSA) in public education grew tremendously beginning in the mid-1980s, flourished during the 1990s, and unquestionably continued to grow with the implementation of assessment and accountability requirements (DePascale, 2003;Kifer, 2001). Today, assessments of student performance are more prevalent than at any other period in history. National and international assessments have become increasingly frequent as has the use of college admission and placement tests. The development of LSA has been an ongoing process with many aspects still in need of improvement. For example, "the contents of standardized tests and the examination can assess only limited forms of competence, and teachers are quite able to predict which aspects of competence will be assessed" (Wiliam, 2001, p. 165).
LSA is an important component of determining achievement in education. It should be carried out in a systematic fashion (Taylor & Tubianosa, 2001) and shows the general public the effectiveness of students, schools, and teachers where there is increased accuracy and objectivity in examining achievement (Crundwell, 2005).

Cognitive(ly) Diagnostic Assessment (CDA) makes explicit the cognitive models of learning beyond
the logical taxonomies and discrete content specifications that are prevalent in traditional LSAs. CDA is designed to measure specific knowledge structures and processing skills in students to provide information about their cognitive strengths and weaknesses; it provides formative diagnostic feedback through fine-grained reporting of test takers' skill mastery profiles (Buck & Tatsuoka, 1998;DiBello et al., 1995;Jang, 2009a;Javidanmehr & Anani Sarab, 2017;Nichols, Chipman, & Brennan, 1995).
Cognitive Diagnostic Models (CDMs) have been developed "to help construct diagnostic assessments and estimate students' attribute mastery patterns associated with different cognitive skills" (Cui et al., 2006, p. 4). The rapid advance and a growing interest in CDA have resulted in extensive research and theories in the past two decades. Because of its relative infancy and technical complexities, researchers interested in CDA may find it daunting to comprehend the entire picture of this body of research. In this project, we analyzed the relevant studies on CDA and CDMs using concept maps. Concept maps are used to visually represent concepts in a body of knowledge and their relationship to each other.
Concept maps show the hierarchy of these concepts and emphasize the links between them. They are valuable pedagogical tools Clark & James, 2004;Novak, 1990).
The purpose of this study was to conduct a systematic review of CDA and its accompanying statistical modelling approaches through concept mapping. This paper specifically focuses on the application of a

Literature Review
CDA is designed to measure specific knowledge structures and processing skills that students have acquired and to provide fine-grained diagnostic information about their cognitive strengths and weaknesses (DiBello, Stout, & Roussos, 1995;Jang, 2009;Nichols, Chipman, & Brennan, 1995). CDA can give instructors and policy makers more information about the strategies students use to attack problems, the relationships students perceive among concepts, and the principles students understand in a domain. It provides useful information for educators and policymakers to draw conclusions about students' instructional needs and the effectiveness of instructional programs (Nichols, 1994). By measuring different cognitive processes and the knowledge required to solve test items in a domain of interest, CDA can provide a profile of students' mastery and non-mastery of particular cognitive skills.
CDA can have a significant impact on the improvement of instruction and learning by giving students and teachers reliable feedback on student strengths and weaknesses (Roussos, Templin, & Henson, 2007) and "further help design effective interventions for individual students" (Cui et al., 2006, p. 4).
CDA provides relevant procedures for application such as cognitive foundations of structured item response models, the Fusion model skill diagnosis system, diagnostic inferences about cognitive skills, and cognitive-diagnostic score reporting.
CDA is "still in its infancy, but its parentage is fairly well established" (Leighton & Gierl, 2007, p. 1) because it can be traced back to the 1950s when prominent researchers, such as Messick, Snow, and Lohman, put these perspectives forward. Over the past three decades, a growing number of researchers have argued that cognitive science and psychometrics should be tightly aligned in the service of instruction (Messick, 1984;Snow & Lohman, 1989). In the 1990s, some influential work was published in books and journals, such as A Framework for Developing Cognitively Diagnostic Assessments (Nichols, 1994) and Cognitively Diagnostic Assessment (Nichols, Chipman, & Brennan, 1995). Recently, Cognitive Diagnostic Assessment for Education: Theory and Applications (Leighton & Gierl, 2007) and Diagnostic Measurement: Theory, Methods, and Applications (Rupp, Templin, & Henson, 2010) have become important sources of new ideas leading to innovations in CDA. Much attention has been paid to developing psychometric models that are capable of estimating students' cognitive profiles. CDA has also drawn much attention from practitioners and policy makers who strive for specific information about students' knowledge states and proficiencies that are necessary for making decisions about the allocation of resources as well as planning effective instructional interventions for individual students (Cui et al., 2006;Nichols, 1994;Roussos, Templin, & Henson, 2007). There have also been many studies of relevant theories and models, such as the Fusion model skills diagnosis system, the IRT-based parametric latent class models, the rule space model, the attribute hierarchy method (Cui, 2007;Cui & Leighton, 2008;Cui et al., 2006;Gierl, 2007;Jang, 2008;Roussos et al., 2007) and studies of the broad application of CDA such as skill diagnostics (Jang, 2005(Jang, , 2009Jang et al., 2008;Leighton & Gierl, 2007;Zhang, 2013). During the last decade, evidence in over 600 publications in various forms has shown a rapid growth in the use of CDA (Zhang, Jang, & Chahine, 2011). These studies reported high numbers of broadly classified theoretical debates, psychometric modeling, and applications to a range of testing contexts including both large-scale and classroom assessments when using CDAs. With this rapid growth and CDA's technical complexities, many researchers have expressed a need for a clear framework that can guide test design and validation processes. Despite the large body of published work about CDA, beginning researchers often find it difficult to discern the salient issues that await further research.
Therefore, our intent was to examine the status quo of research on CDA and offer a systematic roadmap that could be useful both for developing a much-needed framework of CDA and for beginning researchers.
A systematic literature review is "designed to locate, appraise and synthesize the best available evidence relating to a specific research question in order to provide informative and evidence-based answers" (Boland, Cherry, & Dickson, 2017, p. 2). The systematic review of literature in a narrow area is very important and has "recently become a major area of methodological development" (Gough, Oliver, & Thomas, 2017, p. 9). This idea arose in the 1980s in studies featuring statistical meta-analysis (Glass et al., 1981) and meta-ethnography (Noblit & Hare, 1988). Although the literature review as a form of study has been continuously developed over many years (Boland, Cherry, & Dickson, 2017;Bolin, 2012;Shadish & Lecy, 2015;Sheble, 2017), the systematic literature review is "still a young and rapidly developing field of study and methods of reviewing have not been developed for all areas of science" (Gough, Oliver, & Thomas, 2017, p. 9). There are still many challenges to be overcome (Dickersin, 2015;Oakley et al., 2005) such as conceptual and methodological challenges, resource constraints, the constraints related to the use of reviews, and broader political challenges (Gough, Oliver, & Thomas, 2017). However, many scholars have continued to improve the process used for conducting a systematic literature review in areas such as planning, finding materials to include in the review, describing them, assessing them, analyzing them, synthesizing results, and appraising them (Boland, Cherry, & Dickson, 2017;Lockwood, Munn, & Porritt, 2015;Gough, Oliver, & Thomas, 2012).
This study applied a systematic literature review to the development of CDA to help us to understand its holistic and longitudinal content and relevant theories. Also, this process helped us to enhance a particular research and learning method via concept mapping.

Data Source
We intentionally started to collect relevant references three years ago when we undertook some related studies. After we had accumulated an applicable number of references, we systematically searched them for studies related to CDA. Our research began by locating published sources on CDA research through a university library system. We were able to identify 538 sources in various types of publications (see Table 1). Among them, 40.5% were journal articles, 17.5% conferences papers, and 16.4% book chapters. Twenty-three books dealt with issues related to CDA, and among them, five focused specifically on CDA or CDM. Initially, we identified sources that spanned the period between 1957 and 2010. Figure 1    identifying key concepts in a collection of documents or a research area, comprehending the structure of a literature review in preparation for writing the review, and understanding theory and concepts as well as the relationships between them. It can serve as a tool for establishing the relationships among areas that are not apparent and are often hierarchical (Novak, 1984). The content, structure, links, and organization of the concept map can be used as the framework for the literature review. There are many kinds of software for concept mapping, both commercial software tools (e.g., 3D Topicscape, Inspiration) as well as free software (e.g., Compendium, Cmap Tools). In this review, a program called Inspiration 9 was used.

Result
Based on the relevant literature, the concept mapping in this study took place in three steps: mapping the preliminary structure of CDA and CDM, creating a comprehensive map, and deeply analyzing this map.
In the first stage, we created a preliminary structure map of CDA and CDM based on four main books and four articles (Jang, 2008;Leighton & Gierl, 2007;Nichols, 1994;Nichols, Chipman, & Brennan, 1995;Rupp, Templin, & Henson, 2010;Tatsuoka, 2009). This preliminary structure map included a center (CDA) and four sub-groups, as shown in the following graphic.

Figure 2. The Preliminary Structure Map
Subsequently, we created a comprehensive CDA and CDM map based on the basic map from step 1.
We reviewed all 187 studies and mapped them according to the expanded key words. It was necessary to revisit the original articles to gain some additional information. If an article was involved in a particular area, we would input this study's code into a relevant circle. If there was no relevant circle, a new one would be created based on its relationships with other areas. As a result, the map in Figure 3 successfully represented areas covered by all 187 sources. In our last step, we analyzed each area more substantively and identified distinct issues associated with it.

Figure 4. Detail of a Subsection of the Comprehensive Map
After we obtained the comprehensive map in Figure 3, we clarified its hierarchical structures in Microsoft Word format by outputting them from Inspiration 9. For example, we could focus on particular subsections of our comprehensive map, as shown by the dark lines in Figure 3. Figure 4 details the subsection of the comprehensive map that related to the real-life data application of CDA. Table 2 shows its hierarchical structure as it was outputted in Microsoft Word format. We gained many benefits from the concept mapping process beside the visual maps generated by Inspiration 9. After we created the maps, Inspiration 9 could export additional results in different formats such as PDF, Word, or image files. For example, as mentioned above, Table 2 shows the output of maps in Microsoft Word format. Each file included two parts: at the top of the file, we could retrieve an image of the relevant map; then, the structure of the map became available in text format, which showed the relevant concepts, nodes, and studies in the different categories and levels. This allowed us to recognize the categories and hierarchical structure of the relevant studies; we also could easily track the original studies. In addition, we could establish the areas in which each individual study was involved.
By applying concept mapping, we broadly analyzed the relevant studies on CDA and CDM; we acquired many interesting results. We also considered this process to be a learning experience. By collecting references, analyzing these references, creating maps, and summarizing the maps and relevant outputs, we continued to learn more knowledge and skills that allowed us to adjust our maps continually. Through comparing the many maps made during the process, we found that we also enhanced our research skills and extended our knowledge in more broad areas beside CDA and CDM.

Discussion
The project developed over a two-year period. visually compartmentalize the large body of literature. We categorized the literature into three main classifications: a) real-life data application, b) theoretical and conceptual issues, and c) psychometric modeling (see Figure 2). It appears that much attention has been given to the development of psychometric models, and more research is necessary to examine the validity of CDA in specific learning contexts. The field will benefit from a systematic and comprehensive framework for developing and validating CDA. Two books (Leighton & Gierl, 2007;Rupp, Templin, & Henson, 2010) have done excellent work in this direction.
In the technical and academic support for CDA and CDM, the approach used in this project might prove to be a very useful practice (Zhang, 2016). Concept mapping, as we used it, can be a deductive method for building a conceptual model. 2) Reference pools would include links to all part and points in these maps. Users interested in some parts or points could click on them to track the relevant references.
3) Key elements such as reference lists in APA style, key words, and abstracts for each reference could be included in the reference pools. We could post the relevant articles on the web system after receiving copyright permission for their use in this project. The academic online platform, which could be named The Systematic Review of CDA and CDM, could be an open system. We could set up a function that would allow for group discussion. It would be open to all researchers and learners.
Members could discuss any question and share ideas after registering to obtain a user account.
Using this system, we hope to build an online and virtual learning community that would be helpful for researchers, learners, and CDA's development in the future (Kernan, Basch, & Cadorett, 2018).
In summary, we conducted the first systematic review of the literature relevant to CDA and CDM using a concept mapping method. This study contributes to the field by applying this new method of literature review to CDA and CDM studies. This study showed this method is useful for building an entire picture of a complex body of research and opening a window to future studies in theoretical and technical support. Posting these maps on-line could provide beginners, researchers, and others with information on relevant studies and clarify the framework of CDA and CDM.