Designing Cognitively Diagnostic Assessment for Algebraic Content Knowledge and Thinking Skills

This study explored a diagnostic assessment method that emphasized the cognitive process of algebra learning. The study utilized a design and a theory-driven model to examine the content knowledge. Using the theory driven model, the thinking skills of algebra learning was also examined. A Bayesian network model was applied to represent the theory model and the quantitative assessment structure. Simulated data was applied to the model to illustrate the purpose. The diagnostic assessment model was represented by a Bayesian network model. Four explanatory variables and nine evidential variables were identified. These were developed to describe the content domain and cognitive structure in an algebra learning process. The diagnostic assessment model both exhibited learning progresses and provided diagnostic feedback. Through students’ performance examples model-based achievement scores were reported at three levels: 1) evidential variable level, 2) explanatory variable lower level, and 3) explanatory variable higher level. This study revealed that the diagnostic assessment model can effectively report learners’ progress in algebra learning in both content knowledge and thinking skills.


Research Problem Statement
How to elaborately discern and effectively assess algebra learning progresses and difficulties has increasingly become a serious problem for algebra educators and algebra learning researchers.Many algebra teachers face the situation that high school students do not comprehend algebraic and mathematical concepts and cannot proficiently establish connections among these algebraic concepts.Students had difficulty understanding expressions and translations among algebraic relations (Perry & Dockett, 2005;Telese, 2007).These same students had trouble solving problems involving equations.They also had difficulty with functions and non-linear relations and applying relevant algebraic knowledge specific in problem contexts.

Study Purpose
The purpose of the study is to explore a cognitively diagnostic assessment procedure of algebra content knowledge and thinking skills.The cognitive diagnostic assessment (Mislevy, Steinberg, Breyer, Johnson, & Almond, 2002;Zhang, 2007) provides algebra teachers with a set of procedures and tools to recognize algebra students' learning progresses, difficulties and obstacles.These teachers have the opportunity to facilitate students' acquisition of algebraic knowledge and development of algebraic problem skills, and enhance students' ability in choosing sensible strategies to tackle complicated algebraic problems (Drijver, Dekker, & Wijers, 2011;Telese, 2000).The study is also concerned with key difficulty factors which may impede students' algebraic skill development.The key difficulties in algebra learning include abstraction, generalization, procedural fluency, and symbol sense (Arcavi, 2005;Drijver, Dekker, & Wijers, 2011).
This study examines how to combine cognitive diagnostic assessment theories and Algebra I content knowledge effectively.Efforts to solve Algebra I students' learning difficulties and measurements of students' learning progress are examined as well.
Based on observations through cognitive task analysis, the investigators discern the factors that facilitate and impede algebra learning.This is achieved through the administration of a set of assessment tasks to students.These tasks can be used in both learning and assessment environments.These tasks consist of three dimensions: (a) mathematical cognitive dimension (Kilpatrick, Swafford, & Findell, 2001), (b) algebraic content dimension, and (c) algebraic approach dimension (Drijvers, Goddijn, & Kindt, 2011;National Council of Teachers of Mathematics, 2011).

Literature Review
This literature review consists of three parts: algebra learning, cognitive task analysis and assessment.In the algebra learning part, we present conceptual understanding, procedural skills, strategic problem solving, reasoning skills and symbol sense.Understanding these aspects well can foster understanding reasons for students' mistakes, difficulties and obstacles.Cognitive tasks and analysis describe the rationale of how high quality cognitive tasks connect learning and assessment effectively.

Algebra Learning, Procedural Skills and Conceptual Understanding in Algebraic Learning
Algebra is an important core subject area at the middle school level.Algebra and other related mathematics courses provide students with knowledge and basic problem solving skills (Kieran, 2012;Cai & Brooke, 2006) which directly influence success in advanced mathematics and sciences courses (Li, Shavelson, Kupermintz, & Ruiz-Primo, 2002;Rech & Harrington, 2000).Deficiency of preparation of algebraic curriculum in secondary schools will impede mathematics learning and achievement.
Researchers (e.g., Rittle-Johnson & Siegler, 1998;Zhang & Telese, 2012) believe that algebra learning and progress can be examined in both conceptual understanding and procedural skills in a given algebraic sub-domain.Current debates about this perspective are on the relations of conceptual understanding and procedural knowledge.Several researchers claim that conceptual understanding is more important than procedural knowledge in algebraic learning.The conceptual understanding dominates cognitive processes while learning algebra and conceptual understanding is intertwined with the development of procedural skills (Rittle-Johnson & Siegler, 1998).Students with a greater conceptual understanding have better procedural knowledge in solving a math problem.Conceptual understanding precedes procedural skills, further evidence indicates that conceptual understanding predicts student problem solving proficiency with their procedural knowledge (Rittle-Johnson & Siegler, 1998).
A second perspective is that conceptual and procedural knowledge may develop iteratively (Karmiloff-Smith, 1992;Schneider & Stern, 2010), meaning that procedural knowledge could influence conceptual understanding.An iterative process model (Rittle-Johnson, Siegler, & Alibali, 2001) suggests that one type of knowledge gained first will improve another type of knowledge through problem representation.This perspective introduces a potential factor between the relations of conceptual and procedural knowledge development, and problem representation.In other words, cognitive tasks are a critical component to explore the relations of development of conceptual understanding and procedural knowledge.
The relation between procedural skills and conceptual understanding is a well-explored field.Kilpatrick, Swafford and Findell (2011) postulated the concept of mathematical proficiency based on five components: (a) conceptual understanding (b) procedural proficiency (c) strategic competence (d) adaptive reasoning, and (e) productive disposition.

Development of Strategic Problem Solving, Reasoning Skills and Symbol Sense
Some researchers claim that the core goal of school algebra education is to develop problem solving strategies and reasoning skills rather than procedural fluency (Drijvers, Goddijn, & Kindt, 2011).However, symbol sense and flexibility are important in solving algebraic problems.Symbols are broadly defined by Cobb (2000) as objects to "denote any situation in which a concrete entity such as a mark on paper, an icon on a computer screen, or an arrangement of physical materials is interpreted as standing for or signifying something else" (p.17).
Mathematizing and thus, algebratizing a situation requires the use of symbol sense (van Oers, 2000).Symbol sense includes strategic work, global focus and algebraic reasoning.It is not enough that students can use symbols and complex symbolic representations, they also need to be able to judge which symbols are relevant to a given situation (van Oers, 2000).Combining symbol sense, conceptual and procedural knowledge and strategic problem solving will lead to a focus on deeper understanding of algebraic concepts.

A Macro-Strategic Approach to Algebra
Several researchers explore algebraic pedagogical strategies in chunking patterns.Bednarz, Kieran and Lee (1996) argue algebraic knowledge is necessary to make generalizations, in problem solving, and modeling real-world situation using functions.The National Council of Teachers of Mathematics (NCTM, 2011) suggests that algebra instruction focuses on (a) understanding patterns, relations and functions; (b) representing and analyzing mathematical situations and structures using algebraic symbols; (c) using mathematical models to represent and understand quantitative relationships; and (d) analyzing change in various contexts.Similarly, Drijvers, Goddijn, and Kindt (2011) believe that school algebra should also focus on patterns and formulas, restrictions, functions and representing symbols.This brief and logical summary can be applied to both algebraic content and cognitive domains.Patterns and structures of algebra have been a focal point of many studies (Kindt, 2004).These are about "investigating, identifying and formulating similarities relating to general patterns and underlying algebraic structure" (Drijvers, Goddijn, & Kindt, 2011, p. 12).
Restriction is a word used to describe a category that includes solving equations and inequality problems, and calculating values of unknown variables that satisfies the required conditions (Webb & Abels, 2011).These kinds of problems not only exist in algebra, but are also found in economics and health sciences.Function is an approach to dealing with relations between variables.Two features are highlighted: covariance and dynamics.Changes are observed between two variables.Thus, algebra can be used in expressing relationships between two or more quantities.
Hence, symbolic representation uses a set of signs, symbols and rules to express the variables and relationships among them.Thus, algebra problems require translation from the problem context to algebraic language in a compact manner.Representing symbols is also involved in developing a structure by which algebraic relationships such as variable relations can be represented (Cobb, 2000;Gravemeijer, Cobb, Bowers, & Whitenack, 2000).

Critical Difficulties in Algebraic Problem Solving
Some researchers (Drijvers, Goddijn, & Kindt, 2011;Doorman & Drijvers, 2011) examine "stumbling blocks" related to (a) abstraction, (b) overgeneralization, and (c) procedural fluency and symbol sense.Abstraction is the first difficulty.Students could understand a concrete example such as a formula.However students may have difficulty transferring the formula to a general function, i.e. from abstract to concrete.The second difficulty is overgeneralization (Kirschner & Awtry, 2004).In this situation, students struggle with identifying generalizations and their limits.Students can work well in a simple situation.However, they make mistakes again when they solve different problems in a complex situation.The third difficulty is related to procedural fluency and symbol sense.Although students have mastered the algebraic problem rules, and are able to apply them in a simple problem situation correctly, they still make mistakes in other similar situations.Students lack structure knowledge of algebraic problems and further lack insight for certain algebraic problem steps.

Algebraic Expertise in Algebraic Problem Solving
Algebraic expertise involves different kinds of knowledge skills and strategies.In terms of Drijiver, Goddin, and Kindt's perspective ( 2011), algebraic expertise model consists of basic skills and symbol sense.Basic skills contain procedural work, local focus and algebraic calculation.The symbol sense (Arcavi, 1994(Arcavi, , 2005) ) includes strategic work, global focus and algebraic reasoning.Strategic skills are about arriving at a problem approach; it is the capacity to maintain an overview of the problem solving process, and to view expressions and formula globally.Drijver et al. (2011) continue their model in algebraic reasoning that the reasoning involves "qualitative reflections on terms and factors in expressions, symmetry considerations or reasoning with particular or extreme cases" (p.22).

Cognitive Task Development and Analysis
Cognitive tasks, also called learning tasks, are usually defined as a series of objects which are implicitly contained in a learning environment (Clark & Estes, 1996).These cognitive tasks are "wrapped" into content knowledge of a given domain.In a pedagogical context, the instructor and learners rarely place the focus on appropriate cognitive tasks.The instructor can basically target some contents, but they do not precisely tailor the content into knowledge categorization and different problem solving features.In other words, the instructor cannot elaborately design cognitive tasks for both (Clark, Yates, Early, & Moulton, 2010).At the end of a learning period, the instructor collects a piece of content knowledge arbitrarily or randomly to develop them into test items.Learners' response to these items can be represented at a score level.However, these scores do not report what knowledge these learners acquired, what problem solving skills the learner developed, what kind of issues and errors they made, and what reasons led to these problems.Any progressive details in the learning process cannot be recorded and recognized.Thus, an effective representation of a cognitive task in learning is a crucial step towards establishing an assessment that reports diagnostic information of knowledge acquisition and skill development.

Cognitive Task Analysis and Influencing Factors
Learning involves learning tasks and researchers in the learning sciences seek to examine them from diverse angles.Chipman, Schraagen, and Shalin (2000) describe learning tasks in terms of a covert-overt continuum.The covert learning tasks, explored in this study, involve unobservable knowledge corresponding to psychological activities, thought processes and goal structures as opposed to overt learning tasks which involve observable performance.
Cognitive task analysis is a method for specifying the covert cognitive structures and processes that are associated with overt task performance.Cognitive task analysis focuses on covert cognitive processes (Clark & Estes, 1996;Gray, 2000;Prasanna, Yang, & King, 2009).According to Redding (1992) cognitive task analysis has three steps: (1) develop visual representations of knowledge structures, (2) describe cognitive processes underling performance, and (3) determine implications of results.Olson and Biolsi (1991) argue that cognitive representations are critical for conceptual and performance features of cognitive task analysis.Multivariate techniques such as multidimensional scaling can characterize quantitative features of cognitive representations (Rider & Redding, 1993).Researchers have suggested various frameworks, such as representing problem solving skills and strategies (Redding, 1995), knowledge structures and measurement modeling (Benysh, Koubek, & Cakvez, 1993), and knowledge modeling classification (Essens, Fallesen, McCann, Cannon-Browers, & Dorfel, 1994).

Algebraic Learning Task Development
Based on the understanding of cognitive task analysis, development of algebra learning tasks becomes an important theme in algebraic education.There is insufficient research study report on algebraic task development, Ainley, Bills, and Wilson (2005) reported that they developed a sequence of tasks for the learning and teaching of algebra at ages 11-13.In the study the researcher focused on three activities: a) general activities, b) transformational activities and c) global, meta-level activities.These researchers attempted to establish a relation between the algebraic tasks and activities.However, they did not do cognitive explorations of these tasks and activities.Kieran (2012) examined algebra learning tasks to connect algebra learning and computer algebra system (CAS).The algebra cognitive tasks have been discussed based on a dichotomy in algebra between procedures and concepts.In other words, researcher belief the algebra task design is important as the algebra learning realizes in computer-based tutorial system.They argue that algebra task design is a crucial element of learning environment.The nature of algebra learning task influences the activity of students.Sequential characteristics of algebra learning task as an example has been discussed (Kieran & Drijvers, 2006;Hitt & Kieran, 2009).In brief, several groups of researchers have already targeted their research problems on the algebra learning task in different learning and pedagogical environments.

Cognitive Diagnostic Assessment
Cognitive diagnostic assessment (CDA) was developed by Mislevy, Almond, and Lukas (2004) in their evidence-centered assessment design.This framework is an effective structure for designing, producing and delivering assessments.These can be used to provide cognitive diagnostic assessment information.This model contains three logically connected models: student model, evidence model and task model.
Student models represent student knowledge, skills and expertise.These cognitive aspects cannot usually be observed, but they can be elicited through what students say or performance in their learning processes.After cognitive analysis, an assessment construct can be established.
In fact, evidence models consist of two sub-models: (a) the evaluative sub-model, and (b) the statistical model.The evaluative sub-model is composed of a set of evidence rules by means of which feature of student's performance are extracted.The statistical model is applied to make inferences about assessment constructs (student models) based on the evidence variables.
Task models provide a framework for establishing the contexts and tasks which will be used to observe individual performances.A task model is crucial to the assessment process because it determines what kinds of task variables can be extracted from data.
There are many different variations of diagnostic cognitive assessment models.However the essence and crucial elements are on the examination of the learning process based on cognitive and learning theory.Thus learning progresses, difficulties and obstacles can be identified.

Diagnostic Assessment of Algebra Learning
In recent years, there have been increasing research studies focusing on mathematical learning and diagnostic assessment.For example, Ketterlin-Geller and Yovanoff (2009) summarized approaches to diagnostic assessment in three categories: cognitive diagnostic assessment, skill analysis and error analysis in mathematical learning.They also evaluated diagnostic assessment from pedagogical purposes in identifying persistent misconception, skills and errors.From content referent perspective, they also looked at diagnostic assessment as theory of cognitive processing in mathematics, broad skills across the curriculum, and procedural skills across the curriculum.From score estimation, cognitive diagnostic assessment represents knowledge state towards mastery of multidimensional cognitive attributes; skill analysis examines skill aggregation towards mastery of unidimensional subskills; and error analysis scrutinizes distractor element exploration in looking for error patterns.
Diagnostic assessment in mathematics learning can be applied to web-based or on-line based learning environments (Heffernan, Koedinger, Junker, & Ritter, 2012;Kieran, 2012).Heffernan, Koedinger, Junker, and Ritter (2012) used web-based cognitive assessment systems for predicting student performance on state exams.Kieran (2012) examined the relations between technological application in mathematical learning and algebraic learning theories.
Recent exploration and advancements in learning theories, cognitive diagnostic assessment and mathematical learning provide new knowledge and experience for our multidimensional diagnostic assessment research study.

Research Questions
Based on our research purposes in this study we pose research questions as follows: 1) How are the cognitive dimension elements related to the content dimension elements in Algebra I learning tasks?
2) What cognitive aspects can we examine in student difficulties when completing Algebra I learning tasks?
3) How do the diagnostic assessment models provide diagnostic information for each student when they have learning difficulties?

Research Design
There are two parts of the research design: learning task design, and data for simulation.The purpose of the learning task design is to explore characteristics of learning tasks used in both learning and assessment.The data collection design further clarifies factors that influence students' algebra learning when students expose to these learning tasks and solve these algebraic problems.

Learning Task Design
Learning tasks are usually defined as a series of objects which are implicitly contained in a learning environment (Clark & Estes, 1996).Algebra I was taken as content domain.In algebra I learning, a learning task can be an algebraic concept, a topic, or a problem statement followed by several questions.Some examples can be referred to in Algebra I Assessment (Hall, Reed, & McGowan, 2011).An instructor emphasizes one or two aspects which reflect student progress in content and cognitive dimensions.In an assessment context, a learning task is elaborately tailored to a measureable object which reveals content, cognitive and psychometric characteristics.In this study, we develop learning tasks for assessment.The characteristics of these learning tasks in assessment will provide solid algebraic contents and cognitive evidences further develop similar learning tasks utilized in pedagogical environments.The tasks in assessment can be any formats such as essay questions, multiple choice questions, and other algebraic problems.
The learning tasks in assessment design pursue a framework consisting of content criteria and cognitive characteristics.The content criteria include Algebra I Assessment, Eligible Essential Knowledge and Skills.These content topics include functional relationship, properties and attributes of functions, linear functions and inequalities, quadratic and other nonlinear functions.Algebra I contents are classified into several topics which all reflect the functions: (a) general formulas, (b) forms of linear equations, (c) circumference, (d) area, (e) surface area, and (f) volume.These content domains will be integrated into the algebraic learning tasks in assessment.
Cognitive characteristics will be examined in conceptual understanding, basic skills and symbol sense.The basic skills consist of procedure work, logical focus and algebraic calculation; the symbol sense includes strategic work, global focus and algebraic reasoning.

The content dimension indicates what kinds of knowledge students learn; the cognitive dimension reveals what
cognitive characteristics the task focuses on.

Data for Simulation and Bayesian Network Representation
An example data set is presented below for the purposes of illustration and simulation.This data simulated 3 students' performance in a given Algebra I learning task.The variables were students' successful performances (or errors) in each cognition and content dimension.In cognitive assessment analysis, students' competences to complete Algebra I tasks can be represented in theoretical framework which consists of explanatory variables and evidence variables (Mislevy, Almond, & Lukas, 2004;Mislevy, Steinberg, Breyer, Johnson, & Almond, 2002;Zhang, 2007).These variables can be further developed into a Bayesian network, a quantitative diagnostic assessment model.Student individual progress information can be provided by the quantitative Bayesian network model (Darwiche, 2009;Koski & Noble, 2009).The following is one example of the Bayesian network model of Algebra I learning task completion (see Figure 1).

Figure 1. The representation model of the algebra learning task
The Bayesian network model exhibits a hierarchically cognitive process.The model consists of Conceptual Understanding, Basic Skills and Symbol Sense at exploratory level.The evidential variables are in the bottom of the model.The Bayesian network model is a quantitative format of the cognitive diagnostic assessment model used to describe a student's learning trajectory (Zhang, 2007;Zhang & Leung, 2007;Zhang & Lu, 2014a;Zhang & Lu, 2014b, Zhang, 2016).
Thus, a Bayesian network and a cognitive diagnostic assessment are integrated into a unified model called Bayesian cognitive diagnostic assessment model.This model basically consists of two kinds of variables: explanatory variable and evidence variable.An evidence variable is an observable variable and an explanatory variable is an unobservable variable.The evidence variables can be assembled into different cognitive diagnostic assessment models.
Figure 1 exhibits three explanatory variables, conceptual understanding, basic skills and symbol sense in the diagnostic assessment model; basic concepts, rule applications, term relations, procedural work, local focus, algebraic calculation, strategic work, global focus and algebraic reasoning are evidence variables.Both explanatory variables and evidence variables consist of a Bayesian cognitive diagnostic assessment model.Students' achievement scores as evidence variable values update explanatory variables: conceptual understanding, basic skills and symbol sense.Thus, a contingent table can be set up between each individual student achievement score and three explanatory variables.We will set up students' achievement scores in the horizontal level (row) and explanatory variables in the vertical level (column).The horizontal level provides a student with individual diagnostic information to inform student's progresses and obstacles; the vertical levels provide teachers and assessors with patterns of diagnostic information.

Initializing Values of the Bayesian Network Diagnostic Assessment Model
BayesiaLab (Jouffe, 2011)

Conclusion and Discussion
This study developed a diagnostic assessment model for Algebra I learning with Bayesian network model.The Bayesian network model represented the diagnostic assessment model in both conceptual and quantitative representation.The model had four exploratory variables which represent the learners' proficiency in solve Algebra I learning tasks.The proficiency was represented in three aspects: Conceptual Understanding, Basic Skills and Symbol Sense.There were 9 evidential variables which generated the students' scores, which were updated in the Bayesian network assessment model.It noticed that the hierarchical diagnostic assessment model was non-linear.From student B's score in Algebra I learning we found that even though the student performed with relatively lower score in Conceptual Understanding, Basic Skills and Symbol Sense, the Algebra I Learning scored higher scores at 50.29 of 100.Briefly this was related to that the assessment model statistically is nonlinear.
The diagnostic assessment model was a dynamic model which recorded the all possible learning trajectories.Thus the assessment model can effectively differentiate different cognitive aspects: Conceptual Understanding, Basic Skills and Symbol Sense.This diagnostic assessment model provided an example for how to effectively assess Algebra I Learning.The content domains and cognitive structures always vary.The diagnostic assessment model should be specified to stress on different contents and cognitive structures.

Limitations
The diagnostic assessment model for Algebra I Learning was simulated with 3 learners' data.With a larger sample size, this assessment model will become more robust.
The assessment model was also cognitive task structure specific.The researchers should have sufficient domain and learning science knowledge.Thus the assessment model can provide effective diagnostic assessment information.

Figure
Figure 4.I , a Bayesian network software, was used to build up the cognitive diagnostic assessment model.In this model, there are three explanatory variables: Conceptual Understanding, Basic Skills and Symbol Sense which consist of Algebra I Learning Competence.In other words, cognitively we have to