Conference logo
[ ASCILITE ] [ 2004 Proceedings Contents ]

Promoting metacognition through negotiated assessment

Joe Luca and Mark McMahon
School of Communications and Multimedia
Edith Cowan University
Metacognition is widely considered integral to effective learning. However environments that support metacognition can be difficult to develop. This paper proposes an approach to assessment through student contracts that are designed to both address issues of fairness of assessment as well as promote the planning, monitoring and evaluation integral to enhancing metacognition. By negotiating their assessment in ways that involve feedback that is internal, parallel and external to students, they have the opportunity to develop clearer understandings of themselves as learners and their own learning processes. The model is supported with online technology to help create an easy and confidential manner in which peer feedback can be collated. Initial findings suggest that students perceive themselves to be metacognitive, however, they may not be fully engaged in the processes that underpin this psychological state. It is proposed that online environments that support negotiated assessment expose these processes in ways that may enhance metacognitive outcomes, and lead to further research that identifies the nature of such processes and their value in the assessment approach.


Introduction

Metacognition is one of the holy grails of education. Defined as "knowledge and beliefs about thinking and the factors affecting thinking" which regulate "the articulation of strategy and knowledge" (Pressley, 1998), it is the primary enabling state for students to be able to work independently and flexibly. The high level of awareness that characterises metacognition is associated with a desire for self knowledge, whereas low self consciousness breeds intellectual defensiveness. Metacognition is often associated with stable psychological states such as IQ. However, rather than being developmentally fixed, research is showing that the acquisition of metacognition may be subject to instructional intervention (Boekaerts, 1997). The question then becomes one of how to promote it.

Weinstein & Mayer (1986) see all metacognitive activities as partly the monitoring of comprehension, and it would appear that this ability to monitor oneself is what distinguishes metacognitive activity from domain specific cognition. Wilson (1999) argues that the term metacognition can be used in 'vague, confusing, and often contradictory' ways and can be used to describe a range of disparate higher level cognitive skills. She goes on, however, to define metacognition as "awareness individuals have of their thinking and their evaluation and regulation of their thinking". In this definition it is both a state and a process, with 3 functions:

Shraw et al (1995) state, "Examples of general metacognitive awareness include evaluating the adequacy of relevant domain knowledge, selecting strategies that are situationally appropriate, and allocating cognitive resources to a degree that matches task demands" (p. 444). This position is further reinforced by Jacobson (1998) who defines metacognition both as "knowing the process by which one learns" (p. 3) and, in citing Borokowski, Carr, and Pressley (1987) as "the self monitoring of, and conscious use of learning strategies" (p. 4).

This apparently contradictory position of being a both a state and a process can be reconciled by acknowledging the dependence of metaknowledge upon domain dependent cognitive processes. In their research on metamemory, Nelson and Narens (1994) identify the relationship between the meta-level and the object level of cognition through a reciprocal flow of control and monitoring (Figure 1). While the model itself is perhaps a little simplistic, it does give some hope to those floundering in the problem of how to encourage metacognition in students. In Nelson and Naren's concept, one can view metacognition as a pet puppy - in order to grow and become strong it must be fed and exercised. The process of monitoring nurtures metacognition, likewise the activation of control processes exercises it.

Figure 1

Figure 1: A model of metacognition

It can be argued, then, that metacognition is not something that can be tackled as a discrete entity - just as well, since it is quite inaccessible as such - but can be enhanced through engaging at the subordinate levels of monitoring and control of thinking processes.

Blakey and Spence (1990) cite Dirkes' synthesis of much of the literature on metacognition into the following features:

Each of these define some aspect of monitoring and control "Connecting" new information with former knowledge is primarily driven by the context of learning, and within a framework of skills inherent in a specific task. Thus it is integral to domain specific skills. The second involves the actual development of use of metacognitive strategies applied to a task. Planning, evaluating and monitoring, however, define the internal processing used to support the acquisition of domain specific skills and inform the application of regulatory strategies. These can therefore be considered key to the whole process of metacognition.

One of the ways of promoting metacognition is through assessment. Haefner (2004) describes an approach to assessment that engages planning monitoring and evaluation, through three different mechanisms of assessment feedback. These engage students in setting goals, evaluating their performance and monitoring their understandings through techniques that are internal, such as self assessment, parallel such as through peer collaboration, and external, such as tutor feedback.

This study builds on this approach by engaging these forms of feedback in a formative way, where the criteria for students judging the value of their work is negotiated over a semester. The study is based around a final year undergraduate unit in Project Management Methodology for Interactive Multimedia development. As with most final year courses (both graduate and undergraduate), teamwork is often needed to complete developmental projects that illustrate the students' technical/content skills learnt throughout the course of their study, as well as professional skills needed by industry practitioners. However, students often complain about team based assessment, as they perceive that if they work harder than others, the whole team is still given the same mark. A learning environment was developed to help promote fair and equitable teamwork, while at the same time integrating the planning, monitoring, and evaluation inherent in metacognitive development through internal, external and parallel feedback.

An approach to negotiated assessment

Context and unit assessment requirements

This project was conducted with a group of 16 final year students enrolled in the Interactive Multimedia course at Edith Cowan University (IMM3228 "Project Management Methods"). These students are required to develop web sites for "real" industry clients through teamwork to help meet industry needs, as well as support the development of students' professional skills (Collis, 1998; Klemm & Snell, 1996; English & Yazdani, 1999). A custom built online courseware management system was used to help deliver this unit (see http://www.scam.ecu.edu.au/).

Students are encouraged to select their teams and allocate their own tasks based on their skill strengths. Teams are required to create a project proposal, design specification, metrics, evaluation report, post mortem and a web site. Decisions need to made on:

By the end of this unit teams compile an online CV that can be used for employment purposes with the following components (see http://studentprojects.scam.ecu.edu.au/): Student teams are also required to complete eight problems using an online application to post solutions. These represent the topics being covered in the lectures, and are designed to encourage students to use the given resources to research solutions (book, readers, lecture material URL's, library and expert opinioin). Teams make decisions about what resources to use, what "angle" to take in solving the problem and which team members are involved (as negotiated in the Team Contract). Completed tasks are then posted online and assessed by other teams and tutors. The topics form the basis of the final exam and are assessed according to the following criteria: Overall, the assessment for this unit comprised of:

Designing the learning environment for negotiated assessment

Learning activities are designed to support fair and equitable teamwork by focusing on authentic, self regulated and reflective activities. Figure 2 illustrates the team based setting in which the study was conducted and highlights key aspects of the learning environment:
Figure 2

Figure 2: Learning environment design

Within this learning environment, students use the "Team Contract" (Table 1) to negotiate their assessment items. Each row in the Team Contract represents a key assessment point that students can consider how heavily they want to contribute to, based on their aims for future employment and current skill sets.

If there are four students in a team, then each student should contribute 25% of the overall marks. However, this is not mandatory, and students can specify how much of the "assessment pie" they want! This team based negotiated assessment process is completed in two separate stages. Students are to consider:

Having students negotiate each assessment item helps promote responsibility within the team, as well as the expected quality expected from each team member. So, when the actual assignments are submitted, it was clear how much effort and quality has been contributed by each team member. Also, the fact that the assignment components are authentic, and aligned with multimedia project management good practice, helps motivate students contribute to this process.

Beyond the issues of fairness and equitability of marks distribution, however, this negotiation also involves students in planning their learning, by setting goals and estimating their performance both in terms of outcome (mark) and process (quality of work), they are then required to evaluate these goals against actual achievement. Through a cyclical process and through the internal, parallel and external feedback mechanisms of peer, tutor, and self assessment, students are engaged in a continuous process of self monitoring.

Table 1: Team contract

Assessment
Items
%Name 1Name 2Name 3, etc
EM    EQ    AM    AQEM    EQ    AM    AQEM    EQ    AM    AQ
Online Tasks16


Project Proposal10


Design Specifications5


PM Doc 12


Application Development5


Presentation & Online CV2


Evaluation Report3


Metrics Report3


Post Mortem2


PM Doc 22


Total50


We agree that the assigned assessment values & quality are acceptable (Tutor ____________________)
Signatures


EM = Estimated Mark, EQ = Estimated Quality, AM = Actual Mark, AQ = Actual Quality

The implementation of the team contract is based on planning, monitoring and evaluation (Dirkes, 1985). By week 3 students must plan and negotiate with their team members which assessment items they are responsible for ("EM" in Table 1), as well as predicting the quality of these ("EQ" in Table 1). As the semester progresses, students are required to continually monitor their own performance in terms of their stated plans as well as their team members' commitments (as agreed to in the contract). If they feel the team isn't progressing as agreed, they can inform the tutor through the peer assessment tool. As well as ongoing modification of their initial plans, students formally evaluate their performance and that of their peers when the assessment item is submitted. They do this by entering the actual mark and quality of their contributions into the team contract ("AM" and "AQ" in Table 1).

Rationale for the development of a means of negotiated assessment and initial support for student contracts

As an initial step to exploring the value of negotiated assessment, students were required to complete a five point Likert scale questionnaire that examined their existing perceived existing orientations to planning, monitoring and evaluating their learning, as well as their interpretations of the value of negotiated assessment as implemented through the strategy of a student contract. Students were asked the extent to which they agreed to the statements shown in Table 2. The greyed area shows the category and associated Figure to which each question is related to.

Table 2: Metacognition questionnaire results (n=16)

QuestionSD  D  N  A SA Metacognitive
Category
See
Figure
1.I know how I learn best003103General
Metacognition
2
2.I know what I am good at as well as the things I have difficulty with001105General
Metacognition
2
3.It is useful for me to think about my studies before I go to class00286Planning3
4.I find it useful to set myself goals for learning003112Planning3
5.I understand concepts better when I imagine them in practice001510Monitoring4
6.It is important for me to find ways of applying what I am
studying to real settings
00295Monitoring4
7.Making notes helps me understand what I am studying01492Monitoring4
8.I find it helpful to compare my ideas with other students to
make sure I'm on the right track
00385Evaluation5
9.I compare what I've achieved in learning to what I planned to achieve02482Evaluation5
10.Using the student contract will help me plan my learning006100Value of student
contracts
6
11.Using the student contract will help me monitor my performance003121Value of student
contracts
6
12.Using the student contract will help me evaluate my performance03121Value of student
contracts
6
13.Using the student contract will help me understand myself
as a learner better
024100Value of student
contracts
6
SD = Strongly Disagree, D = Disagree, N = Neutral, A = Agree, SA = Strongly Agree

As can be seen from Table 2, students on the whole perceived themselves to be operating at a reasonably high metacognitive level. Figures 3 to 7 show the extent to which they support each concept of overall metacognition, planning, monitoring, evaluation, and the value of the student contract as a tool to enhance their monitoring processes.

Figure 3

Figure 3: Perceived overall metacognition (Q 1-2)

Figure 4

Figure 4: Perceived planning (Q 3-4)

Figure 5

Figure 5: Perceived monitoring (Q 5-7)

Figure 6

Figure 6: Perceived evaluation (Q 8-9)

The positive overall response suggests that most students perceive themselves to be operating at a high level with regard to their ability to plan, monitor and evaluate their learning processes. The higher reported response to monitoring indicates that most students are comfortable with engaging in strategies that involve monitoring their understandings although as Table 2 (question 7) demonstrates, one student

Figure 7

Figure 7: Perceived value of student contracts (Q 10-13)

disagreed with the concept of note taking as a useful approach. It was also intriguing that two students disagreed with the statement regarding the evaluation of goals (question 9). While comparing what one has achieved with what one has planned to achieve is integral to the process of metacognitive monitoring it would appear that for some students at least, it is not highly valued.

The student contracts, as with the concepts inherent in metacognition also received support overall. Again, some disagreement was received, in particular to the statement regarding the value of the tool as a means of making students better learners (question 13). It would appear that while all subjects can see how the strategy will engage them in planning, monitoring and evaluation, the generalisation of these processes to a general state of metacognitive awareness is not considered an integral outcome of negotiated assessment for all of the students.

Conclusions about negotiated assessment as a tool for promoting metacognition

With an increasing need for graduates to have a broad base of generic skills that can be applied to the diverse and constantly evolving world of work, the value of metacognition as a means for students to regulate their understandings and develop new strategies for learning has become paramount. This paper has proposed negotiated assessment as a means of providing online support for the development of the planning, monitoring, and evaluation inherent in metacognition. The proposed approach also addresses many of the issues of teamwork based projects where a single shared mark often fails to capture the complexities of group dynamics and the relative contributions made by each member of the team.

Initial findings suggest that students perceive themselves to be operating at a reasonably high metacognitive level. They can also see the value of a tool for negotiated assessment in promoting self monitoring. Nevertheless, the accuracy of student perceptions must be questioned, when at least two students fail to see the value of evaluating performance against goals; an activity integral to the planning, monitoring and evaluation feedback loop that is self monitoring. This is further exacerbated by a tendency for a couple of students to fail to draw the link between the student contracts and the processes that underpin them, with a general concept of metacognition involving enhanced self awareness in the learning process.

Further study will be required to explore the actual patters of self monitoring in which students engage. This may involve analysis of the discourse involved in negotiating assessment to identify the actual nature of self monitoring compared to the students' self perceptions. While this represents the beginning of a project that is necessarily iterative and exploratory over a period of time, initial findings suggest that negotiated assessment through an on line student contract system has the potential to provide a means to expose the processes of assessment that will ensure self monitoring on the part of students, that should lead to outcomes that are fair, representative of the work that students engage in, and above all, lead to the development of students' awareness of the processes that underpin their own learning.

References

Blakey, E. & Spence, S. (1990). Developing Metacognition. ERIC Digest. [viewed 19 Jan 2002 at http://www.ed.gov/databases/ERIC_Digests/ed327218.html, verified 30 Sep 2004 at http://www.ericfacility.net/ericdigests/ed327218.html]

Boekaerts, M. (1997). Self-regulated learning: A new concept embraced by researchers, policy makers, educators, teachers, and students. Learning and Instruction, 7(2), 161-186.

Collis, B. (1997). Supporting project-based collaborative learning via a World Wide Web environment. In B. H. Khan (Ed.), Web-Based Instruction (pp. 213-219). New Jersey: Educational Technology Publications.

English, S. & Yazdani, M. (1999). Computer-supported cooperative learning in a virtual university. Journal of Computer Assisted Learning, 15(2), 2-13.

Haefner, T. (2004). Assessment as a magnification of internal, parallel, and external reflection. Action in Teacher Education, 25(4), 14-19.

Nelson, T. O. & Narens, L. (1994). The role of metacognition in problem solving. In J. Metcalfe & A. Shiminura (Eds.), Metacognition (pp. 207-226). Cambridge: MIT Press.

Jacobson, R. (1998). Teachers improving learning using metacognition with self monitoring learning strategies. Education, 118(4), 579-589.

Klemm, W. R. & Snell, J. R. (1996). Enriching computer-mediated group learning by coupling constructivism with collaborative learning. Journal of Instructional Science and Technology, 1(2). [verified 30 Sep 2004] http://www.usq.edu.au/electpub/e-jist/docs/old/vol1no2/article1.htm

Pressley, M., Van Etten, S., Yokoi, L., Freebern, G. & Van Meter, P. (1998). The metacognition of student scholarship: A grounded theory approach. In D. J. Hacker, J. Dunlosky & A. C. Graesser (Eds), Metacognition in Educational Theory & Practice (pp. 347-366). New Jersey: Lawrence Earlbaum Associates Inc.

Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26, 113-125.

Weinstein, C. E. & Mayer, R. (1986). The teaching of learning strategies. In M. Wittrock (Ed), Handbook of research on teaching (pp. 315-327). New York: Macmillan.

Wilson, J. (1999). Defining metacognition: A step towards recognising metacognition as a worthwhile part of the curriculum. Proceedings AARE Conference, Melbourne. [verified 30 Sep 2004] http://www.aare.edu.au/99pap/wil99527.htm

Authors: Joe Luca, School of Communications and Multimedia. Edith Cowan University. j.luca@ecu.edu.au
Mark McMahon, School of Communications and Multimedia. Edith Cowan University. m.mcmahon@ecu.edu.au

Please cite as: Luca, J. & McMahon, M. (2004). Promoting metacognition through negotiated assessment. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 562-570). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/luca.html

© 2004 Joe Luca & Mark McMahon
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.


[ ASCILITE ] [ 2004 Proceedings Contents ]
This URL: http://www.ascilite.org.au/conferences/perth04/procs/.html
HTML created 29 Nov 2004. Last revision: 29 Nov 2004.