Conference logo
[ ASCILITE ] [ 2004 Proceedings Contents ]

Creating authentic assessments: A method for the authoring of open book open web examinations

Jeremy B. Williams
School of Business
Universitas 21 Global
The central premise of this paper is that, as an assessment instrument, the final examination has become something of an anachronism. Little has changed in the centuries that universities have been in existence and, even with the advent of sophisticated information and communication technologies, come the end of semester, the vast majority of students still find themselves sitting in rows in invigilated examination halls, answering questions with paper and a ball-point pen. Quite aside from the fact these are tools that are rarely used these days, the assessment tasks that students are presented with are seldom representative of any authentic, 'real world' setting. This paper outlines a process for the construction of open book open web type examinations, and argues that this is a format more in keeping with modern learning theory.

Introduction

There has long been a question mark over the usefulness of examinations - at least in the way they have been traditionally delivered. Entwistle and Entwistle (1991), for example, are in no doubt that examinations do not assess deep conceptual understanding and process skills. To support this contention, many a student would no doubt testify, when quizzed about their examination strategy, that it is often a case of 'cramming' the night before and 'data dumping' on the day, with little knowledge retention thereafter. In spite of this criticism, there is little in the way of a counter argument mounted from those who speak in favour of the status quo. A search of the major educational databases for an article in a refereed journal published in the last 30 years that extols the virtues of closed book, invigilated final examinations produces a nil return.

One might reasonably ask the question, therefore, as to whether the closed book, invigilated final examination belongs to some bygone era. Some universities have been doing the same thing now for hundreds of years, with perhaps the main innovation during this period being the transition from the ink well and the quill to the ball-point pen! The computer assisted assessment (CAA) movement has made some in-roads (see, for example, CAA, 2004) but perusal of the work in this area reveals that in those cases where universities have experimented with CAA, it seldom extends beyond a 'point and click' (or 'drag and drop') multiple choice question (MCQ) format which - an increasingly sophisticated interface notwithstanding - treats knowledge and its acquisition as little more than a memory test. It is also typically used for formative purposes. Rarely, if ever, is CAA used to validate students' learning in a summative fashion, specifically their ability to handle complex, unstructured problems in authentic settings.

Open book open web examinations

Universitas 21 Global (U21G) is an online business school backed by sixteen of the world's leading universities. A new examination format currently being trialed on the MBA program at U21G is best described as an 'open book open web' (OBOW) examination. An example of such an exam can be found at Williams (2004a). OBOW examinations represent a significant departure from the conventional, closed book, invigilated model for examinations in that they take full advantage of the rich media resources available through the World Wide Web. In keeping with the constructivist tradition, students are presented with an opportunity to produce a response that clearly delineates 'what they know' rather than 'what they don't know'. Importantly, students engage with a more authentic and meaningful assessment task, which enhances their learning experience and provides a solid grounding for the demands of the business world. Indeed, as the literature on authentic assessment reveals, this approach is solidly based on constructivism, acknowledging the learner as the chief architect of knowledge building (see, for example, Wiggins 1998; Herrington & Herrington 1998; and Herrington & Standen 2000).

Following Biggs (1999), there is a constructive alignment between this examination instrument and U21G pedagogy in that it builds upon and extends the case oriented, problem based learning approach of the MBA curriculum. It is fitting summative assessment in that, having completed a series of cases during the course unit that require the learner to apply specific key concepts at certain stages, this final assessment item invites the student to draw on all that they have learnt (determining what is relevant), in order to respond to a semi-structured 'mini-case'. This 'mini-case' (or 'caselette') differs appreciably from the case studies used for the purposes of continuous assessment in that it is typically less voluminous in terms of text, it is broader in focus, and it harnesses the power of multimedia to emphasise currency and real world authenticity.

Getting started

A useful way to commence with the groundwork for an OBOW examination is to pick up the local newspaper, browse national newspaper or periodical websites, or flick through the pages of a magazine (e.g. The New York Times, The Guardian, Time Magazine or National Geographic). Alternatively, one can keep an eye on the television news or current affairs magazine programmes (e.g. BBC World, CNN and CNBC). While one might start off with something quite definitive in mind, this is not essential because it is quite common to come up with something better while browsing.

It is also true that some of the best topics crop up when one least expects them. To this end, rather than think about an exam theme just before exam time, it is a good idea to keep a look out for material all the time. An in-flight magazine, a newspaper delivered to the hotel room, or a professional journal lying around in a waiting room all constitute rich sources of ideas for exam topics.

While it is not beyond the realm of possibility, good ideas for exam topics are not usually to be found in text books or academic journals. Looking for leads in these types of publications defeats the object of a commitment to the construction of authentic, 'real world' assessment items. The prose is likely to be overly theoretical, not terribly current and - with the greatest of respect to academic authors - a little 'dry'. Articles written for the general reader or news items, on the other hand, have the capacity to engage the student more readily.

What to look for

The first objective is to come up with a 'story' that is presented in a largely non-academic way. This could be a story about a government, a company, or an individual. As the subject matter expert, the author of an OBOW exam has the capacity to read a newspaper article or watch a television news report through the lens of their academic discipline. The lay person, meanwhile, will view the same article or report quite differently. In constructing an authentic assessment task, the object of the exercise should be to get students to think conceptually about an issue that they would likely have approached quite differently if they had not had the benefit of formal learning in the discipline in question. While this does not have to be stated explicitly, what one is seeking to do is to provide students with an opportunity to demonstrate that they, too, have subject matter expertise. Presenting them with a case that they can easily relate to in lay terms, and then setting them up as 'expert witness' is an effective mechanism for the validation of their learning in their own minds.

Ideally, therefore, what one is looking for is a story that essentially reports on 'the facts'. It need not necessarily be devoid of theory, but it is important the story not be so prescriptive that it does all the thinking for the student! The key, simply, is to provide a stimulus for the student's own, independent thinking. Importantly, this stimulus does not have to be terribly complex. Indeed, the more unstructured the scenario the better, as this affords a student greater latitude in the construction of their response.

Creating a scenario

Having settled on a theme, the next step in the process is to gather together various media that can bring the case to life. While this may appear quite trivial to some, the inclusion of photographs and/or streaming video adds a human dimension to the task that does much to add to the authenticity of the case. Hyperlinks to company web sites and news portals also attest to the legitimacy and genuineness of the case.

If this process seems a little daunting, it is important to remember that the object is to create a scenario, not to write a full blown case study. The text and media that are included need to describe the situation that currently confronts the central character in the scenario (be it a government, a company or an individual) and little more. The inclusion of macro data is quite appropriate to put the case into context, but it is inadvisable to include too much micro data as this might cause the learner to become unnecessarily 'bogged down' with case related content issues when the main aim is to have the student construct a critically analytical response within the broadly defined parameters of the course unit as a whole.

If there is one principle to be observed in the scenario creation process it is to try and use a real life scenario rather than a fictitious one - not least because dreaming up fictitious scenarios generally requires more work. In addition, there is always the chance that an imaginary case will inadvertently include an inconsistency that has the potential to undermine the success of the case. Where it is possible to introduce an element of fiction (and do so with the minimum of risk) is in the setting of the task that the learner has to perform in order to complete the examination.

Setting the task

There are likely to be relatively few occasions during their course of study when a student will be able to 'kill two birds with one stone' and complete an assessment item that can simultaneously be of professional benefit and be credited to their summative assessment for a course unit. This notwithstanding, it is clearly desirable in terms of student motivation if there is an easily discernible link between a learner's education and their professional practice. A useful mechanism for making this bridge is role play.

Having fashioned a scenario for the assessment task, taking full advantage of the various media to create an authentic context, the next and most critical step in the process is the definition of the assessment task itself. Placing the learner in the role of the key decision maker, the expert advisor, or the auditor is a good strategy as it serves to validate the student's learning; that is, they can see the point of what they are doing, and that studying the subject throughout the semester has not just been an academic exercise. However, the brief that one gives the key decision maker, the expert advisor, or the auditor requires careful thought. At this juncture it is advisable to revisit the stated learning outcomes for the course unit as they have been defined in the course unit syllabus. The object of the assessment task should be to provide the learner with the maximum opportunity to demonstrate that they have achieved these learning outcomes.

Defining the parameters

The definition of the assessment task (the role that the learner is to fulfil) might amount to no more than a paragraph, and ideally it should invite a wide of variety of equally correct answers. However, one has to exercise care not to set a task that is so unstructured a student is either struck by 'writers' block' and is unable to start, or they start too hastily and go off on a tangent, not getting to the crux of the matter. Reflecting once again on the stated learning outcomes and offering some broad guidelines on how the student might approach the task is therefore a good idea, provided, of course, these guidelines are not so specific the student is 'spoon fed' with an answer plan. The examination author treads a fine line here. Ultimately, it will be a question of professional judgement.

Laying down the ground rules

The final step in the process is essentially administrative, and can be standardised for all course units. Importantly, there should be some very specific instructions designed to combat plagiarism and cheating, including a clear statement that, while the question is a broad one likely to invite a variety of acceptable answers, high marks will be awarded for good, critical analysis rather than content cut and pasted from web sites and other electronic sources. It is also a good idea to keep the time period allowed for the examination sufficiently tight (i.e. no more than 24 hours) and to insist that, to secure a passing grade, students must draw on the concepts and analytical tools referred to during the course unit in question; evidence of which must take the form of direct references to the subject materials (i.e. the prescribed text, courseware, and other resources). These conditions, together with the fact the assessment task is very contextualised, make it extremely difficult for students to cheat (Williams 2002; 2004b).

Summary and conclusion

University examinations continue to be dominated by invigilated, closed book, pen and paper tests, yet there is no peer reviewed academic literature in existence (or at least none that the author of this paper can find) that extols the virtues of such a format. The defining characteristic of the OBOW approach is a commitment to authentic assessment. It is a form of assessment that fosters understanding of learning processes in terms of real life performance as opposed to a display of inert knowledge. The student is presented with real world challenges that require them to apply their relevant skills and knowledge, rather than memorise and regurgitate, or select from predetermined options as is the case with MCQs, for example (Laurillard, 2002). The scope for cheating is limited if the task is appropriately structured (Morgan & O'Reilly, 1999, p. 80), and most important of all, it is an approach that engages students because the task is something for which they will have empathy. This, in turn, elicits deeper learning.

References

Biggs, J. (1999). Teaching for Quality Learning at University. Oxford: Oxford University Press.

CAA (2004). CAA Conferences at Loughborough University. [viewed 23 Sept 2004] http://www.lboro.ac.uk/service/ltd/flicaa/conferences.html

Entwistle, N. J. & Entwistle, A. C. (1991). Contrasting forms of understanding for degree examination: The student experience and its implications. Higher Education, 22, 205-227.

Herrington, J. & Herrington, A. (1998). Authentic assessment and multimedia: How university students respond to a model of authentic assessment. Higher Education Research and Development, 17(3), 305-322.

Herrington, J. & Standen P. (2000). Moving from an instructivist to a constructivist multimedia learning environment. Journal of Educational Multimedia and Hypermedia, 9(3), 195-205.

Laurillard, D. (2002). Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies (2nd edn). London and New York: Routledge.

Morgan, C. & O'Reilly, M. (1999). Assessing Open and Distance Learners. London: Kogan Page. Wiggins, G. (1998). Educative Assessment. San Francisco: Jossey Bass.

Williams, J.B. (2002). The plagiarism problem: Are students entirely to blame? Proceedings 19th ASCILITE Conference, vol. 2, pp. 721-30. http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/189.pdf

Williams, J. B. (2004a). The Authentic Assessment Web Site. [verified 23 Sept 2004] http://www.jeremybwilliams.net/authentic/

Williams, J. B. (2004b). The five key benefits of on-line final examinations (with three free bonus benefits). In R. Ottewill, E. Borredon, L. Falque, B. Macfarlane & A. Wall (Eds), Educational Innovation in Economics and Business VIII: Pedagogy, Technology and Innovation. (pp. 347-358). Dordrecht: Kluwer Academic Publishers.

Please cite as: Williams, J.B. (2004). Creating authentic assessments: A method for the authoring of open book open web examinations. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 934-937). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/williams.html

© 2004 Jeremy B. Williams
The author assigns to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author also grants a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the author.

[ ASCILITE ] [ 2004 Proceedings Contents ]
This URL: http://www.ascilite.org.au/conferences/perth04/procs/williams.html
HTML created 1 Dec 2004. Last revision: 4 Dec 2004.