|[ ASCILITE ]
[ 2004 Proceedings Contents ] |
The central premise of this paper is that, as an assessment instrument, the final examination has become something of an anachronism. Little has changed in the centuries that universities have been in existence and, even with the advent of sophisticated information and communication technologies, come the end of semester, the vast majority of students still find themselves sitting in rows in invigilated examination halls, answering questions with paper and a ball-point pen. Quite aside from the fact these are tools that are rarely used these days, the assessment tasks that students are presented with are seldom representative of any authentic, 'real world' setting. This paper outlines a process for the construction of open book open web type examinations, and argues that this is a format more in keeping with modern learning theory.
One might reasonably ask the question, therefore, as to whether the closed book, invigilated final examination belongs to some bygone era. Some universities have been doing the same thing now for hundreds of years, with perhaps the main innovation during this period being the transition from the ink well and the quill to the ball-point pen! The computer assisted assessment (CAA) movement has made some in-roads (see, for example, CAA, 2004) but perusal of the work in this area reveals that in those cases where universities have experimented with CAA, it seldom extends beyond a 'point and click' (or 'drag and drop') multiple choice question (MCQ) format which - an increasingly sophisticated interface notwithstanding - treats knowledge and its acquisition as little more than a memory test. It is also typically used for formative purposes. Rarely, if ever, is CAA used to validate students' learning in a summative fashion, specifically their ability to handle complex, unstructured problems in authentic settings.
Following Biggs (1999), there is a constructive alignment between this examination instrument and U21G pedagogy in that it builds upon and extends the case oriented, problem based learning approach of the MBA curriculum. It is fitting summative assessment in that, having completed a series of cases during the course unit that require the learner to apply specific key concepts at certain stages, this final assessment item invites the student to draw on all that they have learnt (determining what is relevant), in order to respond to a semi-structured 'mini-case'. This 'mini-case' (or 'caselette') differs appreciably from the case studies used for the purposes of continuous assessment in that it is typically less voluminous in terms of text, it is broader in focus, and it harnesses the power of multimedia to emphasise currency and real world authenticity.
It is also true that some of the best topics crop up when one least expects them. To this end, rather than think about an exam theme just before exam time, it is a good idea to keep a look out for material all the time. An in-flight magazine, a newspaper delivered to the hotel room, or a professional journal lying around in a waiting room all constitute rich sources of ideas for exam topics.
While it is not beyond the realm of possibility, good ideas for exam topics are not usually to be found in text books or academic journals. Looking for leads in these types of publications defeats the object of a commitment to the construction of authentic, 'real world' assessment items. The prose is likely to be overly theoretical, not terribly current and - with the greatest of respect to academic authors - a little 'dry'. Articles written for the general reader or news items, on the other hand, have the capacity to engage the student more readily.
Ideally, therefore, what one is looking for is a story that essentially reports on 'the facts'. It need not necessarily be devoid of theory, but it is important the story not be so prescriptive that it does all the thinking for the student! The key, simply, is to provide a stimulus for the student's own, independent thinking. Importantly, this stimulus does not have to be terribly complex. Indeed, the more unstructured the scenario the better, as this affords a student greater latitude in the construction of their response.
If this process seems a little daunting, it is important to remember that the object is to create a scenario, not to write a full blown case study. The text and media that are included need to describe the situation that currently confronts the central character in the scenario (be it a government, a company or an individual) and little more. The inclusion of macro data is quite appropriate to put the case into context, but it is inadvisable to include too much micro data as this might cause the learner to become unnecessarily 'bogged down' with case related content issues when the main aim is to have the student construct a critically analytical response within the broadly defined parameters of the course unit as a whole.
If there is one principle to be observed in the scenario creation process it is to try and use a real life scenario rather than a fictitious one - not least because dreaming up fictitious scenarios generally requires more work. In addition, there is always the chance that an imaginary case will inadvertently include an inconsistency that has the potential to undermine the success of the case. Where it is possible to introduce an element of fiction (and do so with the minimum of risk) is in the setting of the task that the learner has to perform in order to complete the examination.
Having fashioned a scenario for the assessment task, taking full advantage of the various media to create an authentic context, the next and most critical step in the process is the definition of the assessment task itself. Placing the learner in the role of the key decision maker, the expert advisor, or the auditor is a good strategy as it serves to validate the student's learning; that is, they can see the point of what they are doing, and that studying the subject throughout the semester has not just been an academic exercise. However, the brief that one gives the key decision maker, the expert advisor, or the auditor requires careful thought. At this juncture it is advisable to revisit the stated learning outcomes for the course unit as they have been defined in the course unit syllabus. The object of the assessment task should be to provide the learner with the maximum opportunity to demonstrate that they have achieved these learning outcomes.
CAA (2004). CAA Conferences at Loughborough University. [viewed 23 Sept 2004] http://www.lboro.ac.uk/service/ltd/flicaa/conferences.html
Entwistle, N. J. & Entwistle, A. C. (1991). Contrasting forms of understanding for degree examination: The student experience and its implications. Higher Education, 22, 205-227.
Herrington, J. & Herrington, A. (1998). Authentic assessment and multimedia: How university students respond to a model of authentic assessment. Higher Education Research and Development, 17(3), 305-322.
Herrington, J. & Standen P. (2000). Moving from an instructivist to a constructivist multimedia learning environment. Journal of Educational Multimedia and Hypermedia, 9(3), 195-205.
Laurillard, D. (2002). Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies (2nd edn). London and New York: Routledge.
Morgan, C. & O'Reilly, M. (1999). Assessing Open and Distance Learners. London: Kogan Page. Wiggins, G. (1998). Educative Assessment. San Francisco: Jossey Bass.
Williams, J.B. (2002). The plagiarism problem: Are students entirely to blame? Proceedings 19th ASCILITE Conference, vol. 2, pp. 721-30. http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/189.pdf
Williams, J. B. (2004a). The Authentic Assessment Web Site. [verified 23 Sept 2004] http://www.jeremybwilliams.net/authentic/
Williams, J. B. (2004b). The five key benefits of on-line final examinations (with three free bonus benefits). In R. Ottewill, E. Borredon, L. Falque, B. Macfarlane & A. Wall (Eds), Educational Innovation in Economics and Business VIII: Pedagogy, Technology and Innovation. (pp. 347-358). Dordrecht: Kluwer Academic Publishers.
|Please cite as: Williams, J.B. (2004). Creating authentic assessments: A method for the authoring of open book open web examinations. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 934-937). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/williams.html|
© 2004 Jeremy B. Williams
The author assigns to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author also grants a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the author.