Full Paper

David Jones

d.jones@cqu.edu.au

Department of Mathematics and Computing

Bruce Jamieson

b.jamieson@cqu.edu.au

Information Technology Division

Central Queensland University, Rockhampton

The resource intensive nature of assessment can often decrease the quality of the feedback provided to students (Oliver and Mitchell, 1996). Providing timely, meaningful feedback on student progress is essential and can influence student results. Problems associated with traditional assignment management approaches contribute significantly to assignment turnaround time while much of the process of performing assessment is mechanical, repetitious and a perfect candidate for the application of information technology. It is the combination of these factors that is driving interest in online assignment management systems by many educators and institutions throughout the world.

This paper describes experience at CQU with three generations of online assignment management over the last three years by on-campus and distance computing students. The paper starts by describing the problems associated with the traditional form of assignment management. The next section describes in some detail the experience of using each of the three online assignment management systems. In closing the paper describes the next generation system to be used at CQU.

The move to an online assignment management system at CQU was driven by a number of problems with the traditional approach including

• packaging
  One of the most laborious tasks in marking an assignment is dis-entombing the disks and print outs from assignment packaging (which can include removing staples from disks). At the other extreme are badly packaged assignments from which components fall out of during delivery.
• slow delivery time
  A significant proportion of the turnaround time for distance students is the time taken to deliver assignments via the postal service.
• process duplication
  CQU units are now being offered via a number of different modes including: traditional on-campus, commercial campus, distance and overseas campus. Each mode uses a different form of traditional assignment submission system. This variety makes it difficult to document the appropriate procedure for students to follow and to identify problems.
• moderation problems
  There is a trend, especially for overseas campuses, for campuses to use local markers for all units. This leads to moderation and consistency problems.
• human error
  The traditional assignment management process includes a very complex process that includes a large amount of handling by a number of different people. The complexity of the process increases the likelihood of errors.
• corruption
  Many of the units offered by the Department of Mathematics and Computing requires that students submit programs on floppy disk. It is not unusual for these disks to be corrupted either by staples or other mistreatment during the delivery stage.
• storage and retrieval
  It is not uncommon for a student to contact a lecturer to query a returned assignment. In most cases the only record of the assignment a lecturer will have is the final mark (if that). With only this information it is difficult to make meaningful comments.

availability of markers
Central Queensland does not have a large computing industry that means that most CQU computing graduates move to capital cities to work. This means that the pool of qualified markers, especially for advanced level computing subjects, is very small.

Online assignment management systems have been used within the Department of Mathematics and Computing at CQU since 1995, mainly in two advanced level computing units. The following section describes in detail the experience of using three different generations of the system. Due to space limits only three of the six semesters which have used online submission will be described. An overview of these semesters is shown in Table 1.

The first online assignment submission system was entirely manual, email-based and used by distance students only. The submission process consisted of the following steps

1. Student emails assignment to the lecturer.
1. Lecturer manually acknowledges receipt, saves the assignment and forwards a copy to the marker.
1. The marker marks the assignment and returns it to the student with a copy going to the lecturer for archival purposes.

In 1995 widespread use of the Internet was still rare and relatively primitive. For example 80% of assignments submitted using email were sent using text-based email programs with only 20% using graphical user interface (GUI) based tools such as Pegasus. Table 2 provides an overview of the performance of this approach.

As can been seen in Table 2 the turn-around time on assignments was far from good. The major cause of this was problems with the marker being unavailable due to personal reasons at important times during the semester. This influenced turnaround times for both online and traditional assignment management systems.

This initial foray into online assignment management led to the following observations

• the technology was new
  Using this system was the first experience many students (and the marker) had of email attachments and sending large binary files by email. Inexperience with the technology led to students and the marker having to spend significant amounts of time working out how to perform simple tasks. Appropriate training would be necessary for future use of online assignment management.
• the technology was of low quality
  The difficulty of using the text-based tools and the poor speed and quality of connections led to many frustrations on the part of students and the marker. It was hoped that the trend towards GUI-based tools would address this problem.
• automation is necessary
  Even with only 20 students using the online system, the lack of automation in the process generated more work for the lecturer than with the traditional approach.

While not a great success the first attempt did provide some promise of the benefits that a suitable system combined with appropriate support could offer. The next online assignment management system was entirely automated, using Perl scripts, but still relied on email for assignment submission. The process was:

1. Student sends the assignment to an email address using a specific format for the subject line of the email message.
1. A Perl script saves the assignment in an archive, forwards a copy to the marker, updates a results Web page (Figure 1) to indicate receipt and sends an acknowledgment back to the student.
1. Once marked the marker returns the assignment to the student with a copy going to another email address.
1. At this other email address a Perl script saves the marked assignment in an archive and updates the results Web page (Figure 1).

This process was used in the second half of 1995 and the first half of 1996. Table 3 provides a summary of the results from the first use of the system in 1995. Other changes from the manual online assignment management system included a new marker and the fact that all the online assignments were submitted using GUI email programs such as Eudora or Pegasus.

It should be noted that assignments submitted before the due date were not marked until the due date. This contributes somewhat to the turn-around times listed in Table 3.

Observations from the first use of this automated email system included

The advent of the file upload capability for Web browsers (introduced in Netscape version 2.0) led to the development of a completely Web-based assignment management system. This system has been used since the second semester of 1996. Table 4 provides a summary of the systems use during the first semester of 1997.

With this system assignment management consisted of the following steps

1. Student connects to a specific Web page, enters their student number and chooses the assignment to submit.
1. The system presents a page specific to the assignment that contains specific instructions and the correct number of file upload elements.
1. Student selects the location of the assignment files on their local computer and hits submit.
1. The files are placed onto the Web server (in a secure area) and the student sees a page that lists the size of the assignment files on the server (used by the student to ensure safe uploading).
1. A password protected Web page provides the marker with access to all student assignments.
1. After downloading an assignment, comments and marks are entered via another Web page.
1. Assignments are returned via another Web page.

Like the email system this system uses Perl scripts and maintains a results page similar to Figure 1.

During the use of this system and number of observations were made including:

• Word viruses
  It was during this time that Word viruses became prevalent. To address this problem, students were instructed to submit assignments as Text, RTF or HTML documents, however a quarter or more were submitted in a word processed format. During the early stages of this problem it was policy to open documents if possible, requesting resubmits only as a last resort - this changed quickly.
• online marking was quicker than traditional marking
  With three to four semesters of online marking experience, better resources and a better system which significantly reduced the overhead of online marking it was found that online marking was taking 20-30% less time than marking traditional assignments.

The system used at CQU is undergoing further development and expansion including

• increased security
  Each student will have a unique username/password that will enable them to review and check on their assignments. Forms of encryption may also be included.
• automatic allocation of assignments to markers
  Lecturerís will be able to program the system to automatically channel assignments to appropriate markers. For example, a marker based in Hong Kong will receive all the assignments from Hong Kong students. This feature could be used to support peer assessment.
• automated event notification
  This feature will automatically generate email messages to appropriate people in response to events including students not submitting assignments by the due date or markers taking too long to mark assignments.
• automated file checking
  This feature would check the files submitted by students to ensure that they are the specified format and eventually that they are virus free.
• data mining
  With all assignments stored on a central server it will be possible to perform numerous types of analysis to identify trends, student performance across degrees and plagiarism.
• automatic marking systems
  There are a number of automated marking systems, such as Ceildh ( http://www.cs.nott.ac.uk/~ceilidh ), that could be incorporated into the system to further automate the system.
• improved assignment annotation
  The current Web-based system uses a fairly primitive method to allow markers to make annotations on student assignments. Hypertext, Word templates (Price and Petre, 1997) or systems such as CleverX ( http://www.cleverx.com/) and Markin ( http://www.net-shopper.co.uk/creative/education/languages/martin/markin.htm) could be incorporated into the system to improve this.
• integration with other University databases.
  While CQU uses a central database for student records, which includes a Web interface, transcription of student results into this database is still primarily done via paper and physical data entry. Integrating the assignment management system with these databases will greatly decrease the resources required and the possibility of human error during transcription.

The system being used and under development at CQU is part of Webfuse ( http://webfuse.cqu.edu.au). A freely available system for creating and maintaining Web sites which runs on both UNIX and Windows NT/95 servers.

As many expect, online assignment management makes it possible to reduce assignment turn-around time, can reduce the time taken to mark individual assignments and provide the opportunity to implement a number of new approaches to assessment. However achieving this is not a straight forward process and cannot be achieved without appropriate technical, training, and administrative support. It is important to note that the technical difficulties associated with online assignment management are simple in comparison to the organisational and administrative changes necessary.

(c) David Jones and Bruce Jamieson

The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.