Spanner Back Home

Full Paper

Back to List of papers

Evaluation of Computer Assisted Learning Across Four Faculties at Queensland University of Technology: Student Learning Processes and Outcomes

Lynn Wilss

Division of Information Services

Queensland University of Technology



This paper details results of an evaluation of Computer Assisted Learning (CAL) at the Queensland University of Technology. Qualitative methodology was employed to investigate factors that constituted cognitive, relevance, and accessibility dimensions of learning with CAL. In addition, processes of developing and integrating CAL programs within courses were examined. CAL programs considered were from the following schools: Mathematics, Optometry, Humanities, and Nursing. Data were gathered from three groups: lecturers, students, and instructional designers. Analysis of the results indicated the CAL programs fostered student engagement and learning that matched educational objectives set by lecturers; CAL facilitated a constructive approach to learning for students; and CAL initiated deeper learning for some students. It also became apparent that multimedia features assisted comprehension and understanding and students were afforded opportunity to practise flexible learning strategies. Perceived limitations of CAL included lack of relevance of CAL to the rest of the course, in some instances limited access to computers, the lengthy process of development, and relationship issues between members of development teams. Findings relating to cognitive and relevance dimensions of CAL as well as developmental procedures are reported in this paper. Some recommendations regarding development and integration are suggested.


Computer assisted learning (CAL) is becoming increasingly prominent in education and as a result is changing the face of teaching and learning. While the driving force behind such innovations is multifaceted, the desire to increase teaching efficiency (Davies & Crowther, 1995) and quality (Coopers & Lybrand et al., 1996) have generally been regarded as the major contributing factors. This paper details some results of a comprehensive evaluation undertaken to determine the efficacy of CAL at QUT. Specific objectives of the evaluation were: cognitive - to determine how CAL accommodated students learning needs; relevance - to determine studentsí perceptions of CAL as part of their course of study; accessibility - to determine how CAL was integrated into courses; developmental procedures - to determine the reasons for adoption of CAL and subsequent processes of development. Findings relating to cognitive, relevance and integration, and developmental processes are reported in this paper.

Development of CAL

Higher education institutions are making greater use of CAL. This had led to the establishment of ëcentresí in some institutions where customized software is developed. Jacobs (1996) proposed conditions for the establishment of such centres stating they should be:

While these conditions may facilitate development of CAL programs, the actual process of development is complex and time consuming (Reid, 1994). Production of effective educational CAL programs entails incorporating elements that will ensure student engagement. This means addressing specific design considerations such as:

Taking into consideration the factors mentioned above, it becomes apparent that development of educationally sound IMM is a complex process. Some (Reid, 1994) consider the process of developing CAL to be fraught with difficulties. Factors that may contribute to developmental difficulties include: lack of educationally sound examples on which to base development, departure of staff involved in development, the lengthy time of development and the high costs involved. Darby (1992) suggested developing software that can be used over a number of institutions as part of a ìcollaborative consortiaî (p. 194) to alleviate the problem of high costs. Allen et al. (1996) consider the working relationship between developers and subject matter experts to be complex and a potential source of difficulty. If advantages of learning with CAL are to be realized, integration must be planned carefully.

Relevance and Integration of CAL

Successful development and integration of CAL within courses in higher education institutions is dependant, to some extent, on the culture of the institution. Darby (1992) maintained that major changes to the organization of higher education would have to occur for CAL to be used on a wider basis. He suggested making more funds available for courseware development and that lecturers accept a changing role to become more involved in interacting with students as managers of learning. Reid (1994) reported that CAL developed by isolated enthusiasts without recognition or support from peers or the institution can result in the demise of a CAL project if the enthusiast left. To avoid this he advised that development become a part of ìthe academic life of the institutionî (p. 8). Integration must also ensure that students make links between the different types of learning they are experiencing. As Wills and McNaught (1993) contend ìclear information needs to be given about the objectives of any course in which CBL is being usedî (p. 699). The influences of CAL on teaching and learning are explored in the following section.

Facilitation of Learning Processes with CAL

Ultimately instruction is conducted so that students will learn. Some caution that use of IMM does not necessarily mean learning will follow (Bruder, 1991; Allen & Booth, 1996). De Laurentiis (1993) believes that for learning to occur the surface structure of a curriculum, such as the activities and graphics, should match the deep structural issues of the knowledge domain. He maintained IMM has the potential to do this as it incorporates every educational medium except human contact and suggested considering the following points when developing IMM to ensure it is educationally sound: it should be adaptable to studentsí current knowledge and skills, IMM should utilize different media depending on the content and educational objectives, and it should include feedback that is immediate and serves to focus learning activities.

In addition, it is essential that CAL programs are supported by a clear model of learning and teaching (Coopers & Lybrand et al., 1996). For example contemporary theories of learning place emphasis on gaining knowledge through appropriation of information. This is central to a constructivist tradition of learning that emphasizes the role of the learner in knowledge creation (Biggs, 1991). Implications for teaching involve encouraging students to plan and define their goals for learning based on their existing knowledge; allowing students opportunity to examine their processes of thinking and problem solving, that is to become metacognitive processors of information; and providing learning experiences that engage the learner. Hedberg et al. (1994) proposed a method for developing IMM, based on the principles of constructivism, that incorporated three phases: 1) describe the project space, based on the content, and give consideration to structure of the material and what the target audience understands about the material; 2) review the description, make links through an appropriate instructional presentation strategy, and identify metaphors that will facilitate information structure; 3) link the design ideas into an interactive structure.

In summary, productive learning is dependent upon considerations such as content, delivery, the needs of the learner, and specific aspects of the learning environment. Evaluation of courses that incorporate technological innovations helps to illuminate factors that may contribute to educationally sound practices and productive learning outcomes. This evaluation set out to do that as indicated in the objectives detailed above. Methods employed and subsequent findings are presented below.



This evaluation was summative as it set out to determine if CAL programs used across four schools at Queensland University of Technology had achieved intended instructional and integrative objectives. Qualitative methodology was employed. Data were collected from students, lecturers, and instructional designers (IDs).

Description of the Sample

The sample comprised three groups: four lecturers, one involved in development and integration of each CAL program; four instructional designers (Ids), one from each program; students who used the programs. The student profile is detailed in Table 1. In summary students were undergraduates with three groups of students in the first year of their course and optometry students in their final year (4th year). The majority of students were aged in their early twenties with ages ranging from 17 years to 36 years. Ages of students who worked through ëClick On To Mathsí were not available (see Data Collection for ëClick On To Mathsí below).

School and

CAL Program


Student Year Level

and Course

Number of Students

Mean Age In Years

Standard Deviation

School of Mathematics

ëClick On To Mathsí

1st year, Bachelor of Engineering


1996 - 29

1997 - 3



School of Optometry

ëEye Managerí

4th year, Bachelor of

Applied Science Optometry





School of Humanities, Language Studies - Japanese

ëLanguage Masterí


1st year, Bachelor of Business




School of Nursing

ëProblem Based Learning - Module 1, Robertís Situationí


1st year, Bachelor of Nursing




The CAL Programs, Student Use, and Educational Objectives

The CAL programs and educational objectives for each program were:

ëClick On To Mathsí - Mathematics, was developed to provide students with a standard level of understanding of trigonometry and complex numbers;

ëEye Managerí - Optometry, was developed to provide individualised instruction in crucial areas of clinical decision making and patient management;

ëProblem Based Learningí (PBL)- Nursing, was developed to allow students to develop and practise decision making skills and to allow students to develop and integrate knowledge;

ëLanguage Masterí - Japanese, was developed to allow students to practise aural comprehension, conversation, and reading skills in beginning Japanese.

Each program contained elements of multimedia and offered interactive learning. Data were gathered after students had used the program during semester 1, 1997; data for ëClick On To Mathsí was also based on student use during semester 1, 1996. Student use for ëPBLí - Nursing and ëEye Managerí comprised one session lasting approximately 45 to 60 minutes. ëLanguage Masterí - Japanese was timetabled for a one hour session each week for weeks 4 to 12 of the semester. Students worked individually, in pairs, or in small groups. ëClick On To Mathsí was used during a tutorial session by a group of students in semester 1of 1996. During 1997 use was voluntary, as explained below.

Qualitative Data Collection Methods

Some methods of data collection spanned all respondent groups while other methods were specific to a particular group. The methods used for each group are as follows:

Data Collection for ëClick On To Mathsí.

The lecturer who had instigated the development of ëClick On To Mathsí retired at the end of 1996, consequently use by students during the semester this evaluation occurred was voluntary. Only 11 of 500 students used the program and 3 students completed the questionnaire. Students had been surveyed after use of the program during a tutorial in Semester 1, 1996. These surveys were made available as a source of data for the current evaluation.

Analysis of Data

Qualitative data were analysed, in line with Richardís and Richardís (1994) belief that qualitative research involves ìrecognition of categories in the data, generation of ideas about them, and exploration of meaningsî (p. 446). The analysis, carried out using the computer program (NUD.IST) version 3.04, revealed categories which are delineated below along with interview extracts that illustrate each category.


Cognition - Learning With CAL

Lecturers were asked for the educational objectives of their specific program; IDs were asked what they perceived students would learn as a result of using the program; students were asked what they had learnt. The main response categories ranged from learning nothing, to subject specific learning, and finally to learning content and skills. A response was categorised as subject specific if it did not relate generally to learning from the CAL programs but concerned specific aspects of a particular program. A sub-category of skills was defined as working through a process. Examples of responses for each CAL program are presented below.

Click On To Maths

Most responses for learning from Click On To Maths related to learning content. This was evidenced in the lecturerís response and two student responses. One student felt they had learnt nothing, another studentís learning was subject specific, and the ID saw learning as skills based.



Subject Specific


It helped me clear some points.




They will learn the basics of this topic whatever the topic is.


Complex numbers and how they will work.




Theyíll learn very basic, basic for university level, skills in complex numbers and trigonometry and they may get a better understanding of certain concepts.

Eye Manager

There was consensus among the three groups about learning from Eye Manager with the lecturer, ID, and 16 students stating that learning involved gaining the skills to work through a process. This process involved gaining skills in clinical decision making and management of patients. Other student responses included learning nothing (3) and subject specific learning (6) which indicated that students had gained a greater awareness of their diagnosis and management capabilities.

Subject Specific


I need to think more broadly. I get too focussed on a single direction. I need to learn to think about more than one thing at once.

Skills - Work Through a Process



They should have a far better knowledge of clinical decision making processes and management outcomes.


Going through the whole program youíre encapsulating clinical decision making.



Awareness of different management options, logical reasons for performing tests, relative importance of diagnoses.


PBL - Nursing

The majority of responses about learning with PBL indicated that the skills of working through a process to solve a problem were gained. This was evidenced in the Lecturer, ID, and eight student responses. One student felt they learnt nothing.



Very little, it was an evaluation system that is common sense or with experience.

Skill - Work Through a Process:


Iím hoping that they will be able to understand the process behind addressing a problem, so I guess giving them some skills and background in how to address a problem situation.


Basically what we wanted to make explicit was the process that was involved in going through PBL. That it isnít a linear process, it wasnít a lock step, that there are strategies that you can actually go through when you are solving a problem which may not be the answer to it all but they can give you strategies to actually achieve an answer to a problem.


I have learnt that thinking logically in a series of steps as well as thinking carefully about the choices or information along the way enhances my understanding and learning.

Language Master - Japanese

The lecturer stated that use of Language Master - Japanese would improve studentsí listening and reading skills. While nine students concurred with this, the ID and majority of studentsí responses (16) were subject specific and indicated cultural aspects of Japan would be or had been learnt. Four students felt they learnt content such as meaning of words and grammar.

Subject Specific



They will learn things like how they dress. The lecturers were very aware that this would be a cultural experience and they chose clips accordingly.


Learning more cultural background of Japan, how the Japanese people see foreign people and the polite way of talking to people.




What some words mean having seen them in real-life situations.


Has helped me to improve my grammar and understanding where all of these particles are used.



It will probably mainly improve listening skills but also reading because they can see the text on the screen.


Itís really fast the way they speak. It takes me a little while to - I call it ëdrop iní. You have to listen for a while before your ear wants to pick up the Japanese.

Facilitation of Comprehension and Understanding

Each group was asked to identify features of the program that would or did facilitate comprehension and understanding. Four main categories of responses became apparent: the interface; in-built features - that is specific elements of the program that were incorporated to promote student engagement; the interactivity; student strategies. In-built features included pictures, the HELP option, feedback, and the logical sequence of content. Students also reported using other strategies, that is student strategies, to assist their comprehension and understanding, these were asking someone and self-initiated strategies such as trial and error. (Examples of responses will be included at the conference presentation).

Relevance and Integration of CAL

Factors pertaining to relevance and integration of CAL emerged from responses about the limitations of CAL. Some felt the content was either too directed in approach while others felt that it was too limited. Socially, CAL was regarded as being addictive as well as not allowing for social interaction as the human element was missing. Integrative limitations included technical problems and limited access. Some students expressed uncertainty as to the relevance of the program in relation to the rest of the course as well as specific program objectives. (Examples of responses will be included at the conference presentation).

Development - Processes and Relationships

Lecturers and IDs were asked about positive experiences as well as difficulties faced during development of the CAL programs. Positive aspects comprised two main categories: personal and technology related and derived mainly from lecturersí responses. The personal category was defined further as gaining satisfaction from the process of development and enjoying the peer collaboration. Two IDs responded that they had gained nothing or could not think of anything that was positive about the process of development. Technology related positive aspects included gaining a greater understanding of the process of development, being able to incorporate deeper issues, and developing leading technology. Negative responses comprised money issues in that developing CAL was an expensive process, time in that development was a lengthy process, procedural difficulties, staff problems as staff were working on other projects or ID staff left during development, and technology related matters.

Another negative aspect of development concerned the working relationship between lecturers and IDs. Negative categories constituted disagreements and feeling a lack of support, lecturers and ID teams having different ideas about development, and a lack of communication between the groups. (Examples of responses will be included at the conference presentation).


Student responses to questions about learning with CAL revealed that each of the programs facilitated learning according to the lecturersí objectives. Students also reported that learning with CAL had allowed them to gain a broader knowledge of the subject content. This implies that students used knowledge they already had as a basis for further learning and indicates that the CAL programs fostered a constructive approach to learning (Biggs, 1991). Features of the programs that lecturers and IDs stated were included to foster comprehension and understanding were identified by students as doing just that. In particular multimedia elements like graphics and the notepad were reported as assisting comprehension and understanding. As de Laurentiis (1993) stated multimedia features, such as graphics, enhance the processes of learning as they help students to clarify topics and allow for development of constructive learning interactions. Students also used their own strategies, such as asking someone, thus reinforcing the flexibility of learning with CAL. While the CAL programs achieved educational objectives, some limitations became apparent. The most commonly reported limitations from students were that the programs lacked relevance to the rest of the course and there was uncertainty about objectives for learning with CAL .

The majority of positive comments about development of CAL were reported by lecturers. Most stated they were satisfied with development and pleased with the completed program. Allen and Booth (1996) noted that difficulties of development can relate to relationship issues. Results of this evaluation indicated that this was the case particularly for IDs. In fact one ID experienced great relief when the project finished. Other negative aspects were similar to those identified in the literature and included high cost, the lengthy time taken (Reid, 1994), insufficient staff working on projects, and communication difficulties (Phillips, 1996). Integration was also problematic for one program as the lecturer involved left and there was no support within the school to use the program.


The results of this evaluation indicated that the CAL programs fostered student engagement and learning and allowed students to develop a constructive approach to their learning. The multimedia features assisted comprehension and understanding of information and in addition to this afforded students the opportunity to practise flexible learning strategies. The need to make closer links between CAL and the rest of a course, including objectives for learning with CAL became evident. It also became apparent that the process of development is complex and can be fraught with difficulties that include both technological considerations as well as relationship issues. Additionally development and integration should involve support from within a school. Some recommendations are suggested:


Allen, P., Booth, S., Crompton, P., & Timms, D. (1996). Added-value: quality rather than quantity. Active Learning, 4, 14-18.

Biggs, J. B. (Ed.). (1991). Teaching for learning - The view from cognitive psychology. Hawthorn: The Australian Council for Educational Research Ltd.

Bruder, I. (1991). Multimedia: How it changes the way we teach and learn. Electronic Learning, 1(1), 22-26.

Coopers & Lybrand, Institute of Education , & Tavistock Institute. (1996). Evaluation of the teaching and learning technology programme (TLTP). Active Learning, 5, 60-63.

Davies, M., & Crowther, D. (1995). The benefits of using multimedia in higher education: Myths and realities. Active Learning, 3(December), 3-6.

Darby, J. (1992). The future of computers in teaching and learning. Computers Education, 19(1/2), 193-197.

De Laurentiis, E.C. (1993). How to recognise excellent educational software. U.S. Department of Education, Office of Educational Research and Improvement, 2-9.

Hedberg, J. G., Harper, B., Brown, C., & Corderoy, R. (1994). Exploring user interfaces to improve learning outcomes. In K. Beattie, C. McNaught, & S. Wilss (Eds.), Interactive multimedia in university education: Designing for change in teaching and learning, (pp. 15-29). New York: Elsivier.

Jacobs, G. (1996). Editorial: Is there a way? Alt-J, Association for Learning Technology Journal, 4(3), 2-3.

Park, I., & Hannafin, M. J. (1993). Empirically-based guidelines for the design of interactive multimedia. Educational Technology Research and Development, 41(3), 63-86.

Pea, R. D., & Gomez, L. M. (1992). Distributed multimedia learning environments: Why and how? Interactive Learning Environments, 2(2), 73-109.

Reid, T. A. (1994). Perspectives on computers in education: The promise, the pain, the prospect. Active Learning, 1, 4-10.

Richards, T. J., & Richards, L. (1994). Using computers in qualitative research. In N. K. Denzin, & Y. S. Lincoln (Eds.) Handbook of qualitative research (pp. 445-462). London: Sage Publications.

Somekh, B. (1996). Designing software to maximize learning. Alt-J, Association for Learning Technology Journal, 4(3), 4-16.

Wills, S., & McNaught, C. (1993). Evaluating computers in learning in tertiary education. Reaching Out With IT. 691-703.


(c) Lynn Wilss


The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.


Back to List of papers

This page maintained by Rod Kevill. (Last updated: Friday, 21 November 1997)
NOTE: The page was created by an automated process from the emailed paper and may vary slightly in formatting and layout from the author's original.