Full Paper

Robert Kemm, Debbi Weaver, Agnes Dodds*, Glen Evans* Deirdre Gartland*, Tom Petrovic, Leanne Delbridge and Peter Harris

R.Kemm@physiology.unimelb.edu.au

Department of Physiology, and Multimedia Education Unit*, University of Melbourne

In the Physiology Department, we have often observed that Medical students are very capable of rote-learning their lecture material, but that when their knowledge of physiological concepts is tested by presenting them with a new problem, we find their conceptual logic is lacking.

The objectives of our multimedia development evaluated here were to produce a tutorial tool that aids students to understand some fundamental physiological mechanisms and to be able to apply these concepts to new or different situations.

Pivotal to this approach was the active involvement of students in constructing their own working model of a cell secreting acid and their subsequent interpretation of the effects of hormones and drugs in controlling this cellular function. The pedagogical principles and interface design issues considered in its development have been described in detail at the previous ASCILITE meeting (Weaver et al, 1996). The tutorial is based on a constructivist approach to student learning, within the constraints of the scientific knowledge on the topic.

The aims of this article are principally to assess how well the tutorial meets the learning objectives, and to ascertain whether there are more general outcomes arising from this evaluation that apply to other multimedia developments.

This highly-interactive tutorial program was designed to complement both lectures and practical classes for second-year Medical and Physiotherapy students. Students complete a practical class on gastric acid secretion, using one of their number as a subject, and analyse gastric contents. They test the effects of several common drugs known to be involved in controlling the level of acid secretion. The practical class is popular, but they have difficulty in relating the results obtained to lecture material on the intracellular acid secretion.

After the practical class, students then work in groups of 2 or 3 to complete this tutorial. Their first task is to construct a balanced acid-secreting cell, by selecting from a palette of membrane-transport molecules, moving them on to a simple cell template, and then testing the operation of the cell. Feedback is provided both visually, by animation of the processes, and textually, with hints that encourage appropriate decisions. Students thus proceed to assemble their own model of the specific cell involved in acid secretion. Many students choose to experiment with the program, constructing "impossible" models to better understand the homeostatic concepts involved.

The students then investigate the intracellular activity of the various hormones and drugs used during the practical class. For this second part of the tutorial, a tasks sheet is provided to give some direction to their study, including the predictions of changes induced by these agents.

It is envisaged that this program will enable students to correlate the theories presented to them in traditional lectures, with the practical class results.

Although this program was specifically designed as part of the second-year Physiology course for Medical students, we have also used it for Physiotherapy students, who undertake the same practical classes and lecture course as our Medical students. Science students used the tutorial without completing the practical class.

The tutorial was developed in an iterative process with feedback from formative evaluation at many stages. We focussed particularly on learning outcomes, but recognised the need for data to be collected during the design and development phases in any project.

We started with an analysis of student misconceptions, assessed from a written examination question set in the previous year in which students had access only to conventional lecture and tutorial teaching. The documented list of misconceptions led to the development of a flow-chart to map student errors in dealing with the topic. The development group met regularly throughout the development phase, with input from content experts and educational evaluators and a beta version was reported at ASCILITE (Weaver et al, 1996). This article reports the use of a completed version, with a screen shot shown in Figure 1.

When introduced into the classroom, a comprehensive set of evaluation procedures were used to triangulate information, and to gain both a broad understanding of student reactions and a more detailed examination of student learning processes and actions when using the tutorial. These procedures were cleared with the human experiments ethics committee. The results of the following procedures are reported in this article:

1. a questionnaire to assess student perceptions of the usefulness of the program in assisting their understanding of acid secretion,
2. a questionnaire to determine the students' general approaches to study in this subject
   (Science Students only)
3. observations of selected pairs of students as they worked on the program. These observations could be matched with the exit questionnaires and the audit trails, to give a complete picture of student use for a selected group of students

Additional procedures have been undertaken and are incompletely analysed, so will be reported in a later communication. These include: an audit trail of important student decisions as they progress through the program; a comparative analysis of the misconceptions in an answer to an examination question on the topic, in years before and after the tutorial was available; and voluntary interviews with students to ascertain their comprehension of key concepts.

In March 1997, the first target group of students (second-year Medicine and Physiotherapy students) completed the tutorial as part of their practical class, in 6 sessions over a 2 week period. Various forms of evaluation were conducted for these sessions.

In July-August 1997, the tutorial was also used in the second year Science Physiology course, as a stand-alone tutorial in one of their fortnightly 3-hour computer-based learning workshops in the Science multimedia laboratories. These students did not undertake the practical class, but were supplied with copies of the practical class notes and a sample set of class results. Additional questionnaires were issued to evaluate student approaches to learning.

This involved a comparison of the two groups of students taught in two separate courses, thus the medical (N=216) and physiotherapy (N=72) were compared with science (N=86) students.
The tutorial was a compulsory part of the course for the medical and physiotherapy students. and the rate of participation was high. Participation was optional for science students, and the proportion participating was much lower. Medical and Physiotherapy students had to use the tutorial on outdated computers that made the animations very slow in operation, perhaps affecting their views on the operation of the tutorial compared with Science students.

All three groups of students were asked if the features of the program helped them to understand gastric acid secretion as shown in Table 1. A multivariate analysis of variance (MANOVA) showed an overall difference for these features in these courses (F (8, 716) = 10.35, p < 0.001). Further investigation revealed that the differences were significant for all four variables. In summary, science students were more positive than the other two groups, although all means were high. All students reported less use for the concentration bars. In fact, most of the information in the concentration bars could be seen in a qualitative way from the pictorial information accompanying the animations, and the concentration bars are not used in later versions of the program.

Ratings for medical and physiotherapy students were similar (mean, 4.05) and slightly more favourable than for science students (mean, 3.74). This difference may have occurred because the latter did not actually undertake the gastric acid lab measurements.

Overall ratings of student experience of using the software (1 = bad, 5 = good) were high, although there was a group difference, F(2, 366) = 22.73, p < 0.001 Medical students were less positive on average (mean, 4.047) than Physiotherapy (4.155) and Science (4.767) students.

Once we were satisfied that the tutorial laboratory was suitable for the purpose for which it had been designed, we moved to the next stage of evaluation, beyond student satisfaction and developmental issues. In round 2 of the evaluation, the group of 86 science students were presented with a modified exit questionnaire that sought information on attitude change, and strategy use, challenge and effort.

They also answered a second questionnaire in two parts. The first part comprised 21 items from the Study Process Questionnaire (Biggs, 1987). These items assessed three study strategies, termed 'deep,' 'surface,' and 'achieving,' with seven items each and internal consistency reliabilities (alpha coefficients) of 0.72, 0.69, and 0.74 respectively (Biggs, 1987). The second part was a specially constructed set of items designed to assess three other main self perceptions considered important in the way in which students might tackle the tutorial. These were perceived self efficacy in aspects of the physiology subject of which the tutorial formed a part (efficacy, seven items, alpha = 0.77); an orientation to learn from experiences in the course, eg from discussion or mistakes (learning orientation, seven items, alpha = 0.70); and assessment orientation, eg use of examinations and assignments (assessment orientation, six items, alpha = 0.60); (Evans and Dodds, in development). Our main interest was in how these self perceptions correlated with each other, and with perceptions of the tutorial. It was possible that they might moderate students perceptions of, and approaches to, this constructivist oriented tutorial in predictable ways. For example, it might be expected that deep strategies and a learning orientation might enhance students' use of thinking strategies in the actual tutorial, and also affect the pattern of the audit trails recorded.

The science students were asked to what extent the program changed their attitude to the subject matter of acid secretion (1 = worse, 3 = same, 5 = better). The mean rating was 3.98 (SD = 0.81), indicating a strong change in attitude to the subject as a by-product. The mean attitude rating after the program 'attitude now' was 3.87 (SD = 0.76, 1 = bad, 3= alright, 5 = good). This represents a generally favourable attitude to this subject matter.

Science students rated the strategies they used in working through the program. Table 2 shows that guessing was used least and 'think out' strategies most. The more constructive approaches of applying theory, working out new ideas, and thinking out moves were rated more favourably on average than were guessing and trial and error, although none of the strategies are mutually exclusive. Most significantly for this tutorial project, the highest mean was for 'work out'.

Ratings for the question: "How challenging did you find the tasks in the program?" grouped closely around the midpoint of the 5-point scale (1 = too easy, 3 = good challenge, 5 = too hard; mean = 3.01; SD = 0.50). For the question: "How much effort did you put into the program?", the mean rating was 3.40, SD = 0.71 (1 = very little, 3 = moderate, 5 = intense). The students found the task provided an optimal challenge and it required or elicited reasonable effort.

We were also interested in the extent to which the students reflected on what they had done during the tutorial immediately after they had completed it.. We asked: "How much do you keep going over in your mind what you did in the program?" (1 = very little, 5 = very much). the mean rating was 3.36, with SD 0.76.

The mean scores on the six 5-point scales concerned with efficacy and learning approaches are presented in Table 3. The mean ratings on only the efficacy and learning orientation scales depart appreciably from the scale mid-point of 3.00, with the efficacy mean score representing a comfortable but not overly confident view of efficacy in the course.

The correlations among these self perceptions of learning characteristics are shown in Table 4. For this sample, deep and achieving strategies and learning orientation were moderately intercorrelated, while surface strategy had low negative correlations with each of these. Efficacy was also related to all of these variables except achieving strategy, and also to assessment orientation.

Software help, software use, and 'attitude now' were moderately intercorrelated (correlations of 0.45, 0.38, and 0.49. Similarly, movement, animation, and feedback screens were intercorrelated (correlations of 0.54, 0.39, 0.39). The strategies of 'apply', 'work out', and think out also formed a moderately intercorrelated group (correlations of 0.22, 0.26, and 0.44). These three groups of variables were themselves moderately intercorrelated, giving evidence of a general factor of enthusiasm for the tutorial. Factor analysis also resulted in such a factor.

The personal learning characteristics were also moderately correlated in different ways with these sets of responses to the tutorial. The correlations are shown in Table 5.

Learning orientation is quite strongly correlated with applying theory and facts and with reflection. It also has low but significant correlations with 'work out' and 'think out' as well as with favourable views of the tutorial. Efficacy similarly has low to moderate correlations with 'apply', 'work out', 'think out' and reflection, and a negative correlation with guessing. Deep strategies have low but significant correlations with 'apply', 'think out', and reflect. By contrast, surface strategies are negatively correlated with software help, 'work out', and effort, while being positively correlated with guessing. Achievement strategies are positively related to reflection, in contrast to assessment orientation, which is negatively related. Deep strategies, learning orientation, and efficacy as a group are positively related to constructive and reflective approaches in the tutorial, while surface strategies are positively related to guessing. Interestingly, guessing and trial and error are quite highly correlated (0.51).

Table 5. Correlations* between learning characteristics and responses to the tutorial.

*Only correlations of 0.21 or greater are shown. These are significant at p < 0.05 level

The results of the questionnaires indicate considerable engagement for this tutorial. The students, particularly those from the science course, were impressive in their support for the control over movement of objects in the model, the animation, the feedback screens, and the overall experience of using the software. The medical and physiotherapy students, who actually carried out laboratory experiments, also saw the computer tutorial as of considerable help in interpreting the lab results, more so than the science students, who only saw the results of the laboratory work.

The extra data from the science students indicated that the tutorial effected considerable changes in attitude toward this subject matter resulting in a quite favourable impression. According to the questionnaire results, it provided an optimal challenge and elicited reasonable effort and reflection. The results also showed that the main strategies used in the tutorial were the actively constructive ones of applying theory and facts, working out new ideas, and thinking out moves, rather than guessing or trial and error only. These were precisely the outcomes hoped for in designing the tutorial.

There were some interesting correlations between self assessed personal characteristics and the responses by the science students to the tutorial. Self efficacy and an orientation to learning rather than to just performance for assessment were moderately correlated with active constructive strategies in the tutorial, whereas surface strategies tended to be related to guessing. These relationships, which need to be researched more fully, suggest that the tutorial provides scope for the use of constructive strategies, but the extent to which they are actually used depends also on personal learning characteristics.

More qualitative aspects of the evaluation suggest that active learning extends beyond merely working out how to make the cell animation work successfully. A number of students reported trying out moves they knew to be incorrect to find out their effects, and so increase their understanding of the physiological processes involved. Others were able to think critically about the model itself; they not only reached conclusions about the possible limitations of the model, but tried to extend it to cover other circumstances. Some students expressed a desire to do further reading to answer the questions they had formulated. The tutorial thus shows promise of offering a stimulus to further exploration, partly within the model itself, and partly beyond it.

Our evaluation analysis so far indicates that the tutorial is very successful as a tool to improve student understanding and application of fundamental physiological processes in cellular physiology, and that its style has importance for the development of truly interactive multimedia tutorials in other disciplines.

It has been demonstrated widely to Australian and international academics, and there has been great interest in obtaining it for use in their teaching programs. This has further encouraged our next plan to produce a generic cell model builder to deal with a range of other cell types and functions in health and disease.

The next stage in our continuing evaluation would then be to assess how well the understanding of the principles involved in a cellular operation could be transferred to other cell types, or to hypothetical situations.

We believe that this style of decision making and accountability for the choices is an important step in students' understanding of broader issues. We will be using the structural design of this tutorial as a model for other tutorials dealing with contemporary health issues, such as appropriate treatment of high blood-pressure, obesity and sleep-disorders.

This program was written in SuperCard©, and is currently only available in Apple Macintosh version. It is anticipated that a SuperCard player for Windows will be released later this year, which will enable the program to be used on a PC platform. Discussions are being held with potential publishers for its distribution.

This project was prepared with funding from CAUT (Committee for the Advancement of University Teaching), in a grant to A/Prof. Peter Harris, Dr. Robert Kemm, and Mr. Tom Petrovic of the Department of Physiology, University of Melbourne, in 1996. Additional support from the University of Melbourne, GlaxoWellcome Australia Ltd. and from the Apple University Development Fund is gratefully acknowledged.

(c) Robert Kemm, Debbi Weaver, Agnes Dodds, Glen Evans Deirdre Gartland, Tom Petrovic, Leanne Delbridge and Peter Harris

The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.