|[ ASCILITE ]
[ 2004 Proceedings Contents ] |
Many researchers have focused on the question of whether or not there is a difference in learning effectiveness between online and on campus courses. Using the SOLO taxonomy, we explore the quality of learning outcomes of students enrolled in "Computer Network and Internet" classes offered by an institution in wholly online and face to face (blended) learning modes. Students enrolled in the F2F course made much less use of the e-learning resources than those in the virtual course. Although there were significant differences between the two classes with respect to pre-test scores, at the end of the semester, there was no significant difference in post test scores or SOLO ranking. This is not to say that the two classes resulted in the same individual improvement in learning outcomes. This methodology, if adapted to include more assessment items and larger cohorts of students might be a useful model for evaluating the pedagogical effectiveness of various e-learning courses.
The pedagogical effectiveness of computer based and online learning experiences has been under active research for many years (Hartley, 1996). Whilst pre- and post-tests have often been used as measures of instructional effectiveness, they often focus on recall if facts and when used alone do not reveal sufficient information about the quality of learning, or depth of understanding reached by students. This is in part because of different types of assessment in the different courses, or because assessment items do not involved higher order learning. One framework for considering the quality of learning outcomes is the Structure of Observed Learning Outcomes (SOLO) taxonomy (Biggs & Collis, 1982). Bloom's taxonomy can be used for planning examinations, but SOLO is best used to review learning outcomes (Biggs, 2002). The five level SOLO taxonomy, when applied to student responses to examination questions, can be used to see if the learner has remembered only a number of points without relating them in a meaningful way or has built meaningfully related structures in the newly learned material, that has been extended by relationship with prior knowledge related to the topic. Thus the primary goals of this study were to use the SOLO taxonomy to explore the quality of learning outcomes from students completing a wholly course with those completing a parallel on campus course.
|F2F Class (Blended Course)||Virtual Class (e-Course)|
The pre-PT results of the students in the F2F class were significantly higher than those enrolled in the virtual class (Table 2). The students in the virtual class had lower scores, but there was a greater range of scores than that seen in the F2F class. On the post PT, students in the F2F had higher mean scores, but there was no significant difference between the mean scores for the two classes. In the case of post PT scores, there was a greater range of scores in the F2F class than in the virtual class. The SOLO based analysis of questions in the pre-test revealed that students in the F2F class had a deeper understanding of the subject matter on entry to the course compared with the students in the virtual class (p<0.05) (Table 3). On exit from the course, there was no significant difference between the groups in terms of their depth of understanding of the examination topics. This is not to say that the overall learning 'effect' of the two courses was identical. Plotting each student's performance on the pre and post questions (Table 4) shows a clustering in the lower right corner. This reflects increased post-test scores compared with the respective pre-test scores. In the Virtual and F2F classes, 33% of students increased one SOLO rank from SOLO 3 to SOLO 4. It is noteworthy that some students in the F2F class did not reach SOLO 3 and some regressed.
|* Significant difference between e-class and Blended class t=-2.387, df= 56, p<0.020|
|SOLO Rank||Virtual Pre|
|Mean SOLO score||3.21||3.68||3.57||3.76|
Learning outcomes for the two courses were comparable when SOLO was used to inform assessment planning and review learning outcomes. The use of grades in evaluating educational outcomes is based upon the often incorrect assumption that the course content is adequately sampled in the assessment regimen and assesses understanding rather than lower order cognitive skills (Biggs, 200). In this case, multiple choice questions and two open ended questions used a pre- and post-tests provided some standardisation for comparison between the two courses. However there were other assessment data that could have been analysed using SOLO to determine more broadly the extent of learning achieved in this course. But since different assessment tasks were used in the two courses, for the purposes of direct comparison between the courses it was inappropriate. If the exercise was to be repeated, it would be useful to consider the use of a broader range of assessment items common to both courses that could available for analysis.
The majority of students in the virtual class demonstrated improvements in learning compared with those in the F2F class. Given that the participants in the virtual course entered from industry, often with a limited formal knowledge of the subject matter, this was a welcome sign. Course website logs show that these students made extensive use of online materials, including the replaying of recorded lectures and the use of the discussion forum. The greatest potential for e-learning lies in harnessing the power of technology to support students from non-traditional backgrounds (Veronikas & Shaughnessy, 2004). In this virtual course, the orchestration of a range of learning experiences and resources appeared to be near optimal in supporting students less prepared for academic study.
The differences between these two courses point to some factors necessary for the educational effectiveness of online courses. Communication tools - discussion forums, email and interactive video and whiteboard - were infrequently used by students in the F2F course. However in the wholly online course they were almost over used. Morris & Zuluaga (2003) and Koory (2003) emphasised the importance of a range of mechanisms to facilitate staff and student interactions for successful learning in online courses. For students (in this case in different cities in another country), these mechanisms appeared to provide equivalent academic and peer interactions that would normally occur in the F2F classroom. Examining the relationship between these communication mechanisms and learning outcomes is an area for further research.
Benigno, V. & G. Trentin. (2000). The evaluation of online courses. Journal of Computer Assisted Learning, 16, 259-270.
Biggs, J.B. (2002). Teaching for quality learning at university. Buckingham: Oxford University Press.
Biggs, J.B. & Collis, K.F. (1982). Evaluating the quality of learning. Academic Press: New York.
Dutra, J., Gibbons, J. F., Pannoni, R. L., Sipusic, M. J., Smith, R. B. & Sutherland, W.R. (1999). Virtual Collaborative Learning: A Comparison between Face-to-Face Tutored Video Instruction (TVI) and Distributed Tutored Video Instruction (DTVI). A Sun MicroSystems Research Technical Report. [verified 22 Nov 2004] http://www.sunlabs.com/technical-reports/1999/smli_tr-99-72.pdf
Hartley, S. et al. (1996). Enhancing Teaching Using the Internet: Report of the Working Group on the WWW as an Interactive Teaching Resource. In ACM SIGCSE Integrating Technology into Computer Science Education, Vol 28, Special Issue, 1996.
Jones, S. & Madden, M. (2002). The Internet goes to college. Pew Internet & American Life, 15 Sept. [verified 10 Oct 2004] http://188.8.131.52/pdfs/PIP_College_Report.pdf
Koory, M. A. (2003). Differences in learning outcomes for the online and F2F versions of "An introduction to Shakespeare". Journal of Asynchronous Learning Networks, 7(2). http://www.sloan-c.org/publications/jaln/v7n2/v7n2_koory.asp
Morris, E.J.S. & Zuluaga, C.P. (2003). Educational effectiveness of 100% online IT courses. In G. Crisp. D. Thiele, I. Scholten, S. Barker and J. Baron (Eds), Interact, Integrate, Impact: Proceedings 20th ASCILITE Conference. Adelaide, 7-10 December. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/353.pdf
Veronikas, S.W. & Shaughnessy, M.F. (2004). Teaching and learning in a hybrid world: An interview with Carol Twigg. EDUCAUSE Review, 39(4), 50-62. http://www.educause.edu/apps/er/erm04/erm0443.asp
Whelan, A.M., Mansour, S. & Farmer, P. (2002). Multiple choice progress examination: Comparison of traditional and PBL-based curricula. Interim analysis. PE6. Association of Faculties of Pharmacy of Canada Annual Conference, Montreal.
Zimitat, C. & I. McAlpine. (2004). Student use of CAL and associated learning outcomes. Biochemical and Molecular Biology Education, 31, 146-150.
|Authors: Nian-Shing Chen, National SunYat Sen University, Taiwan. Email: firstname.lastname@example.org|
Craig Zimitat, Griffith Institute for Higher Education, Griffith University, Brisbane. Email: email@example.com
Please cite as: Chen, N.S. & Zimitat, C. (2004). Differences in the quality of learning outcomes in a F2F blended versus wholly online course. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 175-179). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/chen.html
© 2004 Nian-Shing Chen & Craig Zimitat
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.