Conference logo
[ ASCILITE ] [ 2004 Proceedings Contents ]

A tale of two cities: A study on the satisfaction of asynchronous e-learning systems in two Australian universities

Nadira Hisham, Paul Campton
School of Information Systems
University of Tasmania

Des FitzGerald
School of Mathematics and Physics
University of Tasmania
The aim of this paper is to present a critical analysis of the satisfaction of first year Information Systems students with asynchronous e-learning systems at the University of Tasmania and the University of Adelaide. It measures whether factors such as content, learner interface, feedback and assessment, personalisation, learning community and access affect students' satisfaction with asynchronous e-learning systems. This paper also compares whether there is any difference in satisfaction with the different systems between the two universities. It found that there is no difference in satisfaction between the different asynchronous e-learning systems used in the two universities but found that factors such as content, personalised feedback, interface and learning community were significantly related to students' satisfaction with asynchronous e-learning systems. These findings may influence how lecturers and designers utilise asynchronous e-learning systems for first year University students. Well structured quality content that is presented in an easy to understand format along with receiving personalised feedback on their progress are important elements of effective e-learning. Coupled with these factors is the need to learn in a community and the ability to select resources from the asynchronous e-learning system to suit their personal needs.


Background

There are many definitions for what constitutes e-learning. Jackson (cited in Parker, 2002) defines e-learning as learning which is technology delivered or technology enhanced. Heppell (2003) takes a student centred approach to e-learning with a focus on the enabling nature of the technology. Stager (2004) asserts that computers offer "learners a rich intellectual laboratory and vehicle for self expression. Although computing has transformed nearly every aspect of society, schools remain relatively untouched". Much has been made of the potential of using the Internet and other electronic means for enabling learning. John Chambers, CEO of CISCO, (cited in Friedman, 1999), said, "The next big killer application for the Internet is going to be education. Education over the Internet is going to be so big it is going to make email usage look like a rounding error in terms of the Internet capacity it will consume." The rapid development of Internet technology has caused a radical change in the way learning is occurring. Many other benefits have been associated with e-learning. These include the ability to cater for individual needs, greater leverage in terms of resources and the ability to create collaborative leaning environments and communities (Zhang & Nunmakeern, 2003: Salmon, 2002a).

The University sector is now embracing e-learning not only for economic reasons but also for the flexibility and opportunities it provides (DEST, 2002). There exist a number of e-learning systems that are commonly employed within Universities; these include WebCT, BlackBoard and numerous in-house systems. Within these e-learning systems there are a number of different tools, including discussion boards, email, slideshows, streaming audio and video (Ashley 2003). Communication tools form an integral part of an e-learning system as they provide a means for communication and learning to occur between students as well as between students and lecturer. Feedback is integral to the learning and assessment process.

E-learning systems can either operate in an asynchronous or synchronous mode. Essentially the asynchronous mode is where communication, collaboration and learning can occur in "different time - different place" manner and where users can select when they wish to communicate (Ashley, 2003). This may be very useful where lecturers need to manage large numbers of students. University students face a number of issues (Schrum & Hong, 2002), such as balancing the competing demands of work, family and study. The ability to access and communicate in asynchronous mode can meet many of their needs of a "just for me" learning environment (Hisham, 2004; Rogers, 2000). The synchronous mode allows people to interact with each other in a "same time - different place" manner.

As with traditional paper/print based learning there are a number of factors to be considered in developing asynchronous learning systems. These include learning styles, communication and feedback, and access. Indeed e-learning is often being used in a blended manner (Valiathan, 2002; Wenting et al., 2000) to complement traditional methods of delivery. Successful systems often require a multifaceted approach to satisfy students' requirements.

The need to measure satisfaction is critical in order to evaluate whether the systems that are currently being employed actually meet the users' needs. Whilst attention has been paid to designing and evaluating e-learning systems (Lambert, 2003; Trinidad, 2004) there has been limited research into University students' satisfaction with asynchronous e-learning systems in Australia. There exist a number of instruments for measuring user satisfaction with Information and Communication Systems. Wang (2003) undertook an international study, which specifically examined students' satisfaction with asynchronous e-learning systems. He sought to examine whether five factors - content, learner interface, learning performance (feedback and assessment), personalisation and learning community influence user satisfaction. He found that all of these factors apart from learning performance were related to student satisfaction. Content and learner interface remains key factors for determining satisfaction. Swan (2001) reported about the relationship of course structure to students' satisfaction. It is important to have a suitable learner interface that meets their needs when considering a learning system. Material must be presented in a way that is user friendly. Sharing and learning from others, whether it be other students or the lecturer in a learning community is important. Much has been made in the literature of the constructivist nature of learning and the need for student to interact in an enriching manner (Salmon, 2002b). Constructivism is where students learn by constructing new concepts in an active manner (Gery, 2002).

Whilst Wang (2003) presented five variables that could influence learner satisfaction, he did not specifically include access, which is of considerable importance to students (Johnson & Rupert 2002). Wang's instrument was based on previous research undertaken by Doll & Torkzadek (1988). This study seeks to build on the research undertaken by Wang (2003). Additionally, access will be considered, in this study, as a possible sixth factor that could affect student satisfaction.

Given the importance of e-learning to Australian universities and the value of understanding the manner in which it is being used, we are faced with the need to measure whether students are satisfied with e-learning systems. This study aims to evaluate six factors that may affect the level of satisfaction with asynchronous e-learning systems for first year Information Systems students within the Commerce faculty at the University of Tasmania and the University of Adelaide, as well as any differences in student satisfaction between the different e-learning systems used in the two universities.

Methodology and data collection

This study will be objective in nature, taking a positivist epistemological position, and a quantitative approach to studying students' satisfaction with e-learning systems. The population surveyed were students from two first year Information Systems units, one at the University of Tasmania and the other at the University of Adelaide for Semester 1, 2004. The students at the University of Tasmania used WebCT and the students at the University of Adelaide used MyUni (BlackBoard). The total population sizes for the two groups were 490 and 453 respectively. A web based questionnaire hosted at the University of Tasmania with an invitation sent electronically through the course unit co-ordinator. The questionnaire was adapted from Wang (2003) with an additional section on access. The questionnaire contained a section on demographic data as well as sections on content, learner interface, feedback and assessment, personalisation, learning community and access. There was also a section on overall satisfaction. The reader is referred to Appendix A for the questions that were considered in this study. A 7 point Likert scale was used for items other than the demographic data. The questionnaire was piloted on a group of 21 students in order to test for reliability, and minor modifications were made. The students piloted were enrolled in Business Information Systems at the University of Tasmania and the results were not included in the data set. Validity was considered by Wang (2003; p. 80) and this research relies on his instrument's validity.

A number of hypotheses were considered initially.

H01:Students' satisfaction with asynchronous e-learning systems is not related to the content provided through the e-learning systems.
H02:Students' satisfaction with asynchronous e-learning systems is not related to the learner interface displayed by the e-learning Systems.
H03:Students' satisfaction with asynchronous e-learning systems is not related to the feedback and assessment provided through the e-learning Systems.
H04:Students' satisfaction with asynchronous e-learning systems is not related to the personalisation option provided through the e-learning Systems.
H05:Students' satisfaction with asynchronous e-learning systems is not related to the learning community provided by the e-learning System.
H06:Students' satisfaction with asynchronous e-learning systems is not related to gaining access to the e-learning system.
H07:There is no difference in students' satisfaction with asynchronous e-learning systems between the two universities.

The researchers also included this seventh hypothesis to investigate if there is a difference in the students' satisfaction level between the two universities participating in this research.

Findings and discussion

There were 82 responses from the University of Tasmania and 55 from the University of Adelaide. The response rate was 14.5%. The data were cleaned using the options available in SPSS.

Most students had attended university for less than one year (80%), were aged 18-25 years (80%), were currently using e-learning in their first year units (96%) and considered themselves to be intermediate users (71%).

Responses were obtained in two waves or groups, early and late responders, which permitted an assessment of any systematic differences associated with time of response (early or late). The Mann-Whitney Test (Table 1) considered University, Study Duration, Age and Computer Experience. Given the P-values were 0.075, 0.324, 0.537, 0.317 respectively, and were all greater than 0.05 we can reasonably conclude that these variables are not affected by time of response, which gives us some confidence that non-response is not an important source of bias

Table 1: Mann-Whitney test for non-response bias (N=137)


UniversityStudy durationAgeComp. exp. level
Mann-Whitney U1975.0002164.5002224.5002140.000
Z-1.783-.986-.618-1.001
Asymp. Sig. (2-tailed).075.324.537.317

Given there were 137 respondents, this enabled a factor analysis to be performed (Foster, 2001). A factor analysis is an exploratory technique that reduces a large number of variables to the underlying component factors (Coakes, 2003). An extraction process using SPSS was undertaken, with Principal Component Analysis using Varimax Rotation Method with Kaiser Normalization. The rotation converged in 6 iterations with exclusion of single-item factors. The 28 items were reduced to 26 items (Table 3), which were then grouped into component factors identified as personalised feedback, learning community, learner interface, content and access. Refer to Appendix A for the specific questions in each factor.

Cronbach's Alpha was used to test for reliability of the items. Reliability is used to ensure the consistency of the results for the various items being tested within each component (Foster, 2001). The values are reported in Table 2. As the values were greater than 0.7, the data could be retained and undergo further analysis. It should be noted that the factors for Personalisation and Feedback were shown by the Factor Analysis to be contained within the same component, Personalised Feedback. Thus, the hypotheses H03 and H04 were combined to form a new hypothesis:

H03:Students' satisfaction with asynchronous e-learning systems is not related to the personalised feedback provided through the e-learning systems.

Table 2: Cronbach's alpha reliability coefficients

VariablesN of itemsCronbach's Alpha
Personalised Feedback110.943
Satisfaction30.919
Learner Interface50.911
Learning Community40.894
Content30.820
Access30.718

Table 3: Factor analysis for principal components


Component
Personalised
feedback
Learning
community
Learner
interface
ContentAccess
FACTOR306.811



FACTOR304.754



FACTOR303.743



FACTOR405.741



FACTOR404.680



FACTOR302.669



FACTOR305.640



FACTOR401.614



FACTOR403.592



FACTOR402.547



FACTOR301.520



FACTOR502
.827


FACTOR503
.810


FACTOR504
.772


FACTOR501
.762


FACTOR203

.813

FACTOR205

.755

FACTOR204

.696

FACTOR202

.564

FACTOR201

.536

FACTOR102


.761
FACTOR101


.730
FACTOR103


.633
FACTOR61



.880
FACTOR62



.786
FACTOR63



.644

Normality was not established and indeed as the variables were measured on an ordinal scale non-parametric testing was necessary (Foster, 2001; p. 7). Non-parametric testing involving a bivarate correlation using Spearman's Rho correlation was employed to test hypotheses H01 to H06 (excluding H04). The values are reported below in Table 4.

From Table 4, the P-values for association of satisfaction with (respectively) content, learner interface, personalised feedback, learning community and access were 0.00, 0.00, 0.00, 0.00 and 0.016. All P-values were less than 0.05 so we can reject all the hypotheses. We conclude there is a relation between each of the factors (content, learner interface, personalised feedback, learning community and access) and the criterion of students' satisfaction with asynchronous e-learning systems. Clearly, personalised feedback is of major concern to students, followed by content, learner interface and learning community. The means for achieving personalised feedback with existing e-learning systems is limited, as the assessment criteria cannot be linked across tasks. Given the importance of reflection in the learning process (Stansfield et al., 2004) it is important to have systems that provide this level of feedback.

Table 4: Summary of Spearman's rho correlation test for the five hypotheses

Null
hypotheses
Independent
variables
NDirectionSpearman's rho correlationResult
Correlation
coefficient
Significance
(2-tailed)
H01Content122Positive0.698**0.000Reject
H02Learner
Interface
121Positive0.689**0.000Reject
H03Personalised
Feedback
91Positive0.715**0.000Reject
H04Note: Personalisation and Feedback combined as a result of the factor analysis
H05Learning
Community
103Positive0.652**0.000Reject
H06Access73Positive0.282*0.016Reject
** Correlation is significant at the 0.01 level (2-tailed)
* Correlation is significant at the 0.05 level (2-tailed)

We employed the Mann-Whitney test to examine the seventh hypothesis on the difference in students' satisfaction between the two universities. As the value, 0.466 was greater than 0.05 we cannot reject H07. The Null Hypothesis, H07, was accepted and hence we can say there is no evidence of difference in satisfaction between the asynchronous e-learning systems in the University of Tasmania and the University of Adelaide. It would appear that the issue is, not what propriety asynchronous e-learning systems are employed, but how it is employed. The issues of having a well-designed interface, the capacity for personalised feedback and appropriate content are more important than what system is employed.

Table 5: Mann-Whitney U test
Differences in the students' satisfaction between the two universities (N=126)

TestSatisfaction
Mann-Whitney U1756.000
Z-.730
Asymp. Sig. (2-tailed).466

Conclusion

From the research presented in this paper and those of Wang (2003) it is clear that there exist similar factors, which can affect students' satisfaction. However, Wang found that learning performance (feedback and assessment) was not related to users' satisfaction. This study found personalised feedback to be an important factor. Students require feedback on their progress (Soon et al., 2000; Thurmond et al., 2002) in order for learning to be successful. Personalised feedback to students is a critical component of a successful asynchronous e-learning system. There exists some practical ways of providing feedback to students using existing office automation tools (Warner, 2004). More research and development needs to be undertaken in order to allow existing Learning Content Management Systems, such as WebCT or BlackBoard, to incorporate rubrics or at least automated processes that allow detailed feedback on the criteria being assessed and for these criteria to be measurable across learning tasks. Currently, there is little capacity with existing asynchronous e-learning systems to give detailed feedback on a range of criteria across multiple tasks (Campton, 2003).

Students surveyed in this research had spent less than one year at university, were aged 18-25 years and hence relatively new to a University learning environment. Students require a supportive learning community in order to be satisfied with their asynchronous e-learning environment. There exists a need, supported by the research findings, to use tools such as discussion boards and virtual groups in order to establish a learning community. Asynchronous e-learning systems need to provide a suitable interface for students and as well as provide students with the opportunity to easily access the content at a time and place that meets their individual learning needs. Whilst access had a low correlation this might be taken to mean that many issues with access and bandwidth have been resolved, rather than access' lack of significance. From this research there appears little evidence to suggest one propriety asynchronous e-learning system is better than another.

The importance of asynchronous e-learning systems to universities is well established. The question is not if, but how asynchronous e-learning systems should be used. The findings are significant, in that they provide lecturers with clear directions on what needs to be undertaken when, in order to satisfy students' needs. On the basis of this research, it is clear that further work needs to be undertaken to develop asynchronous e-learning systems that provide for personalised feedback through automation tools or rubrics that cater for the assessment criteria across tasks.

References

Ashley, J. (2003). Synchronous and asynchronous communication tools. Executive Update. The Center for Association Leadership, Washington DC. [viewed 23 Mar 2004, verified 17 Oct 2004] http://www.centeronline.org/knowledge/article.cfm?ID=2587

Campton, P. (2003). E-learning-'Trick or treat'? Using technology for teaching and learning in a tertiary setting. Proceedings 20th ASCILITE Conference. Adelaide, December 7-10. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/585.pdf

Coakes, S. & Steed, L. (2003). SPSS: Analysis without Anguish. Queensland: Wiley.

Department of Education, Science and Training (2002). Universities Online: A survey of online education and services in Australia. Occasional Paper Series, J. S. McMillan Printing Group, Canberra. http://www.dest.gov.au/highered/occpaper/02a/02_a.pdf

Doll, W.J. & Torkzadeh, G. (1988). The measurement of end-user computing satisfaction. MIS Quarterly, 12(2), 259-274.

Friedman, T.L (1999). Foreign Affairs: Next, It's E-ducation. The New York Times, 17 November 1999. Late Edition, Section A, p. 25:col.5.

Foster, J. (2001). Data analysis using SPSS for Windows version 8 to 10: A beginner's guide. London: Sage Publications.

Gery, G. (2002). Task support, reference, instruction, or collaboration? Factors in determining electronic learning and support options. Technical Communication, 49(4), 420-427.

Heppell, S. (2003). The Future of E-Learning. Connected, 6. [viewed 23 July 2003, verified 17 Oct 2004] http://www.ngflscotland.gov.uk/connected/connected6/E-Learning/FutureELearning.asp

Hisham, N., (2004). Measuring First Year Information Systems Students' Satisfaction with Asynchronous E-learning Systems in Three Universities within Australia. Unpublished Masters Thesis, University of Tasmania, Hobart.

Jackson, R.H. (2001). Defining e-Learning - Different Shades of "Online". Web Based Learning Resources Library. http://www.outreach.utk.edu/weblearning/ [viewed 2 Jul 2004, not found 17 Oct 2004]

Johnson, A. & Ruppert, S. (2002). An evaluation of accessibility in online learning management. Library Hi Tech, 20(4), 441-451.

Lambert, S. (2003). Learning Design at the University of Wollongong. Proceedings 20th ASCILITE Conference. Adelaide, 7-10 December. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/643.pdf

Parker, M. B (2002). Three Pillars of Technology-enhanced e-Learning. Proceedings 4th Annual World Wide Web Conference, University of Stellenbosch. 4-6 September. [viewed 17 July 2004, verified 17 Oct 2004] http://general.rau.ac.za/infosci/www2002/Full_Papers/Parker%20M/Parker_TechEnhE-learning.pdf

Rosenberg, M.J. (2001). E-learning: Strategies for delivering knowledge in the digital age. McGraw-Hill, USA.

Rogers, J. (2000). Communities in practice: A framework for fostering coherence in virtual communities. Educational Technology and Society, 3(2). http://ifets.ieee.org/periodical/vol_3_2000/e01.html

Salmon, G. (2002a). Computer mediated conferencing for management learning at the Open University. Management Learning, 31(4), 491.

Salmon, G. (2002b). Learning from the future. Proceedings of the Annual Conference of the Australian Computers in Education Conference (ACEC). Hobart, 11-13 July. [verified 17 Oct 2004] http://www.pa.ash.org.au/acec2002/uploads/documents/store/conferences/conf_185_gilly_salmon.pdf

Schrum, L. & Hong, S. (2002). Dimensions and strategies for online success: Voices from experienced educators. Journal of Asynchronous Learning Networks, 6(1), 57-67. http://www.sloan-c.org/publications/jaln/v6n1/v6n1_schrum.asp

Soon, K.H., Sook, K.I., Jung, C.W. & Im, K.M. (2000). The effects of Internet-based distance learning in nursing. Computing in Nursing, 18(1), 19-25.

Stager, G. (2004). The Case for Computing. [viewed 15 July 2004, verified 17 Oct 2004] http://stager.org/articles/thecaseforcomputing.html

Stansfield, M., McLellan, E. & Connolly, T. (2004). Enhancing student performance in online learning and traditional face-to-face class delivery. Journal of Information Technology Education, 3, 173-188.

Swan, K. (2001). Virtual interaction: Design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22(2), 306-331.

Thurmond, V., Wambach, K., Connors, H.R. & Frey, B.B. (2002). Evaluation of student satisfaction: Determining the impact of a web-based environment by controlling for student characteristics. The American Journal of Distance Education, 16, 169-190.

Trinidad, S. (2004). Helping teachers successfully use e-learning environments. Proceedings Australian Computers in Education Conference (ACEC). Adelaide, 5-8 July. http://www.acec2004.info/uploads/documents/store/conferences/conf_P_78_finalacec2004-oles.doc

Valiathan, P. (2002). Blended learning models. Learning Circuits. ASTD. [viewed 24 April 2004, verified 17 Oct 2004] http://learningcircuits.org/2002/aug2002/valiathan.html

Warner, D. (2004). Assignment management: A bottom-up approach. Proceeding Australian Computers in Education Conference (ACEC). Adelaide, 5-8 July. [verified 17 Oct 2004] http://www.acec2004.info/uploads/documents/store/conferences/conf_P_43_assignmentmanagement2004-02-19.doc

Wentling, T.L., Waight, C., Gallaher, J., La Fleur, J., Wang, C. & Kanfer, A. (2000). E-learning - A review of literature. http://learning.ncsa.uiuc.edu/papers/elearnlit.pdf [11 Mar 2004, verified 17 Oct 2004].

Appendix A: Questionnaire

Demographic data (not listed)

Content (factor1)

101.   The e-learning system provides content that exactly fits my needs.
102.   The e-learning system provides useful content.
103.   The e-learning system provides sufficient content.
104.   The e-learning system provides up-to-date content.

Learner interface (factor 2)

201.   The e-learning system is easy to use.
202.   The e-learning system makes it easy for me to find content I need.
203.   The content provided by the e-learning system is stable.
204.   The e-learning system is user-friendly.
205.   The operation of the e-learning system is stable.

Feedback and assessment (factor 3)

301.   The e-learning system responds to my requests fast enough.
302.   The e-learning system makes it easy for me to evaluate my learning performance.
303.   The testing methods provided by the e-learning system are easy to understand.
304.   The testing methods provided by the e-learning system are fair.
305.   The e-learning system provides secure testing environments.
306.   The e-learning system provides testing results promptly.

Personalization (factor 4)

401.    The e-learning system enables me to control my learning progress.
402.    The e-learning system enables me to learn the content I need.
403.    The e-learning system enables me to choose what I want to learn.
404.    The e-learning system records my learning progress.
405.    The e-learning system records my learning performance.
406.    The e-learning system provides the personalized learning support.

Learning community (factor 5)

501.   The e-learning system makes it easy for me to discuss questions with my lecturers and/or tutors.
502.   The e-learning system makes it easy for me to discuss questions with other students.
503.   The e-learning system makes it easy for me to share what I learn with the learning community.
504.   The e-learning system makes it easy for me to access the shared content from the learning community.

Access (factor 6)

601.   The e-learning system is difficult to access.
602.   The speed of access to the e-learning system is slow when accessed from the university.
603.   The speed of access to the e-learning system is slow when accessed from home.

Overall satisfaction

701.   As a whole, I am satisfied with the e-learning system.
702.   As a whole, the e-learning system is successful.
703.   Using e-learning systems to enhance my educational experience is valuable.

Authors: Nadira Hisham, School of Information Systems, University of Tasmania, Australia. nhisham@utas.edu.au Paul Campton, School of Information Systems, University of Tasmania, Australia. Paul.Campton@utas.edu.au Des FitzGerald, School of Mathematics & Physics, University of Tasmania, Australia. D.FitzGerald@utas.edu.au

Please cite as: Hisham, N. Campton, P. & FitzGerald, F. (2004). A tale of two cities: A study on the satisfaction of asynchronous e-learning systems in two Australian universities. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 395-402). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/hisham.html

Print version errata: Tables 3 to 6 in the print version should be numbered Tables 2 to 5.

© 2004 Nadira Hisham, Paul Campton & Des FitzGerald
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.


[ ASCILITE ] [ 2004 Proceedings Contents ]
This URL: http://www.ascilite.org.au/conferences/perth04/procs/hisham.html
HTML created 27 Nov 2004. Last revision: 27 Nov 2004.