Does a traditional learning environment provide better student engagement than a virtual environment?

Engagement; Virtual Learning; Traditional Learning; Human-Computer Interaction; Graduate students; Higher education.

INTRODUCTION

Our topic seeks to determine if there is a difference in the level of engagement for virtual learning versus traditional learning. Virtual learning is becoming an increasingly popular option for those seeking to pursue education. In 2016, the proportion of students who enrolled exclusively online increased by 14.7% while further increasing up to 15.2% in 2017. Student growth is also increasing for students taking both traditional and virtual classes concurrently, up 5.4% in 2017 (Ginder, 2018). While many topics related to virtual learning have been studied, not much has been researched about the differences in student engagement between virtual and traditional learning.

ABSTRACT

Virtual learning is becoming an increasingly popular option for education (Ginder, 2018). While education has traditionally required students to attend educational institutions in person, advancements in technology now allow students to learn and attend classes remotely. While there is a lot of research that has been done for virtual learning (Pellas N., 2014; Boschman F., 2017), not much has been studied about the differences in engagement levels between virtual and traditional learning environments. This study seeks to determine if traditional learning is more effective at keeping students engaged than virtual learning. We sought to accomplish this by measuring levels of engagement gained through an online survey through Qualtrics. Graduate students enrolled in Human-Computer Interaction at Iowa State University were recruited as participants for this study. Data was analyzed using a Mann-Whitney U test, which showed that traditional learning provided higher levels of engagement for students over virtual learning. This research provides further evidence that key elements of engagement exist at higher levels in traditional learning environments, which research has shown (McCarthy 1990), impacts the likelihood of student success.

Online space has become the mainstream for Computer-Supported Collaborative Learning. With more Web-based and virtual world tools, online learning has become commonly accepted by the global academic community. Although dynamics of emotions during an online learning program might be less clearly understandable than a traditional classroom environment, with present online learning success as evidence it is reasonable to presume that several socio-cognitive constructs of motivation may be contributors in students’ engagement and learning achievements in the virtual world. (Antino, 2008; Hartnett, 2012) A Comparison of Online and Traditional Computer Applications Classes was conducted in 2001 at Macon State College (Georgia) in terms of both student perceptions and student performance as measured by grade distribution. (Cooper, 2001). We are interested in understanding student engagement not just in terms of grade distribution, but in-class vs online activities and student participation.

Does a traditional learning environment provide better student engagement than a virtual environment?

With growing online education, it would be interesting to compare the user experience in both traditional and virtual environments. Our research hypothesis is that (1) traditional learning environment provides the same engagement level as virtual learning environment, or (2) traditional learning environment provides better student engagement compared to virtual learning environment. Through existing research factors, we are leaning towards the alternative hypothesis. In the hypothesis, the Independent Variable is the delivery methods, which contains two conditions: traditional learning environment and virtual learning environment.

For verifying our research hypothesis, we designed our research as a between-subjects comparison, which required statistics to provide an objective and convinced result. We planned to recruit around 30 participants. These participants were expected to have both the options of traditional and virtual learning methods, but only to choose one as their preference. Then we had two groups of participants: one group included half of the participants who prefer to take traditional learning method and the other included half of the participants who prefer virtual learning method. An online anonymous survey was the method we collected data from our participants, and a between-group statistic was the data analysis for our findings.

TeamGreen_FinalPresentation_Spring2019

METHOD

Participants

32 participants were recruited by email for the research. We randomly selected 16 male and 16 female masters students from the Human-Computer Interaction program at Iowa State University. Since HCI courses consist of both virtual and traditional delivery methods, students are exposed to both learning environments and they already have experience with the use of technology. Among the participants, 8 male and 8 female graduate students were learning in a traditional environment and 8 male and 8 female graduate students were learning in a virtual environment. The age range is 21-36, and the mean age is 28 years (Mean = 27.66, SD=3.44).

Figure 1. The structure of the research design

Design

The research was designed as a between-subjects design. Since our data for traditional environments was not normally distributed, the statistic method for data analysis applied was the Mann-Whitney U test, which is a non-parametric equivalent for independent samples t-test. The dependent variable is student engagement and our operational definition is “engagement scale”, which is perceived engagement as measured through an anonymous survey on Qualtrics under ISU. While our independent variable is delivery methods with 2 levels: traditional learning and virtual learning. Gender and age were not considered as conditions of independent variables in the research design. However, we kept the number of participants for both females and males as the same for avoiding bias from gender.

32 participants were recruited from the Human-Computer Interaction program at Iowa State University and split into 2 groups of 16, each with a balance of 8 male and 8 female. Each group was asked to finish an anonymous online survey of self-evaluation about their student engagement in their condition. An online survey was chosen because (1) data is collected directly from the user’s perspective, (2) strong privacy-protection, (3) the research data is more objective. A survey was created around 3 categories found in our research to be key elements of student engagement: Sense of community, frequency of interaction, and perceived connectedness. All 3 categories were broken down further into 3 sub-categories and questions were conceptualized around those 3 sub-categories. After the survey, our researchers analyzed the student engagement levels from the answers and scored their student engagement from 0-4 for each student. A datasheet was formed after that with each participant’s number, age, gender, the chosen delivery method, and the engagement score.

The goal of the research design and setting is to test whether there is a significant difference among students’ class engagement levels when they choose different delivery methods and if the difference exists, which method would bring a higher class engagement level.

Materials

The online survey was made available to complete online through a mobile device or computer. All the participants were recruited as volunteers with no awards or gifts. However, all the participants were told the purpose of the experiment, which was to test the learning engagement scales for improving the learning engagement in different delivery methods. Thus, the stimuli for recruiting participants was contributing to the improvement of the learning experience for graduate courses. To perform the data analysis, a Mann-Whitney U test was applied using JASP, an open-source statistical program.

Procedure

For this study, we recruited 32 participants from Human-Computer Interaction Graduate program at Iowa State University. The participants were asked to complete an engagement survey online using Qualtrics software under ISU. The participants can get access to the questionnaire through both cell phones and desktop/laptop computers. The survey evaluated each student engagement level based on the sense of community, frequency of interaction and perceived connectedness. Higher scores reflect a stronger student engagement measured by the constructs of the sense of community, frequency of interaction, and class engagement. The scale was measured between 0 and 4 (0 - strongly disagree, 1 - disagree, 2 - neither disagree or agree, 3 - agree, 4 - strongly agree). The questionnaires chosen were used in previous studies and they were validated. We made slight modifications to adapt the questions to both learning environments (traditional and virtual).

Figure 2. Dividing the participants

Evaluation criteria of the survey

Sense of Community was based on sense of belonging and emotional connection with peers (for example, Everyone appreciates and loves the class environment), satisfaction of needs and opportunities for involvement (for example, people in this class collaborate) and, support and emotional connection with the classroom peers and teachers (for example, if I need help I can speak with someone in this class). (Selection of Classroom Sense of Community Scale (SoC-C), self-report questionnaire).

Frequency of Interaction is based on intellectual/instructional interactions (for example, instructor provides meaningful feedback on my assignments), organizational/procedural interactions (for example, I feel that I am graded fairly) and, social interactions (for example, I am encouraged to participate in discussions with my classmates).

Perceived connectedness is based on behavior (for example, I devote my full attention while participating in the learning activities), emotional (for example, the instructor is cognizant of my interests and needs) and, cognitive (for example, the activities help me learn).

After organizing the results of the surveys into a spreadsheet document, it was imported into JASP and analyzed with a Mann-Whitney U test.

Figure 3. Survey Evaluation Criteria

RESULTS

For the analysis of the data collected from the online survey, assumption checks were done for Independent Samples T-Test. Shapiro-Wilk displayed a deviation from normality for traditional learning environment (p < .05). Levene’s test for equality of variances showed a value of p = .41, which validates the equality of variances (Figure 4). Because of the deviation from normality in the Shapiro-Wilk test, the authors decided to select the non-parametric Mann-Whitney U test to determine the significance of the results. In this test, the values of mean and standard deviation are not relevant, instead, medians and interquartile range are taken in consideration, the Mann-Whitney U test dependent variable is considered ordinal.

Figure 4. Assumption checks

As shown in Figure 7, a Mann-Whitney U test indicated that engagement levels were significantly greater in a traditional learning environment (Mdn = 3.5, IQR = 1.0, n1 = 16) than in a virtual learning environment (Mdn = 1, IQR = 1.0, n2 = 16); U = 241, p < .05. The median difference between the two groups found with Hodges-Lehmann estimate, a value of 2.0, rejects the null hypothesis. The rank biserial correlation (rB = .88) showed a large effect size, which validates the significant result.

Figure 5. Descriptive Statistics
Figure 6. Boxplots
Figure 7. Mann-Whitney U test

DISCUSSION

Through conducting a Mann-Whitney U test, we found the data showed significant indicators that engagement level was greater in a traditional learning environment (Mdn = 3.5, IQR = 1.0, n1 = 16) than in a virtual one (Mdn = 1, IQR = 1.0, n2 = 16); U = 241, p < .05, which gives support to our hypothesis. As what we analyzed from the 32 participants, traditional learning provides a more engaged learning environment than the virtual learning methods. This finding could contribute to educational areas, particularly for development of virtual learning in instructional design. Sense of community, frequency of interaction, and perceived connectedness are the three aspects that we can probably find advantages from traditional learning and apply to virtual learning.

There are some limitations in our research. (1) The most critical one is such as the limited participant pool (2) not considering gender and age, (3) the study was considered for a random selection from all of the graduate students in Human Computer Interaction at Iowa State University, since different courses experiences provide different engagement levels. For further research, we plan to recruit participants from multiple majors from different institutions. Moreover, we plan to include more independent variables in the future, such as the device for virtual learning method. The independent variable shall be considered under more conditions for analyzing the data, such as age and gender. Based on possible improvements, we need to consider other statistical methods that are more appropriate for the comparison among multiple groups, such as factorial ANOVA. For further development of students’ learning experience to assist the research of learning engagement, adding qualitative methods for user experience would be recommended.

CONCLUSION

Through this study we were able to understand the student engagement level of graduate students in Human Computer Interaction at Iowa State University. The survey lead towards our hypothesis, that a traditional learning environment does provide higher student engagement than a virtual learning environment. This research process helped us understand that advancement of technology does not mean traditional practices needs to be eradicated. Through this research we can also see that both virtual and traditional learning benefit students at different levels. In the future, it will be interesting to evaluate satisfaction levels of students who go through both these environments in a single course setting.

REFERENCES

  1. Artino, R., & McCoach, B. (2008). Development and initial validation of the online learning value and efficacy scale. Journal of Educational Computing Research, 38, 279–303.

  2. Boschman, F., McKenney, S., Pieters, J., & Voogt, J. (2017). Design talk in teacher teams: What happens during the collaborative design of ICT-rich material for early literacy learning?

  3. Cooper, L.W. (2001). A Comparison of Online and Traditional Computer Applications Classes. T.H.E. Journal, 28(8),. Retrieved May 9, 2019 from https://www.learntechlib.org/p/94124/.

  4. Fuller, K. A., Karunaratne, N. S., Naidu, S., Exintaris, B., Short, J. L., Wolcott, M. D., Singleton, S., & White, P. J. (2018). Classroom Engagement Measure [Database record]. Retrieved from PsycTESTS. doi: http://dx.doi.org/10.1037/t70512-000

  5. Ginder, S.A., Kelly-Reid, J.E., and Mann, F.B. (2018). Enrollment and Employees in Postsecondary Institutions, Fall 2017; and Financial Statistics and Academic Libraries, Fiscal Year 2017: First Look (Provisional Data) (NCES 2019- 021rev). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved May 5th, 2019, from http://nces.ed.gov/pubsearch.

  6. Goss-Sampson, M. (2018). Statistical Analysis in JASP: A Guide for Students. JASP 0.9.1.

  7. Gunuc, S., & Kuzu, A. (2015). Student Engagement Scale [Database record]. Retrieved from PsycTESTS. doi: http://dx.doi.org/10.1037/t45975-000

  8. Hartnett, M. (2012). Relationships between online motivation, participation, and achievement: More complex than you might think. Journal of Open, Flexible and Distance Learning, 16(1), 28–41

  9. In M. Orey, & R. M. Branch (Vol. Eds.), Educational media and technology yearbook: Vol 40, (pp. 27–52). Switzerland: Springer International Publishing.

  10. JASP Team (2018). JASP (Version 0.9.2) [Computer software].

  11. McCarthy, M. E., Pretty, G. M. H., and Catano, V. (1990, May). Psychological sense of community and student burnout. Journal of College Student Development, 31, 211 – 216.

  12. Pellas, N. (2014). The influence of computer self-efficacy, metacognitive self-regulation and self-esteem on student engagement in online learning programs: Evidence from the virtual world of Second Life. Computers in Human Behavior, 35, 157-170.

Survey

Student Engagement Scale

What questionnaires are you going to use, if any? For instance, if your participants are going to fill out the System Usability Scale, you need to describe it here and explain why you are using it. Have a separate sub-subsection for each questionnaire.

Higher scores reflect a stronger student engagement measured by the constructs of the sense of community, frequency of interaction, and class engagement.

Scales: 0 to 4

0 - strongly disagree

1 - disagree

2 - neither disagree or agree

3 - agree

4 - strongly agree

Questionnaires

The questionnaires chosen were used in previous studies and they were validated. We made slight modifications to adapt the questions to both learning environments (traditional and virtual). For the frequency of interaction, we were not able to access a previous questionnaire due to lack of permission, but we were able to create an initial structure based on previous research and create our own questions.

Sense of Community

Resource: Selection of Classroom Sense of Community Scale (SoC-C), self-report questionnaire

Sense of belonging and emotional connection with peers

  1. I feel I belong in this class.

  2. I feel I have a lot in common with my peers.

  3. Everyone appreciates and loves the class environment.

  4. This class, compared to others, has many positive aspects.

Satisfaction of needs and opportunities for involvement

  1. In this class, there is a willingness to help one another.

  2. People in this class collaborate.

Support and emotional connection with the classroom peers and teachers

  1. People in this class are an important source of moral support to me.

  2. If I need help I can speak/contact with someone in this class.

Frequency of Interaction

Intellectual/Instructional interactions

  1. The information taught is easy to understand and beneficial to my studies.

  2. Instructional materials provided are helpful in following learning activities and homework assignments.

  3. Instructor provides meaningful feedback on my assignments.

  4. The rubrics for the assignments are well defined and easy to follow.

  5. Instructor provides satisfying responses to questions asked in learning activities

Organizational/Procedural interactions

  1. My professor/TA have set sufficient office hours.

  2. I am able to interact with my professor/TA through email in case of clarifications.

  3. I feel that I am graded fairly.

  4. I feel the course follows all university regulations.

  5. Through the course syllabus, I am well informed about university policies.

Social interactions

  1. When there is open time, I am able to chat with my classmates about our interests.

  2. I am encouraged to participate in discussions with my classmates.

  3. I feel my peers are open to collaborate.

Class Engagement

Behavioral

  1. I devote my full attention while participating in the learning activities
    (Resource: School Engagement Measure, SEM)

  2. I complete my assessments on time
    (Resource: Student Engagement Scale, SES)

Emotional

  1. I enjoy learning new things
    (Resource: Engagement vs Disaffection, EvsD)

  2. The instructor is cognizant of my interests and needs
    (Resource: Student Engagement Scale, SES)

Cognitive

  1. The activities help me learn
    (Resource: Motivated Strategies for Learning, MSLQ)

  2. I have tried a new approach or way of thinking about the content

(Resource: Classroom Engagement Measure, Psyc TESTS)