Online Course Research

This past week, we read a series of papers on analyzing the effectiveness of online courses. Sarah Haavind sums it up best, "Rarely is it possible to compare circumstances where variables such as class size, student background, curriculum and even the level of enthusiasm of the instructor are effectively controlled." The individual studies themselves are not irrefutable. Studies of classes are sociological in nature and the research methods are not purely scientific. This leaves the conclusions open to interpretation and makes the results from different studies difficult to compare. These issues are especially pronounced in meta-analysis (studies of other studies). I would rather see the research focus on identifying students that will excel in online courses and how to make online courses more effective.

Issues raised in the studies:

• it is difficult and time-consuming to codify and quantify comments from discussion boards.
• differences between instructor experience, training, and teaching approaches (e.g., "craft-oriented practices" (Saba, 2001)
  
• variations among student achievement and characteristics
• normal "ebb and flow" class variations
• "Courses are participatory experiences shaped by the learners themselves, making each individual offering a unique entity" (Sener - pg 1). Examples include media attribute theory, social presence theory, content analysis, systems dynamics and discourse analysis
  
• studies include different subjects
  
• varying student demographic data and prior knowledge
• varying student technical skills
• difficult and impractical to assign learners randomly to classes for studies (Sener - pg 1)
  
• varying student and teacher knowledge of collaboration techniques
• varying teacher knowledge of online moderation techniques
• varying perceptions about online and f2f learning
  
• varying assessments. The Rice report (pg 432) cites Bernard et al. (2004) who found that studies that involved researcher-made tests favored distance learning over face-to-face, while studies using teacher-made tests favored face-to-face classrooms over distance learning.
• "various alternative approaches to evaluating online learning within its own frame of reference" (Sener - pg 3)
• studies use different technology tools and online pedagogies
• the technology tools and pedagogy may shift over time and between studies
  
• varying degrees of learner support
  
• what was the social environment like for students (this may not be visible by the teacher)
• varying degrees of course material quality

Meta-Analysis

The Effects of Distance Education on K-12 Student Outcomes and Evaluation of Evidence-Based Practices in Online Learning are meta-analysis comparing online learning and traditional face-to-face (f2f) students. The reports show that online learning is at least as good as f2f learning. In some cases, online is more effective. At the same time, I am wary of this type of analysis. I find it useful as an approximation, but it lacks sufficient detail to provide an irrefutable conclusion. Many of the studies involved students from a wide range of grade levels, including K-12, undergrads, and post-grads. The studies also included people who were not students at all - professionals in the workforce. The studies needed the age ranges for a reasonable sample size. As the same time, I don't find much logical in mixing such a broad range of learners. Some of the studies were older and prior to the advent of web2.0 tools. I could argu that older studies are an even more positive for online learning because the tools were cruder and teachers had less knowledge of what to do. There is concern that there may be survivor bias because students that completed the online course were motivated and performed well.

Two algebra studies were also positive for online education and were more thorough than the meta-analysis mentioned above. Algebra Achievement in Virtual and Traditional Schools and Effectiveness of Online Algebra Learning found that, "Online students consistently outperformed traditional students across the AAU subscales". These studies were recent (2007) used 85 traditional and 25 online students in three schools. I have no doubt that online will continue to evolve faster than traditional curriculum and pedagogy. Online courses have not yet begun to leverage social networking tools. In addition, we don't have enough data to It is likely that not all subject and grade levels will be appropriate for online instruction. Hopefully, there will be a new round of research that identifies which subjects are the best candidates for online courses. There also needs to be more research on what makes an online course successful.