Online Course Research

This past week, we read a series of papers on analyzing the effectiveness of online courses. Sarah Haavind sums it up best, "Rarely is it possible to compare circumstances where variables such as class size, student background, curriculum and even the level of enthusiasm of the instructor are effectively controlled." The individual studies themselves are not irrefutable. Studies of classes are sociological in nature and the research methods are not purely scientific. This leaves the conclusions open to interpretation and makes the results from different studies difficult to compare. These issues are especially pronounced in meta-analysis (studies of other studies). I would rather see the research focus on identifying students that will excel in online courses and how to make online courses more effective.

Issues raised in the studies:

• it is difficult and time-consuming to codify and quantify comments from discussion boards.
• differences between instructor experience, training, and teaching approaches (e.g., "craft-oriented practices" (Saba, 2001)
  
• variations among student achievement and characteristics
• normal "ebb and flow" class variations
• "Courses are participatory experiences shaped by the learners themselves, making each individual offering a unique entity" (Sener - pg 1). Examples include media attribute theory, social presence theory, content analysis, systems dynamics and discourse analysis
  
• studies include different subjects
  
• varying student demographic data and prior knowledge
• varying student technical skills
• difficult and impractical to assign learners randomly to classes for studies (Sener - pg 1)
  
• varying student and teacher knowledge of collaboration techniques
• varying teacher knowledge of online moderation techniques
• varying perceptions about online and f2f learning
  
• varying assessments. The Rice report (pg 432) cites Bernard et al. (2004) who found that studies that involved researcher-made tests favored distance learning over face-to-face, while studies using teacher-made tests favored face-to-face classrooms over distance learning.
• "various alternative approaches to evaluating online learning within its own frame of reference" (Sener - pg 3)
• studies use different technology tools and online pedagogies
• the technology tools and pedagogy may shift over time and between studies
  
• varying degrees of learner support
  
• what was the social environment like for students (this may not be visible by the teacher)
• varying degrees of course material quality

Meta-Analysis

The Effects of Distance Education on K-12 Student Outcomes and Evaluation of Evidence-Based Practices in Online Learning are meta-analysis comparing online learning and traditional face-to-face (f2f) students. The reports show that online learning is at least as good as f2f learning. In some cases, online is more effective. At the same time, I am wary of this type of analysis. I find it useful as an approximation, but it lacks sufficient detail to provide an irrefutable conclusion. Many of the studies involved students from a wide range of grade levels, including K-12, undergrads, and post-grads. The studies also included people who were not students at all - professionals in the workforce. The studies needed the age ranges for a reasonable sample size. As the same time, I don't find much logical in mixing such a broad range of learners. Some of the studies were older and prior to the advent of web2.0 tools. I could argu that older studies are an even more positive for online learning because the tools were cruder and teachers had less knowledge of what to do. There is concern that there may be survivor bias because students that completed the online course were motivated and performed well.

Two algebra studies were also positive for online education and were more thorough than the meta-analysis mentioned above. Algebra Achievement in Virtual and Traditional Schools and Effectiveness of Online Algebra Learning found that, "Online students consistently outperformed traditional students across the AAU subscales". These studies were recent (2007) used 85 traditional and 25 online students in three schools. I have no doubt that online will continue to evolve faster than traditional curriculum and pedagogy. Online courses have not yet begun to leverage social networking tools. In addition, we don't have enough data to It is likely that not all subject and grade levels will be appropriate for online instruction. Hopefully, there will be a new round of research that identifies which subjects are the best candidates for online courses. There also needs to be more research on what makes an online course successful.

Project Reflection

I enjoyed working on the project with my partner developing a sample online physics course. There is a broad range of rich multimedia content for our topic that included animations, videos, documents and simulations. We started using a wiki to brainstorm about the project. After a few days of independent brainstorming, we collaborated synchronously using Skype. Wikis can only be edited by one person at a time. This did not pose a problem for us, since there was only two of us. If there were more members of the team, then we could have looked at Google Docs, Etherpad, or some other multi-user word processor.

During our first Skype call, we edited the wiki to convert the brainstorming into a "to do" list and we discussed who was responsible for each item. As we worked on the project, we added and subtracted from the "to do" list. Sometimes, we would add an item without who would be responsible. It was a friendly way to ask the other person to do a task without being bossy. We continued the cycle of independent work culminating with a synchronous call. I liked the independent work because I looked forward to seeing what my partner added on a daily basis. We would also leave comments and questions directly in the lesson wiki. In the end, we had four Skype calls with a total time of 9.75 hours. I'm not sure that I would have been anymore productive in a f2f setting.

Week 7 Reflections

What did you think about, or worry about, as you began to grapple with creating a unit for an online course?

I am more enthusiastic than worried about the project. The partner finding process was unstructured, but I found a partner with the same interests. We have already Skyped and started a wiki for collaboration. My only concern is addressing the breadth of best practices for online teaching. We have read a great deal on the attributes of what makes an online course engaging - frequent communication, structured interactions, teacher online training, and student online tool training. We will not have an opportunity to actually use the course, so we will not get feedback to improve the course. Lastly, if I were doing this for real, I would work with a small group of students to design it and try ideas.

Week 6 Reflections

Has your view of the schools you chose to research changed from the first week you looked at them? If so, how?

My views of Virtual Virginia (link, wiki) and Pennsylvania Virtual Charter School (link, wiki) have been consistent. They are medium-quality schools (compared to other wiki entries) and suffer from a dearth of available information. VV and PVCS both use asynchronous learning, and teachers assist students in a supportive role. Both schools use curriculum developed by third-parties and the sample courses were disappointing. I believe that schools need to develop their own online curriculum and have teachers assist the instructional designers to continually improve them. Technology is evolving rapidly and courses need to adapt to use the most popular and productive social networking technologies. The class forum discussions this week highlighted that managing online discussions is tricky. Online discussions are challenging to organize well, maintain organization, keep on track, and grade. This makes me worry about the teachers at VV and PVCS with respect to how well they are trained to manage their online classrooms - even in a supportive role. VV was more encouraging than PVCS with professional development. Before teaching an online class, teachers take a seven-week online training course, and a week-long f2f workshop.

Miscellaneous
I have a wish list of two items after this week. The first is that there is a new generation of open source discussion software that is highly configurable. The second is that there are better online courses and that the first few lessons can be taken for free. I think about Amazon.com, which lets people preview the table of contents and some of the book before purchase.

Week 5 Reflections

I enjoyed taking the TPI assessment – I don’t think teachers are introspective enough about their own teaching strengths and weaknesses. I don’t mind taking a survey or two to force the issue. If the role of education includes teaching citizenship, then we are missing the boat. I chatted with the TPI creator (John Collins) and he said that less than 3% of the 100,000+ teachers who took the test were Social Reform dominant (including group 2). We have non-academic advisory and life skills at my school, but it has a fraction of time compared to major subjects.

Evaluating online teachers is different than evaluating teachers (f2f) in a brick-and-mortar environment, although it may be more thorough. F2f teachers are generally evaluated qualitatively through direct observation. Sometimes students surveys are conducted too. Online teachers operate in a highly transparent environment. There is lots of data on interactions with students. Data can be found in course management system logs, e-mail and discussion board logs, phone records, and shadow students. Other industries have long used data to improve their product/service and become more efficient. I am optimistic that this data will be beneficial to both students and teachers. Given the digital nature of the online learning environment and interactions, I am hopeful that schools can start to mke a direct connection between what teachers do and student achievement. New courses and teaching techniques could be tested to see if they improve student achievment. NACOL provides a comprehensive template for creating an evaluation of online teachers. It is a good starting point, although some of the questions are too specific to a technology (rather than a skill) and some questions don’t apply if teachers don’t have control over the curriculum.

I believe that professional development is an important component of great schools (online or f2f). Online school teachers have potentially more time to collaborate on students, pedagogy, and curriculum. By contrast, f2f teachers usually cite lack of time and their number one job dissatisfaction complaint. This collaboration can occur when teachers are geographically dispersed. If the teachers are physically located in the same place, then there is even greater collaboration potential. And, the possibility to build a common culture and teaching philosophy. Although culture seems like a minor intangible, the most successful businesses point to a common mission and culture as keys to their success.

Thank you to Alejandro Heyworth (link) for a post that stirred me to write a comment that lasted several paragraphs. One of Alejandro’s many excellent points was about teachers being actors if they had no involvement in the curriculum design. I think that online schools should have their own instructional designers and that teachers should be required to collaborate with them to make the courses better.

I continue to be very optimistic about the future of online, but the notion of a hybrid schools intrigues me more. Teacher evaluations should be based on observation of teaching skills, curriculum design (or collaboration with designers), responsiveness to students, commitment to continually improve (and help others), and student surveys.

Random Thoughts

1. I wonder if f2f instruction should be a prerequisite for online instruction?
2. At what age has a child socialize enough to be able to handle online interaction productively?

Week 4 Reflections

This week I learned that it is difficult to compare online schools. It is even more complicated to compare state-run schools and charter or for-profit schools. In all cases, the websites lack complete information. The information that is available is related to how they are marketing the school. It seems like the online curriculum and related tools should be a major factor in selecting a school. Yet, in many cases, there was not even a single lesson to demo. The charter and for-profit schools seem to be a little more responsive to potential students. Virtual Virginia (VV) has little information other than a course directory and some faculty information. Pennsylvania Virtual Charter School (PVCS) discussed their instructional strategy that includes the parents and socializing with other online students. The only area where I felt VV had an advantage over PVCS was in the online curriculum. PVCS uses courses from K12 Inc, which I felt were very lightweight. VV designs their own and uses some third-party providers (although I was not able to review them).

What did you learn this week that struck you as particularly important or interesting? I was disappointed at the general level of online course sophistication. The vast majority of online courses were no more than online versions of traditional textbooks. I would have expected more progressive pedagogy given the innovation occurring in the area of Web2.0 tools.

Has your thinking changed as a result of what you learned this week? I believe that online classes hold great promise for individualization, investigation-based learning, and access to higher quality teachers. In addition to teachers, students can also gain access to experts. For example, it might be impractical for a school to have a doctor on staff, but an online doctor could be available for a few hours a week. The doctor could make teaching his full-time job and make enough money by working with several schools. As I mentioned above, the current generation of online schools are teaching 19th (or 18th) century skills instead of innovating to deliver 21st century skills.