Right now I’m supposed to be writing an article about evaluating e-learning. It is the only assignment in a seminar course on e-learning evaluation. The spring term ends on June 18th, and with barely over two weeks before my deadline and all I’ve produced is an outline with which I am incredibly dissatisfied. I could blame writer’s block. I could claim that my brain hasn’t yet returned from DisneyWorld, where I celebrated my birthday last week. I could point fingers at my 60+ hour a week work schedule that I’ve only just recently pared back. I could easily name many other distractions. The simple truth that I am forced to admit to myself is that I haven’t consistently wanted to think long and hard about evaluating e-learning. At least not until this week.
As the instructor of an online course and a student who is attempting to complete the final stages of a doctoral degree from a distance, I feel that I have a unique perspective on e-learning and its evaluation. Certainly, my perspective differs from my classmates; I am the only student in the e-learning evaluation seminar for whom the seminar itself is an e-learning experience. I want my article to reflect what I have learned as I’ve participated in e-learning experiences, both formal and informal, and in both of my roles, instructor and student.
At the same time, my professor is expecting an academic article and I’m struggling with how best to weave my personal narrative into the evaluation framework that Dr. Williams has provided for us. The e-learning evaluation framework given by David Williams and Charles Graham in their soon to be published article hangs on the following (rather generic) questions:
- What is the context/background?
- Who are the stakeholders?
- What is the evaluand?
- What are the criteria for judging the evaluand?
- What questions will answer how well the evaluand meets the criteria?
- What methods should be used to answer the questions?
- What do you get when you collect and analyze the data?
- How does ‘what is’ compare to ‘what should be’?
- What recommendations does the study yield?
- How well was the evaluation conducted?
As I review these questions yet again, I realize that this framework matches the logic that I used to design and evaluate the online course that I taught. At the same time, this methodology does not quite reflect the informal process that I, as a student, use to evaluate my e-learning experiences. I know that I have found some of my e-learning experiences more valuable than others, which indicates that I evaluated those experiences on some level, but how?
Some of the questions from the framework seem to apply to both perspectives. I can describe the context for each of my varied e-learning experiences, and in all my informal evaluations of e-learning there has only ever been but one stakeholder: me. (Even now, in the evaluating e-learning seminar, I am relatively unconcerned with the learning experiences of my classmates. I hope they are learning, but my evaluation of the experience is independent of what they feel they are gaining from the experience. I am somewhat concerned about how Dr. Williams will evaluate me, but my evaluation of the seminar will not be impacted by whether or not Dr. Williams feels that it has been a success.) I have never explicitly stated my criteria for evaluating e-learning experiences as a student and I’m not sure that I could articulate each criterion now, or even if I’ve applied the same criteria to each experience. I have definitely never framed evaluation questions based on my criteria, contemplated methods of data collection, or analyzed data that I’ve collected during my e-learning experiences. Still, I know that as a student in each of my e-learning experiences I’ve come to conclusions about how ‘what is’ compares to ‘what should be’ and have used these conclusions to inform the design of e-learning opportunities for the courses that I teach. I’ve done this without ever meta-evaluating the process that resulted in my conclusions.
I find myself wondering: (1) how do most students evaluate e-learning experiences? and (2) if students were to consciously apply the Williams-Graham framework would their evaluations of their e-learning experiences change?
What do you think? What process do you use to evaluate e-learning experiences as either an instructor or a student?