I have often heard people refer to their dissertation as “their baby”.  You will never hear that here; there’s nothing tragic about the premature death of a dissertation.

Over the past few months, I’ve occasionally wondered if I would have found it easier to work on my dissertation if I hadn’t changed my topic.  During the past week, I’ve decided that it really doesn’t matter.  I’m interested in how people learn and in the efforts people make to help learning happen; from that perspective, either topic provides plenty of interest.

This week’s progress

In the past week, I’ve read and outlined seven articles related to my dissertation.  Two of the articles related to a specific problem I am trying to solve.  Specifically, my conditionally approved prospectus states that I will use a rubric to score student responses on a pre- and post- assessment.  One of the conditions of approval is producing the rubric that I will use.  Two of the articles I read this week will help me craft my justification for a particular rubric.  (At the time of my defense, I was waiting for the release of the results of the study containing this rubric.)

Two of the articles were literature reviews that I wish I had read before designing my study.  I feel that these literature reviews provide perspective on the larger field of teacher education related to technology integration.  Someday when I am advising graduate students, I will require them to read literature reviews related to their area of interest before selecting their topic.  If they can’t find one, then I believe they have prematurely narrowed their topic.  In my case, I feel that (as of December 2009–I’m currently playing catch up) I had read most of the literature  related to technological pedagogical content knowledge (also known as TPACK or TPCK), but I should have looked for literature reviews on the broader topic of teacher education and technology integration in order to better understand the context of my study.  I may have made some adjustments to the design of my study or framed my questions differently.  If nothing else, I would have had a stronger defense.

The remaining three articles were reports on various attempts people have made to teach teachers how to teach with technology.  Some articles provide useful citations for my literature review, some are of note for methodological or contextual similarities to my study.

As I’ve read, I’ve remembered that I like reading articles, which has been helpful for regaining motivation and momentum.  I met my dissertation progress goals last week and got to enjoy my reward.  I’m currently waiting on four specific books from the library, so while I’m waiting, I’ll start transcribing the interviews from my study.  I expect transcription take most of the next two weeks.

Some important facts:

1. The data for my dissertation have been collected.
2. My son (pictured above) is now 5 1/2 months, takes regular naps, and sometimes sleeps through the night.

My conclusions based on the above facts:

1. I need to devote time to analyzing my dissertation data.
2. I have time to devote to analyzing my dissertation data.

For the past couple of months, I’ve been half-heartedly trying to re-dedicate myself to the work of being a PhD student.  It’s been tough-going as half-heartedness and dedication are antagonistic concepts.  (Besides, my whole heart is already devoted to the pint-sized person pictured above.  He really is a sweet baby.)  Still, I’ve recently decided that finishing my dissertation and being a good mother are not mutually exclusive goals.  I just need to manage time differently than I have in the past.

To help myself reach my dissertation goals, I mapped out deadlines for the completion of important dissertation steps and placed them on my Calendar.   I brainstormed a list of all of the required tasks and sub-tasks related to my dissertation, and I decided that I will devote two hours each weekday to dissertation tasks and will read and outline at least one article daily. Each Saturday I will summarize my progress in a blog post, to try to hold myself accountable for meeting my goals.  As a reward for meeting my process goals, I will earn time that I can spend on my hobbies once my son is in bed for the night.  When I complete tasks from my to do list, I will “earn” a small financial reward (really, give myself permission to spend money previously earned).  The rewards for this upcoming week–knitting time and the registration fee for an event in which I want to participate.

Check back next week to see how the plan is working.

My son, day 1

Last year, blogging took a back burner to other priorities in my life.  I was working at multiple jobs for most of the year, I was PhD student, and I was pregnant.  I am no longer pregnant, and I’ve stopped working outside the home, but blogging will still be on the back burner.  The little guy pictured above is priority #1 and will be for the foreseeable future.  I managed to defend my dissertation prospectus the week before he was born, and am still plugging away at collecting and analyzing my data, but I have to admit that educational technology isn’t as relevant in my life as it was even a few months ago.  My son is nearly seven weeks old.  He’s learning some important things right now, like how to hold up his head, smile, and use his hands.  The world of bits and bytes has very little to offer him right now, and I’m OK with that.

Obviously I am biased, but I am a strong advocate of a scientific (in the sense of “occupied or concerned with science”) approach to teaching at all levels.  My reasoning for a scientific approach is not because I have complete faith in the primacy of the scientific method as a way knowing, but because I believe science (in its broadest sense as “systematized knowledge in general” or “knowledge gained by systematic study”) to be an interdisciplinary study.  Art, Music, Language, Literature, the various Social Sciences, and Mathematics would benefit from what I would term, a scientific approach to teaching, or perhaps, a scientific approach to curriculum.

Below I share what I believe are the crucial components to a “scientific approach to teaching”.

1. Foster a sense of wonder

I would challenge all educators to become familiar with Rachel Carson’s short work, The Sense of Wonder.  All science, all true knowledge and understanding, begins with a sense of wonder, an intense curiosity about “why” and “how”.  Too often, our teaching at all levels focuses on the “who, what, and where” of the world around us and treats the potentially awe-inspiring matters of “why” and “how” as simply another “what” to memorize.    A scientific approach to teaching encourages learners to wonder about “why” and “how”.

2. Encourage observation

A sense of wonder is grounded in a keen awareness of one’s surroundings.  Scientific knowledge is based in empirical data, data gathered from experience and observation.  Encourage learners to attend to details and use their senses of sight, hearing, touch, taste, and smell whenever appropriate.  Challenge learners to record these details qualitatively and quantitatively.  A scientific approach to teaching gives learners many appropriate opportunities to use their senses and record their observations.

3.  Push for analysis

Wonder grows with understanding and understanding comes of analysis.  Quantitative analysis is the heart of much of what we traditionally consider the realm of science and mathematics.  Quantative analysis also reaches deep into the realms of the social sciences, of economics, psychology, sociology, and many others.  But qualitative analysis is also valid; it has a place in the sciences, the social sciences, and is especially important in the arts and literature.  Provide time for learners to revisit their observations and search for patterns, in whatever form they may take.  Push learners classify, connect, and count.  A scientific approach to teaching pushes learners to seek for patterns.

4. Require communication

Like most things in life, wonder is best appreciated when shared.  Our languages provide us with means of communication, but human language is far from our sole source of expression.  Prod learners toward precision in their descriptions of the world around them and the findings of their analyses.  But also instruct learners on the use of ambiguity in the creative expression of emotion and personal experiences.  A scientific approach to teacher requires learners to communicate their learning.

I see a scientific approach to teaching as a way to guide learners in the process of transforming the concrete objects and artifacts in the world around them into the abstract concepts that inhabit the human mind.  Each step in the process is crucial.  Without a sense of wonder, the learner will have no desire to continue the journey.  If a teacher fails to take the time to sufficiently ground the learner in concrete observation, the learner will be lost in unrecognizable terrain.  The skills of analysis are what will enable learners to chart future courses for independent learning, while the powers of clear communication and creative expression are what will bind the learner to the rest of the human race.

So how about it?  Do you agree, disagree?  What would a scientific approach to teaching look like in a Pre-K science lesson or  a college art history class?  I’d love to know what others think.

I’m 16 weeks pregnant and my mother has Stage 3 thyroid cancer.  My mother’s prognosis is still good, but I’m nervous; my paternal grandmother had thyroid cancer and died when I was only 6.  Like my mother now, Grandma Smith was only in her 50′s.  Right now, I’m supposed to be working on my dissertation research study.  In fact, I should have been working on my research study for the past several weeks, but I can’t seem to care about it.

Having previously experienced two early miscarriages and an ectopic pregnancy, I don’t take my baby’s life as a guarantee and I know that a diagnosis of cancer doesn’t necessarily mean death.  Still, my mom is edging closer to death than I want her to just as my husband and I prepare to welcome a new life into our home, our family.  If my life were a novel, it would feel like a contrived literary device.

When I was very young, my Grandmother Smith made me a quiet book that I took to church with me every Sunday for years.  When I got too old for it, my younger siblings played with it.  After she was gone, the book was tangible evidence that I was important to my grandmother.

I have relatively few memories of time spent with Grandma Smith.  I grew up on the east coast and she lived in Colorado.  Still, she and grandpa made several trips back east to visit my younger brother and I (my other siblings weren’t born yet).  On her last visit, I remember her taking me for a walk to the park and watching me play on the swings.  When we got home, she took a nap.  At the time, I didn’t think much of it, but when I got a little older and realized that she was in the late stages of cancer during that visit, her simple act became one of my most cherished memories; I knew she loved me because she spent what little time and energy she had that day on me.

I don’t know how much time my mom has left on earth.  I don’t know how much time I have left, or that my child will have for that matter.  I do know that my mother wants me to can blackberry jam with her this summer and I have a stack of patterns for baby blankets and clothes.  I want them both to know I love them.

As the instructor of an online course and a student who is attempting to complete the final stages of a doctoral degree from a distance, I feel that I have a unique perspective on e-learning and its evaluation.  Certainly, my perspective differs from my classmates; I am the only student in the e-learning evaluation seminar for whom the seminar itself is an e-learning experience.  I want my article to reflect what I have learned as I’ve participated in e-learning experiences, both formal and informal, and in both of my roles, instructor and student.

At the same time, my professor is expecting an academic article and I’m struggling with how best to weave my personal narrative into the evaluation framework that Dr. Williams has provided for us.  The e-learning evaluation framework given by David Williams and Charles Graham in their soon to be published article hangs on the following (rather generic) questions:

1. What is the context/background?
2. Who are the stakeholders?
3. What is the evaluand?
4. What are the criteria for judging the evaluand?
5.  What questions will answer how well the evaluand meets the criteria?
6. What methods should be used to answer the questions?
7. What do you get when you collect and analyze the data?
8. How does ‘what is’ compare to ‘what should be’?
9. What recommendations does the study yield?
10. How well was the evaluation conducted?

As I review these questions yet again, I realize that this framework matches the logic that I used to design and evaluate the online course that I taught.  At the same time, this methodology does not quite reflect the informal process that I, as a student, use to evaluate my e-learning experiences.  I know that I have found some of my e-learning experiences more valuable than others, which indicates that I evaluated those experiences on some level, but how?

Some of the questions from the framework seem to apply to both perspectives.   I can describe the context for each of my varied e-learning experiences, and in all my informal evaluations of e-learning there has only ever been but one stakeholder: me.  (Even now, in the evaluating e-learning seminar, I am relatively unconcerned with the learning experiences of my classmates.  I hope they are learning, but my evaluation of the experience is independent of what they feel they are gaining from the experience.  I am somewhat concerned about how Dr. Williams will evaluate me, but my evaluation of the seminar will not be impacted by whether or not Dr. Williams feels that it has been a success.)  I have never explicitly stated my criteria for evaluating e-learning experiences as a student and I’m not sure that I could articulate each criterion now, or even if I’ve applied the same criteria to each experience.  I have definitely never framed evaluation questions based on my criteria, contemplated methods of data collection, or analyzed data that I’ve collected during my e-learning experiences.  Still, I know that as a student in each of my e-learning experiences I’ve come to conclusions about how ‘what is’ compares to ‘what should be’ and have used these conclusions to inform the design of e-learning opportunities for the courses that I teach.  I’ve done this  without ever meta-evaluating the process that resulted in my conclusions.  

I find myself wondering: (1) how do most students evaluate e-learning experiences? and (2) if students were to consciously apply the Williams-Graham framework would their evaluations of their e-learning experiences change?

What do you think?  What process do you use to evaluate e-learning experiences as either an instructor or a student?

First, I know that my graduation goal is April 2010 and that I need to have my data collected during the Fall 2009 semester in order to meet this deadline.  Ideally, I should complete my prospectus this summer.  However, professors like to take vacations during the summer, which means that I will not be able to gather my committee in one place until the Fall.  As a result, I need to plan on defending my prospectus at the very beginning of the Fall 2009 semester.  The first day of Fall classes is August 31st.  I need to schedule my defense no later than September 4, 2009.

My committee will need time to read and review my prospectus prior to my defense.  Really, 2 weeks is plenty of time for them to review one prospectus.  However, they will be very busy at the beginning of the semester, so I plan to give them 4 weeks.  My prospectus must be completed by August 7, 2009.

I have approximately 12 weeks to complete my prospectus.  

I could say that I will work on the introduction for 4 weeks, then the literature review for 4 weeks, and then finish with the methodology for final 4 weeks.  This approach seems artificial to me.  I think a better, more organic approach would be to allow the prospectus to emerge from my reading.  I think I should start by intensely focusing on the literature.  I also need to follow up on the contacts that I made at the SITE conference.  To be realistic, I need to take into account that I will be on vacation with my husband for at least two weeks this summer.  As a result, I am proposing the following time line:

• May 21 – Prepare reading list from references of articles already read, and the recommendations of those in the field.  Begin reading the articles/books and take notes on each reading.  Also, begin a concept map connecting the themes from all of the readings.  The purpose of my reading between today and June 4 will be to better understand the terminology.
• June 4 – Prepare an outline based on my synthesis of my readings to date.  Frame broad research questions.  Edit (most likely augment) the original reading list.  The purpose of my reading between June 5 and June 11 will be to identify the types of research studies that have been conducted previously.
• June 11 – Revise the outline to include specific research questions.  Identify the types of study most appropriate for the research questions.  The purpose of my reading between June 11 and June 18 will be to identify specific research methodologies that are appropriate for my study.
• June 18 – Revise the outline to include a description of my proposed research methodology.  Contact any available committee members for feedback at this point.  The purpose of my reading between June 18 and June 24 will be to search for possible inspiration from tangential areas of research.
• June 24 – Revise outline to reflect any feedback received from the committee and any inspiration from reading.   Honestly, I’ll be attending a family reunion in late June/early July, so I’m not likely to do much academic reading during the weeks that immediately follow June 24.  Still, I will take my articles with me for the plane rides.  My purpose for reading between June 24 and July 9 is to re-read the literature with fresh eyes, looking for insights I may have missed previously.
• July 9 – Prepare my first prose draft of the literature review of my prospectus.   Provide copies of the draft to peers and members of my committee for feedback.
• July 16 – Prepare my first prose draft of the methodology section of my prospectus.  Provide copies of the draft to peers and members of my committee for feedback.
• July 23 – Prepare my first prose draft of the introduction section of my prospectus.  Provide copies of the draft to peers and members of my committee for feedback.  Between July 24 and August 6th, my purpose for reading will be to discover solutions to issues raised during the feedback process.
• July 30 – Prepare a list of constructive criticisms offered by peers and committee members.  
• August 6 -Revise all sections of my draft, incorporating feedback from my peers and committee members.
• August 7 – Provide all committee members with a copy of the prospectus in preparation for the defense on September 4.
• September 4 (or before) – Defend the prospectus.  I welcome feedback.

Am I being realistic?  Any suggestions?

Thanks!

Brief Review

I was assigned to read Assessing Affective Characteristics in Schools by Lorin Anderson and Sid Bourke.  I found the text to be less technical than Summated Rating Scale Construction, but often more detailed in its advice.  (This shouldn’t be particularly surprising, since Anderson and Bourke used far more pages than Paul Spector.)  Anderson and Bourke also dedicated far more pages to convincing the reader of the necessity of assessing affective characteristics than Spector did trying to convince the reader of the necessity of constructing summated rating scales.  Over the past few years, I’ve become increasingly convinced of the importance of affective characteristics in learning, particularly in the role of motivation.  As a result, I sometimes felt that Anderson and Bourke were preaching to the choir, and wished I could read a less evangelical version of the text that would simply tell me what I needed to know to get the job done.  

Summary of Content

In the first chapter, Anderson and Bourke define the terms that comprise their title. They enumerate five features that they claim define affective characteristics, specifically, that affective characteristics are typical ways of feeling that are directed toward some target with some intensity. Anderson and Bourke define assessment as “the gathering of information about a human characteristic for a stated purpose.” The authors choose to focus on affective characteristics of students in the context of school settings. According to Anderson and Bourke, affective characteristics have value as means to ends and as ends in themselves. In the latter sections of the first chapter, Anderson and Bourke address common beliefs that sometimes impede the assessment of affective characteristics in schools. According to Anderson and Bourke, affective can and should be assessed in school settings.

Chapter two of Assessing Affective Characteristics in Schools focuses further on definitions, detailing the importance of clearly defining the specific affective characteristic or characteristics that one intends to assess. Anderson and Bourke also point out the importance of carefully defining the target to which the affective characteristic is directed. Conceptual definitions provide an understanding of abstract meaning while operational definitions specify behaviors that allow observers to make inferences about affective characteristics. The authors believe that conceptual and operational definitions must be closely aligned in order to provide useful information about a particular affective characteristic. The chapter provides a description of two major approaches for developing operational definitions of affective characteristics, the mapping sentence approach and the domain-reference approach. Whether one is creating a new assessment instrument or selecting a previously created assessment instrument, one should begin with a precise definition of the affective characteristic in question.

The third chapter discusses the major methods for collecting data about human characteristics, the observational method and the self-report method. Both methods have strengths and weaknesses. The observational method is limited by the observer’s powers of observation as well as their powers of interpretation. The self-report method is limited by respondent’s memory and/or integrity as well as the questioner’s ability to ask the right questions. Some studies have shown that observational and self-report methods that claim to assess the same characteristic provide dissimilar results. Anderson and Bourke believe that, at least in the context of schools, self-report methods are generally superior. However, the authors also state that they do not intend the chapter to be interpreted as a complete rejection of observational methods.

Good affective scales must have communication value, objectivity, validity, reliability, and interpretability. A questionnaire has communication value if the respondent can easily understand what the questionnaire is asking them. A scale has objectivity when it has minimized scorer or coder bias. An instrument has validity when it actually measures what it purports to measure. Scales are considered reliable when they have internal consistency, stability , and equivalence. Internal consistency is often measured by Cronbach’s alpha, stability may be measured using test-retest results, and equivalence may involve a comparison of multiple measures of the same affective characteristic. Questionnaires are considered to have interpretability when the results are reported in such a way that primary audience of the data can understand the results. Anderson and Bourke describe a number of common practices in the assessment of affective characteristics included the use of several varieties of Likert scales.

Anderson and Bourke provide advice for either selecting or designing assessment instruments for affective characteristics. When possible, they recommend selecting an existing an instrument over designing one. They enumerate several potential sources for locating existing assessment instruments,

• electronic databases,
• commercial publishing houses,
• professional associations,
• research institutes and laboratories, and
• compendiums.

 They also provide a list of six steps for designing a new instrument:

• preparing a blueprint,
• writing the items,
• writing directions,
• having the draft instrument reviewed,
• pilot testing the instrument, and
• readying the instrument for administration.

 However, whether an individual will select an existing instrument or design a new one, Anderson and Bourke emphasize that the first steps are to determine the purpose of the assessment, identify the target population, and define the affective characteristics and targets. The authors list four common categories of purposes for affective assessment,

• enhancing student learning,
• improving the quality of educational programs,
• evaluating the quality of educational programs, and
• conforming to administrative or legislative mandates.

 Data analysis is the main focus of chapter six. The authors provide a list of five steps for developing and analyzing scale scores:

• coding,
• entering and checking data,
• dealing with missing data,
• recoding items as necessary,
• checking scale validity and reliability, and
• creating and reporting scale scores.

Anderson and Bourke address the importance of good data and provide advice for error checking, such as dual coding, as well as methods for dealing with small amounts of missing data. The authors also discuss using factor analysis to address empirical validity in multiscale instruments.

 The authors describe the process of interpreting assessment data for affective characteristics in chapter seven. They suggest using absolute and/or relative comparisons to assist in the interpretation of the data. Absolute comparisons require the identification of a neutral point and the creation of a neutral range as well as a range above the neutral range and a range below the neutral range. Relative comparisons may involve a normative sample or it may involve comparisons between known groups whose scale scores are expected to differ. Interpretations will depend on the comparison method used.

 Anderson and Bourke use chapter 8 to argue the importance of affective assessment in finding solutions to common education problems including student motivation, the design of effective learning environments, and character building.   

Brief Review

Spector uses the Work Locus of Control Survey throughout this work to exemplify the process of constructing summated rating scales. I found it more useful to consider how the advice given applies to the instrument that Dr. Graham has developed to assess pre-service teachers’ assessment of their own Technological Pedagogical Content Knowledge. Also, since Stata, not SPSS, is my preferred statistical package (and because this text was published in 1992) I found the information on computer software irrelevant or obsolete. Still, I think the text helped me to better understand information that I had previously read in survey methodology texts.

Summary of Content

One of the defining characteristic of a summated rating scale is the presence of multiple items. Multiple items provide reliability and precision. Additionally, the individual items that comprise a summated rating scale must be measured using a continuum and written so that there is no single answer. Individuals responding to a summated rating scale must answer each item with its own rating.

The process of developing a summated rating scale is iterative. The primary step involves defining the construct. Only after construct definition, can a researcher hope to design and then pilot a scale. Once a scale has been piloted, the next step is to administer the instrument and conduct a thorough item analysis. The results of the analysis may lead the refine his or her original construct definition. Once the researcher is satisfied with the construct definition, he or she may begin to validate and norm the assessment.

Three common categories of response categories include agreement, evaluation, and frequency. According to Spector, the optimum number of responses for an item ranges between five and nine. Negative responses should be re-scaled before the data is analyzed. The formula for re-scaling negative data is R = (H + L) – I where H is the largest number, L is the lowest number, I is the response to an item, and R is the score for the reversed item.

Spector shares several rules of thumb for item writing:

1. Items should express single ideas.
2. Some items should be worded positively, others negatively.
3. Items should avoid the use of colloquialisms, expressions, and jargon.
4. Item-writers should remember the reading level of the target audience for the scale.

A main purpose of item analysis is to determine the items that contribute to the internal consistency of the instrument. Coefficient alpha is a common measure for describing internal consistency and 0.70 is a minimum target. Coefficient alpha is used in tandem with item-remainder coefficients to identify potentially troublesome items. One strategy for selecting items for inclusion are to decide on a number, for example, m, and then select the m items with the highest item-remainder coefficients. Alternatively, you can set an item-remainder coefficient criterion and include all items that meet the set criterion. A researcher may consider other, external criteria, such as social desirability, hen selecting items. The Spearman-Brown prophesy formula can provide a useful estimate of the number of items needed to reach internal consistency.

There are many different ways to study the validity of an instrument. Criterion-related validity includes concurrent, predictive, and known-groups validity. Each of these criterion-related validity techniques involves a comparison between the scores from the summated rating scale in question and a set of other variables. In concurrent validity studies, the scale scores are collected at the same time, from the same individuals, as the other variables. In predictive validity, the scale scores are collected and then used to predict the value of a variable in the future. In known-groups validity, the researcher tests one or more hypotheses about differences between the scores to two or more groups.

Convergent and divergent validity studies are based on the principle that measures of the same construct will correlate strongly while measures of different constructs will correlate less strongly. Researchers use the Multitrait-Multimodal Matrix (MTMM) in order to explore convergent and divergent validity.

Factor analysis is another tool that researchers use to explore the validity of instruments. Exploratory factor analysis helps to determine the number of constructs that might describe a particular data set. Confirmatory factor analysis can help determine if a set of constructs in a theoretical framework fits the empirical data.

Spector suggests that researcher validate instruments by collecting as many different types of evidence as possible . Spector also addresses the importance of determining the reliability of the instrument, not only internally, but across time, as in test-retest reliability. Additionally, Spector points out that instruments should be normed with samples from the appropriate target population, not simply with samples of convenience found on college campuses. When calculating norms, mean and standard deviation are of primary importance, as is the overall shape of the distribution.

Finally, since scale construction is a recursive, iterative process, it is never-ending. The goal is not perfection, but to get a scale that behaves consistently within its own theoretical framework.