Another book summary in partial fulfillment of my independent reading assignment for graduate school.

Brief Review

I was assigned to read Assessing Affective Characteristics in Schools by Lorin Anderson and Sid Bourke.  I found the text to be less technical than Summated Rating Scale Construction, but often more detailed in its advice.  (This shouldn’t be particularly surprising, since Anderson and Bourke used far more pages than Paul Spector.)  Anderson and Bourke also dedicated far more pages to convincing the reader of the necessity of assessing affective characteristics than Spector did trying to convince the reader of the necessity of constructing summated rating scales.  Over the past few years, I’ve become increasingly convinced of the importance of affective characteristics in learning, particularly in the role of motivation.  As a result, I sometimes felt that Anderson and Bourke were preaching to the choir, and wished I could read a less evangelical version of the text that would simply tell me what I needed to know to get the job done.  

Summary of Content

In the first chapter, Anderson and Bourke define the terms that comprise their title. They enumerate five features that they claim define affective characteristics, specifically, that affective characteristics are typical ways of feeling that are directed toward some target with some intensity. Anderson and Bourke define assessment as “the gathering of information about a human characteristic for a stated purpose.” The authors choose to focus on affective characteristics of students in the context of school settings. According to Anderson and Bourke, affective characteristics have value as means to ends and as ends in themselves. In the latter sections of the first chapter, Anderson and Bourke address common beliefs that sometimes impede the assessment of affective characteristics in schools. According to Anderson and Bourke, affective can and should be assessed in school settings.

Chapter two of Assessing Affective Characteristics in Schools focuses further on definitions, detailing the importance of clearly defining the specific affective characteristic or characteristics that one intends to assess. Anderson and Bourke also point out the importance of carefully defining the target to which the affective characteristic is directed. Conceptual definitions provide an understanding of abstract meaning while operational definitions specify behaviors that allow observers to make inferences about affective characteristics. The authors believe that conceptual and operational definitions must be closely aligned in order to provide useful information about a particular affective characteristic. The chapter provides a description of two major approaches for developing operational definitions of affective characteristics, the mapping sentence approach and the domain-reference approach. Whether one is creating a new assessment instrument or selecting a previously created assessment instrument, one should begin with a precise definition of the affective characteristic in question.

The third chapter discusses the major methods for collecting data about human characteristics, the observational method and the self-report method. Both methods have strengths and weaknesses. The observational method is limited by the observer’s powers of observation as well as their powers of interpretation. The self-report method is limited by respondent’s memory and/or integrity as well as the questioner’s ability to ask the right questions. Some studies have shown that observational and self-report methods that claim to assess the same characteristic provide dissimilar results. Anderson and Bourke believe that, at least in the context of schools, self-report methods are generally superior. However, the authors also state that they do not intend the chapter to be interpreted as a complete rejection of observational methods.

Good affective scales must have communication value, objectivity, validity, reliability, and interpretability. A questionnaire has communication value if the respondent can easily understand what the questionnaire is asking them. A scale has objectivity when it has minimized scorer or coder bias. An instrument has validity when it actually measures what it purports to measure. Scales are considered reliable when they have internal consistency, stability , and equivalence. Internal consistency is often measured by Cronbach’s alpha, stability may be measured using test-retest results, and equivalence may involve a comparison of multiple measures of the same affective characteristic. Questionnaires are considered to have interpretability when the results are reported in such a way that primary audience of the data can understand the results. Anderson and Bourke describe a number of common practices in the assessment of affective characteristics included the use of several varieties of Likert scales.

Anderson and Bourke provide advice for either selecting or designing assessment instruments for affective characteristics. When possible, they recommend selecting an existing an instrument over designing one. They enumerate several potential sources for locating existing assessment instruments,

• electronic databases,
• commercial publishing houses,
• professional associations,
• research institutes and laboratories, and
• compendiums.

 They also provide a list of six steps for designing a new instrument:

• preparing a blueprint,
• writing the items,
• writing directions,
• having the draft instrument reviewed,
• pilot testing the instrument, and
• readying the instrument for administration.

 However, whether an individual will select an existing instrument or design a new one, Anderson and Bourke emphasize that the first steps are to determine the purpose of the assessment, identify the target population, and define the affective characteristics and targets. The authors list four common categories of purposes for affective assessment,

• enhancing student learning,
• improving the quality of educational programs,
• evaluating the quality of educational programs, and
• conforming to administrative or legislative mandates.

 Data analysis is the main focus of chapter six. The authors provide a list of five steps for developing and analyzing scale scores:

• coding,
• entering and checking data,
• dealing with missing data,
• recoding items as necessary,
• checking scale validity and reliability, and
• creating and reporting scale scores.

Anderson and Bourke address the importance of good data and provide advice for error checking, such as dual coding, as well as methods for dealing with small amounts of missing data. The authors also discuss using factor analysis to address empirical validity in multiscale instruments.

 The authors describe the process of interpreting assessment data for affective characteristics in chapter seven. They suggest using absolute and/or relative comparisons to assist in the interpretation of the data. Absolute comparisons require the identification of a neutral point and the creation of a neutral range as well as a range above the neutral range and a range below the neutral range. Relative comparisons may involve a normative sample or it may involve comparisons between known groups whose scale scores are expected to differ. Interpretations will depend on the comparison method used.

 Anderson and Bourke use chapter 8 to argue the importance of affective assessment in finding solutions to common education problems including student motivation, the design of effective learning environments, and character building.   

A summary of Summated Rating Scale Construction: An Introduction by Paul E. Spector.  This summary is provided in partial fulfillment of the requirements for my independent reading course this semester.  

Brief Review

Spector uses the Work Locus of Control Survey throughout this work to exemplify the process of constructing summated rating scales. I found it more useful to consider how the advice given applies to the instrument that Dr. Graham has developed to assess pre-service teachers’ assessment of their own Technological Pedagogical Content Knowledge. Also, since Stata, not SPSS, is my preferred statistical package (and because this text was published in 1992) I found the information on computer software irrelevant or obsolete. Still, I think the text helped me to better understand information that I had previously read in survey methodology texts.

Summary of Content

One of the defining characteristic of a summated rating scale is the presence of multiple items. Multiple items provide reliability and precision. Additionally, the individual items that comprise a summated rating scale must be measured using a continuum and written so that there is no single answer. Individuals responding to a summated rating scale must answer each item with its own rating.

The process of developing a summated rating scale is iterative. The primary step involves defining the construct. Only after construct definition, can a researcher hope to design and then pilot a scale. Once a scale has been piloted, the next step is to administer the instrument and conduct a thorough item analysis. The results of the analysis may lead the refine his or her original construct definition. Once the researcher is satisfied with the construct definition, he or she may begin to validate and norm the assessment.

Three common categories of response categories include agreement, evaluation, and frequency. According to Spector, the optimum number of responses for an item ranges between five and nine. Negative responses should be re-scaled before the data is analyzed. The formula for re-scaling negative data is R = (H + L) – I where H is the largest number, L is the lowest number, I is the response to an item, and R is the score for the reversed item.

Spector shares several rules of thumb for item writing:

1. Items should express single ideas.
2. Some items should be worded positively, others negatively.
3. Items should avoid the use of colloquialisms, expressions, and jargon.
4. Item-writers should remember the reading level of the target audience for the scale.

A main purpose of item analysis is to determine the items that contribute to the internal consistency of the instrument. Coefficient alpha is a common measure for describing internal consistency and 0.70 is a minimum target. Coefficient alpha is used in tandem with item-remainder coefficients to identify potentially troublesome items. One strategy for selecting items for inclusion are to decide on a number, for example, m, and then select the m items with the highest item-remainder coefficients. Alternatively, you can set an item-remainder coefficient criterion and include all items that meet the set criterion. A researcher may consider other, external criteria, such as social desirability, hen selecting items. The Spearman-Brown prophesy formula can provide a useful estimate of the number of items needed to reach internal consistency.

There are many different ways to study the validity of an instrument. Criterion-related validity includes concurrent, predictive, and known-groups validity. Each of these criterion-related validity techniques involves a comparison between the scores from the summated rating scale in question and a set of other variables. In concurrent validity studies, the scale scores are collected at the same time, from the same individuals, as the other variables. In predictive validity, the scale scores are collected and then used to predict the value of a variable in the future. In known-groups validity, the researcher tests one or more hypotheses about differences between the scores to two or more groups.

Convergent and divergent validity studies are based on the principle that measures of the same construct will correlate strongly while measures of different constructs will correlate less strongly. Researchers use the Multitrait-Multimodal Matrix (MTMM) in order to explore convergent and divergent validity.

Factor analysis is another tool that researchers use to explore the validity of instruments. Exploratory factor analysis helps to determine the number of constructs that might describe a particular data set. Confirmatory factor analysis can help determine if a set of constructs in a theoretical framework fits the empirical data.

Spector suggests that researcher validate instruments by collecting as many different types of evidence as possible . Spector also addresses the importance of determining the reliability of the instrument, not only internally, but across time, as in test-retest reliability. Additionally, Spector points out that instruments should be normed with samples from the appropriate target population, not simply with samples of convenience found on college campuses. When calculating norms, mean and standard deviation are of primary importance, as is the overall shape of the distribution.

Finally, since scale construction is a recursive, iterative process, it is never-ending. The goal is not perfection, but to get a scale that behaves consistently within its own theoretical framework.