A summary of Summated Rating Scale Construction: An Introduction by Paul E. Spector. This summary is provided in partial fulfillment of the requirements for my independent reading course this semester.
Spector uses the Work Locus of Control Survey throughout this work to exemplify the process of constructing summated rating scales. I found it more useful to consider how the advice given applies to the instrument that Dr. Graham has developed to assess pre-service teachers’ assessment of their own Technological Pedagogical Content Knowledge. Also, since Stata, not SPSS, is my preferred statistical package (and because this text was published in 1992) I found the information on computer software irrelevant or obsolete. Still, I think the text helped me to better understand information that I had previously read in survey methodology texts.
Summary of Content
One of the defining characteristic of a summated rating scale is the presence of multiple items. Multiple items provide reliability and precision. Additionally, the individual items that comprise a summated rating scale must be measured using a continuum and written so that there is no single answer. Individuals responding to a summated rating scale must answer each item with its own rating.
The process of developing a summated rating scale is iterative. The primary step involves defining the construct. Only after construct definition, can a researcher hope to design and then pilot a scale. Once a scale has been piloted, the next step is to administer the instrument and conduct a thorough item analysis. The results of the analysis may lead the refine his or her original construct definition. Once the researcher is satisfied with the construct definition, he or she may begin to validate and norm the assessment.
Three common categories of response categories include agreement, evaluation, and frequency. According to Spector, the optimum number of responses for an item ranges between five and nine. Negative responses should be re-scaled before the data is analyzed. The formula for re-scaling negative data is R = (H + L) – I where H is the largest number, L is the lowest number, I is the response to an item, and R is the score for the reversed item.
Spector shares several rules of thumb for item writing:
- Items should express single ideas.
- Some items should be worded positively, others negatively.
- Items should avoid the use of colloquialisms, expressions, and jargon.
- Item-writers should remember the reading level of the target audience for the scale.
A main purpose of item analysis is to determine the items that contribute to the internal consistency of the instrument. Coefficient alpha is a common measure for describing internal consistency and 0.70 is a minimum target. Coefficient alpha is used in tandem with item-remainder coefficients to identify potentially troublesome items. One strategy for selecting items for inclusion are to decide on a number, for example, m, and then select the m items with the highest item-remainder coefficients. Alternatively, you can set an item-remainder coefficient criterion and include all items that meet the set criterion. A researcher may consider other, external criteria, such as social desirability, hen selecting items. The Spearman-Brown prophesy formula can provide a useful estimate of the number of items needed to reach internal consistency.
There are many different ways to study the validity of an instrument. Criterion-related validity includes concurrent, predictive, and known-groups validity. Each of these criterion-related validity techniques involves a comparison between the scores from the summated rating scale in question and a set of other variables. In concurrent validity studies, the scale scores are collected at the same time, from the same individuals, as the other variables. In predictive validity, the scale scores are collected and then used to predict the value of a variable in the future. In known-groups validity, the researcher tests one or more hypotheses about differences between the scores to two or more groups.
Convergent and divergent validity studies are based on the principle that measures of the same construct will correlate strongly while measures of different constructs will correlate less strongly. Researchers use the Multitrait-Multimodal Matrix (MTMM) in order to explore convergent and divergent validity.
Factor analysis is another tool that researchers use to explore the validity of instruments. Exploratory factor analysis helps to determine the number of constructs that might describe a particular data set. Confirmatory factor analysis can help determine if a set of constructs in a theoretical framework fits the empirical data.
Spector suggests that researcher validate instruments by collecting as many different types of evidence as possible . Spector also addresses the importance of determining the reliability of the instrument, not only internally, but across time, as in test-retest reliability. Additionally, Spector points out that instruments should be normed with samples from the appropriate target population, not simply with samples of convenience found on college campuses. When calculating norms, mean and standard deviation are of primary importance, as is the overall shape of the distribution.
Finally, since scale construction is a recursive, iterative process, it is never-ending. The goal is not perfection, but to get a scale that behaves consistently within its own theoretical framework.