|Survey management of Work disability|
The workshop sessions were designed to discuss issues relating to:
This chapter summarizes the workshop participants' discussions that flowed from the presentations of the two papers presented in Chapters 2 and 3. Participants identified many unanswered questions about measurement that need to be researched. Some of the key issues that surfaced during the discussions are summarized in this chapter.
One of the major challenges related to the measurement of persons with disabilities and persons with work disabilities concerns the translation of the various complex conceptual models into questions that can be comprehended by the general public and that produce both valid and reliable measures. The Social Security Act defines disability (for adults) as ". . . inability to engage in any substantial gainful activity by reason of any medically determinable physical or mental impairment which can be expected to result in death or which has lasted or can be expected to last for a continuous period of not less than 12 months. . ." (Section 223 [d]). As stated, the definition appears to presuppose a simple relationship between a medical condition and the ability to work and one that is amenable to measurement via the survey process. In contrast, most contemporary theory concerning disability focuses on the complex nature of the relationship between medical conditions and disability and the importance of environmental factors, including the physical and social environments, in mediating the relationship between a medical condition and work. The first session of the workshop focused on a review of the two background papers, addressing the various conceptual frameworks, the complexity inherent in the measurement of disability, and the empirical evidence of measurement error associated with instruments that have been used to measure disability and work disability.
As discussed by Jette and Badley (see Chapter 2) both the Nagi model of disability and the model proposed in the second revision of the International Classification of Impairments, Activities, and Participation (ICIDH-2) emphasize that the accomplishment of particular social roles, such as work, involves not only the accomplishment of the particular activities related to the work task but also the ability to perform with respect to several other activity domains. For example, participation in work activities may also involve the ability to use transportation systems to get to and from work. Therefore, the inability to find accommodating transportation may result in a "work disability," even though the individual is quite capable of performing the tasks associated with his or her occupation.
Both the Nagi and ICIDH-2 conceptual models view the disablement process as a function of the physical and social environments, accommodations and barriers within those environments, and personal attributes and resources. Work disability therefore is unlikely to be related either directly or only to a health condition or impairment. Rather, from the perspective of these conceptual models, work disability (or any other social role) is a function of the interaction of the physical and social environments, accommodations and barriers within those environments, together with the psychosocial and other attributes of the person. Therefore, the measurement of work disability should take into account personal attributes, as well as the social and physical environments in which the person operates.
The complexities of the conceptual models of disability lead to tension in the measurement of disability. The measurement of disability is further complicated by the often ambiguous and varying uses of the terms used in conceptual models and measurement instruments both within and outside the field. In addition, in light of the passage of the American with Disabilities Act, the language is evolving. These factors suggest the use of a survey instrument in which multiple questions concerning disability are asked to set the context for the measurement process, clarify terminology, and tap the multiple domains of interest. This conflicts with the desire of the Social Security Administration, as well as numerous other federal agencies, to identify a short battery of questionnaire items that can identify persons with disabilities and work disabilities through the survey process. Of primary concern is the fact that the questions used to identify persons with disabilities should be both valid and reliable and should be robust when used under varying survey design features.
The empirical literature addressing the measurement error properties of disability and work disability, albeit limited, provides evidence of low reliability and questionable validity. For example, questions concerning disability were asked in the 1991 Canadian census; individuals were sampled on the basis of their responses to the census and administered the Health and Activity Limitation Survey (HALS). The sample included both individuals who indicated a disability and those who did not in response to the census questions. Of the 35,000 persons classified as disabled on the basis of answers to the census questions, 20 percent were classified as not disabled on the basis of responses to HALS. Among the 113,000 persons classified as not disabled on the basis of the census data, 5 percent were classified as disabled on the basis of the data from HALS. Similar examples of an apparent low reliability are evident in the 1990 U.S. census and the CRS (see examples provided in Chapter 3, p. 51). Factors such as the wording of the question, the context of the question, the mode of data collection, the nature of the respondent (self-response versus proxy response), and the sponsorship of the survey all appear to affect estimates of disability.
Even if it is well measured, work disability is a matter of degree, suggesting that the measurement of work disability should be on a continuum as opposed to the dichotomous measurement currently used in most surveys. The categorization of an individual as "work disabled" or not could then be based on clearly defined and established thresholds.
The survey challenges associated with the measurement of persons with disabilities are not limited to measurement error but must also address errors of nonobservation. Particular physical or cognitive disabilities may affect an individual's likelihood of inclusion in the sample or participation in the process if sampled. For example, the use of telephone for data collection limits participation among deaf persons.
Several key issues were identified during the discussion, including the following:
The development of valid and reliable measures of persons with work disabilities requires that one identify differences in the various conceptual models and the implications of those differences for survey measurement. Workshop participants were asked to consider the following questions:
Despite apparent differences between the conceptual models, workshop participants noted that there is, in fact, a lot of commonality among the models. Of critical importance with respect to the operationalization of the various conceptual models is a clear distinction between the measurement of capacity and participation. Both need to be measured, as do the factors that illuminate the relationship between capacity and participation. Further research is needed on the development of measures of capacity and participation.
One of the gaps identified by workshop participants is the lack of research addressing the relationship between measures of quality of life and general health status and measures of disability and work disability. One means by which the relationship between measures of health status and measures of disability can be addressed and the measurement of capacity can be disentangled from performance within health status measures is to foster cross-disciplinary research between the health status research community and the disability research community.
The disablement process is described in the conceptual models as a continuous, dynamic, interactive process, implying that longitudinal measurement is imperative to understanding the relationship among the individual's characteristics, the environment, and participation in the workforce.
Participants stressed the need to develop valid, reliable measures of workforce participation, regardless of the number of questions required to do so. Once a gold standard is developed, the predictive reliability of any subset of those questions could be established.
In summary, the sessions identified the following key issues:
Errors of nonobservation are of potential concern for any data collection effort. Of particular interest with respect to the sampling and measurement of persons with disabilities are issues related to coverage error (i.e., the exclusion of persons with disabilities from the frame used for sampling) and nonresponse error (i.e., the extent to which those who do not participate--because of an inability to be contacted, an inability to participate, or an unwillingness to participate--differ with respect to disabilities from those sampled individuals who agree to be interviewed). Design decisions related to the mode and method of data collection, usually viewed as decisions that affect measurement error, also potentially affect both coverage error and nonresponse error.
Obtaining estimates of the population of persons with disabilities raises particularly challenging issues with respect to the development of a frame from which individuals can be sampled for inclusion in a study. The workshop participants noted that persons with disabilities are probably more likely to live in institutions or in group quarters or to be homeless; hence, reliance on a sample frame consisting of households will result in noncoverage of the population of interest.
Design decisions concerning mode of data collection, sample frame, and nonresponse are closely interconnected, perhaps more so for the measurement of persons with disabilities than for other measures. For example, if one is interested in conducting a telephone survey, then the sampling frame is not all households but rather households with telephones. Persons living in households without telephones will have a zero probability of selection. To the extent that individuals who live in households without telephones differ from those who live in households with telephones, the resulting estimates will suffer from coverage error. However, the choice of the telephone as the mode of data collection further assumes that within the households with telephones there is at least one person who can hear well enough to participate.
Findings from the National Comorbidity Study were discussed as illustrative of the interplay among decisions concerning mode of data collection and potential nonresponse error. This study was designed to measure the prevalence of disabling psychotic disorders. The survey was conducted as a household-based face-to-face survey, viewed as the best mode of data collection. The survey achieved a respectable response rate of 85 percent. That meant, however, that 15 percent of the sample selected for participation did not participate, and the prevalence of this type of disability is estimated to be about 1 percent. To address this issue, the investigators surveyed 20 percent of the nonrespondents. They were offered incentives, alternative modes of data collection (e.g., telephone), or alternative locations for data collection (e.g., public place). The nonresponse study found relatively high rates of anxiety disorders among those respondents who were willing to complete the nonresponse interview but who were not willing to have a stranger in their homes. The findings suggest that, for this particular study, the choice of mode of data collection was related to the nonresponse error. Permitting flexible modes of data collection may have resulted in a reduction of the nonresponse rate and a reduction of the nonresponse error.
Workshop participants also discussed the following topics:
Cognitive psychology provides a theoretical framework for addressing the processes involved in the question-answer interchange between an interviewer and a respondent. This framework is informative with respect to where the process may be problematic for the measurement of persons with disabilities. The question-answer process is most often categorized as a four-stage process involving comprehending the question, retrieving the information, making a judgment concerning whether the retrieved information is relevant, and formulating a response. Given the complexity of the concept of interest, workshop participants indicated that comprehension represented the largest challenge in the development of effective questions to measure disability and work disability. Survey questions are probably most effective in screening out the large number of persons without disabilities as well as identifying persons who are current beneficiaries of Supplemental Security Income or Social Security Disability Insurance. Questionnaires were viewed as most problematic in identifying disabled persons who currently received no benefits and who may or may not be employed.
Potential comprehension problems associated with the measurement of persons with work disabilities were illustrated by using questions scheduled to be included in the long form of the decennial census for the year 2000. The proposed question is "Because of a physical, mental, or emotional condition lasting 6 months or more, does this person have any difficulty in doing any of the following activities? Working at a job or business?" The response categories are yes or no. The question includes at least three distinct elements: (1) a causal connection between the disability and an underlying medical or psychological condition; (2) the condition is enduring; and (3) the condition results in an inability to do something, in this specific case, the inability to work. To answer the question the respondent must determine what is meant by work, must determine whether the inability to work meets the criteria related to the time element, and must attribute the inability to work to an underlying condition. In addition, the question involves an implicit comparison to a standard, for example, the ability or inability to perform some set of tasks associated with a particular job.
The complexity of the question contributes to problems of comprehension. The empirical literature indicates that many of the questions designed to measure disability are subject to context effects, providing evidence of comprehension problems. Workshop participants called for research in which the reliability of complex multidimensional questions designed to measure persons with disabilities is compared with the reliability of a series of shorter items. The hypothesis is that the use of several less complex questions may improve comprehension with little or no additional administration time for the interview.
Many survey questions that attempt to measure disability include a dimension of duration (e.g., "lasting six months or more"). For some questions, duration is expressed in the past tense; other items express duration either in the past tense or as a duration that is expected to continue for a specific time period into the future. Although the intention of including a dimension of duration is to eliminate short-term disabilities (e.g., a broken leg that will limit the respondent for the next 6 weeks), workshop participants expressed concerns as to respondents' understanding of the dimension of duration as well as concern with the determination of an appropriate duration interval. This was of particular importance for the measurement of persons with intermittent disabilities. Given the dynamic nature of disability, what is the appropriate time period to be used in the survey question? Workshop participants reiterated the need for research related to respondents' comprehension of the questions designed to measure work disability. For example, among persons with episodic mental disabilities, questions of duration become quite difficult to answer. Should the respondent report with respect to date of first onset, total duration of episodes, or duration of most recent episode?
What information concerning disability can and cannot be retrieved during the course of an interview? For many kinds of questions dealing with autobiographical memory, retrieval of the information is the most difficult task facing the respondent. This is most likely not the case with respect to work disabilities. Persons with disabilities and their immediate families are very likely to be aware of a disability if there is one and to be able to retrieve it from memory with little difficulty, if the respondent understands what is being asked. Similarly, the process of making a judgment concerning the retrieved information or formulating a response does not present the respondent with cognitively difficult tasks, but, rather, presents the respondent with tasks that may have implications with respect to issues of social desirability. For example, in a setting in which a respondent is applying for disability benefits, there may be a strong motivation to edit responses toward disability. In household interviews, on the other hand, individuals may wish to deny or minimize their disabilities. Workshop participants suggested the need to conduct research on how to reduce question threat through the use of self-administered modes of data collection, including the use of audio computer-assisted self-interviewing procedures.
The nature of the phenomena of interest implies that information often will be obtained from proxy respondents, that is, one person reporting for another individual. Although there is a small body of empirical literature indicating that responses concerning functional assessments or disabilities obtained by proxy differ from those obtained from a respondent reporting for him- or herself, the direction and magnitude of the response error are not well understood. For items perceived to be threatening, responses obtained by proxy may be subject to lower levels of response error than responses obtained from the target individual.
The discussion resulted in the identification of several additional research activities related to questionnaire development. These include the following:
The literature offers at least two frameworks for the measurement of the impact of environment on participation among persons with disabilities. The Quebec model (Fougeyrollas, 1998) focuses on obtaining ratings of the individual's life habits, for example, determining the level of accomplishment for a certain activity (from no difficulty to not performed) or ascertaining the type of assistance that a person uses, including assistive technology, environmental modification, and personal assistance. Respondents are also asked to rate the environment, where environment includes such factors as the physical environment, social attitudes, and rules related to the workplace.
An alternative framework is offered by the ICIDH-2, which can be used to classify the environment on several levels, including the personal level (e.g., the immediate environment of the person including, but not limited to, the home, school, and workplace), the community level (including economic and social institutions), the broader cultural and social customs and structures, the physical, human-made environment, and the natural environment.
To measure environment, researchers need to have a shared understanding of activities and participation, using the ICIDH-2 vernacular. Activity, sometimes referred to as "naked capacity," is the ability to perform an activity without any kind of technology or personal assistance. In measuring activity, one could consider a continuum ranging from "extremely well" to "not at all." Performance could then be defined as performance of activities with all types of assistance. Respondents could be asked whether they do the activity with any type of assistance and, if yes, what kind of assistance. This type of question sequence would allow researchers to understand the types of assistance or environmental accommodation that facilitates the performance of an activity. Such an approach allows people an opportunity to express their views of their level of performance using available assistance and technology.
Environment also alters an individual's perception of disability. For example, the number of applications for SSI and SSDI benefits often varies as a function of the economy, suggesting that individual perceptions may, in part, be a function of the labor force options available. The passage of the Americans with Disabilities Act most likely will change the social climate concerning disabilities and entitlements of persons with disabilities; hence, over time, there may be an increase in the number of individuals who classify themselves as disabled.
Workshop participants expressed concern about the validity of the perspective of the respondent as reporter of the environment, as well as the validity of reports of the environment provided by proxy respondents. To alleviate these concerns, participants recommended comparison of both self-reports and proxy reports of the environment with unbiased measures of the environment to provide empirical data on the validity of the reports by the two types of respondents. For example, work environments could be sampled by trained observers who rate the environment and by comparisons of those ratings with reports obtained from self-reporters and proxy respondents.