Measurement issues

Ken Black

Survey Cycle

Selected issues in instrument development

Instrument development

• Disability is a complex, multidimensional concept
• It is perceived differently by different people, especially across age groups, cultures and time
• A good survey instrument overcomes the subjectivity and individual perceptions of disability and produces measures which are:
  • valid (measure what they say they measure) and
  • reliable (give consistent results over repeated measures)

Instrument development

• Consistency with the ICF important
• Data items and questions can attempt to cover a broad range of domains, or just specific ones, depending upon client needs
• Once domains chosen, need to consider whether to attempt to measure the whole domain, or just parts of it.
  • For example, if decide to include limitations in activities related to mobility, will it include:
    • changing and maintaining body position
    • carrying, moving and handling objects
    • walking and moving,
    • moving around using transportation?
  • How can this concept be described to respondents in a simple manner?

ADL and IADL

• The concepts of Activities of Daily Living (ADL) and Instrumental Activities of Daily Living (IADL) are sometimes used to classify types of disability
• ADLs can broadly be seen as the activities:
  • Communication
  • Self-care
  • Mobility
• IADLs can broadly be seen as the more complex activities:
  • domestic activities
  • home maintenance and gardening

Testing and validation

• Development of questionnaires can take between 12 to 18 months depending on the topics and the amount of testing required to assure quality
• Two broad categories of testing:
  • Pre-Testing
  • Field Testing

Pre-testing of survey instruments

• Pre-test properties:
  • Small, targeted sample
  • Occurs before field tests
  • Controlled environment
  • Qualitative analysis
    
• Pre-tests focus on:
  • the interviewers administration of the questions, and
  • the respondents ability to answer questions without error
• Two main types of pre-tests:
  • Focus groups and
  • Cognitive testing

Aims of pre-testing survey instruments

• The main aim of pre-testing is to identify the sources of potential non-sample error, including:
  • the effect the interviewer has on question responses;
  • error the respondent makes due to the questionnaire;
  • errors due to the method of delivery or errors caused by sequencing; and
  • errors caused by the interaction of these factors
• Pre-testing can also:
  • reduce survey development time
  • reduce survey costs
  • improve the quality of data collected
• Ideally all survey development programs should incorporate some degree of pre-testing

Focus Groups

• Used to explore respondent understanding of, and reaction to terms, concepts and issues
• Moderator led, informal small group discussion
• Moderator remains neutral at all times
• Several groups may be necessary to represent different population subgroups

Cognitive interviews

• Cognitive interviews
  • ask respondents questions about the survey questions
  • are conducted one to one with an interviewer in a standardised environment, usually taped
• Cognitive interviews identify problems in:
  • question comprehension
  • memory recall
  • selecting responses
  • interpretation of reference periods
  • reactions to sensitive questions
• Cognitive interviews can use:
  • retrospective and concurrent probes
  • think alouds
  • paraphrasing
  • vignettes
  • card sorting tasks

Field testing

• Two broad types:
  • Pilot tests
  • Dress rehearsals
• Objectives are to test:
  • questions / instrument design
  • written instructions for interviewers
  • interviewer training manuals and delivery of training
  • timing / costing of the survey
  • processing systems
• Sample size of about 300 households sufficient
• 8 to 12 interviewers sufficient

Field Testing

• Pilot testing
  • Small random sample to test
    • identification of populations
    • understanding of questions (broad)
    • question sequencing and overall survey flow
    • interviewer instructions
    • respondent reaction to questions
    • editing procedures
• Dress rehearsal
  • Final test covering:
    • interview times
    • training of interviewers
    • field procedures
• Debriefing reports
  • Subjective, anecdotal, based interviewer feedback

Context and mode effects

Disability prevalence rates vary according to type and mode of collection used to collect information

• Disability rates were higher in ABS collections when disability questions were asked after detailed health questions (such as the SF36).
• Disability rates generally lower in self-completion forms compared to personal interviews
• General disability rates also lower where proxy interviewing allowed

Sample design

• Samples are generally designed by statistical methodology experts
• Sample size is determined
  • to achieve estimates of desired accuracy (sampling error) or
  • the sample that can be achieved with fixed budget
• Sample size needed depends upon level of accuracy required for particular data items at specified levels
  • For example, a quality estimate of number of people with an employment/school participation restriction at the State level
• Sample frame
  • area
  • list
  • combination of both

Selected collection issues to consider:

• Proxy vs non-proxy interviews
  • The strong preference is to interview respondents about themselves, however in some cases, proxy interviews might be necessary, for example
    • where respondents are not competent to provide information about themselves
    • for children
    • in screening questions (for cost reasons)
• Personal versus telephone interviews
• Special questions versus questionnaires for sub-populations

Selected collection issues to consider:

• Methods to improve collection of information from people with disabilities
  • interviewer training
  • letters to respondents
  • survey brochures (outline the need for the information and encourage cooperation
  • support from advocacy groups

Interviewer training

• ABS disability survey training sessions were three days for all interviewers
• Interviewers already well trained in general interviewing procedures, including computer assisted interviewing
• Training covered:
  • major users and uses
  • structure of the questionnaire
  • main survey populations
  • questions and sequencing
  • coding of conditions
  • practice interviewing
  • issues relating to interviewing people with disabilities

Interviewing people with disabilities

• Understand that there are myths and stereotypes about people with disabilities, but many of them are not true!!
• Always interview in a professional and sensitive manner
• Language is important
  • Needs to be acceptable to people with disabilities, for example, neutral and avoiding terms like disability, handicap, impairment etc.
  • Needs to be accurate and respectful of individuals with disabilities
    • Examples

Interviewing people with disabilities

• Cultural issues
  • This is a very difficult area
  • Being considered by the Washington Group on disability statistics
  • Sometimes there is no overall word for "disability"
  • Reluctance in some cultures to identify people in their household with a disability
• Ask all questions
• Do not assume or anticipate answers
• Be patient and let the respondent finish

Data processing and checking

• Different for the different collection methods such as Computer Assisted Interviewing (CAI) or paper questionnaires
• Create output or derived items
• Editing
  • input (done as part of CAI instrument, or when punched or captured electronically)
• Validation
  • of all output items
  • compared with other data sources, previous data etc

Tabulation

• Determining the table content should be specified early in the development or collection phases of the survey
• Publish output
  • based on key user needs
  • showing the most important information
  • demonstrating the range of data available

Consider range of important tabulations. Issues include:

• Frequency and distribution of disability in the population
• Prevalence of different types of disability, such as, learning, communicating etc,
• Underlying cause of the disabilities in the population
• Severity and estimated duration of the disability
• Needs of people with disabilities in terms of
  • aids and equipment, assistance, services
  • met and unmet needs
• Quality of life issues/comparisons such as, living conditions health, social and economic conditions
• Barriers to full and equal participation in society in areas such as education, work, housing, transport etc.

Dissemination methods

• Methods include:
  • Paper publication
  • Published on Website
  • Electronic tables/spreadsheets
  • Microdata
• Promote survey data releases through:
  • Media releases
  • Seminars
  • Analytical articles etc

Evaluation

• Post survey review
  • Review all aspects of the survey
  • Use interviewer feedback on issues relating to the questions, response categories, procedures etc.
  • Survey processing staff feedback on issues of processing data items, validation etc.
  • Key user feedback on whether survey data met their information needs. Identify:
    • Gaps
    • Deficiencies in collected items
• Data quality analyses
• Non-response evaluation

Types of pretesting (drop??)

• Literature reviews
  • database and library searches
  • other statistical agencies and organisations
• Expert reviews
• Focus groups
• Cognitive testing
• Interviewer debriefing
• Observational interviews
• Behavior coding