Slide
1 Measurement
issues
Ken Black
Slide
2 Survey Cycle
Slide
3 Selected issues
in instrument development
Slide
4 Instrument
development
- Disability is a complex,
multidimensional concept
- It is perceived differently
by different people, especially across age
groups, cultures and time
- A good survey instrument
overcomes the subjectivity and individual
perceptions of disability and produces measures
which are:
- valid (measure
what they say they measure) and
- reliable (give
consistent results over repeated measures)
Slide
5 Instrument
development
- Consistency with the
ICF important
- Data items and questions
can attempt to cover a broad range of domains,
or just specific ones, depending upon client
needs
- Once domains chosen,
need to consider whether to attempt to measure
the whole domain, or just parts of it.
- For example, if
decide to include limitations in activities
related to mobility, will it include:
- changing and maintaining body position
- carrying, moving and handling objects
- walking and moving,
- moving around using transportation?
- How can this concept
be described to respondents in a simple
manner?
Slide
6 ADL and IADL
- The concepts of Activities
of Daily Living (ADL) and Instrumental Activities
of Daily Living (IADL) are sometimes used
to classify types of disability
- ADLs can broadly be seen
as the activities:
- Communication
- Self-care
- Mobility
- IADLs can broadly be
seen as the more complex activities:
- domestic activities
- home maintenance
and gardening
Slide
7 Testing and
validation
- Development of questionnaires
can take between 12 to 18 months depending
on the topics and the amount of testing required
to assure quality
- Two broad categories
of testing:
- Pre-Testing
- Field Testing
Slide
8 Pre-testing
of survey instruments
- Pre-test properties:
- Small, targeted
sample
- Occurs before field
tests
- Controlled environment
- Qualitative
analysis
- Pre-tests focus on:
- the interviewers
administration of the questions, and
- the respondents
ability to answer questions without error
- Two main types of pre-tests:
- Focus groups and
- Cognitive testing
Slide
9 Aims of pre-testing
survey instruments
- The main aim of pre-testing
is to identify the sources of potential non-sample
error, including:
- the effect the
interviewer has on question responses;
- error the respondent
makes due to the questionnaire;
- errors due to the
method of delivery or errors caused by
sequencing; and
- errors caused by
the interaction of these factors
- Pre-testing can also:
- reduce survey development
time
- reduce survey costs
- improve the quality
of data collected
- Ideally all survey development
programs should incorporate some degree of
pre-testing
Slide
10 Focus Groups
- Used to explore respondent
understanding of, and reaction to terms, concepts
and issues
- Moderator led, informal
small group discussion
- Moderator remains neutral
at all times
- Several groups may be
necessary to represent different population
subgroups
Slide
11 Cognitive
interviews
- Cognitive interviews
- ask respondents
questions about the survey questions
- are conducted one
to one with an interviewer in a standardised
environment, usually taped
- Cognitive interviews
identify problems in:
- question comprehension
- memory recall
- selecting responses
- interpretation
of reference periods
- reactions to sensitive
questions
- Cognitive interviews
can use:
- retrospective and
concurrent probes
- think alouds
- paraphrasing
- vignettes
- card sorting tasks
Slide
12 Field testing
- Two broad types:
- Pilot tests
- Dress rehearsals
- Objectives are to test:
- questions / instrument
design
- written instructions
for interviewers
- interviewer training
manuals and delivery of training
- timing / costing
of the survey
- processing systems
- Sample size of about
300 households sufficient
- 8 to 12 interviewers
sufficient
Slide
13 Field Testing
- Pilot testing
- Small random sample
to test
- identification of populations
- understanding of questions (broad)
- question sequencing and overall
survey flow
- interviewer instructions
- respondent reaction to questions
- editing procedures
- Dress rehearsal
- Final test covering:
- interview times
- training of interviewers
- field procedures
- Debriefing reports
- Subjective, anecdotal,
based interviewer feedback
Slide
14 Context and
mode effects
Disability prevalence rates vary according
to type and mode of collection used to collect
information
- Disability rates were
higher in ABS collections when disability
questions were asked after detailed health
questions (such as the SF36).
- Disability rates generally
lower in self-completion forms compared to
personal interviews
- General disability rates
also lower where proxy interviewing allowed
Slide
15 Sample design
- Samples are generally
designed by statistical methodology experts
- Sample size is determined
- to achieve estimates
of desired accuracy (sampling error) or
- the sample that
can be achieved with fixed budget
- Sample size needed depends
upon level of accuracy required for particular
data items at specified levels
- For example, a quality estimate of
number of people with an employment/school
participation restriction at the State
level
- Sample frame
- area
- list
- combination of
both
Slide
16 Selected collection
issues to consider:
- Proxy vs non-proxy interviews
- The strong preference
is to interview respondents about themselves,
however in some cases, proxy interviews
might be necessary, for example
- Personal versus telephone
interviews
- Special questions versus
questionnaires for sub-populations
Slide
16 Selected
collection issues to consider:
- Methods to improve collection
of information from people with disabilities
- interviewer training
- letters to respondents
- survey brochures
(outline the need for the information
and encourage cooperation
- support from advocacy
groups
Slide
18 Interviewer
training
- ABS disability survey
training sessions were three days for all
interviewers
- Interviewers already
well trained in general interviewing procedures,
including computer assisted interviewing
- Training covered:
- major users and
uses
- structure of the
questionnaire
- main survey populations
- questions and sequencing
- coding of conditions
- practice interviewing
- issues relating
to interviewing people with disabilities
Slide
19 Interviewing
people with disabilities
- Understand that there
are myths and stereotypes about people with
disabilities, but many of them are not true!!
- Always interview in a
professional and sensitive manner
- Language is important
- Needs to be acceptable
to people with disabilities, for example,
neutral and avoiding terms like disability,
handicap, impairment etc.
- Needs to be accurate
and respectful of individuals with disabilities
Slide
20 Interviewing
people with disabilities
- Cultural issues
- This is a very
difficult area
- Being considered
by the Washington Group on disability
statistics
- Sometimes there
is no overall word for "disability"
- Reluctance in some
cultures to identify people in their household
with a disability
- Ask all questions
- Do not assume or anticipate
answers
- Be patient and let the
respondent finish
Slide
21 Data processing
and checking
- Different for the different
collection methods such as Computer Assisted
Interviewing (CAI) or paper questionnaires
- Create output or derived
items
- Editing
- input (done as
part of CAI instrument, or when punched
or captured electronically)
- Validation
- of all output items
- compared with other
data sources, previous data etc
Slide
22 Tabulation
- Determining the table
content should be specified early in the development
or collection phases of the survey
- Publish output
- based on key user
needs
- showing the most
important information
- demonstrating the
range of data available
Slide
23 Consider
range of important tabulations. Issues include:
- Frequency and distribution
of disability in the population
- Prevalence of different
types of disability, such as, learning, communicating
etc,
- Underlying cause of the
disabilities in the population
- Severity and estimated
duration of the disability
- Needs of people with
disabilities in terms of
- aids and equipment,
assistance, services
- met and unmet needs
- Quality of life issues/comparisons
such as, living conditions health, social
and economic conditions
- Barriers to full and
equal participation in society in areas such
as education, work, housing, transport etc.
Slide
24 Dissemination
methods
- Methods include:
- Paper publication
- Published on Website
- Electronic tables/spreadsheets
- Microdata
- Promote survey data releases
through:
- Media releases
- Seminars
- Analytical articles
etc
Slide
25 Evaluation
- Post survey review
- Review all aspects
of the survey
- Use interviewer
feedback on issues relating to the questions,
response categories, procedures etc.
- Survey processing
staff feedback on issues of processing
data items, validation etc.
- Key user feedback
on whether survey data met their information
needs. Identify:
- Gaps
- Deficiencies in collected items
- Data quality analyses
Slide
26 Types of pretesting
(drop??)
- Literature reviews
- database and library
searches
- other statistical
agencies and organisations
- Expert reviews
- Focus groups
- Cognitive testing
- Interviewer debriefing
- Observational interviews
- Behavior coding
|
About Us | Media Centre | Members | Programmes | Documents | Publications | Jobs | 
Bangkok, Thailand