Analytic Quality Glossary
A B C D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
Citation
reference: Harvey, L., 2004, Analytic Quality Glossary, Quality
Research International, http://www.qualityresearchinternational.com/glossary/
This is a dynamic glossary and the author would welcome any e-mail
suggestions for amendments or additions.
____________________________________________________________________
core definition
Performance indicators are data, usually quantitative in form, that
provide a measure of some aspect of an individualÕs or organisationÕs
performance against which changes in performance or the performance of others
can be compared.
explanatory context
Although
performance indicators have relatively precise meaning the term has grown to
mean any statistical data related to the activity of higher education
institutions, whether or not they really do throw any light on performance.
Furthermore, a decade ago, Yorke (1995, p. 15) noted a tendency for performance
indicators to be collected irrespective of the policy framework within which they are
to be used Ñ this was particularly evident in the United Sates in the late
1980s and early 1990s.
analytical review
CHEA (2001) defines performance indicators as:
Representations (usually numeric) of the state of,
or outcome from, an education organization, its programs, or processes.
Sometimes called "management indicators." Regarded as a set of
tangible measures designed to provide public accountability. Often includes
admission and graduate data, research records, employment of graduates, cost
per student, student/staff ratios, staff workloads, student relevance, class
size, laboratory and other equipment, equity, libraries, information
technology, and other learning resources. Should be subject to informed
interpretation and judgment.
The UNESCO definitions is:
Performance Indicators: A range of statistical
parameters representing a measure of the extent to which a higher education
institution or a programme is performing in a certain quality dimension. They
are qualitative and quantitative measures of the output (short-term measures of
results) or of the outcome (long-term measures of outcomes and impacts) of a
system or of a programme. They allow institutions to benchmark their own
performances or allow comparison among higher education institutions. (Vl‹sceanu
et al., 2004, p. 39)
There is considerable variation in the use of performance indicators in
quality review (Cave et al., 1997). Woodhouse (1999, p. 33) notes that:
Most
commonly, institutions are invited to specify their performance indicators,
indicating why and how they use them. The external quality review agency,
through its independent review team, then forms its own interpretation of the
results. In other systems, however, higher education institutions are expected
to report against a system-wide set of performance indicators, which are then
available to the external quality review process.
Vl‹sceanu
et al.
(2004) argue that:
Performance
indicators work efficiently only when they are used as part of a coherent set
of input, process, and output indicators. As higher education institutions are
engaged in a variety of activities and target a number of different objectives,
it is essential to be able to identify and to implement a large range of
performance indicators in order to cover the entire field of activity. Examples
of frequently used performance indicators, covering various institutional
activities, include: the number of applications per place, the entry scores of
candidates, the staff workload, the employability of graduates, research grants
and contracts, the number of articles or studies published, staff/student ratio,
institutional income and expenditure, and institutional and departmental
equipment and furniture. Performance indicators are related to benchmarking
exercises and are identified through a specific piloting exercise in order to
best serve their use in a comparative or profiling analysis. (p. 40)
Kells (1993, p.
7), in a study of 12 OECD countries noted:
the primary initiative and source of interest in
performance indicators remains the government agencies and ministerial
officials who are responsible for higher education
The Linke Report (1991), in an early study, identified the following
performance indicators:
Institutional context:
Equivalent full-time academic staff; Student/staff ratio; Average
student entry score; Academic activity cost per student
Performance in teaching and learning
Student progress rate; Mean completion time; Research higher
degree productivity rate; Perceived teaching quality (from CEQ)
Performance in research and professional service
Number of research grants; Value of research grants; Average
publication rate; Professional service activity
Participation and social equity
Academic staff gender ratio; Commencing student gender ratio;
Academic programme diversity.
In the early
1990s there was much research on Ôperformance indicatorsÕ, most of which
suggested that statistical indicators, whether reliable or not, are rarely
valid
operationalisations of quality (Klein & Carter, 1988; Cave & Kogan,
1990; Goedegebuure et al.,
1990; Head, 1990; Johnes & Taylor, 1990; Pollitt, C, 1990; Cave et al., 1991, Gallagher, 1991; Yorke, 1991;
Murphy, 1994). Furthermore, despite being ÔindicatorsÕ it is unclear, exactly,
of what performance they are indicative.
What,
for example, does an increase in percentage of ÔgoodÕ degree classifications
tell us about quality? Does it indicate that the student learning performance
has improved? Does this mean that the teaching staff have performed better, or
are the students learning more despite the teachers? Or does it mean that
academic standards have fallen? Similarly, what does the employment rate of
graduates within the first six months after graduation tell us about the
performance of the institution? Perhaps it says more about the vagaries of the
recruitment process and the differential in take-up rates between different
subject specialisms than provide any indication of the performance of the
institution.
Harvey
(1998, p. 243) argues that, in practice, performance indicators are usually
simplistic, convenience measures that bear no relation to any notion of
quality.
Yorke (1998)
suggested that the benefit that might accrue from improving statistical
measures to make them into really meaningful performance indicators is
outweighed by the cost that would accrue.
sources
Cave, M. and Kogan, M., 1990, ÔSome
concluding observationsÕ, in M. Cave, M. Kogan, & R. Smith (Eds.), 1990, Output
and Performance Measurement in Government: The state of the art, pp. 179Ð87 (London, Jessica Kingsley).
Cave,
M., Hanney, S. and Kogan, M., 1997, ÒThe Use of Performance Indicators in Higher
EducationÓ, 3rd ed., Higher Education Policy, Series 3, Jessica Kingsley
Publishers.
Council For Higher Education
Accreditation (CHEA) 2001, Glossary
of Key Terms in Quality Assurance and Accreditation, http://www.chea.org/international/inter_glossary01.html,
updated 8 May 2001.
Gallagher, A., 1991, ÔComparative value
added as a performance indicatorÕ, Higher Education Review, 23 (3), pp. 19Ð29.
Goedegebuure, L.C.J., Maassen, P.A.M.
& Westerheijden, D.F. (Eds.), 1990, Peer Review and Performance
Indicators: Quality Assessment in British and Dutch Higher Education (Culemborg, Lemma).
Harvey, L., 1998. ÔAn assessment
of past and current approaches to quality in higher educationÕ, Australian
Journal of Education,
42(3), pp. 237Ð255.
Head, P., 1990, Performance
Indicators and Quality Assurance. Information Services Discussion paper, 4 June, 1990. (London,
CNAA.)
Johnes, J. and Taylor, J., 1990, Performance
Indicators in Higher Education.
(Buckingham, Society for Research into Higher Education (SRHE)/Open University
Press).
Kells, H.R. (Ed.), 1993, The Development of Performance
Indicators for Higher Education: a compendium for eleven countries. 2d ed. Paris: Organization
for Economic Cooperation and Development. ED 331 355..
Klein, R. and Carter, N., 1988,
ÔPerformance measurement: a review of concepts and issuesÕ, in D. Beeton (Ed.),
1988, Performance Measurement: Getting the concepts right, (London, Public Finance Foundation).
Linke Report (1991) Performance Indicators in Higher Education.
Report of a Trial Evaluation Study Commissioned by the Commonwealth Department
of Employment, Education and Training, Volumes I and II, AGPS: Canberra.
Murphy, P., 1994, ÔResearch quality, peer
review and performance indicatorsÕ, The Australian Universities Review, 37 (1), pp. 14Ð18.
Pollitt, C., 1990, ÔMeasuring
university performance: never mind the quality, never mind the widthÕ, Higher Education
Quarterly, 44 (1):, pp.
60Ð81.
Vl‹sceanu, L., GrŸnberg, L., and P‰rlea, D.,
2004, Quality Assurance and
Accreditation: A Glossary of Basic Terms and Definitions (Bucharest, UNESCO-CEPES)
Papers on Higher
Education, ISBN 92-9069-178-6. http://www.cepes.ro/publications/Default.htm
Woodhouse, D., 1999, ÔQuality
and Quality AssuranceÕ in Organisation for Economic Co-Operation and
Development (OECD),, 1999, Quality and Internationalisation in Higher
Education, pp. 29Ð44,
Programme on Institutional Management in Higher Education (IMHE), Paris, OECD.
Yorke, M., 1991, Performance
Indicators: Observations on their use in the assurance of course quality, Council for National Academic Awards
Project Report 30, January (London, CNAA).
Yorke, M., 1995,
'Taking the odds-on chance: Using performance indicators in managing for the
improvement of quality in higher education', Tertiary Education and
Management, 1(1),
pp. 49Ð57.
Yorke, M., 1998, ÔPerformance
indicators relating to student development: can they be trusted?Õ Quality in
Higher Education, 4(1), pp. 45Ð61.