Analytic Quality Glossary

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Citation reference: Harvey, L., 2004, Analytic Quality Glossary, Quality Research International, http://www.qualityresearchinternational.com/glossary/

This is a dynamic glossary and the author would welcome any e-mail suggestions for amendments or additions.

____________________________________________________________________

Peer Review

core definition

Peer review is the process of evaluating the provision, work process, or output of an individual or collective operating in the same milieu as the reviewer(s).

explanatory context

Operating in the same milieu’ may be a work colleague, or someone in a similar academic discipline either as a teacher or researcher, or (rather more widely) someone also engaged in higher education.

analytical review

The Freedictionary.com (2004) the Hyperdictionary (2004) and Websters On-line (2004) all define peer review as a verb:

to evaluate professionally a colleague’s work

Peer review is, thus, originally a term in academia that relates to establishing a scholarly standard for a discipline. Thus:

Peer review (known as refereeing in some academic fields) is a scholarly process used in the publication of manuscripts and in the awarding of money for research. Publishers and agencies use peer review to select and to screen submissions. At the same time, the process assists authors in meeting the standards of their discipline. Publications and awards that have not undergone peer review are liable to be regarded with suspicion by scholars and professionals in many fields. (Free definition, 2004)

The Parliamentary Office of Science and Technology (2002) in Post Notes is concerned with peer review in the refereeing sense:

Peer review is a system whereby research — or a research proposal — is scrutinised by (largely unpaid) independent experts (peers). In general, the process serves a technical (ensuring that the science is sound) and a subjective function (is the science interesting, important and/or groundbreaking?).

Peer review is used in the UK for three main purposes:

· Allocation of research funding. The main funding bodies such as the research councils and biomedical charities all use peer review for advice on which research projects should be funded in the first place and to assess the progress of funded projects....

· Publication of research in scientific journals. Peer review is used to assess the quality of research submitted for publication and to assess its importance. The process thus influences what science enters the public domain, where it is published and what impact it will have (the more prestigious the journal, the greater the likely impact of the publication).

· Assess the research rating of university departments. Peer review has been used as part of the Research Assessment Exercise (RAE) to judge the quality of research conducted by each department. The results are used to direct the distribution of public funds (£5 billion following the 2001 RAE) to each institute. In addition to the above, peer reviewed science is playing an increasingly influential role in the formulation of UK policy and decision making.

Peer review is designed to improve the quality of research reporting and to prevent poor research from taking place. It is generally regarded as having the confidence of the research community. Processes such as the RAE are widely accepted as having raised standards, but there is surprisingly little evidence on the effectiveness of peer review from formal studies. One recent review found some evidence that the accuracy and readability of manuscripts is improved between submission and publication, although it was not clear whether this was due to peer review, or to technical editing. There is also some evidence that it is effective at weeding out poor quality research both at funding and at publication. In general, peer review is held to be beneficial to the scientific community and has become central to the process by which science is conducted.

Bruce (1997) defines peer review as a collegial self-improvement process::

Peer review is a staff development process that is widely used in training and other professional contexts. The basic idea is that the person who is concerned about some aspect of their own work invites a colleague to review the quality of what he or she is doing. In practice we are doing this all the time. It is very common for someone to say - ‘Do you have a little time to tell me what you think of this?’; or to ask - ‘Has anyone thought of a better way of doing…?’

When an individual chooses to formalise this process, strategies must be implemented so that maximum benefit is gained. The usual framework is for the two individuals concerned to sit down together and discuss what the reviewee is interested in receiving collegial feedback about. The person who has asked for the review explains what they are doing, and what they would like their ‘critical friend’ to look for. The critical friend asks any questions that may be required for clarification. The next step is for the critical friend to ‘observe’ whatever her colleague wants her to review. This may be a ‘live’ event such as a class or a client interaction, or it may be a recording of such an event, or perhaps a package of some kind. It is very important that the person being reviewed also takes the time to critique or reflect on the event herself.

At the University of Queensland (2002), peer review of teaching is seen in formal terms:

Peer review of teaching occurs whenever one (or more) peer(s) observes, examines, discusses, analyses, dissects, or just talks about the teaching practices of a colleague, with that colleagues express consent and blessing.

Peer review can be conducted to rate a colleague’s performance against standards or criteria, but here at UQ peer review is a way of getting and giving structured feedback to each other in order to improve teaching practices. The name ‘peer review’ sometimes sounds too much like an assessment of performance, so in some departments and faculties peer review is called ‘peer development’. Peer review can be done in pairs, or in small teams. Either way there are important things to keep in mind about how peer review ought to be conducted. Some of these are discussed below.

In the context of quality evaluation processes, peer review is one of the core methods most often used in assuring, assessing, auditing or otherwise checking quality. Most accreditation, quality assessment and quality audit approaches include peer review teams who undertake on-site visits. CHEA (2001) defines peer review in these terms:

External review and evaluation of the quality and effectiveness of an institution’s academic programs, staffing, and structure, carried out by a team of external evaluators who are specialists in the fields reviewed and knowledgeable about higher education in general. Reviews may be based on standards set by the accrediting organizations or on quality standards set more broadly.

The UNESCO definition of peer review is:

Assessment procedure regarding the quality and effectiveness of the academic programmes of an institution, its staffing, and/or its structure, carried out by external experts (peers). (Strictly speaking, peers are academics of the same discipline, but in practice, different types of external evaluators exist, even though all are meant to be specialists in the field reviewed and knowledgeable about higher education in general.) The review may [also] vary the source of authority of peers, types of peers, their selection and training, their site visits, and the standards to be met. A review is usually based on a self-evaluation report provided by the institution and can itself be used as a basis for indicators and/or as a method of judgment for (external) evaluation in higher education.

(Vlãsceanu, et al., 2004, p. 44)

The UK Research Assessment Exercise also uses peer reviewers, although they do not undertake visits but review research outputs. The external examiner system, used in many countries is another form of peer review.

Woodhouse (1999, p.32) notes that peer review is:

a term with a long tradition in academia, and it has usually denoted an evaluation by another academic or academics, usually in the same discipline (Frederiks et al., 1993). Increasingly, the membership of quality review teams is not restricted in this way, and in many systems they now include people outside academia and people from other countries [e.g. Denmark, Hong Kong, China, New Zealand, the European pilot projects]. Since, to the world outside academia, the term “peer review” has rather cosy connotations, it may be better to drop it in favour of, for example, “independent review”.

Diana Green (1994) had argued that:

Peer review ‘is an umbrella term used indiscriminately to describe all the methods that involve human judgement; whether or not the judgements are informed by less subjective data and irrespective of whether those making the judgments are peers. (Green, 1994, p. 11)

In the same volume, Malcolm Fraser defined peer review as:

The involvement of people as active university teachers, as researchers or as practicing professionals to offer advice and to make judgements and/or decisions about proposals for new programmes, the quality of research programmes, the continuation or modification of existing programmes, the quality of institutions is described as a peer review. All the countries mentioned in this paper use peer review to some extent in their processes for quality maintenance and enhancement in higher education. (Fraser, 1994, p. 107)

The Open University Validation Service (2004) discusses peer review in the context of validating submissions. ‘Peer review’ means that:

all submissions are considered by specially convened panels of expert advisers drawn, as appropriate, from the fields of quality assurance, senior management, teaching in higher education, commerce, industry, the professions or subject expertise.

The aim is to provide a panel that can offer a range of perspectives, take a critical but sensitive approach and make suitable comparisons with standards and practice elsewhere in higher education in the UK. The proposed membership of accreditation and validation panels will be discussed with the institution.

associated issues

Issues raised by peer review [refereeing].

Post Notes (Parliamentary Office of Science and Technology, 2002, prepared by Harding) suggested the following issues arise with peer review of papers and proposals for funding:

1. Fraudulent research: Different types of fraud Peer review relies on mutual trust and honesty: researchers must entrust their data/ideas to referees while referees must trust that researchers are telling the truth. Because of this reliance on trust, the peer review system is open to abuse. Recent years have seen a small number of high profile cases where the system has failed to detect fraudulent research, although these cases are thought to account for only a tiny proportion of peer reviewed research. Fraudulent research can take a number of forms including:

· Fabrication – where data or cases in manuscripts submitted for publication are simply invented…

· Falsification – where data in manuscripts submitted for publication are distorted or manipulated in some way. This can include ignoring ‘inconvenient’ results and analysing data in inappropriate ways.

· Plagiarism – copying of data, papers or ideas. This can occur in manuscripts submitted for publication and in research proposals for which funding is sought

· Failure to disclose conflicts of interest. The increasingly close links between science and industry have lead to concerns that commercial interests may bias the scientific literature.

· Other forms of scientific misconduct. These can include (undisclosed) redundant publication (where authors publish the same paper in a number of different journals) and gift authorship (where senior members of staff lend their names to papers with which they have had little or no involvement)…

2. Bias: It has been suggested that peer review may introduce a number of different biases to decisions on funding and publication. For instance, a 1997 investigation by the Swedish Medical Research Council reported that female applicants had to be 2.5 times more productive than their male colleagues to get the same peer-review rating. … A related concern is that research funding committees tend to be male dominated as there is a relatively small pool of senior female scientists from which to select reviewers. There are also concerns that peer review tends to favour publication of positive results. One possible reason for this may be that editors are under pressure to publish results that generate big impact factors (e.g. as measured by the Science Citation Index 6). This has led to concerns that the non-publication of negative results leads to bias in the scientific record. Other possible biases that may be introduced by peer review include language (with publication being biased in favour of papers written in English) and institutional bias (with some studies suggesting that reviewers favour submissions from researchers at prestigious institutions)

3. Preserving the status quo: It has been suggested that peer review is an inherently conservative process, that encourages the emergence of self-serving cliques of reviewers, who are more likely to review each others’ grant proposals and publications favourably than those submitted by researchers from outside the group. This could have a number of consequences. For instance, it may

· discourage researchers from moving into new fields in which they have no track record;

· make it difficult for junior researchers to obtain grants or publish their research;

· present difficulties for multidisciplinary work, since peer review committees that do not contain individuals qualified to judge all aspects of a proposal may be less likely to approve the funding;

· result in the funding/publication of ‘safe’ research that fits neatly into the conventional wisdom and work against innovative, ‘risky’ or unconventional ideas.

· Inefficiency. Peer review can be relatively slow and inefficient both for funding and publication. Reasons for this may include:

· failure of referees to keep to deadlines -reviewers are commonly given 3-4 weeks to complete and submit reviews, but typically only 50% keep to this deadline;

· inconsistency between referees often means that more must be sought, thus slowing the process;

· recruiting and retaining referees is increasingly difficult (acceptance rates are typically as low as 50%);

· the lengthy time taken for editors and funding bodies to reach a decision regarding the fate of an application (sometimes up to six months).

Concerns about peer review in quality review and evaluation

Harvey argues in various places (inc Harvey, 2002) that, although self-assessment is often taken seriously only if peer review follows, the peer reviews themselves are not particularly an effective or efficient means of review. His concerns reflect those of Harding, above. In the main, peer-review teams make judgements based on what they are told and tend to look for discrepancies in the story. They attempt to relate what they hear (and sometimes see) to the self-assessment document. However, in practice, there tends to be a significant gap in the perceptions of peers and the authors of self-assessment documents. Peer groups see relatively little as they spend most time in a limited number of locations with group after group of ‘selected’ discussants. It is also unlikely that peer reviewers have time to read and thoroughly evaluate it the documentation, which is often extensive if not revealing. Even if the peer team has appropriate documentation, which allows some form of cross-checking, and they observe facilities and practices first-hand, they tend to see and assimilate only a tiny fragment of the entire institutional operation.

Peer reviewers are encouraged to ask questions but they are not trained as investigators. Sometimes they are not trained at all. There is very little attempt to challenge the preconceptions and prejudices of peers — after all their views are to be ‘respected’. The little ‘training’ or ‘briefing’ given to peer groups is usually about what areas need to be examined and the sorts of things on which to focus. Peers are rarely trained how to identify and interpret what they see (Harvey, 2002). A study in Chile, for example, suggested that, even in the newly developing private university sector, peer reports, in 90 per cent of cases were simply confirming what the institutions already knew and, furthermore, the prior experience of peer reviewers tends to influence the outcome of reports (Silva, Reich & Gallegos, 1997, p. 31).

related terms

See

site visit

peer

sources

Council For Higher Education Accreditation (CHEA), 2001, Glossary of Key Terms in Quality Assurance and Accreditation, http://www.chea.org/international/inter_glossary01.html, updated 8 May 2001.

Fraser, M., 1994, ‘Quality in higher education: an international perspective’ in Green, D. (Ed.), 1994, What is Quality in Higher Education? pp. 101–111 (Buckingham, Open University press and Society for Research into Higher Education)

Free definition, 2004, ‘Peer review’ http://www.free-definition.com/Peer-review.html

Freedictionary.com, 2004, ‘Peer review’, http://www.thefreedictionary.com/peer%20review

Green, D., 1994, ‘What is quality in higher education? Concepts, policy and practice’, in Green, D. (Ed.), 1994, What is Quality in Higher Education? pp. 3–20 (Buckingham, Open University press and Society for Research into Higher Education).

Harvey, L., 2002, ‘Evaluation for What?’, Teaching in Higher Education, 7(3), pp. 245–264.

Hyperdictionary, 2004, ‘Peer review’, http://www.hyperdictionary.com/dictionary/peer+review

Silva, Reich & Gallegos, 1997

Open University Validation Service (2004) What is peer review? http://www.open.ac.uk/validate/pop17711.shtml, undated page, accessed November, 2004

Parliamentary Office of Science and Technology, 2002, Peer Review, Post Notes, September, Number 182, www.parliament.uk/post/home.htm, note prepared by Dr Emma Harding.

University of Queensland, 2002, Teaching and Educational Development Institute, Introduction to peer review of teaching, http://www.tedi.uq.edu.au/Evaluations/PeerReview/guidebook.html, modified, 8 March, 2002.

Websters On-line, 2004, ‘Peer review’, http://www.webster-dictionary.org/definition/Peer%20review

Woodhouse, D., 1999, ‘Quality and Quality Assurance’ in Organisation for Economic Co-Operation and Development (OECD),, 1999, Quality and Internationalisation in Higher Education, pp. 29–44, Programme on Institutional Management in Higher Education (IMHE), Paris, OECD.

Vlãsceanu, L., Grünberg, L., and Pârlea, D., 2004, Quality Assurance and Accreditation: A Glossary of Basic Terms and Definitions (Bucharest, UNESCO-CEPES) Papers on Higher Education, ISBN 92-9069-178-6. http://www.cepes.ro/publications/Default.htm