Orientation Observation In-depth interviews Document analysis and semiology Conversation and discourse analysis Secondary Data Surveys Experiments Ethics Research outcomes



Social Research Glossary

About Researching the Real World



© Lee Harvey 2012–2019

Page updated 25 January, 2019

Citation reference: Harvey, L., 2012–2019, Researching the Real World, available at
All rights belong to author.


A Guide to Methodology

8. Surveys

8.1 Introduction to surveys
8.2 Methodological approaches
8.3 Doing survey research

8.3.1 Aims and purpose
8.3.2 Background to the research
8.3.3 Feasibility
8.3.4 Hypotheses
8.3.5 Operationalisation
8.3.6 How will data be collected and what are the key relationships?
8.3.7 Designing the research instrument
8.3.8 Pilot survey
8.3.9 Sampling
8.3.10 Questionnaire distribution and interviewing
8.3.11 Coding data
8.3.12 Analysis Response rate Frequency tables Graphical representation Measures of central tendency (averages) Levels of measurement Crosstabulation Measures of dispersion Generalising from samples Dealing with sampling error Confidence limits Statistical significance Chi-square test Association Summary of significance testing and association: an example

8.3.13 Hypothesis testing
8.3.14 Significance tests
8.3.15 Report writing

8.4 Summary and conclusion

Activity Statistical significance
Statistical significance is the measure of the probability that deviation in a sample from the population parameter is likely to be the result of sampling error. It is thus delaing with the same issues as confidence limits but just framing the difference as a probabilistic outcome rather than constructing the confidence limits and seeing whether a specified value lies within the confidence interval.

For example, Table indicates that older people in the sample are more likely to know about Clause 28 than younger people. This looks to be quite marked, with twice the percentage of over-eighteens as under-eighteens having heard of the clause. However, once again we are dealing with a sample and need to consider sampling error. the example reveals a difference in knowledge about Clause 28 for each age group. What is the probability that such a difference could be the result of taking a random sample from a population where in fact all the three age groups have the same knowledge about Clause 28?

We can work this out by using a statistical significance test. A significance test operates in the same sort of way as confidence limits. It tells us what the probability is that any observed difference within a sample could be the result of sampling error.

All significance tests examine a null hypothesis. A null hypothesis is simply a statement that says there is no difference between two samples (or sub-samples) or that there is no relation between two variables. The significance test then works out the probability that this null hypothesis is true. The resercher decides on a cut-off point called the significance level. In the social sciences this is usually 5% (p=0.05) (which is equivalent to 95% confidence limits).

If the probability that results from the test is less than 5% then we reject the null hypothesis. This means that we reject the idea that the difference is due to sampling error. In other words, if the test probability is less than 0.05 then the difference in the sample reflects a difference in the population.

A widely used significance test for crosstabulated data is the chi-square test. It is applicable to all size tables (provided that there are sufficient numbers in each cell) and is fairly easy to work out and understand (although it is not always the best test in all the circumstances where it is used). (Information on how to calculate a Chi-square statistic can be found at QMSS About the Chi-square test (accessed 23 October 2016))

See section 8.

However, it is important to know how to interpret a chi-square result. The chi-square test operates by comparing the actual frequencies in the different cells of the table with the frequencies that would be expected if the null hypothesis were true.

Taking Table as an example, the null hypothesis in this case would be that age makes no difference to knowledge about Clause 28. What this would mean in practice is that we would expect the same percentage of 'No' answers for each age group. There are very nearly twice as many 'No' answers overall as 'Yes' answers, so we would expect twice as many of each age group to say 'No' rather than 'Yes' if the null hypothesis that there was no difference was true.

So the expected frequencies for Table would thus be approximately those in Table


Table Approximate expected values given the null hypothesis of no difference for Table V1 'Had you heard about Clause 28?' by V30 'Age in years'

Expected frequencies
16 or 17
19 or 20
Row total

Yes (1)





Yes (1)






In practice, to calculate chi-square you need to work out the expected frequencies exactly.

The observed and expected frequencies are then compared. A simple visual check in this case suggests that there is a difference between the observed frequencies and the expected ones, with fewer younger people having heard of Clause 28 than expected. But is this difference statistically significant or just the result of sampling error.

Top Chi-square test
The chi-square test procedure provides the probability that the difference is the result of sampling error. The test procedure produces a value for chi-square. This can then be converted into the probability of the differences in the sample being the result of sampling error.

If you are working it out by hand, there are chi-square tables that provide the conversion from the statistic to a probability figure depending on sample size and the number of cells in the table (degrees of freedom, as it is known). A computer program a probability will provide both the value of chi-square and the corresponding probability. In the example, (Table and Table the calculated chi-square was 5.90 and this gives a probability (or significance value) of 0.052.

What does that mean? It tells us that there is only a 5.2% chance that the observed sample came from a population in which there was no difference between age groups. This means, conversely, that there was a 94.8% chance that the differences in the sample reflect real differences in the population. Is this good enough? The answer to that depends on convention.

In the social sciences, as was noted above, it is conventional to take 5% as the cut-off point for all significance tests. If there is less than a 5% chance that the observed difference in the sample is due to sampling error then the difference is said to be statistically significantly different9the null hypothesis of no change is rejected). This means that you can generalise the difference in the sample to the population from which the sample is taken.

In the example, it means that although the sample shows a difference, this is not significant because probability is just over 5%. So it is possible that the differences in the sample are caused by random sampling error at the conventional 5% significance level.

Do males and females have significantly different views as to whether homosexuality should be part of sex education in schools? (compare V7 with V31) (See CASE STUDY Data File and Coding Frame.)

You should note that there are a large number of significance tests that suit different situations. What this Section has concentrated on is the meaning of significance tests and how to make sense of the statistics that you might come across, either in books or generated by computer programs.


To sum up, in general a statistical significance test will give you a value for the statistic and a probability (or significance value), which you may have to look up unless you are using a computer program. If the significance value is less than 0.05 (5%) then the sample difference is significant at the 5% significance level, that is, the null hypothesis is rejected and the differences in the sample are not due to random sampling error but represent a real difference in the population.

So significance tests and confidence limits are statistical procedures used to make generalisations about the population from a random sample. They provide the grounds for making claims about the population. A statistically significant result means that the observed difference or relationship within a sample is due to a difference or relationship in the population from which the sample was taken.

However, statistical significance tells you nothing more than that an observed difference or relationship is unlikely to have been caused by random sampling error. It tells you nothing about the size of any difference or about the degree of any relationship.

Next Association