As I put together a survey design for the Open Data Impacts project I will be blogging about the process and the choices I make, to increase the transparency of the process, invite input from anyone interested in the project, and to hopefully share my learning with any others going through a similar process.
Education, Education, Education
Sometimes the questions you think are going to be easiest to write for a survey turn out to be some of the trickiest to pin down. I thought starting with a few demographic questions on ‘Education’ would ease me into the question-drafting process, but, after hours trawling through categorizations of education levels in other surveys, exploring the rather handy (although in this case it didn’t lead me to any simple answers) Taxonomy Warehouse, and boggling at some of the ‘knowledge’ captured in OpenCYC, I found that there were no ready-made lists of education levels of subjects that I could just slot into the Open Data Impacts survey – and it was time to put together my own.
The Need
For the Open Data Impacts survey I’m interested in identifying some basic facts about respondents – including their level of qualification, and the subjects they have studied. Ideally, I want to be able to get a sense of how the subjects studied, and the level they have been studied to, affect what people do with open data resources.
However, given that I anticipate there is likely to be a technological slant in the skill set of survey respondents, I need to be able to capture the diversity of qualification and educational routes that people take in fields such as technology – and to recognize that many people are self-taught or picked up their skills through non-conventional educational routes.
I’ve got a separate set of questions in draft to try and identify ‘data mashing skills’ rather than ‘qualifications’ – but having some sense of qualifications will be, I think, still useful for the analysis stage.
Although the Open Data Impacts survey will focus on the UK, it would be useful to select educational categories that allow cross-border comparison of results, should a similar survey be run in other national contexts – and it’s important to recognize that not all those completing the survey (and engaged with UK government open data) will necessarily have gained qualifications within the UK education system.
I would also, ideally, like to recognize the different-but-equal status of vocational and academic qualifications, and to avoid suggesting a hierarchy of qualifications that privileges academic over vocational learning. (This turns out to be particularly difficult to do whilst maintaining the usability of a list of possible survey responses, where users often expect a certain ordering of response categories).
Searching for a scale
My first strategy for finding the scales I needed was to turn to existing surveys to see if there was a ready-made scale to re-use. After trawling through a number of surveys in the UK Data Archive, I found the following scale in the 2006 General Household Survey: See here
Variable hrpedlev : Highest education qualification of HRP
Values/Categories:
- -9 Never attended school
- -8 NA
- 1 Higher Degree
- 2 First Degree
- 3 Teaching qualification
- 4 Other higher qualification
- 5 Nursing qualification
- 6 GCE A level in two or more subjects
- 7 GCE A level in one subject
- 8 GCSE/Olevel, standard grades, 5+
- 9 GCSE/Olevel 1-4
- 10 CSE below grade 1, GCSE below grade C
- 11 Apprenticeship
- 12 Other qualification
- 13 no qualification
The Oxford Internet Survey uses a similar scale also. This gave me a good starting point – but looks the General Household Survey was based on an Interviewer, which allows the scale to be fairly complex, whereas I’m developing an online survey – so I’m after a slightly more compressed scale.
As mentioned, I was also after some international comparability of the chosen scale. Searching for that led me to the UNESCO ‘International Standard Classification of Education’ listen below:
-
Pre-primary education
-
Primary education
-
First stage of basic education
-
Lower secondary education
-
Second stage of basic education
-
(Upper) secondary education
-
Post-secondary non tertiary education
-
First stage of tertiary education (not leading directly to an advanced research qualification)
-
Second stage of tertiary education (leading to an advanced research qualification)
This scale is designed to map onto different national education systems and provide for international comparison. Mappings for the UK education system don’t seem to exist – but the descriptions of the education stages given does allow a fairly intuitive mapping across – and with the help of Directgov’s list of UK qualifications I think I’ve got a fairly good sense of how to group UK qualifications with these levels.
The Directgov list was also useful in highlighting some of the qualifications I might anticipate amongst respondents, and of which I was not aware. It also drew to my attention that I needed to be aware of the multiple names by which people may know education levels, even in the UK – where GCSE’s are newer qualifications equivalent to older O-Levels and so-forth.
The draft scale
After trying to find an equilibrium between all these scales, I’ve ended up with a draft six-point scale. Whilst in analysis, this scale may be treated as ordinal, as mentioned before, I had been hoping to avoid a direct implication of hierarchy. This turned out to be a challenge, and the best concession I’ve been able to make to this in the draft is the minor semantic point of talking of the ‘furthest’ rather than ‘highest’ level of education completed. Suggestions on other approaches to meeting this challenge are very much welcome.
Please select the furthest level of education you have completed:
- Secondary Education (GCSE/O-Levels)
- Post-Secondary Education (College, A-Levels, NVQ3 or below, or similar)
- Vocational Qualification (Diploma, Certificate, BTEC, NVQ 4 and above, or similar)
- Undergraduate Degree (BA, BSc etc.)
- Post-graduate Degree (MA, MSc etc.)
- Doctorate (PhD)
In using general titles for each point on the scale, and then giving examples, I’ve tried to make the scale as intuitive as possible, whilst easily readable for respondents. I’ve yet to fully test the usability of the scale for those not educated within the UK system – and that’s the next stage of development and testing.
Subject areas
After a lot of searching through taxonomy databases, my list of subject type has ended up a little more arbitrary, and linked to the focus of the research, where I’m analytically interested in some specific relationships between subjects studied and use of open data.
- Arts & Humanities
- Computer Science
- Engineering & Physical Science
- Life Sciences
- Mathematics
- Politics, Public Policy or Law
- Social Sciences
- Statistics
- Other (please specify)
The use of a relatively short list is also guided by a concern to get a reasonable number of responses in each category – as a fragmentation of response categories would end up requiring responses to be grouped an analysis on the basis of my own judgement – whereas this list of categories leaves the judgement of subject to the respondent, and keeps the question to a management length.
My hope is that respondents can fairly easily fit their subjects into of the categories above, although again this now needs further testing.
Feedback welcome
These scales are still very much being developed (they’ve changed a bit during the writing of this blog post: more evidence for the value of blogging as reflective practice…) so I welcome all feedback.
In particular: do you think you could easily and intuitively answer the questions below? Or do you feel that you don’t fit in any of the categories listed…
Recent Comments