[Summary: A brief critique of the 'openness score']
A recent Cabinet Office press release, picked up Kable Government Computing states that: “The average openness score for all departments is 52%”.
What’s an openness score I hear you ask? Well, apparently it’s “based on the percentage of the datasets published by each department and its arms-length bodies that achieve three stars and above against the Five Star Rating for Open Data set out in the Open Data White Paper”. That is, it’s calculated by an algorithm that looks over all the datasets published by a department on Data.gov.uk and checks to see what format the files linked to are in.
Which seems to display both a category mistake on behalf of the Cabinet Office, and a rather worrying lack of statistical literacy and awareness of how such a number might be gamed.
On the category mistake: the openness core appears to equate openness with file format – but ‘openness’ in general is not equivalent to the use of an ‘open’ file format. Firstly, even when using a machine readable format data can be non-open and non-machine readable depending on how it is formatted: a garbled CSV is to all intents and purposes less accessible and open than a well formatted Excel file. Secondly, openness is not just a technical concept, and is not just about data (I’ve commented on that in more detail here). To take the number of well-formatted datasets as a proxy for departmental openness is reductive and narrow in the extreme. This may just be an issue of communication, such that Cabinet Office should be talking about an ‘open data score’ rather than ‘openness score’, but as an input into narratives on open government this risks creating confusion, and again muddling the relationship between openness of government in general, and open data.
On nonsensical numbers: even as an ‘open data score’, the current number is practically meaningless, as it is just a ratio of the non-machine readable to the machine-readable datasets. The score can be increased by removing non-machine readable datasets from Data.gov.uk, and is skewed according to how many datasets a department publishes. A department publishing two datasets, one machine readable and one not, gets a score of 50%. If they publish an extra dataset, full of meaningful information, but that is not yet machine-readable, their score drops to 33%. This means the score is not only misleading, but potentially creates perverse incentives that run counter to the very notion of Tim Berners-Lee’s 5-Star rating of open data (which I should remind readers is not a rigorously designed set of criteria, but something Tim prepared just before a conference presentation as a rough heuristic for how data should be opened), which calls for people to put data online as a first step, even if it can’t be made machine readable right away.
An openness score constructed as the current score potentially incentivises less data publishing not more.
I hope whoever came up with the idea of the openness score is encouraged to go back to the drawing board and think about both about it’s design, and how it is communicated.
Hi Tim,
This is welcome thought on this matter, and your points are well worth addressing.
Saying openness score equates *just* to file format is a misunderstanding of how the scoring works and the issues that the open data community currently face when using data.gov.uk.
There has been massive criticism by geeks that data has been published in PDF format – impenetrable for analysis – so providing a carrot for publishing in a better format is a very useful heuristic.
The second thing it is based on, which activitists have been widely criticised government for is ensuring the license is open. Now we have a useful carrot to publish under licenses such as OGL, although about 1000 datasets on data.gov.uk still refuse, stating for example ‘terms and conditions apply’, suggesting an activist can’t even include a table or a screen shot of the data in a blog post.
And one more thing the openness score measures – easy access. Again, a serious issue that those trying to browse, analyse or mash-up government data is being blighted by broken links or multiple hops to find the actual data. This year we’ve got public bodies being shut down and merged, hundreds more government websites merged, so keeping track of broken links was never a more important issues. When a direct link is provided to the data file or API, and maintains that then the publisher is rewarded in the openness score.
Tim Berners-Lee might have put the system together on the back of an envelope, but it undeniably addresses the key three problems with published data.
Your main argument that the score is “nonsense” is because “a garbled CSV is to all intents and purposes less accessible than a well formatted Excel file” can you provide any examples? The hundreds of CSVs I’ve personally accessed from data.gov.uk and I’ve not come across a single garbled one. They are all in neat rows, all with headings and nearly all with units. Is this merely a theoretical objection or do you have a list of poignant examples?
You’ve previously commented that a serious flaw in data.gov.uk was documentation for datasets, and that this could be included in the score. I’d like to see if anyone else sees this as a wide issue or not, and particular examples of datasets which are lacking.
You wonder if publishers will ‘game’ the system and remove badly scoring datasets from data.gov.uk. We have controls on deleting, avoiding this specific issue. But more importantly, a metric that has been used a lot more than openness, is the ‘number of datasets’. You could argue it has its problems itself, but it has undeniably been a carrot for publishers to keep opening data.
But I do agree with the concern about the use of the term “openness score” in the wider context. The wording of the press release could be misconstrued aligning “openness score” as a direct measure of “open data commitments”. Although the sentence provides surprising detail in how the percentage is calculated, such a broadly named indicator should probably include other aspects of “open data commitments”, beyond licence, machine-readability and ease of access which are included – it doesn’t take account of how many datasets the public body has failed to open up, or not timely enough.
David Read
data.gov.uk developer
Thanks David for taking the time to respond.
I think however you misread my argument in a number of places. In particular:
I don’t say the score equates openness **just** to format, but that in tangling up openness as a unipolar concept with file format it makes a category mistake. That is, it is not possible on the current openness score to advance beyond a given level without a particular file format, making file format an integral part of openness in a way that privileges one particular dimension of openness over others.
The essence of my argument might be that openness is a multi-dimensional concept: there are many elements of being more open – and to privilege just a few and score on the basis of these – risks creating distortions on where efforts are put, and in the debate around opening government.
I’ve written in the past about the possibility of multi-dimensional analysis of open data and would encourage any assessment of openness to recognise that openness does not reduce to a single number, but needs it’s different qualities to be distinguished.
Secondly, my main argument does not concern ‘scrambled’ CSVs, but is that openness is irreducible to a ‘score’. On the case of ‘scrambled CSV’ I’ve written about an example of government data here that sets out some of the issues – albeit this is a case of a dataset that is not yet listed on data.gov.uk (this case also highlights the importance of documentation).
Openness always involves a question of ‘open to who?’. To the developer, a dataset may be easier to use if it includes just columns and rows of data that can be read straight into a data matrix, with definitions of variables provided in a second CSV file. To a journalist, the same data may be easier to use if presented in an Excel file formatted with clear bold headings and footnotes explaining what each variable means. When we talk about openness we need to be aware which how we are are serving different users – and what their effective access to the data looks like.
Thirdly, even with the caveats above, I’ve got no major objection to displaying a heuristic indicator next to individual datasets of where they score on the five stars of open data, or to exposing this as a way to search datasets (in fact, what you’ve done has been technically impressive and I’m sure useful to many searching the catalogue). My primary objection is to the generation of departmental scores by taking a simple percentage of datasets that meet the third star from all those published. Given the vast differences in amount of datasets published by different departments, and the fact that datasets vary massively in their informational content, I can’t see any way in which this score can be said to meaningfully measure openness in any sense.
Happy to contribute more to discussions around this in the coming weeks if there is scope for the decision about publishing a ‘score’ aggregate to be reviewed.
Thanks Tim. It’s clear we both have strong reservations about using the openness score percentage as an overall indicator for departments.
But I disagree when you rant against the measurement of each dataset against an ‘openness score’. Although you say you have “got no major objection”, you think that the openness of some data is only in the eye of the beholder, and the complicatedness of the multiple dimensions, or indeed any useful sense of quality, cannot be captured in a single number. It’s clear from your article http://www.opendataimpacts.net/2011/11/evaluating-open-government-data-initiatives-can-a-5-star-framework-work/ you’ve done a lot of careful thought about this before coming to your conclusion, which I respect, but I think now we have actual scores it needs re-evaluating.
We need to change the mindset of the average regional care trust that sees no issue with publishing their October 2012 spending in a PDF file, without a licence and allows the file to go missing six months later. The reality is that a decent chunk of data on data.gov.uk has these sorts of fundamental problems. The five star rating is an excellent carrot to improve this, and if publishers do the small changes to get 3 stars, then in 99% of cases they will come up with excellent open data.
I don’t think you should call for the five stars scores to be scrapped on the basis that it ignores details such as whether something has published complete documentation yet (plenty of people leave comments asking for things like this and get answers) or whether a category of datasets should be in Excel, CSV or both. Yes these are good questions, but I don’t think they skew the whole rating for a dataset by anything like as much as you are suggesting.
Thanks for the continued engagement David. This is a really interesting dialogue.
Firstly, I don’t think I’ve called anywhere for the Five Star scores on individual datasets to be scrapped. These can be useful as a heuristic at an individual dataset level.
I’ve would continue to say that calling the percentage of datasets published that pass a technical assessment against the (technical) five stars of open data an ‘openness score’ is a gross distortion, and should be scrapped – but that does not equate to scrapping any use of the five stars at the assessment of individual datasets. The statistic of how many datasets meet the five stars may be interesting – but it should not be elevated to the status of a policy level in the way the open data score has done.
In saying “you’ve done a lot of careful thought about this before coming to your conclusion, which I respect, but I think now we have actual scores it needs re-evaluating.” are you suggesting that I should re-evaluate my critique because someone has now crunched the data? I’m sure sure why the generation of the numbers in any way invalidates the critique. Or have I misread the point you are trying to make here.
I would further question your assertions that “The five star rating is an excellent carrot to improve this, and if publishers do the small changes to get 3 stars, then in 99% of cases they will come up with excellent open data.”. This begs two question:
1) What is the evidence base for this claim?
2) How are you defining excellent open data?
Unless (2) is answered without reference to the five stars, we’ve got a bit of a circular argument.
To try and briefly summarise my position:
I accept that the short-term impact of the perverse incentives built into the ‘openness score’ may be limited, but I would argue that over time, if inadequately balanced by other pressures, their effect will impact on the nature, quality and availability of open data in a negative way.
I think a progressive ‘openness score’ would include organisations or departments and business units open enough to put documents or data online as is to actively and iteratively improve the dataset quality from feedback.
Thanks for your comments so Tim. Really good to get to the bottom of this.
Aggregating star ratings to get a percentage is bad – we’re agreed on that as I’ve said. I’m afraid I have so far misunderstood that nearly all of your comments have been about this because I have long used the term “openness score” not as the aggregated percentage, but a dataset’s star rating. So I’m sorry about that.
But I want to challenge you more about the individual dataset ratings. I think that you’re not against any of the (six) individual dimensions of analysis that make up the Five Stars of Openness, but that you see little benefit of combining them into a single star rating.
In my experience the individual dimensions of analysis hit on the key problems with published data in reality, and you. You seem to be worrying that incentivising a publisher to add an open license, of convert a PDF to CSV, or fix a broken link could somehow go wrong. So wrong, that the incentive is perverse or even damaging.
You challenge me to define excellent open data, and there are of course many things and it varies from dataset to dataset. To a lesser extent it varies from user to user. But I can tell you with surity that providing it with an open licence, make access easy and in a structured non-proprietary format is a bloody good start. I think anyone coming to data.gov.uk would expect the majority of datasets to achieve these criteria, yet that is far from the truth, so let’s quantify how well these are done and provide a summary as a star rating. Is that such a bad thing?
On aggregating the dimensions into a star rating, your other article admits the power of communicating a single number over 6 individual dimensions. In my experience, that sort of power is what gets a lot of data released and improved and is therefore immensely valuable. I don’t think the five stars distort the dimensions very much – more stars is better in all but a few obscure cases. I think everyone would agree that data in PDFs simply would be published in Excel or KML format, say. Excel vs CSV – well there’s not a lot in it to argue over really and its usually easy to include both. The worry that RDF might be published without any more accessible formats is a special case though. There is nothing bad about shared vocabularies or links as such, but RDF gets complicated and suitable tools and knowledge are still rare. But rather than throw out Five Stars, one could tweak it to award four or five stars only if you also provide an accessible version, such as a spreadsheet or web visualisation.
In conclusion I think Five Stars of Openness is an excellent incentive for data publishers. Your concerns are heavily outweighed by the benefits, and you should be celebrating the introduction on data.gov.uk to drive improvement of compliance with basic standards of data publishing.
Graeme, I don’t think anyone would disagree with wanting both quantity and quality .
But how far should say geographic data go though? Is a PDF map of local allotments good enough? KML of their lat/longs would bring lots of benefits to developers reusing. Overlaying them on Google Maps would be general public friendly and help searching and routing to them. Some councils provide commercial web apps with more features, such as comparing with say geology. And RDF with common vocabularies would allow aggregation of allotments nationwide. Where is the line drawn between quality and a waste of local authority cash?
Thanks David.
Glad we’ve come to an understanding on the aggregation point: that is indeed where the majority of my critique about ‘nonsensical numbers’ was directed.
On the five stars itself, I think we’ve still got somewhat of a misunderstanding.
– The six dimensions referred to in this post are not directly part of the current ’5 Stars of Open Linked Data’ as outlined at http://5stardata.info/ and based on Tim Berners-Lee’s original talk on the subject.
The six dimensions (and my commentary) concerns the hypothetical idea that the assessment of an Open Data Initiative (rather than dataset) could be carried out by ranking ODIs with a score along these dimensions. You can see an example of this sort of multi-dimensional analysis in the Web Index, and it is something we’ll be exploring more in the upcoming Open Data Research network project on Open Data in Developing Countries. I linked to my blog post on the whether 5-star frameworks make sense to analyse open data initiatives in the context of the aggregate openness score – rather than a per-dataset scoring.
As such for this discussion I think it’s safe to put aside the six dimensions, and any discussion of whether reducing the assessment of an Open Data Initiative (e.g. the whole of UK Open Data Policy, to be compared, say, so US or Kenyan policy) to a single number or a small collection of numbers, makes practical sense or provides the right incentives.
However, that leaves the question of how effective the current five stars of open linked data is in providing the right incentives for publishers of individual datasets.
On this I’ve got an alternative past post to link to here which argues that how useful the five star model is for generating the online resources that we need for transparency, accountability and innovation depends on when you see stars as cumulative or things to be leapt over.
To give a worked example:
Imagine three potential users of a dataset on educational performance. The first user is a parent who is opposing the closure of their local primary school. The second is the web developer of a local newspaper looking to create a school finder widget for their website. The third is an app developer looking to integrate school information into a mobile phone route-planning application for people with disabilities.
These users may all benefit from different kinds of data.
The parent’s needs may be best met by a statistical first release dataset from DFE, presented in a formatted Excel sheet where they can locate two or three specific statistics to take along to a residents meeting or council meeting (anecdotally I’ve seen evidence that people have greater trust in these ‘official’ spreadsheets than in the same data mediated through other websites / applications, though I’ve not tested this in any studies yet).
The local authority web developer might want a CSV they can pull down and use within a simple PHP application. The learning curve to deal direct with linked data is likely to be beyond them.
The app developer might have identified that linked data provides the best means to integrate across disparate datasets, and so wants RDF or a SPARQL endpoint for access to the data.
Depending on how the five stars is applied, all, or only one, of these users might find their needs being well served. If the five stars is applied so that stars light up when that particular point is achieved (what we might call the ‘toggle’ model), then an authority has an incentive to publish the Excel spreadsheet, the CSV and the RDF. If they are applied so that published RDF automatically lights the stars below (what we might call the ‘totaliser’ model), then many users may not be well served.
The counter-argument I’ve seen given to this – that with the RDF then the CSV and nicely formatted versions of the data can be generated – whilst in most cases (but not all) theoretically possible, hits up against the problem that this rarely happens in practice – and it leads to real needs that exist now being neglected until some technical intermediary steps in to make data accessible.
In practice it seems that most people want the 5 stars to operate on the totaliser model up until star 3 (i.e. once you’ve got to 3, you can forget 1 and 2), and then as a toggle for 4 and 5 (i.e. 4 and 5 are nice bonuses, but you need to have 3 as well) – which is, I think, fairly sensible, but does demonstrate that it has some weaknesses in the model which we need to by critically aware of.
In essential, they key point I’m trying to make is that your suggestion that:
“[these] concerns are heavily outweighed by the benefits”
is an empirical question – and one that needs to be investigated. That is – we should check the data on the benefits and costs of a five star model before (or alongside; as I’m all for an iterative and learning by doing approach) embracing it as a key instrument of policy.
I hope the worked example is clear. If not let me know and I’ll look at putting together a slide deck on this – as it’s something I’ve been meaning to put together a paper or talk on for a while – and this discussion has been really stimulating in helping me get on with some of the thinking for it…
Tim, thanks again. You example is clear, and I think the tweak I suggested to the system would cover this possibility, ensuring a CSV or Excel file was published too, whilst avoiding the complexity of your 4/5 star toggles. In addition I don’t think the parent would have any problem opening the CSV or the developer converting the Excel – the difference is marginal.
I don’t think there are any more hypothetical examples beyond the spreadsheet vs RDF one. Of the 100 odd RDFs in data.gov.uk I don’t think any are missing an easier way to access the data. So your example remains purely hypothetical and of little value.
Now here’s my example of empirical evidence. This week I used the new system on data.gov.uk to discover a central government department with 80 broken links across 26 datasets on data.gov.uk. They had been broken for months with no-one realising, due to a website reorganisation. The department was very happy to hear about the issues in detail, promptly fixed most of them and are pleased to see their 5 star scores go up from 0 to 3 in most cases.
Your empirical evidence is an interesting anecdote in showing that, if someone is checking, and is able to talk to departments, they will fix broken links. And the automated five star tool, in acting as a link checker, was clearly useful to you here. I don’t dispute that.
However, this does not count as empirical evidence concerning the cumulative impact of using a flawed five star score over time. It’s important to also look at unintended consequences. In anecdote territory I would look at where I’ve seen departments put considerable time and effort into exploring RDF before getting their basic releases of data right, because of the perception that the five-stars were a totaliser, where getting straight to the top was the goal.
I also don’t believe the problems I cite are merely hypothetical.
My experience is that the parent may not have a great experiencing with the CSV file. Compare a CSV file of a statistical first release from DFE to the Excel original, and if you’re simply looking through the tables to find a number it is far far easier with the Excel. Working with VCS organisations and non-data specialists to explore how they can use open data this is a very real problem I’ve seen numerous times – where the format and how the data appears really matters to how usable it is to the end-user. Different tools serve different needs, and what might look like ‘trivial’ issues to the technically skilled are not trivial to the under-pressure parent with limited ICT skills. I believe we should be building an inclusive open data infrastructure, and constantly asking ourselves the critical question of who we are empowering by the technical choices we make.
On the RDF question, a simple non-hypothetical example would come in the form of EduBase, although I note that this, whilst a significant government dataset published as RDF, has never had a data.gov.uk record. EduBase is available as (outdated) RDF at http://education.data.gov.uk/ and can be accessed as non-open data in a limited Excel download behind a log-in at http://www.education.gov.uk/edubase – If this was listed on data.gov.uk tomorrow (and I remain puzzled as to why no record exists for it), what automated score would it get?
As I understand the current implementation, if it was listed with a link to the EduBase website (not direct to data), and the openly licensed (outdated) RDF, it would score top?
Again, I re-iterate that I don’t think this means the five stars are meaningless, but that they are one tool amongst many for analysing the usefulness and openness of data. As your anecdote shows, they can be a useful heuristic for locating datasets that need a closer look – but I would maintain that as an objective measurement, the model has weaknesses that we have to take seriously.
Tim, I have sympathy with Excel being useful when viewing complicated ONS datasets. One might tweak the five star rating to also require Excel when a CSV is supplied before awarding 3 stars. I’m not sure you could require the same with KML and Shapefiles, or requiring other proprietary formats, so this would be a one-off exception. It is something we could do some analysis on to see how widespread the problem is, or whether this would encourage more work than it is worth.
The old Edubase dataset is provided in RDF, CSV, JSON and as web pages so it would get 5 stars. The new one with closed license and difficulty of access both limit it to 0 stars. That’s not to say the automatic routine has some tough choices sometimes, and I’ve considered a manual override for datasets which are complicated for an automatic tool. BTW I’ll push for it to be added to data.gov.uk catalogue, as I agree its a surprising omission.
Regarding this weeks improvements to 26 datasets, on this occasion someone centrally needed to take the trouble. As we grow more confident that the reports are correct we plan to make them public, in a similar fashion to the Spend Data reports: http://data.gov.uk/data/openspending-report/index which we’ve seen driving improvement in the data there, without the need for someone central to coordinate.
I guess our views are coming at this from slightly different angles – I’m mainly concerned about getting all the 0 star datasets we have today moving in what we can mostly agree is the right direction (something better than broken links to perhaps PDFs with closed licences), and you’re focused on the accuracy of the goal, for which we need more of this healthy debate.
It’s hard to articulate and agree on where we want to be. Tim B-L set the parameters for the project kicking off and more voices pitching in are needed to refine it. It’s great to have you involved debating this.