‘Open data’ and how we use it, could be one of the most important debates of our time.  How much should a government or a company be ‘open’?  Is transparency a good thing?  What data does an organisation have and what should it share?   What is the value of my old data and should I just give it away?  These are the sort of questions a lot of people, in all sectors are asking themselves right now.

What is open data?

As a concept, Open data is related to big data.  Professor Nigel Shadbolt of the Open Data Institute, has written about its transformative power and recently defined it as ‘Information that is available for anyone to use, for any purpose at no cost and licensed as open data’.

Historically, there are examples of where information has been released (for free) to the general public, and its free release has, generally, driven positive technological, societal and political developments.  For example, back in the 19th century Florence Nightingales research on medical care for injured soldiers from the Crimean War was openly published and analysed, illustrating how a large number of instances of mortality were due to preventable diseases as opposed to deaths from their wounds.  In this example, official data was published and it achieved positive change – existing policy was challenged, using a quantitative model of openly available data, and it was altered as a result, resulting in better medical practices.

Today, there are many new examples of online initiaitives that are using open data for driving positive change.  For example, prescribinganlaytics.comcollects and collates data on NHS drug usage and makes prescription costs available online.  Data made available through this initiative showed that the prices of statins (the medication used to control cholesterol levels) varied considerably between different NHS authorities, because of different policies on procuring either licensed or generic versions of the same drug.  This analysis alone identified £200 million of savings by highlighting how different regional policies are set around spending.  It was estimated that around £1.4bn prescription efficiency could be found in the UK NHS using this data.

At the same time, initiatives like #bluelighthack work to promote a greater sharing and publication of police data; every month different police authorities publish data on particular types of crime types.  The collection and analysis of this data is already beginning to influence insurance premiums and can shape community discussions with local law enforcement.  Different police forces are using applications and systems developed outside ‘official’ government procured systems and are experiencing immediate effects in assessing where crimes occur and working to prevent them.

Such initiatives are strong positive examples of how open data can help achieve positive change.  But, it is worth noting that all of the examples above focus on government data and are generally from departments with a strong social remit (such as health, or policing).  Such areas have close contact with people and they can practice and implement policies that quickly impact on individuals and communities.  But, in other areas of government, and also, within industry,  how much of an ideal is ‘open data’ and what are the challenges it faces – do all organisations wish to disclose their data, will it benefit everyone?  If you’re a ‘closed system’, is it your interest to open up, or is it a naive, perhaps even dangerous, precident that could ruin your company?

‘Closed systems’

The scientific definition of a ‘closed system’ is a physical system that does not allow physical transfers (such as mass or energy) in or out.  These are useful in chemistry or physics because they provide sealed, self-contained systems that can be used to test theories and are important in the development in the development of theories like thermodynamics and chemistry.  They’re also seen in biology, where a closed system that has the right balance of plants, water, light and animal life produces a microcosm, that can function and sustain itself in a kind of self-contained system of balance.

Another definition of ‘closed’ comes from computing, where ‘closed source’ software is applied to computer code that is produced and, as an opposite to ‘open source’ computing, is kept secret and protected from ‘public release’.

Many businesses and government departments, are is in some senses ‘closed systems’.   Due to the tasks they perform, or the business they conduct, they keep their data contained, sealed within systems of classification and intellectual property protection. Historically, there are a number of reasons why such systems have evolved to protect and safeguard data.

  • Security.  Governments and some companies classify their data based on security.  The history of our times has driven this; in the past 50 years as we’ve gone through the Cold War and subsequently, the Global War on Terror, we’ve maintained strong cultures for being ‘closed’ because of the importance of national security.   National security extends across the full range of activities, from the protection of the ‘Nation State’, through to an individual’s safety and their rights.  With regard to security, if somehow all the data a government had, was suddenly released real, physical harm would be caused.
  • Value.  A lot of data and information is extremely valuable.  Software code is an example of this but we can all think of many others – blue prints, recipes,  unpublished manuscripts etc.  These are all instances of pieces of data and information that are commercially valuable.  Increasingly, where value actually resides, is not actually in produced products, but in the ideas that enable the formation of these products.  This means the release of such data unofficially or its theft is, quite simply, a criminal act that devalues someone’s product.
  • Reputation.  The reality of history and changing societal values means that sometimes, institutions hold data and information that could, perhaps reflect badly on our values today. This is really an issue for institutions over a certain age.  Governments for example, declassify documents after a certain amount of time – say 25 years, these are often heavily censored as well (arguments for censorship in the ‘national interest’ are remain an on-going source of controversy).  Older businesses also seek to protect their histories – for example, how does a company maintain its behaviours when, fifty years ago, it was trying to reflect the beliefs of the average ‘person’ in the street?  Through today’s lens, such beliefs will probably appear sexist, rascist and homophobic.  How does a company handle having data that reflects such values, does it disclose or keep it locked up tight so no-one can ever find out?

Does the notion of open data represent a challenge to such closed systems?  In a way, yes, it does.  Releasing data openly, means that governments and companies need to give up a level of control.  And this is a debate is driving a real and present trend to release information, especially in western democracies and many companies.  In the UK, since the Freedom of Information act in 2001, and through other initiatives around the world – the state is being increasingly expected to disclose information, in some ways, the notion of open data is the next evolution of this.  At the same time organisations that traditionally consist of large, closed systems like Glaxosmithkline for example, have been altering their disclosure policies and publishing their data on clinical trials openly since 2012. Such initiatives, are perhaps forward leaning – for many organisations and government departments there is probably apetite for opening up but a lot of people, especially those who traditionally ascribe to the reasons described above for protecting information, probably question the value, especially with the associated risks of ‘showing dirty washing’ in public, regardless of how old the washing is.

4 reasons why closed systems should ‘open up’?

1.  Openness encourages discussion.  Full disclosure of data means that everything comes out at once.  This could be uncomfortable, but for an old department, is it better to be open and honest about the past, rather than treat it with nervousness and attempt to conceal it?  Such attempts generally cause suspicion and accusations of conspiracy.  They also call for leadership that is strong enough to acknowledge the values of the past and discuss them in an honest manner that acknowledges openly that ‘things were different’ then. In doing so the delivery of such data should prompt sympathetic discussions and debates – are things as simple as an ‘apology’ on one hand, or ‘denial’ or the another?  Can everything an organisation has faced be boiled down to a basic, binary ‘black or white’, ‘right or wrong’ argument, or is more complex than that?  Going back to our history, being open on the things we have learned and, the mistakes we’ve made, is perhaps uncomfortable, but it does enable both us as people and the organisations we work for to develop.

2.  It can be difficult to ‘open up’, but any data helps.  Full disclosure may be naive, but it’s important to remember and support organisations that are disclosing any data and also reflect on the fact that a lack of data, doesn’t always equate to conspiracy.  For example, a young department with a positive remit like the Department of Energy and Climate Change (established in 2008) can realise a lot of data quickly, and this will have a lot of positive social impacts.  An older department like the Ministry of Defence, faces a lot more issues in disclosure.  As a department it holds vast archives of secure information often going back over hundreds of years.  The Army for instance has a ‘corporate memory’ that’s almost 500 years old!  This data is also contained in a myriad of forms in a bewildering range of locations.  How such departments can actually ‘know’ what data they have is a significant challenge, let alone how they could convert it into a format that they could disclose.  Such a difficulty is not an excuse for non-disclosure, but it is a challenge.  In such circumstances, acknowledging where information is and perhaps promoting it to others who can make it available for public analysis does offer a way of opening up such old records and get the data flowing.

3.  Open data increases the association with the ‘real world’.  Security is paramount and it remains one of the strongest arguments for keeping certain systems closed – ‘national security’ is one of historic reasons why our governments formed in the first place.  But, we, ‘the people’ are generally singular in our concerns and see security in a different way to governments.  We worry about human issues – like our personal security, health and our families and, generally we can cope with the idea of ‘one big threat’.  Perhaps this is something we’ve grown accustomed to since World War II.  But a large organisation or government has to function on a number of layers and cope with a multitude of demands from a plethora of departments and sources.  Generally, such demands have made it logical to form closed systems with large, well-organised hierarchies sustaining silos and systems of classification.  But, as voters ask for more and more details from their leaders (generally relating to their interests or their rights) and more and more organisations seek to share data, could openess actually improve things? By being clearer on what needs to be classified and secure and what can be open, would things be easier?   Perhaps in the future we will work to a simpler ‘security’, ‘commercial interest’ ‘everything else’ method of classification which means people can more easily request data from governments, but also at the same time governments and organisations could be clearer on what they need to keep in-house and what they can reliably take from the open source marketplace.

4. Openness promotes rational debate.  An often quoted goal for policy makers and politicians is that policy should be ‘evidence based’.  This is admirable and hopefully making data more freely available and analysing it rationally could actually improve things for decision makers.  As was the case with Florence Nightingale’s data – if you collect, group and analyse real information, it can show what policies are working and what policies need changing.  If we went in this direction, could it lead to us judge our leaders and decision makers not on their appearances or TV performances, but instead on the real outputs of their choices and decisions?  Additionally, if our leaders continue to be judged predominantly on their personalities and other emotionally-focused measures as opposed to quantifiable metrics, won’t it be inevitable that we wish to keep our systems closed in order to protect our collective reputation?

What we should share and what we should classify will continue to be debated.  In the meantime, if we recognise that there are a variety of positive reasons for closed systems to ‘open up’ and considerable benefits for them doing so, then perhaps we’ll start to form policies and make decisions based on what’s rational and quantifiable.  Does it really matter what someone looks like when they are eating a bacon sandwich?