Big data: the public sector is still lagging behind

30 October 2012

By Mark Dunleavy

Big data is big news in the public sector at the moment. Cabinet Office minister Francis Maude recently initiated a "digital-by-default" initiative to kick start the big data agenda. This has seen an annual budget of £22m being spent on a team of web consultants to reshape the "direct.gov.uk" portal into a one-stop-shop for members of the public to access government information under the new Gov.uk single domain.

Despite these developments however, the public sector is still lagging behind the private sector when it comes to big data control. A report earlier this year by think-tank Policy Exchange claimed that big data analytics could save the public sector between £16bn and £33bn annually. This is equivalent to between 2.5 per cent and 4.3 per cent of the government's total budget. Big data presents a significant untapped resource for the public sector. If the UK is to see these predictions become a reality however,  there are some basics that need to be addressed from the outset.

The data privacy hurdle
A recent report by the Information Commissioner's Office (ICO) found that only one of the 15 NHS organisations audited provided a high level of assurance to the ICO. The local government showed a similar trend with only one in 19 achieving the highest mark. Unlike the private sector, commercial success is not the aim for the public sector. Its main role should be to serve its citizens, but to do this well it needs to securely analyse and store the public's information.

Mining the mass of big data held across various government bodies is just the start. That data then needs to be protected, and the onus lies on the public sector to make sure its citizens' information is safe at all times.

The need for transparency
Public data has traditionally been siloed, making it difficult to pull together to achieve a clear, centralised view. These silos need to be broken down and data must be made open to ensure it is trustworthy, actionable, and authoritative.

In January this year, the government announced its intention to create a Public Data Corporation (PDC). This set out to bring together data-rich organisations to provide a more consistent approach towards accessibility of public sector information. However, the latest annual departmental reports reveal that nearly every major department could be in breach of the government's own open data licence with both the Ministry of Defence and the Department of Health yet to publish the required annual report.

Achieving data transparency is clearly a work in progress. In order to get the most from big data, while meeting growing compliance and regulatory standards, public sector organisations need to ensure information is continuously standardised so that it can be delivered to any system, in any agency or department, in any format. This process will enable them to share information more effectively among internal and external stakeholders, and the public in order to improve accountability.

But big data does have the potential to become a financial burden if it is not managed, stored and secured appropriately. One way of mitigating the risks and making savings, is to identify old data in a process called data retirement. Inactive data can be removed from applications to manage growth in production environments, streamline backup, reduce hardware costs and improve performance.

The growth of big data provides an opportunity for the public sector to run more successfully. Whether addressing concerns over security or implementing open data standards, greater efficiency will enable public sector organisations to embrace data-sharing across the industry. Armed with a true understanding of citizens' needs the public sector will be empowered to provide a better service.

Mark Dunleavy is managing director UK at Informatica UK