Open Data in India: Policies and Practices

Can Open Data Open Up Internet Governance? 3 November 2013

Sumandro Chattapadhyay and Zainab Bawa, Co-Principal Researchers of the Open Data in Developing Countries project team in HasGeek, were invited to make brief presentations at the ‘Internet Governance and Open Government Data Initiatives‘ workshop (#303) at the Internet Governance Forum 2013, recently held in Bali.

Here are the recorded statement and the slides from Sumandro's presentation:

Towards an Expanded and Integrated Open Government Data Agenda for India 3 November 2013

Sumandro Chattapadhyay, Co-Principal Researcher of the Open Data in Developing Countries project team in HasGeek, presented a short policy paper titled ‘Towards an Expanded and Integrated Open Government Data Agenda for India’ at the 7th International Conference on Theory and Practice of Electronic Governance (ICEGOV), recently held in Seoul on 22-25 October 2013. The paper draws from our ongoing research on the National Data Sharing and Accessibility Policy and data intermediary ecology in India.

Here is the abstract of the paper:

The paper analyses the recently approved and implemented government data policy (National Data Sharing and Accessibility Policy) in India, and identifies the possibility and need of expanding and integrating it within the larger national e-governance and information policy ecosystem. The study draws from an ongoing research project on the Indian government data policy and the roles played by data intermediary organisations. The paper argues that an expanded and integrated open government data agenda will address crucial shortcomings of the national e-governance initiatives in India, and strengthen democratic interactions between the state and citizens through electronic and other means.

The full text can be accessed from Sumandro's GitHub repository.

Also, here are the slides from the presentation made at the conference:

Research Notes from the Asian Regional Meeting of ODDC Network, New Delhi, 16-18 July 2013 21 August 2013

In the study plan reformulation brief, we have already discussed why we decided to shift away from a sectoral focus (that on organisations engaging with issues of urban development) to a more wider perspective of data/information intermediary organisations. In the South and South-East Asian Regional Meeting of the ODDC network held in New Delhi, 16-18 July 2013, we had the opportunity to further discuss the various implications of and methodological challenges generated by this shift of focus. We are very grateful to our mentor Michael Gurstein, Tim Davies and fellow researchers associated with the ODDC network for their critical inputs and insights.

The following notes on our research plan taken from the discussions at the Regional Meeting is structured through a series of questions and answers to reflect the conversations that took place.

What is a 'data intermediary organisation'?

We identify a data intermediary organisation as one that shares data for access, consumption and re-usage (including re-sharing) of the shared data by other organisations and individuals. Three further clarifications are needed here:
• sharing of data by such organisations can either be done on a commercial or a non-commercial basis,
• shared data can either be sourced from an external creator and publisher of data (either government or private agencies), or be created by the data intermediary organisation itself, and
• the data intermediary organisation may or may not add (or reduce) value of the data before sharing it further, that is it may or may not modify the data received by them (by cleaning up, re-organising, re-formatting, aggregating, etc. ) before sharing it.

Can an organisation be categorised as purely a 'data intermediary organisation'? What if the organisation performs multiple data-related tasks, such as creation, sharing and usage?

Instead of categorising organisations that we will study as 'data creating and publishing organisation', 'data intermediary organisation' and so on, we will identify the functions performed by the organisation concerned. We will study various (but surely not all) organisations that mediate access to and use of government data in India. Many of these organisations are expected to perform multiple data-related functions such as creating data, directly using data to inform organisational activities, sharing the data with other organisations and citizens in general, training other organisations and individuals to use (collect, analyse, etc.) data, etc. Though our
primary objective is to study how these organisations undertake their 'data intermediation' function, what social and technical capabilities and decisions are involved, what challenges and opportunities they experience and so on, we will also briefly document the other functions performed by these organisations.

Then, how is a 'data intermediary organisation' identified in this study?

In the context of this study, we consider an organisation to be a 'data intermediary organisation' if the organisation considers (and its activities reveal) 'data intermediation' as its primary (or co-primary) function. Since a direct understanding of the primary function(s) of an organisation is often difficult to obtain, we plan to approach this problem through various questions (in our conversation with the organisation concerned) and triangulate towards the identification of its various functions. Along with asking the organisation about its different activities, (what it considers to be) achievements, and self-description of the functions it performs, we will specifically explore the 'theory of change' that (explicitly or implicitly) informs the activities of the organisation concerned.

Is a 'data intermediary organisation' same as an an 'intermediary organisation'?

No, the former is a subset of the latter. An 'intermediary organisation' is one that mediates activities between government and government-facing non-governmental organisations (or civil society organisations) on one hand, and the people-facing non-governmental organisations working with grass root organisations on the other. 'Data intermediary organisations' are those 'intermediary organisations' whose dominant or primary mode of mediation involves sharing of data (created by the government and/or non-government agencies).

Will the study only engage with Indian 'data intermediary organisations'? Will that be sufficient to understand and map the ecology of government data access and usage in India?

The study attempts to explore and map the organisations in India that perform 'data intermediation' as a primary or co-primary organisational function, and their interconnections with various other (government and non-government) organisations related to creation, sharing and use of data, as well as for creation of capacity (social, technical, legal, etc.) for data-related activities. We understand that given the lack of (hierarchical) depth of the (government and non-government) data access and usage ecology in India, often the same organisation (especially the smaller organisations) are compelled to undertake multiple data-related functions internally. In
this study, we will not limit our exploration only to organisations that purely function as 'data intermediaries' but will converse with a wider range of organisations that perform that task of mediating data, created by government and non-government agencies, and enable the usage of such data by other organisations.

How will the study identify 'data intermediary organisations' in India and their linkages with the larger ecology of (government and non-government) data access and usage?

We do not begin with a fixed pre-defined list of Indian 'data intermediary organisations'. Part of the study is to explore the (government and non-government) data access and usage ecology in India. As mentioned above, we will converse with a wide range of organisations, selected on the basis of our previous knowledge of their work involving data, created by government and non-government agencies. We will depend on both the 'snowballing' and the 'follow the data' methods to identify further organisations, either through direct suggestions from the first set of organisations we converse with, or by tracing the trajectories of data accessed and used by them. The latter method will also help us to map out parts of the (government and non-government) data access and usage ecology in India.

Will the chosen research method lead to specific biases (such as, towards technically sophisticated and larger organisations based in urban areas) in the map of the data access and usage ecology in India?

Given the scope of the study, both in terms of time and resources, we had a choice between either focussing on a particular sector and engage with organisations working at various layers in the (government and non-government) data access and usage ecology, or focussing on a particular stratum (or layer) of the same ecology, that is focussing on organisations that perform an approximately similar set of functions. While the former approach looks at a vertical slice of the ecology, the latter studies a horizontal slice of the same. As we were interested in studying the (potential) impact of the National Data Sharing and Accessibility Policy and the Right to Information Act upon the (government and non-government) data access and usage ecology in India, we decided to take the latter approach and focus on the stratum of 'data intermediary organisations'. These organisations are closest to the sources of data created and published by governmental and non-governmental organisations, and thus are first in the line to be affected in the government's policies and practices of giving access to (government) data. We are sincerely aware of the fact that this decision to focus on the 'data intermediary organisations' will create a distorted view of the data access and usage ecology of India, with potential biases towards technically sophisticated and larger organisations based in urban areas. However, we undertake this study not with the intention of producing a complete account of the ecology concerned, but to begin its exploration, with a specific interest in understanding the impact(s) of government's data and information policies upon this ecology. Further, we will openly share primary findings from our explorations, as much as possible, so as to be utilised in further studies of the ecology and for usage of the findings internally within the ecology.

Invitation to the Survey of Open Data Practices in India 25 June 2013

HasGeek is doing a study on open data policies and practices in India and the role of intermediary organisations.

In this study, we are trying to map how non-government and advocacy organisations collect, store, publish, use and re-use data. We are also attempting to understand the usage scenarios imagined by and implementation challenges faced by policy-makers and data portal implementors. This mapping is being done to identify possible areas of policy modification, capacity building, community organisation, and alignment of efforts.

This study is part of the 'Exploring Emerging Impacts of Open Data in Developing Countries' (ODDC) research network managed by World Wide Web Foundation and supported by the Canadian International Development Research Centre.

You can read more about the study on the research blog, and about the ODDC research network at the Open Data Research Network website.

We are carrying out an initial survey to identify organisations in India that function as 'data/information intermediaries', i.e., those who mediate between creators/publishers of data and users of data, either by mandate or in the way they work.

If your organisation undertakes such data/information mediations or simply collects and publishes government data, please visit the survey page and register your organisation for an in-depth survey that we will undertake during the next months.

We hope you will consider participating in this survey. Should you have any queries or doubts, please write to us for clarifications and suggestions.

Please also circulate this invitation among relevant individuals and organisations.

Mediating Access and Use of Open Data in India: Reformulating the Research Plan 23 June 2013

Original Formulation of the Study

In early 2012, Government of India approved the first policy in the country governing proactive disclosure of government data, and especially of born-digital and digitised data. This National Data Sharing and Accessibility Policy (NDSAP) extends the mandate of the Right to Information (RTI) Act to establish policy and administrative support to enable informed citizenship, better decision-making and heightened transparency and accountability.

The proposed project originally planned to locate these policy documents, especially NDSAP, in the context of their actual implications and uses for non-government data practitioners. We wanted to focus on non-government advocacy and research organisations working in the area of urban development across five cities in India – Ahmedabad, Bangalore, Delhi, Mumbai and Pune – to map their practices of collecting, accessing, sharing and using government and self-generated urban data, such as those related to property records, geospatial data, sanitation and public health, elected representatives etc. The two key questions for the study were:

• How RTI and NDSAP have transformed approaches and processes of data practices, and
• Can the grounded experience of advocacy and research organisations inform shaping of a more effective open data policy for India?

Reasons for Rethinking the Original Formulation

There are two principal reasons for rethinking the formulation of the study.

First: our initial discussions with policy researchers, RTI activists, and open data advocates revealed curious disjunction between the open data and the RTI movements in India. Some of the major differences are:

• Attitudes towards technology
• Relevance of data and the uses of data
• Relationship with the state (and dependence on the state)

These differences in ideology and approaches of RTI activists and open data advocates are especially important in our specific policy context because NDSAP claims to be developed as an extension of the RTI act. Given these early findings, we are keen to revisit our first research question – how RTI and NDSAP have transformed approaches and processes of data practices – and to categorically explore the differences and dynamics between the RTI and open data communities and the implications of this on the present and future of NDSAP.

Second: the focus on ‘urban development’ was originally chosen to systematically tighten the scope of the study as well as to chose a specific governance context to gather more precise insights. During the ODDC Network Meeting in London, we realised that the urban development framing for narrowing the scope of the project was proving to be a distraction.

Instead, we need to look at key organisations across India (and from different sectors of operation) that are gathering, using, and sharing government data either using provisions of RTI act, or NDSAP, or other procedures. We identify these organisations as data intermediaries that are serving certain data needs of the society at large. It is important to us that the activities of these data/information intermediary organisations be understood within their extended ecologies of inter-organisational collaborations and conflicts, instead of having a sector or issue-specific focus (such as urban data, legal data, budget data etc.).

We are keenly interested in studying how they respond to and utilise RTI and NDSAP for their works with government data; whether RTI and NDSAP respond to the challenges faced by data intermediaries, or the data intermediaries face a challenge to work with data opened up by NDSAP. Further we want to study the factors that determined how these organizations accessed, used and opened government data in different ways, and whether (and why) some of the organisations continue to function in the RTI domain while open data remains peripheral to their everyday work.

The New Research Brief and Objectives

The proposed study attempts to map the actual practices around government data by various (non-governmental) data/information intermediary organisations on one hand, and implementation challenges faced by and usage scenarios imagined by the policy-makers and data portal implementors on the other, to identify possible areas of policy modification, capacity building, community organisation, and alignment of efforts.

Further, the issue of the different (and fairly disparate) communities of RTI activists and open data advocates bring in a very country-specific concern about the challenges of organising activism around government data/information. We expect that exploring the differences and commonalities among the two groups will throw a critical perspective on the open data movement in India.

The proposed study aims to achieve the following goals:

• Inducing collaborative efforts within data/information intermediary ecologies by identifying and highlighting the potentials within the existing network, as well as the specific needs for capacity development and resource sharing;
• Enriching government data/information policy discussion in India by gathering evidence and grounded experience of (non-governmental) data/information intermediaries about their actual practices of accessing and using government data, and their utilisation of the provisions of NDSAP and RTI act; and
• Critically reflecting on the nature of open data movement and policy-making in India.

We will soon share our reformulated study plan and survey questionnaire draft. So stay tuned.

Tweets, Photographs and Slides from the ODDC Network Meeting, 24-26 April 2013 28 April 2013

Here is the Storified collection of tweets, photographs and slides from the discussions and the presentations at the ODDC network meeting.

Thanks to Tim Davies for putting it all together.

Below is a video of Sir Tim Berners-Lee announcing the ODDC project at the Open Government Partnership meeting on 24th April.

Word Cloud of #ODDC Tweets 26 April 2013

Here is the word cloud of the #ODDC tweets from the 'Exploring Emerging Impacts of Open Data in Developing Countries' research network workshop held in The Open Data institute, London during yesterday and today. The cloud was created using Wordle. A few common words, such as 'open', 'data' and 'government', have been deleted for clearer representation.

ODDC Network Meeting, London, April 2013 - Slides and Hand-Out 25 April 2013

The first network-wide workshop/meeting of researchers associated with the 'Exploring the Emerging Impacts of Open Data in Developing Countries' project is taking place in London during 24th to 26th April 2013.

Here is a brief description on our proposed study taken from the hand-out prepared for the event.

Accessing, Opening and Using Government Data in Five Indian Cities: Policy Approaches and Non-Government Initiatives

In early 2012, Government of India approved the first policy in the country governing proactive disclosure of government data, and especially of born-digital and digitised data. This National Data Sharing and Accessibility Policy (NDSAP) extends the mandate of the Right to Information (RTI) Act to establish policy and administrative support to enable informed citizenship, better decision-making and heightened transparency and accountability. The project plans to locate these policy documents, especially NDSAP, in the context of their actual implications for non-government data practitioners.

We will focus on research and advocacy organisations working in the area of urban development across five cities in India – Ahmedabad, Bangalore, Delhi, Mumbai and Pune – to map their practices of collecting, accessing, sharing and using government and self-generated urban data, such as those related to property records, geospatial data, public health, elected representatives etc.

The study will allow us to reflect on two key questions: (1) how RTI and NDSAP are transforming approaches and processes of data practices, and (2) can the grounded experience of advocacy and research organisations inform shaping of a more effective and open data policy for India?

This project employs grounded research approach for analysing NDSAP, RTI and the Government of India’s policies and approaches to Public Sector Information access and sharing. By grounded approach, we mean building narratives and accounts from the field to understand how policies align / deviate from the practical, everyday realities of data gathering, sharing, advocacy, and use of government information, and associated legal and technology challenges.

The grounded approach also helps us to understand how these organisations interact with the state, the responses they receive from government officials, the approaches they employ in obtaining government information, and how these approaches inform their understanding of the state and government. We plan to take these findings back to a critical discussion of the challenges and opportunities of open government data in India, with a specific focus on making NDSAP more sensitive about requirements of actually existing data practices among civil society researchers and advocates.

As part of the study, we will generate information about the nature of data that different research and advocacy organisations are collecting in the five Indian cities concerned, and share the tabulated findings as an open archive of data practices.

The hand-out text can be downloaded in .pdf format from here, and the slides to be used in our presentation at the workshop can be downloaded from here.

Notes from Workshop organised by NDSAP-PMU, NIC on 4th April 2013 21 April 2013

The National Data Sharing and Accessibility Policy Project Management Unit (NDSAP-PMU) at the National Informatics Centre (NIC) organised the second workshop for Data Controllers from various ministries on 4th April 2013.

Data Controllers are persons responsible in each Central Government agency for preparing and managing datasets from the respective agency to be shared through the data.gov.in portal. S/he is also acts as the point of contact between the agency concerned and the NDSAP-PMU team at NIC.

While the workshop focused on knowledge exchange, discussing and identifying best practices for sharing data through the data.gov.in portal, and some troubleshooting and process clarifications, the final session of the workshop was assigned as a ‘community interaction’ session. At first, NDSAP-PMU selectively invited individuals involved in open data communities and/or associated with the data industry and academia. Later, participation in the interaction was opened up to all interested people.

Besides officials of various government agencies (such as NIC, Planning Commission, Defense Research and Development Organisation, Border Security Force, Ministry of Panchayati Raj and Ministry of Urban Development), represented non-government bodies included NASSCOM, IBM Research, Tata Institute of Social Sciences, Sharda University, O.P Jindal University, Centre for Internet and Society, Cyber Media Research, DataQuest, Indian Express and Akvo. A number of independent programmers and data enthusiasts were also present.

The interaction session was moderated by Ms. Neeta Verma, Head of the NDSAP-PMU team and Ms. Alka Mishra and Mr. D.P. Misra, members of the NDSAP-PMU team.

The session began with a brief presentation on the existing datasets on the data.gov.in portal and an introduction to the processes of ranking existing datasets, on-site visualisation and multi-format exporting of the datasets, and requesting further datasets from different departments. The discussion immediately began around issues of updating, modifying and augmenting the existing datasets. Concerns were raised regarding the storage of older versions of the same dataset on the portal, whether updated statistics should be put in to the same data files or be shared as new files, etc. It seemed that the Data Controllers are keen to understand effective solutions to these problems and are open to discuss the issues with the various users of the data.

As a question was raised regarding the slow growth of the number of datasets on the portal, it was mentioned that NIC has put in significant effort to ensure even the slow growth, especially by working closely with various departments to clean up their datasets to make them shareable. Further, NIC is aware of the slow growth and has in fact consciously moved slowly to populate the data portal to ensure that capacities for preparing government data to be shared publicly are established across agencies (including NIC). The NDSAP-PMU team feels that the best way to energise proactive data publication by different departments would be by demonstrating various use cases of such opened up data.

Unlike the Right to Information Act, NDSAP is not a statutory directive and hence cannot be implemented by the force of punitive measures. The policy functions by making explicit the benefits of open government data for the citizens as well as for intra-governmental data sharing. It was emphasised that the non-governmental sector, from private data analytics firms to advocacy organisations, has a great role to play in creatin on-the-ground examples of effective solutions based on government data available from the data.gov.in portal.

Several Data Controllers expressed that this initiative is quite a novelty among other data-related initiatives of the Government of India. Hence they are sometimes unsure about carrying out the necessary work, such as identifying high-priority datasets for proactive publication, or establishing a agency-wide process to anonymize datasets and to check whether they can be shared publicly. NIC said that it would continue to engage with such issues as they emerge and support Data Controllers acorss agencies.

Biplab Dasgupta of IBM Research made a small presentation about the possibilities of creating economic value and induce innovation through government data analytics. Sudarshan Rodriguez of TISS emphasised that the data portal should open up Indian government data to facilitate multi-perspective analysis, collaborative research, and transparency in governmental actions.

At the end, the NDSAP-PMU team was asked what the government (or Planning Commission and National Innovation Council) are expecting from the (then) upcoming 12th Plan Hackathon. In response, the team clarified that it is an explorative initiative from the government with a call to produce innovative and heterodox interpretations and representations of the actions being proposed in the 12th Five Year Plan and the statistical evidences behind them.

Note: Re-posted from Sumandro's blog.

Introducing the Project 21 April 2013

We are happy and proud to announce that our proposal to study the National Data Sharing and Accessibility Policy (NDSAP), approved in January 2012, and its implications for non-governmental research and advocacy organisations working in the field of urban development in the cities of Ahmedabad, Bengaluru, Delhi, Mumbai and Pune, was accepted and granted funding as part of the 'Exploring the Emerging Impacts of Open Data in Developing Countries' (ODDC) research programme managed by the World Wide Web Foundation and funded by the Canadian International Development Research Centre (IDRC).

The study will be carried out by HasGeek, with Sumandro Chattapadhyay and Zainab Bawa as principal researchers.

The work comprises of detailed policy studies and interviews of government officials involved in making and implementing of the NDSAP document, advocacy and research organisations engaging with government data as part of their work on urban development issues, and a concluding knowledge sharing workshop with governmental and non-governmental representatives.

We will continue to document and share our research process in this blog.

For further information and/or clarification, please write to Zainab and myself at following e-mail addresses: zainab[at]hasgeek.com and mail[at]ajantriks.net.

            

Note: The funding for this work has been provided through the World Wide Web Foundation 'Exploring the Emerging Impacts of Open Data in Developing Countries' research project, supported by grant 107075 from Canada’s International Development Research Centre (web.idrc.ca). Find out more at www.opendataresearch.org/emergingimpacts