In the past, when government schemes were more on paper and less on the ground, the incentive to conceal was lower, and would usually fit the acceptable margin of error in surveys.(Illustration by C R Sasikumar)“The Household Survey” has always been among the most preferred and reliable tools to collect authentic information on almost anything — who has access to what facilities, which brands are better than others and everything in between. After all, what better way can one arrive at what a person has, wants or feels than just asking the person, right? Wrong.
The Ministry of Statistics and Programme Implementation (MoSPI) has been conducting large-scale and nationally-representative sample surveys, which rely on soliciting information from respondents through canvassing by field enumerators of the National Sample Survey. The Department of Drinking Water and Sanitation (DDWS) also has some experience in running large sanitation surveys over the past few years. In our combined experience, we have observed that over the past few years, the responses received during some of these large surveys are not always realistic. This is especially true when we attempt to gauge access to services for which there are ongoing and popular government schemes which are linked to monetary or other benefits like subsidised food, free clean cooking fuel, housing or sanitation infrastructure. The respondents are not always truthful, and seem to want to conceal the extent of benefits that they have received. But why?
It all boils down to basic behavioural economics. Although field enumerators attempt to elicit genuine responses, people respond favourably to incentives, either direct or perceived. A genuine response based on a sense of nationalism is diminishing.
When a person is asked a question about whether or not he or she has access to a facility that the government has promised to provide, there is an inherent incentive to deny having access to or receiving a benefit for the facility, in a hope that this may allow him or her to receive the benefit of the scheme again. This is even more relevant if the people have seen the actual delivery of the promised benefits for themselves, their kin or neighbour. In the past, when government schemes were more on paper and less on the ground, the incentive to conceal was lower, and would usually fit the acceptable margin of error in surveys. But today, when the benefits of government schemes are actually reaching the intended beneficiaries much more than ever before, the incentive to deny having facilities is at its highest. On the contrary, what does the household gain by acknowledging that it in fact does have access to the facility? Nothing.
We are not the first to be speaking of this phenomenon. Recent articles by columnist Swaminathan Aiyar and academic Neeraj Kaushal have also alluded to this deliberate respondent bias in many national surveys. Both our ministries, however, have experienced this play out first hand. In 2017-18, the DDWS commissioned the Quality Council of India to conduct a national survey on access to sanitation facilities. The QCI did a round of basic training of their field enumerators — typically local youth from the region to be surveyed, before starting the survey of 1.4 lakh randomly selected households. Midway through this exercise, they found that the numbers on access to toilets reported during the survey in some states in North India were significantly lower than the administrative records. They brought the findings to DDWS and decided to send some senior, well-trained enumerators to re-verify if the situation on the ground was really the way it was being reported.
And so, with a fresh round of training, a small group of senior enumerators from the QCI team went back to some of the villages and visited the same households. It is during this exercise that they found clear evidence of the inherent respondent bias during surveys. Many households denied having a toilet, but upon further probing and on convincing them that the findings of the ongoing survey would have no bearing on whether or not they received the government benefit for a toilet, most of them accepted that they in fact did have access to a toilet. Some had a toilet within their household premises, some had it as far as 100 meters away from their home which no enumerator would have known about! The re-survey team returned with photographs of these toilets, and in some villages the number of toilets were more than twice the initial estimate.
This exercise taught us two very important lessons. One, there is no incentive for people to correctly report their well-being during a survey, especially if there are specific ongoing government schemes to provide products and services. In fact, there is a clear incentive to under-report in such circumstances. And two, the enumerators may not have the capacity or the commitment to probe for genuine responses. More often than not, they will simply accept the first response that the household gives and are not trained to extract the facts. If we are serious about soliciting genuine responses, we need to invest a lot more time and effort to train our enumerators to probe for the truth.
The recently released report of the 76th Round of NSS on “Drinking Water, Sanitation, Hygiene and Housing Condition” recognises this problem of deliberate under-reporting and mentions it in the report alongside the results. In this particular case, the under-reporting may have been accentuated even more as the survey also had a question on whether the household has received a benefit for toilets in the past three years. One can safely conclude that the moment a question on benefits received from the government for sanitation was asked, it would have conjured up an image of the Rs12,000 financial incentive linked to a toilet in the mind of the respondent, and this would have likely influenced the response itself, irrespective of what the reality on the ground was. There may also be a need for greater sensitisation of respondents to faithfully report information as responsible citizens, on the assurance that confidentiality of the data reported is maintained.
The MoSPI is committed to further improving the quality of data that it collects and presents and has made it an integral component in its Vision Document. It has also engaged with the Ministry of Skill Development and Entrepreneurship to design skill training courses for survey enumerators, supervisors and data quality personnel who could then be used for such surveys. In addition, the MoSPI is evolving contemporary instruments and tools to better capture information and reduce respondent bias. It will take time for these efforts to bear fruit.
This article first appeared in the print edition on December 4, 2019 under the title ‘Between the lines of a survey’. Srivastava is secretary, Ministry of Statistics and Programme Implementation, and chief statistician to the Government of India. Iyer is secretary, Department of Drinking Water and Sanitation, Ministry of Jal Shakti.Views expressed are personal.
