Afterword

ThimphuTech was the first technology blog in Bhutan. We started writing it in 2009, just as broadband and mobile internet started to take off. (Although internet in Bhutan was launched in 1999, it was either super-slow or super-expensive, and was only used by a selected few).

In the blog, we wrote about technology and food, but also about plenty of other stuff. The blog became popular and influential in Bhutan. A companion bi-weekly column -- Ask Boaz -- was published for many years in the Kuensel, Bhutan's national newspaper. (The complete Kuensel columns are available as an ebook, Blogging with Dragons).

We stopped updating the blog when we left Bhutan in 2014, but the information within the posts can still prove useful, and thus we decided to keep it online.

We thank all our readers.
Tashi Delek,
Boaz & Galit.

Thursday, August 2, 2012

SMS and Internet surveys in Bhutan

Note: this post is aimed at clarifying the statistical aspects of polls and does not take a stance on the Pedestrian Day issue.

Pedestrian Tuesday has raised a debate regarding the public sentiment, with different sides showing contradicting poll results. Following these polls, the issue of the validity of Internet and SMS polls has been raised. Today's Kuensel editorial brought up the critical issue:
SMS-phone polls and online polls, although gather huge pools of respondents, rarely represent the larger population. 
Why is this true?

One reason is that those with no Internet access or mobile phone are not represented in the sample. In Bhutan, most citizens do not have Internet access. An Internet poll is therefore likely to represent office-goers and those who can afford and have access to broadband or mobile Internet. Mobile phones are much more popular, although I don't know the figures regarding population coverage. But here too we must be careful: SMS voting requires understanding how to send an SMS, understanding the question (which language was it in?), affording the cost of an SMS, watching BBS during the polling time frame, etc. 

A second reason that the SMS and online polls that have been carried out do not represent the population is that they are voluntary response samples. In such samples, responders are often those who feel most strongly about the issue. In this case, we'd expect to hear from those who strongly oppose or strongly support Pedestrian Day. If the majority is not strongly opinionated, we will not know. If those opposing Pedestrian Day are more strongly opinionated, our sample will over-represent that group's opinion.

Given that mobile phones are much more prevalent than Internet access, let us consider SMS polls in further detail. Even if we assume that everyone in Bhutan has a mobile and is able to SMS their opinion; and even if BICMA's advice "to permit only one vote or call per telephone number"; How do we take into account the fact that quite a few people have multiple SIM cards? For example, if there is a correlation between the number of SIM cards and taxi ownership, then maybe our SMS sample over-represents taxi drivers. Or maybe the number of SIM cards is correlated with the number of cars owned, thereby leading to a sample over-representing car owners.

The proper way to conduct a poll is to randomly draw a sample of the population (a sample of 1,000 people is sufficient to provide a precise picture of the entire population's opinion), and then contact everyone in this sample and register their response. For instance, we can draw a random sample of 1,000 mobile numbers from the list of all mobile numbers in Bhutan (for simplicity, we ignore issues of multiple SIM cards, people with no mobiles, etc.). We then contact each of these numbers and only these numbers, and register their response either by SMS or an actual call. Because we want to make sure to reach as many of the 1,000 people as possible while minimizing cost, it might make sense to start with an SMS, and then follow up with non-responders via a phone call.

Aside from the sampling issues, there is also the issue of how the question is asked. How do we avoid an imbalanced question? How do we make sure the question is understood correctly? Questionnaire design is yet the subject of a another scientific field, and as crucial for deriving valid inference.

There are scientific methods for designing a poll, so that the results are sufficiently precise in capturing the population's sentiment. Polls have been conducted for many years by governments, market research companies, and other organizations. While technological advances have changed the polling mechanism (from polls by mail or by personal contact to email, SMS and Internet polling), the scientific principles of how to poll and how to analyze the resulting data are still valid.

Lastly, even if the unscientific SMS and online polls are not treated as representing the population, their visibility on public media does influence public opinion, the results of which are now probably too late to assess.