Updated 11/4/15, with responses from the Big Results Now team and the Ministry of Water – see below.
Justin Sandefur of the Centre for Global Development (CGD), writing in the Washington Post, presented five charts that may soon be illegal in Tanzania. He was referring to the Statistics Act, recently passed by the Tanzanian parliament, which makes it a criminal offence to publish false statistics, or statistics “that may result in the distortion of facts.”. This is punishable by a minimum 10m/- ($6,000) fine and/or a minimum 3 year prison sentence.
Here, I have done something similar. But I only refer to statistics produced or cited by the Tanzanian government itself.
My purpose is not to accuse any particular part of the government of deliberately misleading people, but instead to point out some of the difficulties of making it an offence to publish false or distorting statistics.
1. How many people have gained access to clean and safe water under Big Results Now (BRN)?
Both the Ministry of Water and the team behind the government’s flagship BRN initiative report the number of new waterpoints installed under BRN between July 2013 and June 2014, and the number of new people served as a result. Both are reporting on exactly the same thing and the same time period. But the figures are different. At least one must be wrong – i.e. false.
I don’t know which of the two is wrong, but whichever it is (or both), will be illegal under the new law. It will also be illegal for anyone to report the incorrect statistic – whichever it is – in the media.
Here’s the evidence – click on either image to see a larger version:
You could argue that unreliable government statistics are indeed a problem that needs to be addressed. But is it right to punish those responsible with a minimum sentence of three years imprisonment? And is it right to punish a journalist (and their editor) for publishing what appears to be a well-sourced statistic?
2. How well are children doing in secondary school?
The way O-level exams in Tanzania are graded was changed both in 2013 and 2014. First, in 2013, the score required to achieve a grade A in a particular exam was reduced from 81% to 75%, the range for a B was reduced from 60-80% to 50-75%, and so on. The same year, the number of subject passes needed to be awarded a good pass (Division I-III) was reduced. And the following year, the system was changed again, to a Grade Point Average (GPA) system.
The first result of the changes is confusion. It is now very difficult to compare students who sat their exams in different years.
It could be argued that the second result is deception. Look at the chart below, see how the overall pass rate had been sharply declining from 2007 to 2012, then suddenly shot up again when the new grading systems began to be introduced. This holds both for “any pass” (i.e. those achieving Division IV or above under the old system, and a “pass” under the new system) and for “good passes” (Division I, II or III / “credit, merit or distinction”). There is little doubt that the new grading system is responsible for most or all of the increased pass rate.
It is impossible now to meaningfully compare pass rates before and after the changes. As such, any government statement or media report that reports an “improvement in the pass rate”, or uses the figures to claim that “schools are now doing a better job than previously”, would be publishing statistics that are “distorting” or perhaps even “false”.
3. Do Tanzanians want to change the structure of the union between Zanzibar and the mainland?
The Constitutional Review Commission (CRC) reported that 61% of those on mainland Tanzania who expressed a view on the nature of the union preferred a three-government arrangement.
President Kikwete then used the very same data to point out that only 13.6% of those who gave their views to the commission were sufficiently unhappy with the form of the union to raise the issue:
“So people are asking how today 13.6% of all Tanzanians who gave their views has become the majority of Tanzanians!”
In fact, neither the Commission nor President Kikwete’s analysis of the data is statistically correct – those who had the time and motivation to give their views to the commission are not a representative sample. But let’s leave that point aside for the moment.
Instead, we can see this as an example of differing interpretations. Justice Warioba and President Kikwete reach opposite conclusions from the same data. Does that means that either the chairman of the Constitutional Review Commission or the country’s President has communicated statistics that are “false” or “may result in the distortion of facts?”
UPDATE – 11/4/15
1. The Big Results Now communications team tweeted yesterday in response to this post. First, they stated that the Ministry of Water figures quoted in this post covered a longer time period, then accepted that there must be an error in one or other set of figures:
@mtega Takwimu za ripoti ya BRN ni za mpaka kufikia Juni 30,2014 (mwaka wa kwanza wa utekelezaji wa BRN), Wizara inatoa data za mpaka sasa
— BIG RESULTS NOW! (@BRN_TZ) April 10, 2015
@mtega Inawezakana kuna kasoro katika kuweka hilo..wizara inaendelea ku-apdate data kila siku…BRN inaripoti za kila mwaka.
— BIG RESULTS NOW! (@BRN_TZ) April 10, 2015
@BRN_TZ Maana ya "kasoro" ni kwamba ama MoW ama BRN imetoa takwimu zisizo sahihi. Sisemi ni kwa maksudi, ila kwa sheria mpya itakuwa kosa
— Ben Taylor (@mtega) April 10, 2015
2. The figures I cited from the Ministry of Water website have now been removed from the site. The screenshots below show how the text that was there two days ago on the BRN page on the Ministry’s site has now been removed: