A close look at the long-term impacts of mobile money study

In 2008, the New York Times ran a story asking “Can the cellphone help end global poverty?”. Last year, in the concluding sentence of a study published in Science, the authors - Tavneet Suri of MIT and William Jack of Georgetown University - provided an answer: “although mobile phone use correlates well with economic development, mobile money causes it.”

This is a rather bold claim that I think stretches the findings a bit too far beyond the support of the evidence. Not surprisingly, in the days since it was released, the study received widespread attention in both mainstream and social media (For example, here, here and here). This is a very strong and carefully done study and one that deserves attention but it is also important to acknowledge that it is a single study, with certain limitations, in one specific context. Before we run off and declare M-PESA the solution to poverty or the best thing society can do for female headed households, it is important to fully appraise the evidence.

Why this study is important.

On one hand it is clear that M-PESA has been a run-away success. in 2016 it generated USD 415 million in revenues for Safaricom, is used by 78 percent of Kenya’s adult population and in a context where banking infrastructure was until recently very sparse (in 2007, when M-PESA launched there were 1,000 ATMS, or about 5 ATMS for every 100,000 adults) it created a much cheaper, safer and more efficient way of sending and receiving money over distance than the next best alternative, which for many people involved long travel or sending cash through a third party. Indeed, that is the reason why M-PESA became such a success, a job that many people needed doing: sending and receiving money within their social network could be done so much more easily with M-PESA. Now people don’t just sent money, they “M-PESA” it.

On the other hand, many champions of mobile money and digital finance believe that innovations like M-PESA do much more than generate concentrated benefits for service providers in the form of profits, and diffuse benefits for customers in the form of consumer surplus or cost savings. Certainly this belief motivates many of the institutions funding efforts to boost financial inclusion around the world. Along with a series of cross-country studies looking at the impact of access to microcredit and earlier research by Suri and Jack that explored the short run impact of mobile money on the ability of households to cope with unexpected shocks, there aren’t many studies out there that provide good evidence on the effects of commercial financial services on the long-term economic well-being of the households that access those services.

This is an observational study that makes causal claims, so handle with care. This research study pursues the thorniest of questions in the social sciences: a causal one. In a nutshell, the authors ask “does M-PESA cause increases in household consumption and reductions in poverty?”. You might think to yourself, “why not just compare households that use M-PESA versus those that don’t use M-PESA as a way to estimate the effect?” The reason is that if you went out today and took a sample from the 78 percent of adults in Kenya that use M-PESA and compared their consumption levels to a sample from the 22 percent of adults that don’t use M-PESA, much of the difference would be attributable to the fact that on the whole, M-PESA users are more wealthy, more male and more educated than non M-PESA users . You could try your best to compare apples to apples, for example by only comparing users and non-users of M-PESA who are urban and female and who have primary education, but you still would not be sure that you’ve controlled for other factors - such as how socially connected they are - that influence both M-PESA use and future economic outcomes. In any causal analysis, the trick is finding a way to compare economic outcomes among users and non-users of M-PESA (or any other intervention) who are otherwise identical.

Economists have increasingly been using randomized control trials (RCTS) to do exactly this. In an RCT, researchers run experiments in which an intervention or treatment (like a drug or access to a training program) is randomly assigned to some people (a “treatment” group) but not others (a “control” group). Randomization attempts to ensure that people in the treatment and control group are similar across both easily observable characteristics (e.g. height, education, income-level) and unobservable or harder to measure characteristics (e.g. attitude, effort, discipline). As a result, any differences in outcomes between the treatment and control group can be interpreted as being caused by the treatment . For ethical and practical reasons, however, RCTs aren’t always possible or desirable.

In this study, the authors use a “natural” experiment created by the apparently random way the M-PESA agent network was rolled out shortly after its launch. Rather than follow a well-planned demand-led approach of, for example, expanding the M-PESA agent network to fast growing urban towns or rural market centers, the authors argue that in being overwhelmed by agent applications, Safaricom did not “have the time or resources to approve applications based on local conditions”. As they state, to estimate the impact of M-PESA in this study, the “key assumption is the change in agent density between 2007 and 2010 is uncorrelated with household and regional characteristics that might have driven demand for the service and future economic outcomes." A series of statistical tests are presented that provide evidence that households in places where the agent network grew fast were no different than households in places where the agent network grew more slowly . For the most part, this assertion seems to hold, save for one exception that can be seen in Table S2A of the supplementary materials. Agent density growth in 2008-2010 was correlated with bank account use in 2008. This may suggest that agent density growth was related to the banking infrastructure footprint when the study started.

This study does not estimate the long run impact of using M-PESA. More specifically, the study estimates the economic outcomes that materialize four years after exposure to changes in the number of mobile money agents near households that occurred in the initial years following M-PESA’s launch. Perhaps the most important thing to know about the methodology is that the impact of mobile money is estimated using changes in the number of mobile money agents within 1 km of households between 2008 and 2010. In this way, the study is not measuring the impact of the observed use of M-PESA by members of a household, but a proxy - the ability of households to access the cash-in cash-out services provided by agents in their immediate neighborhood. Further, because many changes were implemented in the 4 years between 2010 and 2014 including the passage of the national payments act which allowed banks to use existing agent networks and the integration of new services, such as M-Shwari, with M-PESA, the authors clarify the correct way to interpret their results: “Our estimates of the long-term causal impact of the initial deployment of agents thus include the effect of the subsequent competitive deepening.”

The results of this study do not generalize to all of Kenya. This study is based on five rounds of a household panel survey conducted in 2008, 2009, 2010, 2011 and 2014. Design decisions made when the initial sample was drawn, as well as attrition over time, limit the generalizability of the study. Given that the authors wanted to increase the chances of finding M-PESA users (which were rare at a time when M-PESA was only just starting out), they excluded the poorest locations in the north and north east of the country with inadequate cell-phone coverage, as well as locations across the country without at least one M-PESA agent. Also, given that a high fraction of households in Nairobi were lost to attrition, Nairobi was dropped from the study after 2011. Overall, the sample used in the study represents around 82% of the non-Nairobi, non-North population. It is also worth noting that given the exclusion of locations without an agent in 2008 and the fact that the authors over-sampled locations with higher numbers of agents, the sample is skewed to more urban settings. Among the 3,000 households that participated in the 2008, first round survey, 64.5 percent lived in urban areas which is more than double the national urban population rate. Given attrition and the exclusion of Nairobi, the sub-sample used in this study is made up of the 1,593 households that had data from both the 2008-2010 rounds and the 2014 round.

The big finding in this report is that the impact of the mobile money system on poverty seems to be really big for female headed households. The authors estimate that for each 1 agent difference in the 2008-10 growth in the number of agents within 1km between two groups of households, the difference in the 2014 average level of consumption per person in those households was 1.2 percent. If the household is headed by a woman, the difference in average consumption per person is more than double - 3.1 percent. To recap - because consumption in 2014 could not have caused agent density growth in 2008 and because agent density growth is not correlated with other observable factors that might have influenced changes in household economic welfare, the authors argue that this effect can be given a causal interpretation.

A more grounded way to state the findings can be constructed using some of the values in the data. Between 2008 and 2010, the number of agents within 1km of households increased by an average of 4.7. Households earning near the poverty line (around Ksh 115 per person per day) who were exposed to the typical increase in agent density of 4.7, had consumption levels in 2014 that were KSh 6 higher per person per day (or an additional Ksh 183 per person per month) than households that were not exposed to any change in the agent density. For female headed households, the difference was Ksh 18 per person per day (or Ksh 550 per person per month) almost three times the overall effect. The magnitude of this effect is very large: an extra Ksh 550 per person per month translates to close to Ksh 1,650 per month for a family of three and over the course of the year that adds up to an extra Ksh 19,800 or USD 220.

In a second regression, the authors assess the relationship between changes in per capita consumption between 2008 and 2014 and changes in agent density. Overall, the authors do not find that changes in agent density were related to changes in consumption, however, they find that for female headed households, a 1 agent increase in agent density between 2008 and 2010 is associated with a 1.4 percent change in household per capita consumption between 2008 and 2014. The negligible overall effect seems to be a result of there being a negative association between changes in agent density and consumption for male headed households, which mutes the positive effect for females. For a female head earning near the USD 1.25 poverty line in 2008 who was exposed to the typical change in agent density (of 4.7), the expected change in their consumption per capita as a result would have been on the order of 6.5 percent – or an increase of approximately Ksh 7 - from Ksh 115 to Ksh 122 per person per day.

While the evidence of impacts is strong, the mechanisms that generate them are unclear. Prior research indicated mobile money was able to better insure households against shocks by providing better access to informal remittance networks. This latest study reinforces that finding and also hints that the longer term impact on female headed households might also be a result of enabling additional savings and transitions to higher-return occupations. However, the authors find no evidence that having access to more agents increased the likelihood of female headed households using M-PESA as a savings instrument. What is not widely discussed in the paper is that female headed households are particularly vulnerable and marginalized in the Kenyan context. The 2016 FinAccess household survey estimates that out of all households in Kenya, 22 percent are headed by a female, and out of these, 78 percent are single parent households (due to being unmarried, separated, divorced or widowed). These households face immense pressure as one parent has to provide both income and caretaking duties for the family. The ratio of income earners to school-age children in female headed households is about 20 percent smaller than male headed households on average (Table 1). If there are any households that need social support through transfers, it would be these households and so the effects that are picked up in the study may have more to do with mobile money supporting a social insurance function, rather than a savings and investment function. The negative changes in consumption per capita for male headed households combined with the positive changes for female headed households perhaps may be evidence of redistribution from male headed households with higher purchasing power to female headed households. The authors suggest that “the primary impact on female-headed households was therefore to protect them from falls in consumption in the longer term or to boost it marginally, consistent with our earlier results on risk sharing”.

Table 1: Demographic characteristics of female and male headed households

Lastly, the study finds that female headed households with greater exposure to the mobile money system were less likely to be working in farming or be stitching together incomes from multiple part-time jobs and more likely to be in business or sales. In light of the previous discussion, for female headed households with significant care-taking responsibilities, being able to substitute more remittances for time-consuming farm work would make sense. If these remittances enable additional savings, as the evidence suggests, then, at least for some households heads, they could have been channeled to invest in buying stock or equipment to start a small business.

Several questions remain. It is important to keep in mind that the results of the study are based on comparisons not between households with zero access and some access to the mobile money system, but between households with lesser and greater access. After all, most if not all households in the study had access to at least one agent within 1 km, which is quite close. So was there something about a higher density of agents that enabled the mobile money system to work better? For example, by ensuring that households could access their money when needed by having multiple agent options or did a higher density of agents help ensure that as a whole agents were managing their inventory of cash and e-float more effectively? Did having access to an increasing concentration of agents nearby, promote more usage of M-PESA? Or did denser networks agents help infuse local economies with additional resources that helped improve local economic conditions and purchasing power in the medium run? With mobile money becoming as ubiquitous as cellphones, it will be extremely difficult to untangle these questions going forward but overall the findings do support one overarching conclusion and that is that mobile money and the agent networks that support it have emerged as a cross-cutting institution that carries significant potential to alleviate the day-to-day challenges of poverty and to support poverty reduction over time.