> Posted by Ignacio Mas, Independent Consultant
I guess it happens in all human endeavors; we sometimes get carried away wishing things were the way we think they ought to be. Let me provide three cautionary observations relating to financial inclusion: about how we measure it, how we talk about it, and how we assess it. The point is not to dampen enthusiasm about the possibilities, but to reflect on our progress in a more realistic way.
Industry Showcases and the Numbers Game
Through numerous industry conferences and blogs, certain players get put up as shining examples for the industry to follow. M-Shwari is perhaps the latest one, I guess because it delivers large customer numbers to an industry that is still largely focused on coverage rather than usage, and it represents the kind of telco-bank partnership that many have been fantasizing about.
M-Shwari may indeed be every bit the financial inclusion success that it is made out to be, but how is one supposed to judge that, based on the sparse numbers that have been released? This is pretty much all we know: as of March 2014, 6.8 million registered customers, of which 3.6 million were active, collectively had $46 million in deposits and $14 million in loans outstanding; 15 percent of loan requests were approved, and 2.7 percent of loans were non-performing. Now, is the savings balance total the result of each of the 3.6 million active customers squirreling away $13 for a rainy (more like drizzly) day ($46m ÷ 3.6m = $13), or is it more likely that most of the savings comes from fewer than 100,000 busy traders who are saving $500 ($46m ÷ $500 = 92,000) in order to create more transactional head-room for their linked M-Pesa account? We just don´t know. But then, how can we have an opinion on M-Shwari’s efficacy as a financial exclusion buster?
We must refrain from unduly extolling cases on which we have not been invited to know enough. We need to look beyond average balances, which are typically highly skewed by large balances at the very top of the distribution. Donors, policymakers, and pundits must start asking for customer distributions before parading any savings effort as a success.
Digitization of Payments and the Journey to Cash-Lite
What do we mean when we talk about digitization of G2P payments? Most government payments have for a long time been digitized, at least at source. No ministry holds a huge cash stash to pay pensions and welfare benefits. Now the trend is to pay these directly into beneficiaries’ digital accounts, rather than to intermediary entities for onward distribution of the cash. But when beneficiaries are paid digitally, the practice pretty much universally is for them to withdraw the money in cash immediately and in full at local shops acting as agents, who have the thankless task of procuring the cash. So, in what sense has the payment been digitized? The same amount of cash is still involved, it just got to beneficiaries through a different channel. All we’ve done is outsource last mile cash distribution to retail outlets, via a financial service provider.
This is not to say there is no net benefit: account-based G2P payments are much less prone to corruption, and bank agents may be closer to where beneficiaries live than the old government cash distribution points. But people’s money is for the most part no more electronic than it was before. And by the way, this is true as much for P2P as for G2P payments, and in Kenya as much as everywhere else.
Moreover, because most electronic accounts are largely empty, users do not have a natural preference for paying electronically at the corner store, so local electronic acceptance by local merchants does not take off. Contrary to frequent commentary, digitization of payments is therefore not leading to a cash-lite world. We are confusing the digitization of payments (how money moves around) with the digitization of money (how money is held).
Impact Evaluation and Silver Bullets
Few would argue that there are any silver bullets—understood as simple solutions to complicated problems—in development. We know that progress occurs from the interplay of various forces and interventions—access to education, information, markets, finance, infrastructure, legal and physical security, etc.—and that none of these individually stands a chance to transform lives. The impacts of finance happen mostly through indirect channels, through a process not unlike multiple particle collisions. So when we insist on measuring the impact of financial inclusion programs by carefully isolating single treatments/collisions, aren’t we secretly wishing to find a silver bullet?
While I understand intellectually the need to conduct impact evaluation, how realistic is it to expect to find sustained, systematic impact from narrowly defined and precisely controlled financial interventions? How much mileage will we get from building ever-more precise Hadron Supercolliders in the social sciences?
I’m just sayin’…
Image credit: IPA
Ignacio Mas is an independent consultant. You can learn more about his work here.
Have you read?
Microfinance and Savings Outreach: What Are We Measuring?
Measuring (and Missing) Financial Inclusion
Electronic G2P for the Poor: Does Electronic Social Protection Need Better Consumer Protection?
Facebook
Twitter
8 comments
Comments feed for this article
August 6, 2014 at 9:55 am
James Militzer
Interesting read, thanks Ignacio.
August 6, 2014 at 12:19 pm
Kim Wilson
Great post. As to point number 3, but a tangent, I would want to know if better information actually leads to changes in policies or practices. I was at an interesting lecture yesterday where some new and really outstanding research challenged one-off surveys. But was the data really so different that a decision-maker would make a really different set of decisions? It all reminds me of Borge’s Mapmaker story (http://www.sccs.swarthmore.edu/users/08/bblonder/phys120/docs/borges.pdf)
When is precision silly?
August 7, 2014 at 5:47 am
Y P Issar
The blog correctly challenges the present methods of measuring success of FI. I may be wrong but I find western donee/ research organisations more sympathetic and positive and shying away from taking a hard hitting view of real situations in field. As rightly observed, FI is long drawn process.
August 9, 2014 at 4:58 pm
Julie
Ignacio, as usual, all good points. I particularly liked a rather smallish point about the “drizzly” days. Even when poor people save a substantial portion of their income, it can often only be enough to cover the smallest of risks. Just one more way we need a reality check on around what we can really expect savings to achieve.
August 10, 2014 at 11:40 am
Hugh Sinclair (@MFheretic)
That we over-hyped the benefits of microfinance is now widely accepted. I think most of us privately cringe when we think of Yunus’s bold statement that we’d shortly be opening poverty museums. It’s nice to see, even on an Accion blog, that such humility is creeping in. But the author repeats a worrying comment in the third section, on impact evaluation, by comparing research on the effectiveness of microfinance to the complexity of particle physics. Can it really be that difficult to find at least some credible evidence that microfinance is working? We can sit around criticising the various RCTs on technical grounds, but isn’t it slightly worrying that there is no convincing evidence to date? It reminds me of the 2010 paper by Accion, Opportunity, Unitus, Grameen Foundation and Finca basically rejecting the findings of rigourous academic research in favour of their own carefully selected (and often sponsored) anecdotal quasi-research, “Measuring the impact of microfinance: our perspective”:
http://www.accion.org/Document.Doc?id=794
Pitt and Khandker’s original paper supposedly finding a positive impact of microfinance has been torn to shreds. Duvendack’s seminal literature review, spanning literally thousands of academic papers over three decades, was largely ignored in favour of the sector’s own literature review which, perhaps unsurprisingly, focussed on the quasi-research that found positive impacts. In many cases of papers that had been rejected by Duvendack as failing to meet basic standards of academic rigour. Their occasional book reviews focus on those that support the microfinance mantra:
https://cfi-blog.org/2011/08/23/five-great-books-about-microfinance-and-how-the-poor-use-money/
The featured book is by none other than Elisabeth Rhyne, coincidentally a long-term employee of Accion!
Dean Karlan’s extensive RCT in none other than Accion’s beloved Compartamos Bank apparently discovered some positive impacts, until people actually read the paper and realised all it really discovered was mediocrity:
http://blog.microfinancetransparency.com/disguised-mediocrity-the-quest-for-positive-impact-results-at-compartamos/
This blog’s review of Karlan’s recent book (https://cfi-blog.org/2011/06/14/more-than-good-intentions-by-dean-karlan-and-jacob-appel-book-review-by-guy-stuart/) was fairly critical once again of the academic approach to analysing microfinance, and failed to point out that while the authors discuss microfinance in their book, they do NOT include it as one of their suggestions on how to tackle poverty in their concluding chapter.
Perhaps I am being cynical here, but it seems that those inside the microfinance sector have a strong tendency to reject any research which doesn’t confirm their a priori assumptions of the effectiveness of the tool which ultimately pays their salaries. And the conclusion of this current blog seems to suggest that we shouldn’t even expect to find any positive impact – microfinance just works, simple as that, evidence is not required.
I am currently re-reading “Confronting Microfinance” – a book edited by the infamous Milford Bateman. The opening chapter is fascinating precisely because it describes this group-think, the ideological obsession with microfinance, its almost cult-like following. It then presents 9 papers challenging the accepted wisdom of indebting poor people en masse written by a range of authors. Polarized views of microfinance abound, and the truth likely lies somewhere inbetween. Perhaps Roodman’s quote best summarises this, that the overall average impact of microfinance on poverty reduction is “zero” – not harmful, not beneficial, simply zero. But this is precisely why we need MORE research. Those who suggest that research is, for whatever reason, not reliable, not to be trusted, cannot be expected to uncover the hidden benefits etc. are, in my opinion, not to be trusted.
August 13, 2014 at 10:16 am
imasribo
Hugh, I find your comments to be quaintly uncritical when it comes to academic research.
When you measure something, the result is going to be a function of both the physical quality of the thing being measured and the nature of the apparatus or ruler you are using. Questioning whether you are using an appropriate ruler for a given object (calipers versus a yardstick) is entirely legitimate, and not in itself a negation of the existence of the object. Finding “literally thousands of studies” saying the same thing may not be so powerful if they are all done in the same way, using the same ruler.
Yes, measuring anything emphatically in the social sciences is far, far more difficult than measuring anything in the physical sciences. That´s because particle collisions can truly happen in an isolated environment, and if they collide one way at one point in time-space then we can reasonably expect them to collide in a similar way in any other point in time-space. In the social sciences, the impact of one thing is likely to drown in a myriad other unobserved effects happening at the same time.
No, I´m not surprised that studies don´t show much impact of microcredit. Roodman´s “about zero” impact conclusion that you endorse for microcredit is, sadly, the destiny of the majority of RCTs if they are applied to overly narrow interventions (possibly health-related interventions aside), repeated enough times in different environments, and run over a meaningfully long period of time on sufficiently large samples.
I was amused about how you turned my intrinsically pessimistic view of microfinance, or any other development intervention for that matter, into an implicit defense of microcredit. I´ve never said anything in favor of microcredit and I´ve never worked in that space. Please don´t try to fit us all into your “us versus them” worldview.
The logical conclusion from what I wrote is not that since impact is hard to find, then we can simply dispense with measuring it. Rather, what we need is better analytical tools that are less absurdly reductionist in nature. We are looking for an amoeba with a telescope. In the social sciences, unlike in the physical sciences, there is room to debate what constitutes appropriate rigor.
By the way, since you mention the Compartamos study, have you seen the actual survey instrument that was used in field interviews? Search it online, get yourself a subject, and read the entire thing to him/her out loud. I´ll be curious to hear your views on the appropriateness of that.
August 15, 2014 at 12:39 pm
Hugh Sinclair (@MFheretic)
I am aware of the problems of measurement in (social) science. I did not suggest that Duvendack’s review found “literally thousands of studies saying the same thing”. Their study reviewed 2,643 academic articles, including a few RCTs, pipeline designs, with/without comparisons, natural experiments and general purpose surveys. There is no suggestion that all 2,643 studies found the same thing. To the contrary, there was a wide range of findings, and they attempt to form a generalised conclusion that encompasses this range of findings. That is what a “systematic review” is. The paper can be read in full here:
https://www.givedirectly.org/pdf/DFID_microfinance_evidence_review.pdf
Indeed, Duvendack specifically cites your valid hesistance in accepting RCTs as the pinnacle of research (page 75):
“While there is currently enthusiasm for RCTs as the gold standard for assessing interventions, there are many who doubt the universal appropriateness of these designs. Indeed there may be something to be said for the idea that this current enthusiasm is built on similar foundations of sand to those on which we suggest the microfinance phenomenon has been based.”
I am not aware of the entirety of sources that Roodman based his “zero” reference on, but he is a more qualified econometrician than I, as are Duvendack et al, and I take their conclusions seriously. Roodman reviewed papers that present both positive and negative conclusions regarding the effectiveness of microfinance. Perhaps for this reason he suggests that the impact on poverty reduction is zero ON AVERAGE.
But we should also be aware of what Popper referred to as immunizing strategems. Particularly in social sciences it is possible to maintain a hypothesis (for example, microfinance helps the poor) indefinitely: evidence to the contrary can be blamed on one of the implicit assumptions in the complex underlying hypothesis being tested:
“Although a bundle of hypotheses (i.e. a hypothesis and its background assumptions) as a whole can be tested against the empirical world and be falsified if it fails the test, the Duhem–Quine thesis says it is impossible to isolate a single hypothesis in the bundle.”
http://en.wikipedia.org/wiki/Duhem-Quine_thesis
Given the relatively vast sums of capital deployed in the financial inclusion space it does appear a little disappointing that the results are not more clearly positive. Had we deployed such sums, for example, in distributing vaccines and came up with comparable results of their effectiveness as we do with microfinance, would we not fundamentally question the effectiveness of the vaccine? I am not suggesting that the vaccine need be 100% effective to be deemed worthy, but if there’s still debate on whether it works or not after 30 years or so, surely that would be a slight concern? How long do we blame the measurement tools, the climate, the deployment methods or any other extenuating circumstance (i.e. the bundled assumptions) before we concede that perhaps the vaccine isn’t working?
Indeed, Mal Warwick (co-author of “The Business Solution to Poverty”) made this point very succintly in a recent blog post examining the share of social impact capital directed towards financial inclusion versus other traditional interventions (clean energy, education, healthcare etc):
http://businesssolutiontopoverty.com/where-are-impact-investors-putting-their-money/
They lament that so much capital is directed towards what is essentially an unproven model (we can argue over the extent to which it is unproven, but results are “mixed” at best, and you yourself suggest you are intrinsically pessimistic):
“In our view, microcredit is one of the least worthy fields in social enterprise. That so much of the scarce resources available to address the challenges of poverty and climate change is squandered on for-profit microfinance ventures is a tragedy. While some microcredit programs — principally those operated as nonprofits — have achieved a lot of good, the rush by the big banks and profit-minded entrepreneurs into the field has proven to be a curse, not a blessing.”
Surely the key question is not whether financial inclusion “works”, laden with all the complexities of defining what we mean by “works”, but rather to wonder whether this is the best use of scarce capital?
This may be a “quaintly uncritical” comment regarding academia, but it seems a valid common-sense question to ask in 2014, after the microfinance experiment has been ploughing on for a few decades, evidence is mixed (at best) despite sapping up vast sums of public and private funding, and no one yet seems to be able to offer an answer to the question “does it work?”. Maybe it’s time to return to the proverbial drawing board?
I am not sure I agree with your phrase “we are looking for an amoeba with a telescope” – that does seem to broadly dismiss the entire academic literature on the subject of financial inclusion. But regarding the size of any likely impact of financial inclusion upon the poor, positive or negative, the choice of a microscopic organism in your analogy seems appropriate!
August 15, 2014 at 6:54 pm
imasribo
Hugh, you seem quite determined to interpret that I am arguing that microfinance really has impact even though it´s not showing in the academic studies. I make no such claim. I know that Roodman says its a big zero, all I am explaining is that it could not have been otherwise.
My point on the “looking for an amoeba with a telescope” is not specific to microfinance, it´s probably extensible to pretty much any narrow development intervention (probably with the exception of those that save lives). Again, the point that I was making is that if one narrow intervention did have enduring impact by itself, it would constitute what in common parlance we call a silver bullet. I just doubt very much there are many of those opportunities to improve the world noticeably by just doing one thing (again, outside of purely life-saving interventions like vaccines). Do you believe in silver bullets?
So my critique is not that academic studies should be done with or without randomization, this or that way. Rather, I am highlighting the much broader point that if you look at things too narrowly (i.e. if you are too reductionist) the chances of finding impact are exceedingly small — even though there might be some *joint* impact if the right cocktail of things happened at the same time (for instance, there is a great new teacher assigned to the school, the municipality happens to have built a new road, and yes there is a new microfinance player in town, all of which –let us say, just for the sake of argument– empower some entrepreneurs). But those other things tend to be “controlled away” in the reductionist studies, so you´ll never “catch” the ensuing joint (and hence largely fortuitous) impacts. Again, please don´t interpret that I am saying therefore that microfinance does have impact, because, even if those joint impacts did exist, they could not be attributed specifically to any of the individual things that made the impact possible.
I fully agree with your question about the value of having so much donor money going into microfinance. Of course, as a non-believer in silver bullets, I am not all that sure what else I would spend it on instead (again, with the exception of life-saving health interventions). But I do think there is more to development than people not dying prematurely, so we need to spend more time looking at the impact of combinations of interventions and not individual silver-bullet interventions. I don´t think that the answer will ever be to do only this or only that; the answer is always somewhere in the middle.
On the Duvendack review, let me just point out that of the 1,643 studies that they identified, all but fifty-something were “screened out” and their findings were not at all taken into account in the review that you mention. I´ll bet one thing those fifty-something studies all had in common is that they looked at the impact of microfinance quite narrowly.
So did you look at the Compartamos instrument? What did you think of that?