> Posted by Guy Stuart, Ph.D., Executive Director, Microfinance Opportunities and Fellow, Ash Center, Harvard University

The Financial Inclusion 2020 campaign at the Center for Financial Inclusion at Accion is building a movement toward full financial inclusion by 2020. This blog series spotlights financial inclusion efforts around the globe, shares insights from the FI2020 consultative process, and highlights findings from “Mapping the Invisible Market.”

In the United States, it is financial literacy month, but all the talk these days is of financial capability. Why? Is this another fad to make the topic of financial literacy a little more interesting? Is it an effort by one group or another to differentiate themselves? Maybe. But there are also important, substantive, and practical reasons to be talking about financial capability, which means having the ability and opportunity to manage your money well. And there is no better way to understand this than to look to the pioneer of the capabilities approach, Amtya Sen, to see why it is important, both in the developed and developing world.

In brief, Sen’s capabilities approach:

1. Forces a discussion of a multi-dimensional understanding of what it means to be a fully functioning member of society that takes into account not only what a person knows, owns, and earns, but also what they can do, and what they actually do
2. Opens the door to thinking about financial capability in the broader development context that encompasses other capabilities such as being healthy, educated, and earning a sufficient income to support an individual and their family
3. Recognizes that what it takes to be capable in one context is different from what it takes to be capable in another

The practical implications of this are important:

1. A multi-dimensional understanding of what it means for an individual to manage their money well goes beyond being “literate” (having the right knowledge and skills). It also includes having an individual be able to act on that literacy (financial behavior) and having access to the appropriate financial instruments to fully realize their aspirations. In other words, the capability approach to money management includes a discussion of behavior and access. This means, practically, we have to understand the connection between financial literacy and financial behavior in all its complexity. It also means that talk of financial inclusion and increased access should focus on how they increase financial capabilities, which means a focus on financial service use, not just uptake.
2. Development often happens in silos, and the burgeoning financial capabilities “industry” is no exception. A capabilities approach reminds us that financial capability is just one of many capabilities that people value. Furthermore, being financially capable can impact other capabilities directly, and vice-versa. Managing your money well can help a person better deal with a health crisis, which would otherwise cripple their capability of being healthy. Or being educated generally (being able to read and write) broadens the scope of what a person can learn about managing their money well. As a result, organizations and individuals interested in building the financial capabilities of individuals should be looking across development sectors for ways to cooperate and partner with others. Funders should be doing the same.
3. The understanding that what it takes to be capable in one context may not be the same as what it takes to be capable in another has programmatic implications for those involved in building financial capabilities. So, for example, in a culture where there is a strong expectation of giving and receiving cash gifts to/from family and friends, whether it be for mutual support or other reasons, a financially capable person may be one who is adept at handling their requests for and receipt of cash gifts, and balancing them against their own need to accumulate personal savings. As a result, a financial education curriculum focused on the accumulation of personal savings might also include a discussion of effectively managing gift giving/receiving. Practically, this means that programs that aim to build financial capabilities must be adapted to their local context, not only in terms of language and content emphasis, but also substantively to take into account local understandings of what it means to be financially capable.

So as we celebrate financial literacy month in the U.S., let’s broaden our scope and think about financial capability across the globe.

For more information on Financial Inclusion 2020, sign up for campaign updates.

Guy is Executive Director of Microfinance Opportunities (MFO). Before becoming Executive Director, Guy was a Senior Advisor to MFO and served as Principal Investigator on five Financial Diaries studies and as project leader for the development of the Financial Capabilities Index Web Portal. Guy received his PhD from the University of Chicago in 1994, and subsequently worked for four years in Chicago in the field of community economic development. He then served 13 years as a Lecturer in Public Policy at the Harvard Kennedy School where he taught courses in management and microfinance. He continues to teach in Executive Education programs at the school, and is a Fellow at the Ash Center, Harvard University.

Have you read?

A New Business Model for Savings Mobilization

Financial Capability: Not Many Answers, but Lots of Great Questions

Takeaways from the 2012 Citi-FT Financial Education Summit

Courses featuring microfinance can be found in most colleges or universities, and the development of the syllabi for these courses has begun to create a sort of “microfinance canon.” As the next generation of microfinance professionals enters the market, there are a few things that many of them have read, and a few questions that many of them have been required to consider.

I took 10 syllabi (a small sample size, admittedly) and looked for trends in what is being taught, what is being read, and what is being assigned. Here are a few things I found:

• Students are reading Collins et al., Armendariz and Morduch, and Rhyne more than any other authors (Helms was a close fourth). These three authors represent a nice diversity of method and emphasis. Collins et al.’s Portfolios takes more of a client perspective, Armendariz and Morduch explain the more technical nuances, and Rhyne offers a more practical business approach.

• Guest speakers are a major part of class time. Given that microfinance is still very much practitioner (rather than academic) oriented, it is no surprise that guest speakers are used quite frequently in class time. Most, if not all, of these speakers are in top positions at well-known microfinance institutions.

• Assignments range from case study examples, to policy briefs, business plans, and product designs. In a few of the syllabi, students were required to make presentations, including one that assigned an investor pitch in lieu of a final exam.

Beyond these content-related observations, I noticed that microfinance classes don’t seem to have an agreed-upon disciplinary “home” in academia. Classes are typically hosted in business, economics, public policy, and international relations departments. In addition (and this is probably related), professors tend to not be tenure-track academics. Often, they are involved in microfinance as practitioners in some way.

Despite this lack of an academic “home” or identity, however, microfinance classes are increasingly common. A simple Google search brought up dozens of publicly available syllabi. Compared to the offerings as early as five or 10 years ago, this prevalence is quite encouraging.

To explore these syllabi for yourself, head to the following sources:

• Harvard University John F. Kennedy School of Government: Microfinance in Theory and Practice, taught by Guy Stuart

• New York University Stern School of Business: Building and Financing Microfinance Institutions and their Financial Products, taught by James B. Carlson

• Carnegie Mellon University Heinz School of Public Policy: Microfinance, taught by Tayo Fabusuyi

• University of Michigan: Introduction to Microfinance, taught by Michael Gordon

• Georgetown Public Policy Institute: Introduction to Microfinance for Development, taught by David Roodman

• Berkeley Haas School of Business: Introduction to Microfinance, taught by Sean Foote

• Denver University Josef Korbel School of International Studies: Microfinance and Sustainable Development, taught by Janney Carpenter

• Johns Hopkins School of Advanced International Studies: Microfinance and Development 1, taught by Elissa McCarter and Kate Druschel Griffin

• Johns Hopkins School of Advanced International Studies: Microfinance and Development 2, taught by Victoria White and Isabelle Barres

Image credit: Justin Sullivan/Getty Images

Have you read?

5 Great Books About Microfinance and How the Poor Use Money

Center Advisory and Faculty Councils Meet in DC

Microfinance in Academia – The Center’s Faculty Council Reports on Their Activities

> Posted by Guy Stuart, Faculty Council Member, The Center for Financial Inclusion

Though data I have collected using the Financial Diaries methodology, suggest that almost all people save, even if it is only from week-to-week to smooth consumption, the paper’s approach to segmentation offers a key insight. That is, its approach to market segmentation focuses attention on consumer behavior rather than their socio-economic characteristics, such as age, income or livelihood. Though the latter and the former are related, consumer behavior is far more important for financial service providers to understand than consumers’ socio-economic characteristics.

It is the behavior of the consumer that dictates how a financial service provider can best channel his or her cash flow into a savings account. For example, a sporadic saver, saving money when the normal variance in their cash flow gives them extra to put aside, may not be the ideal candidate for a commitment savings account that requires regular payments. But they might be an ideal candidate for some sort of direct deposit service, if the reason for their sporadic savings behavior is that they occasionally make large sales (such as the sale of a large piece of furniture by a carpenter) in an established market, where buyers could pay them by depositing money directly into their account. (Though see here for an interesting hybrid, where direct deposit is combined with a commitment to freeze some of the money in the account for a specified period.)

Traditional market research often uses the socio-economic characteristics of consumers as a proxy for behavior. But new research tools, such as Financial Diaries or the mining of MFIs’ and banks’ own transactional data, allow us to focus on consumer behavior more directly. Furthermore, Urquizo’s paper has shown that one can use traditional methods, such as cross-sectional surveys, to get at consumer behavior. Regardless of the method, the important point is to keep the focus on behavior. And from that starting point, we should also be looking to understand the factors that drive behavior. For this we should turn our attention to the needs and motivations of consumers and the social structures in which they are embedded.

Guy Stuart is an independent consultant, Senior Advisor to Microfinance Opportunities, and Fellow, Ash Center, Harvard University. Mr. Stuart is also a member of the CFI Faculty Council.

Have you read?

New Credit-Risk Models for the Unbanked

What Financial Diaries Reveal about M-PESA Use

What Do We Really Know About Financial Behavior in Rural Areas?

This post is part of the Center for Financial Inclusion’s Expert Exchange: Building A Movement Toward Financial Inclusion by 2020, cultivating conversation around the goal of reaching full financial inclusion by 2020. For further questions about this series, write to Sonja E. Kelly, Fellow, Center for Financial Inclusion at ACCION International.

My work using the Financial Diaries methodology has made me very cognizant of how much cash low-income people handle daily – more cash, in real terms, than a middle-income person in a developed country handles. If we are to increase financial inclusion by increasing low-income individuals’ use of savings products, we have to find ways to channel some of the large amounts of cash they handle into those products, in ways that they value. Monique Cohen and I have written about this in our recent article “Channeling Cash: Towards a New Business Model for Low-Income Savings Mobilization.”

In the article, we first had to step back and define what we really meant by savings. In the paper, we assert that low-income people use savings for three different purposes:

• Cash flow management in order to manage expenditures that are anticipated and expected
• Risk management in order to manage expenditures that are anticipated but not expected
• Asset accumulation, which can act as either cash flow management with a long time horizon (the old age scenario, for example), or a planned expenditure, at a specified time, for an asset that will improve well-being

Regardless of the purposes for which they are saving, the financial cash flows of individuals will determine the timing, the frequency, and the size of the amounts saved. While it is important that we understand why people save, it is just as critical that we understand how people can save given their cash flow.

Take Bright and Mercy, for example, who have a cooked food business, raise cattle, and maintain a farm. They live in Malawi and were 40 and 36, respectively, at the time of our Financial Diaries study. As you can see below, their income was highly inconsistent, not only because three cattle sales resulted in large infusions of cash at fixed points in time (see weeks 13, 60, and 67), but also because their cooked food business’ net income was highly inconsistent on its own.

Net Income, Cooked Food and Cattle Sales

Stuart and Cohen 2012

When designing a product for this household, therefore, financial service providers need to be at the right place at the right time. If they are going to help Bright and Mercy save in a formal savings account, they have to find a way to physically or technologically meet that income inflow at weeks 13, 60, and 67, as well as help them handle the ups and downs of their ordinary, weekly cash flow.

Collectively, there are ways that governments, businesses, and individuals have already begun to work with financial service providers to channel cash flows. The government of Malawi, for example, pays government employees—including teachers—through direct deposit into a bank account. In Kenya, people are increasingly being paid through M-PESA, even in the informal economy. This direct infusion of funds into the formal financial system automatically increases financial inclusion. Having a bank account that automatically works with one’s income flow allows people to have financial services without the inconvenience of having to cash in or cash out of the system.

These examples suggest a different business model for financial service providers wishing to mobilize savings. Instead of focusing solely on the design of the savings product, service providers should also focus on the value they can add through payment services that reduce the amount of cash the poor handle. If they can get those types of services right, the additional step of then channeling some of the money flowing through them into a savings account will be that much easier.

As we dialogue about reaching financial inclusion by 2020, we need to keep in mind the daily money management challenges facing low-income individuals to ensure we focus not only on access to services but also on the use of services, their quality, and the value they add.  It has to do with understanding client needs, and converting that understanding into products.

So, getting back to Bright and Mercy—what kind of product or delivery channel might you design that would meet their needs?

If you wish to read our full paper, you can view it here.

For more information, sign up for updates from the Financial Inclusion 2020 campaign.

Guy is an Independent Consultant who uses “bottom up” research methods such as Financial Diaries and participatory exercises to identify ways for financial service providers to meet the needs of low-income individuals and households more effectively. 

He is a Senior Advisor to Microfinance Opportunities on their client-focused research projects, including serving as Principal Investigator on five Financial Diaries studies and project leader for the development of the Financial Capabilities Index Web Portal.

He is a Fellow at the Ash Center, Harvard University, where he conducts research on distributed service delivery systems.

He received his PhD from the University of Chicago in 1994, and subsequently worked for four years in Chicago in the field of community economic development. He then served 13 years as a Lecturer in Public Policy at the Harvard Kennedy School where he taught courses in management and microfinance.  He continues to teach in Executive Education programs at the school. 

Have You Read?

Who Needs Financial Education?

It’s Time to Expand the Circle

Who is Your “Plus One”?

Rose lives in a village about 5km from the provincial town of Murang’a in the Central Province of Kenya. Her husband works in Nairobi. During the time when our diaries study interviewed her, between November 2009 and June 2010, she earned very little, less than $10 per week.  But her husband sent her 29 remittances through Safaricom’s M-Pesa mobile money service from Nairobi over a 33-week period.  He also brought cash home on visits. Rose contributed about Ksh.270 ($5.70) to a merry-go-round (a type of ROSCA) about 25 times during the course of our study. At the end of February her husband sent her Ksh.10,500 (about $210) in three remittances in one week. She withdrew all but Ksh. 500 in the same week and spent it on construction materials. Her husband came home that week and stayed for two weeks. He then returned to Nairobi, and on his return he sent Rose more money.

Rose is one of the 92 respondents Microfinance Opportunities (MFO) tracked during an eight-month Financial Diaries study in Kenya.  This study yielded detailed transactional data that enabled us to better understand the economic lives of low-income individuals and households in two rural areas and one urban area.

The diaries also provide a close look at how low-income Kenyans actually use the now-famous mobile money service, M-PESA. M-PESA is an SMS-based mobile money transfer system owned and run by Safaricom, the leading mobile phone operator in Kenya. There are 30,000 M-PESA agents nationwide through whom people can deposit cash into or withdraw cash from their M-PESA accounts. Once they have cash in their accounts, customers can text their money to anyone in the country who has a mobile phone. The recipient can then either on-send the money to someone else or cash it out at an agent. There are over 13.5 million M-PESA accounts. The adult population of Kenya is 20 million.

As with our other financial diaries study in Malawi, and studies performed by other researchers such as Stuart Rutherford, Daryl Collins, and Orlanda Ruthven, the data from Kenya show that the poor handle a lot of cash, often more than twice their income.  Furthermore, they use a wide variety of informal financial services, including intra-family loans and gifts as well as rotating savings and credit associations.

MFO’s diaries methodology captures all the financial transactions of the previous week.  For transactions conducted through M-PESA and other mobile money  providers, we asked respondents the actual date on which they deposited cash into M-PESA, withdrew cash from their accounts, and sent or received e-money.  The data also capture with whom people transacted, and transaction locations. This provides insight into sequences of transactions and how they relate to each other, allowing us to piece together stories like Rose’s. They also allow us to aggregate the data for more comprehensive analyses.

M-PESA and Social Networks

Analysis of the transactions recorded through the diaries gave us powerful insights into how respondents used M-PESA. We found that:

• 53 of the 92 diaries participants used M-PESA at least once.
• 80 percent of M-PESA remittances were within extended families or between friends, and mostly for personal or household use;
• The 20 percent of transactions with “associates” were almost all business transactions;
• A large majority of the remittances crossed distances greater than 20 kilometers.

These findings strongly suggest that Safaricom’s “send money home” marketing of M-PESA has been very effective within the low-income market in Kenya. This is exactly what people appear to be doing. The “send money home” marketing campaign did not create a new behavior, but rather tapped into something people were already doing – giving and receiving money through their social networks. M-PESA just made it easier and cheaper to do this.

M-PESA and Risk Management

We also looked at the role M-PESA played in helping people manage risks like health emergencies.  Of the 60 hospital bills paid by our respondents during the study period, more than a third coincided with a cash withdrawal from M-PESA either in the same or preceding week. Furthermore, when we traced the cash withdrawal back to its source, it was common to find that the money had been recently sent from afar (more than 20 kilometers) to the person paying the hospital bill. This suggests that M-PESA supports the ability to rapidly access health services in times of need.

This use of M-PESA in an emergency was consistent with the way people used M-PESA generally. Our data suggest that they did not keep money in M-PESA. Over two-thirds of the time people cleared out any money they had received through M-PESA or deposited onto M-PESA, nearly always on the day it was received and before any new money flowed into their accounts. The median average daily balance of the respondents in our study was about US$3.70 in purchasing power parity (PPP) dollars, or US$2.20 in exchange rate dollars (All other dollar amounts quoted here are PPP).

Beyond ‘Send Money Home’

Despite M-PESA’s popularity in Kenya, it is a long way from replacing cash: Over 95 percent of transactions were still in cash. This is not surprising, given that many transactions in the low-income market segment are very small and the cost of using M-PESA for tiny transactions is prohibitive. But we also found little use of M-PESA for business purposes, where the transactions tended to be larger. We think this phenomenon may be due to a trust issue: People are willing to send money to friends and family because they can call them to verify receipt of the remittance; they are less trusting of long-distance verification with an “associate” with whom they do business but who is not part of their social network.

M-PESA is a great success. Millions of Kenyans use it regularly. Millions of dollars flow through M-PESA each day. It holds out the hope that it can be the platform for the provision of additional formal financial services such as savings, credit and insurance. But that nut has yet to be cracked, especially in the low-income market. Meanwhile it helps low-income Kenyans manage risk and maintain and reinforce their social networks through which cash gifts and loans flow. That is a good start.

Guy Stuart, Ph.D. is an independent consultant, Senior Advisor to Microfinance Opportunities, and Fellow at the Ash Center, Harvard University

Monique Cohen, Ph.D. is Founder and President of Microfinance Opportunities

Have you read?

5 Tough Questions About Mobile Banking

Protecting Mobile Money Against Financial Crimes

Otero on Mobile Banking

Karlan and Appel’s book More Than Good Intentions is an easy, and often compelling, read.  Initially, you are drawn in by the anecdotes that start each chapter, which Karlan and Appel use to illustrate a broader development issue that they then tackle in the rest of the chapter.  But the stories that really drive the chapters forward, and make the book compelling, are the ones which start with Karlan describing how he teamed up with people with “good intentions” – people trying to deliver goods and services to poor individuals and families in developing countries, to gather data in an effort to address a problem they both want to solve.  In essence, this is the strength of the work Karlan and Appel describe in this book: For researchers to help in efforts to solve the problems confronting poor people, they have to begin with the problems they and the people who are trying to help them face, and gather evidence regarding what works and what does not work in solving those problems so that they then can act on that evidence.

This is a pretty simple message, but it is an important message, because this is not how research is necessarily done.  Professional researchers tend to want to answer a particular question that fits their research agenda or the agenda of the organization for which they work.  It may be the case that these agendas coincide with the problems facing the poor, generally, but even if that is the case, it is unlikely that the research will be specific enough to be useful, nor will the results of the research be reported in such a way that anyone can act on them.

Karlan and Appel want us to believe that Randomized Control Trials (RCTs) are an essential tool in identifying solutions to problems through research.  In fact, there is clear evidence that they think this is the only way to do research.  They continuously, and condescendingly, equate “rigor” with the use of RCTs, which begs the question: Are all the people who generate evidence about what works and what does not work without using RCTs not doing rigorous work?  Are RCTs the only way to get to the truth of the matter?

Given that Karlan and Appel devote so many chapters to microfinance, we can dig deeper into the truth of their assertion about RCTs by looking at what they have learned about microfinance through RCTs.  The first thing to note is that, for those who have been in the field a long time, many of the stories Karlan and Appel tell are not new.

For example, Karlan and Appel report that requiring microcredit borrowers to spend their loan proceeds on a particular item, say business inventory, is a bad idea because it, at the very least, makes liars of your borrowers, and, at worst, forces them to engage in deceptive practices that undermine their ability to repay.  This is not news to many people in the field, and is discussed extensively by J.D. von Pischke in Finance at the Frontier, published in 1991.  Data from Financial Diaries I have done in India, Malawi, and Kenya and data from the book Portfolios of the Poor all show that loan money is fungible, and reveal what a sisyphian task it is to try to monitor its use.

Karlan and Appel also make a big deal out of the fact that microcredit has relied heavily for many years on peer lending to manage the risk inherent in lending in an information-poor environment, one in which most microfinance clients live.  But a cursory review of the history of microfinance tells us that MFIs have not singularly relied on the group lending method to manage risk.  A case study that I wrote on Sogesol in Haiti, one that I have been teaching from for almost 10 years, documents how that MFI makes use of informal knowledge systems to lend to low-income Haitians in Port-au-Prince on an individual basis.  Another case study, this time of Compartamos’ efforts to open a branch in metropolitan Mexico City, revealed the failure of the group-lending method in reaching low-income residents of the metro area, and prompted the organization to change course and offer individual loans.  And, of course, we must not forget that the largest MFI in the world, Bank Rakyat Indonesia (BRI), has been making individual loans since the inception of its KUPEDES program in 1983.

Given how much we know about what works and what does not work in microfinance in the absence of RCTs, we are forced to ask: What insights do RCTs add? Let’s dig deeper into the discussion of peer lending.  As Karlan and Appel note, the evidence from Karlan’s RCT on group lending shows that the joint liability requirement of peer lending is not essential to risk management – in the absence of its enforcement the lender is still able to collect.  Again, those who have been in this field for some time already know joint liability is not always enforced fully, because of the real concerns of practitioners that it puts too much pressure on the borrowers.  Furthermore, as Karlan and Appel note, one of the main proponents of this requirement, Grameen Bank, dropped the practice in 2002 when they instituted their Grameen II reforms.  Did they do so in the wake of the findings from an RCT?  No, Grameen did not use an RCT to make that decision.  Nor was it facing a catastrophic cascade of losses as one group after another collapsed under the weight of the joint liability, which is something that Karlan and Appel argue is a likely consequence of the join liability requirement.   Rather, Grameen talked to its clients and its employees and worked out that individual loans in a group setting could work, and would better serve its clients.

Though this process of deciding how to change the product Grameen was offering its clients did not involve an RCT, it was evidence-based.  But was this evidence good evidence?  Did Grameen just get lucky that Grameen II worked?  Maybe, but more likely, it was able to generate an accurate picture of the needs of the clients and the potential dynamics ensuing from its product changes and on this basis made the changes.  The ability to be accurate is not a given.  First, it requires having the data, which, in this case, were based on years of experience of being in daily contact with clients.  Second, it requires that the organization is willing and able to learn from its data.  Note how these two requirements are key to what Karlan and Appel are arguing for: Have the data; and analyze and learn from it.  The only thing that is different is the nature of the data – experience versus differences in measured results for a treatment and control group.

This seems like a big difference, and in many ways it is.  But, I would argue, even here there is also commonality.  How does a researcher chose what question to answer, what solution to a problem to test, using an RCT?  There are three answers: 1) theory; 2) practice; and 3) a combination of theory and practice, coming out of a conversation between researchers and practitioners.  Karlan is interested in theory, and is especially enamored with behavioral economics, though he also gives a nod to network theory.   But he is also as interested in practice, and speaking to practitioners about the problems they are encountering in delivering goods and services to the poor and what can be done about them.  If Karlan had been hired by Grameen to work with them on reforming their microcredit operations it would have been as likely as not that the bank would have suggested to him that the joint liability requirement needed to be looked at, rather than the other way round.

So, again, what does the RCT bring to the table?  Let’s pretend that Grameen did hire Karlan to help them with their microcredit operations, what would have happened differently.  I assume that Karlan would have suggested: Before you roll out a peer-lending program without a joint liability requirement, let’s do an RCT to see whether removing the requirement does any harm.  So he would have assigned groups to a treatment group, which, in this case, would have been groups no longer bound by joint liability (the “intervention” being the release of the requirement), and a control group, those still bound by joint liability.  One assumes that he would also have insisted that those in the control group be subjected to the proper enforcement of the joint liability requirement.  Over a suitable period of time, Karlan would have observed the performance of the two sets of groups using loan performance records from the bank, and been able to tell Grameen whether removing joint liability was a problem or not.

This seems like it would give a definitive answer, in very much the same way as many of the stories Karlan and Appel tell in their book.  For one thing, those in the treatment group, including the credit officers who served them, would have known that things had changed – unlike medical trials, social science RCTs are very difficult to make “blind” in the sense that the person receiving the intervention does not know they are getting something different.  So if the data had come back showing that removing the joint liability had no effect on repayment rates, one could not conclude that it was because the removal itself was not a problem, but that the removal was not a problem in a trial setting where everyone was “on their best behavior” because they wanted it to work.  So it is likely that the organization would roll out the new process to other parts of the organization, but keep a close eye on the performance data to make sure they had not been misled by the results of the RCT.

It turns out that this approach is not that much different from what an organization might do if it was piloting a new process before rolling it out fully.  In such a scenario, the organization rolls out the new process as a pilot, knowing that people in the pilot might be on their “best behavior,” and then, if no problems emerged in the pilot, gradually roll out it out across the rest of the organization, learning from and adapting to problems as they arose.  This alternative approach does not require an external researcher to work out what is going on, just a managerial team with the capacity to collect and analyze real-time data from their operations.  Sure, a manager trained in doing RCTs would want to choose the site of the pilot randomly, which would be a sensible thing to do.

The discussion thus far has focused on the way in which RCTs might help a manager interested in tweaking their processes to improve customer service.  What about the grander claims made for RCTs?  The study that has received most attention in this regard is the Spandana study, which sought to evaluate the impact of microcredit on poverty – essentially taking on Muhammand Yunus’ claim that microcredit can alleviate poverty.  If nothing else, the study revealed the complexity of trying to evaluate societal effects using any sort of experimental methods.  That does not mean we should not try, but we should be more modest in our approach – not try to answer such a grand question such as “does microcredit alleviate poverty?” with one study.

If we look outside microfinance to a study cited by Karlan and Appel, we can see what a more modest approach might look like.  The Progresa/Oportunidades conditional cash transfer program received a lot of accolades for incorporating an evaluation process into its roll out.  The initial evaluation of the program focused on whether the cash transfers got kids to school, into health clinics, and changing their nutritional intake, rather than whether they were better educated or more healthy.  In other words, they initially focused on operational outputs.  It is only in more recent years, after considerable experience with the program, and after it had been rolled out extensively, did focus turn to the ultimate impacts of the program.  And the most recent evaluation report notes that there are real concerns that Oportunidades is not generating long-term results because of the poor quality of the Mexican educational and health systems.  One could argue that Oportunidades got ahead of the evidence, but it is also clear that without the type of engagement between the practical realities of the program and good research (which, it should be noted, was not limited to RCTs), the Mexican government would have been none-the-wiser as to how to tackle endemic poverty.

Finally, I cannot complete this review without a note on theory.  One of the things researchers bring to the table when they enter into conversations with practitioners is a theoretical perspective.  It may seem odd to say this, as researchers are often criticized for being overly theoretical and unconcerned with practical matters.  But this is more a matter of presentation than of content – my own experience in interacting with practitioners in workshops and other sorts of training programs is that practitioners truly appreciate being given new ways to think about an old problem.  What they want to understand is the relevance of the theory and how it can be used to sort through what they already know.  Karlan and Appel seem to appreciate this fact, and their use of theory in addressing the practical concerns of the people they are working with seems appropriate – at least from this distance.

Ironically for researchers, they fail to draw any theoretical conclusions from Karlan’s studies.  They offer us no coherent way to think about problems of development – or even rules of thumb that might help us tackle specific issues in different contexts.  Even their recommendations at the end seem to miss the boat in some cases.  For example, in their list of “things that work” they highlight de-worming of kids as a way to increase school attendance.  It seems to me that a more general conclusion that one can draw from the experiments the authors describe is “healthy kids are more likely to go to school,” and then a more general list of cost-effective interventions that have been proven to keep kids healthy.  If the work that Karlan’s IPA is doing is going to amount to much more than a large number of isolated experiments, he needs to show that he can draw some general conclusions that might be helpful to practitioners, at least as a starting point for how to think about a problem they are facing.

In sum, this is a highly readable book, if a little glib sometimes.  The main lesson for me was to reaffirm the importance of continuously engaging in the development process as a series of problems to be solved using concrete evidence to determine what works and what does not.  The authors want us to believe that RCTs are the only way to produce concrete evidence, but clearly their own writing suggests that there are other ways.  Furthermore, they want to scale up the use of RCTs.  It is not obvious that this should be done by expanding the work of organizations like IPA.  Rather, they should be in the business (and I believe IPA would argue it is in the business) of training people in developing countries to do this work.  I would go one step further and specify that those being trained in these methods in these countries should be mid-level managers in development organizations, who should be inculcated in the ideas of sound data collection and in the efficacy of a learning organization that is not afraid to test its ideas.  Furthermore, I would recommend that individual and institutional donors look for organizations that are continuously engaged in working to solve the problems faced by the poor in developing countries, and are set up to learn from their mistakes.