This paper critiques recent work that measures the impact of self-help groups, and explains the biases that result from this assessment. Using survey data, it is shown that the methodologies used yield results that misstate the impact. A categorical breakdown is proposed to improve upon these studies, and a simple alternative procedure, the pipeline method, is then estimated to properly correct for selection bias. The results indicate that SHG participation has an impact on assets, livestock income, and salaries. Applying more advanced methods, training is also found to have a positive impact on assets, and empowerment is found to increase with SHG participation.

The authors are grateful to an anonymous referee for comments on an earlier version of this paper.

Ranjula Bali Swain (Ranjula.Bali@nek.uu.se) is at the Department of Economics, Uppsala University and Adel Varghese (avarghese@tamu. edu) is at the Institute for Financial Management and Research, Chennai and Department of Economics, Texas A & M University.

S

Prompted by some of the above reasons, in 2008, the National Council for Applied Economic Research (NCAER) conducted an overarching study on the impact of SHGs (NCAER 2008). Their findings revealed very strong impact of SHG participation on a large number of factors. Unfortunately, their methodology misstates the impact of SHGs. Surprisingly, their pre-post methodology remains similar to the very first impact studies on SHGs. Much of the methodology does not take into account or mention selection biases widely known in the current economics literature.1 Large-scale evaluations such as these by the NCAER have wide impact and reach broad audiences. For example, the recent State of the Sector 2008 Report on Indian microfinance used the NCAER work as “the impact study” on SHGs, without any qualifiers on the methodology (Srinivasan 2009). Thus, it is important to critically reassess the NCAER study in terms of its methods and to provide a more accurate impact study of SHGs.

This paper has two objectives. One, we clearly identify the limitations of pursuing the pre-post methodology that the NCAER and recent impact studies on SHGs follow. Two, we provide easily implementable alternatives for non-experimental data which are more precise in identifying impact. The main alternative, known as the pipeline method, is originally due to Coleman (1999), and has a strong intuitive appeal. It can be implemented directly with proper data collection. The other methods – propensity score matching and structural modelling – are less intuitive, but a software to implement these is easily available. Furthermore, we argue that NCAER’s data set cannot effectively measure impact and provide estimates using a different data set on SHGs. We thus demonstrate the wide differences between the pre-post method and the method we use. We also differentiate among new SHGs, mature SHGs, and non-members. The NCAER paper results are based only on mature SHGs.

march 12, 2011 vol xlvi no 11

EPW

Using the NCAER method, our results do not reveal much difference between new SHGs and mature SHGs for a large number of impact variables. This indicates that the NCAER results may arise from other observable and unobservable elements that are not fully controlled for in the NCAER analysis. When we control for both observable and unobservable changes, we find a positive impact from participation on assets, livestock income, salaries, education and health expenditure. Surprisingly, we find a negative impact on income. In more advanced impact analysis, we find a positive impact of training on assets. We also find a positive impact on empowerment for SHG members.

On Indian SHGs specifically, impact studies consist of the P uhazendhi and Badataya (2002), commissioned by the National Bank for Agriculture and Rural Development (NABARD) with 115 member respondents from three states. Their methods consist of computing the percentage difference of the means of members’ variables pre- and post-SHG membership. This pre-post methodology is very similar to that of NCAER. Thus, their findings suffer from some of the limitations of the NCAER study. Nevertheless, this Puhazendhi-Badataya study has had much policy influence, and has been quoted by many sources, most recently the Rangarajan Committee on Financial Inclusion (2008). The results find that SHG membership significantly increases the asset structure (30%), savings, annual net income, employment (34%), and s ocial empowerment. The study also finds significant increases in the asset structure, savings (14%), annual net income (6%), and social empowerment.

A more recent report by EDA Rural Systems (2006), jointly with Catholic Relief Services (CRS), CARE, and GTZ2 (hereafter EDA) on 214 SHGs from 108 villages does not attempt a formal impact study but provides summary statistics and interviews with focus groups. The authors disclaim their report as an impact evaluation but still the Rangarajan Committee refers to their work as an i mpact study. Due to inappropriate corrections for selection bias, Tankha (2002:55) comments that, “their findings cannot be considered to be conclusive or even convincing”.

Properly accounting for selection bias entails recognising that the sample of SHG members may differ from the population at large. Thus, one needs to effectively extricate the impact of member characteristics, regardless of SHG participation. For example, if we observe that income for SHG members has grown relative to others, is it because of SHG membership or because SHG members are more entrepreneurial? NCAER may have been motivated by these concerns to measure impact for the same household over time. Unfortunately, NCAER’s method creates problems of its own, which we discuss later in the next section.

In the economic impact evaluation literature, randomisation is the preferred methodology (Duflo et al 2007). However, our data is non-experimental and so we cannot apply experimental methods. As an alternative to randomisation, one can follow exclusion rules to find an instrument that affects credit but not the relevant impact variable. Since the choice of these rules or instruments may differ across authors, differences in opinions can arise. A case in point is the academic debate between Pitt and Khandker (1998), who find impact, and Morduch and Roodman (2009), who find no impact, for microcredit programmes in Bangladesh.

Economic & Political Weekly

EPW

Furthermore, SHGs follow no such exclusion rule. Even though SHGs tend to target poorer households, the programme does not follow strict eligibility criteria (as do most microfinance programmes). One can also employ panel data but this suffers from limitations as well. An important objective of this paper is to provide an alternative to randomisation and panel data that is simple for policy use.

The paper proceeds as follows. We first list the limitations of the NCAER study. We then suggest the pipeline method of measuring impact and describe our data set. We then present the results, which apply the NCAER methodology to our data for different categories. We then present and discuss results from the pipeline method. We extend the basic framework to address more advanced impact measurement such as training and empowerment. We end with some policy conclusions, comment on future work, and provide some potential improvements to the current study.

2 Methods and Data

In this section, we outline the basic conditions for an effective impact study, provide an alternative methodology to randomisation, and summarise our data on SHGs. As we argued in the previous section, a pre-post methodology will not adequately account for the changes that SHG member households face. Identifying a comparison or control group which has not received the benefits of SHG membership but which has experienced the same changes as the treatment group (SHG members) will help in this regard.

Assume one has both baseline (beginning period) and current data for the control and treatment groups and that both are from the same area. One can then compute the percentage difference in relevant impact variables for members vis-á-vis the control group. If the percentage change for the treatment group’s variable is significantly greater than the comparison group, then that suggests the programme had impact.

In the ideal scenario, with panel data, one can collect the data yearly and compute these differences. However, like randomisation, panel data faces limitations when applied to SHGs. First, SHG programmes are already widespread, reaching out to 45 million households. One would like to compute the impact of participation for those households relative to previous loans. Second, panel data costs are large and by the time researchers collect and process the data, the programme may have moved on. Third, providing a control group without services for say, one year, is perhaps justifiable, but one cannot deny access and maintain a control group for long. If one wants to measure the impact of a long-term variable such as education, the task is more difficult because one has to hold a control group for much longer. So far, no one has collected panel data for SHGs.

As an alternative to panel data, we can proceed in a different direction. So far we have argued that for a relevant impact study, one would need a control group. Furthermore, ideally one would need to choose a control group with the same external conditions as the treatment group. This selection raises other problems. If one selected both treatment and control groups from the same village, then one would wonder why some households chose or chose not to join SHGs. In other words, SHG members may be special. Regression analysis allows one to control for any observable differences such as age and gender. With cross-sectional data, however, one cannot control for unobservable differences such as entrepreneurial skill and talent.6

A design feature of SHGs provides us with an opening to properly measure impact. By design, members have to wait to receive a loan from the bank (about six months). We can exploit this f eature to identify self-selected SHG members who have not yet r eceived a loan. In particular, our data (which we describe in more detail below) allows the following. In certain districts, some members are currently active members of SHGs. In other villages, in these same districts, members from newly formed SHGs have been selected but have not yet received financial s ervices. Thus, the control group in our sample consists of the new SHGs, while mature SHGs form our treatment group.7 We hypothesise that the old and new SHGs have similar unobservables. We also have information on non-members from these districts so that we can condition on the selection, i e, joining the SHG. In other words, we have two control groups. The first control group includes SHG members who have

march 12, 2011 vol xlvi no 11

EPW

not yet received loans while the second includes non-members who have not received loans and are not members.

Raising the level of aggregation to another level, such as d istricts (where both mature and new SHGs reside) helps hold district-specific conditions constant. Similarly, NABARD’s decision to expand the SHG programme occurs at the district level, without any specific policy targeting certain villages over others.8 This choice of district as the level of aggregation, appropriate for SHGs, differs from most research, which uses villages as the a ggregation unit. To account for the remaining village-level varia bility, we employ village-level characteristics. To the extent that district-wide effects may spill over from mature to new members and nonmembers, the estimates here would underestimate that impact.

One critical check is whether old and new villages differ substantially.9 In other words, did policymakers prefer expanding the SHG programme in certain villages first over others? In our

Table 1: Logit Estimates of Placement discussions with various officials, of SHG Programmes

and our reading of various

Village-level Variables SHG placement

N ABARD and Reserve Bank of

(in km unless noted)

Distance block -2.19 (0.98) India (RBI) documents, we did

Distance haat -0.45 (0.05) not find that certain villages

Distance paved road -11.55 (0.98) were preferred over others. As

Distance bank 1.19 (0.37) an additional check, we can Distance market 7.02 (0. 78)

check the observables at various

Distance healthcare -0.156 (0.02)

levels of aggregation. For the

Distance bus stop 11.30 (1.08)

village level, we first present

Male wage (Rs) 1.78 (1.30)

evidence in Table 1 which esti-

Female wage (Rs) -2.96 (1.44) *** Significant at the 1% level. ** Significant mates a logit regression for old at the 5% level. * Significant at the 10% level. All regressions include district fixed effects.

and new SHGs at that level. Note

Analysis based on 220 observations. Absolute

that none of the village-level

t-ratios in parentheses computed with White heteroskedasticity-consistent standard errors. variables are significant.10

What about non-governmental organisations (NGOs) – do they favour certain types of villages for linkages earlier than others? First, NGOs operate within villages without anticipation of a linkage, i e, they move independently of SHG linkage in their own development work. Second, in comparing linkage models (since some groups are bank-formed and some are NGO-formed), we do not find a discernible difference in linkage choice of villages with old and new SHG members.11 Thus, we do not find any discernible evidence of NGOs favouring certain villages first over others. Table 2: A Comparison of the Observables Finally, to check for differ

between New and Old SHG Households

ences in the observable charac

Household-level Variables Coefficient for Ȗ

teristics for old and new SHGs at

Age -1.94 (1.39)

the household level, we ran re-

Gender -0.001 (0.07)

gressions of the following type:

Dependency ratio -0.01 (0.32)

Number of males 0.07 (0.36) Xijs = ĮD + ȕMijs + ȖTijs ...(3)

s

Primary education -0.08 (1.11) Secondary education 0.06 (0.95) where Xijs is the observable Tertiary education -0.01 (0.40) characteristic, D is a vector of

sLand owned in 2000 0.09 (0.45) district dummies, Mijs is a mem-

Shock dummy 0.03 (0.47)

ber dummy which takes a value

Land cultivated in 2003 0.06 (0.20)

one for members and zero other

*** Significant at the 1% level. ** Significant at the 5% level. * Significant at the 10% level. For the wise, Tijs is a treatment variable full regression equation, see Equation (3) in the

which takes on the value one

text. Results of other coefficients are suppressed. Analysis based on 840 observations. Absolute

for old SHGs and zero for new

t-ratios in parentheses computed with White heteroskedasticity-consistent standard errors. SHGs. Thus, the s ignificance of y

Economic Political Weekly

EPW

march 12, 2011 vol xlvi no 11

savings and lending). Household characteristics include age, gender, dependency ratio, education dummies, and a shock dummy.13 In order to control for initial wealth, we employ land owned three years ago. For village characteristics, in addition to male wage, we include the following distance variables

– paved road, market, primary healthcare indicates any difference over and beyond district and self-selection differences. The results in Table 2 indicate that none of the household-level variables were significant. Thus, at the village, NGO, and household level, we do not find any differences at least at the observable level between old and new SHG members.

This study focuses on a diverse set of ten representative districts from five states.12 The data used for the empirical analysis in this paper was collected from 1,000 households in 2003. Additionally, recall data for the year 2000 was also collected. Within the states, we avoided districts with over- and under-exposure of SHGs and selected SHGs for good operational links with banks.

For this particular study, the collected data was further refined. Of all the respondents, 114 were from villages with no SHGs. Since these households were not provided the opportunity to self-select, these were dropped. Sixty old and new SHG respondents were from the same village and this would contaminate the sample since the earlier signees may have different observable and unobservable characteristics compared to later signees. Of the remaining sample, 604 respondents were from mature SHGs, 185 from new SHGs, and 51 were non-members.

For the critical variable in our test, SHGMON, or the number of months since a member had joined a SHG, we made the following adaptations. Since an SHG is bank-linked only six months after formation, we needed to take those six months into account. A lmost all the new SHG respondents in our data had been members for less than six months, and for these SHGMON was coded as zero. Only 14 of these new respondents had been members for more than six months, so we subtracted six months from the date of formation. Similarly, for the mature SHGs, respondents’ SHGMON was calculated by subtracting six months from their membership. A few mature SHG respondents (46) did not report the date of their SHG formation. For these households, we used the number of the months since they received the first SHG loan for SHGMON.

As suggested by Doss et al (2008), we divide assets into six categories – land owned, livestock wealth, dwelling and ponds, productive assets, physical assets, and financial assets (including

5 More Advanced Impact Evaluation Techniques

The pipeline method outlined in the previous section can be combined with other methods to address other relevant impact questions. In particular, SHGs provide skill formation training for their members. For training, NCAER (2008) does not provide any CAGR calculations but simply states that about half of the members receive no training. NCAER (2008) further states that only 30% of the SHG members receive adequate training and that most of the training need is technical.

One can investigate whether training positively affects members. In this case, we would face a double selection problem, at one level the membership issue and at the second level, whether members choose to undertake training or not. Due to reasons similar to membership selection, for training impact on SHGs, one would need to turn to non-experimental methods. Since much of the choice of training is up to the individual, we need to account for this selection problem.

A popular non-experimental method is propensity score matching. Intuitively, we can first predict (either through a logit

SPECIAL ARTICLE

or probit equation) who will decide to train, and generate a propensity score or a probability for choosing to train. We then can match these probabilities with those who decide not to train and any differential between the treatment and control groups we can attribute to training. We combine these with our previous equation, equation (3) which accounts for member selection. The combination of these methods is referred to as regressionadjusted matching. These methods can be implemented directly with readily available software programmes (Ichino and Becker 2002, Leuven and Sianesi 2009).

By employing regression-adjusted matching, one can actually test the impact of training on outcome variables. Table 9 presents the results of this method using one particular algorithm, the LLR or local linear regression method. Our results indicate that training positively impacts assets but has no effect on income, consonant with our previous results in Table 5, which focused on participation alone. Table 9: Regression-Adjusted Matching Estimates (LLR Algorithm) of Training Impact on Assets and Income

** Significant at the 5% level. * Significant at the 10% level. P-values in parentheses. Figures in parentheses are standard errors created by bootstrap replications of 200. Covariates of regression are the same as Table 3. See text for definitions of variables.

A similar issue arises with measuring the impact of variables such as empowerment. Here NCAER (2008) includes a table ( Table 5.29 specifically) on the impact of social empowerment of women. They ask women whether they feel more empowered a fter joining the SHGs. Not surprisingly, 92% claim that they feel more empowered. Other than the many problematic issues outlined in the previous sections, here the study draws its information from self-reporting households, many presumably knowing that they are asked the information for studying the impact of SHGs.

In Table 10, we report a similar exercise with our data as that done by NCAER, though with an additional category breakdown of mature and new SHGs and non-members. Note that for a number of categories such as increase in self-confidence, mature and new SHG members yield virtually identical results. Since new members have not been members for long, either the act of joining empowers members or respondents believe they have to r espond positively when asked this question. For many of the decision-making variables, we do not observe much difference between the mature groups and non-members. These differences may indicate that in terms of decision-making, membership may not matter.

Apart from being a conceptually complex issue, women’s empowerment is a latent variable and cannot be measured directly. An effort to measure empowerment such as by using a composite index suffers from the arbitrary assignment of weights to variables. An additional difficulty is that many of the variables are

o rdinal in nature and assigning a numerical value may be inappropriate. One possibility is to follow Jöreskog’s LISREL structural modelling method in which the unobserved univariate continuous distribution generates an observed ordinal distribution as a latent response distribution (Jöreskog 2002). The underlying variable therefore assigns a metric to the ordinal variable. In contrast to the pipeline method, Bali Swain and Wallentin (2009) divide the groups into SHG members and control group. They employ the following structural equation for analysing the change in women’s empowerment between 2000 and 2003:

Ș= Į+ Ƚȟ + Ȣ ...(5)

where Ș is the latent women's empowerment of the respondent at year 2003 and ȟ denotes the latent women's empowerment in 2000. The symbol Ȣ indicates the error in the structural equation. However, the estimate of prime interest is Į, the mean change in the latent women's empowerment from 2000 to 2003. The results from the general structural model confirm that the mean difference is highly significant for the programme group but statistically non-significant for the control group. This indicates a s ignificant increase in the level of women’s empowerment over time for the SHG members but no such change for the members of the control group. We need to state two caveats for these results. One, we need to combine LISREL with self-selection correction and two, the 2000 variables were obtained through recall. Still, one can presumably use these variables again with readily available software to better measure the impact of empowerment.

6 Conclusions

We briefly summarise and provide some policy conclusions. In this paper, we have first evaluated the recent NCAER impact study on SHGs, which follows a methodology similar to other impact studies on SHGs. We have argued that the methodology, based on recall data with lack of a proper control group, can lead to misleading impact measurement with the CAGR. Using different data on SHGs, we calculate the CAGR for our data. We show how the CAGR methodology can be improved if these calculations are also made for control groups such as new SHG members and nonmembers. We then show how the pipeline methodology can be readily implemented for evaluation. In our results, we find a strong impact on total assets and an increase in livestock income and salaries. Marginally, both health and education expenditure are affected. We find no impact on other variables.

march 12, 2011 vol xlvi no 11

EPW

2 Now known as Deutsche Gesellschaft für Inter-

3 Thus, the CAGR is undefined when base is greater 13 The shock dummy is coded as one if the respondent K remer (2007): “Using Randomisation in Ecothan the end and when the base is zero. These resreports yes to any one of the following – social nomics Research: A Toolkit” in T Schultz, Paul trictions are problematic for economic impact and religious emergency, failure of crops (includ-and J Strauss (ed.), Handbook of Development variables, less so for financial variables.