The Tarunya Project’s efforts to improve the quality of adolescent reproductive and sexual health services in Jharkhand state, India: a post-hoc evaluation

Introduction

In 1994 at the International Conference on Population and Development (ICPD) the reproductive health needs of adolescents as a group were formally recognized and articulated. The programme of action (PoA) at the ICPD in para 7.47 states,

“Governments, in collaboration with non-governmental organizations, are urged to meet the special needs of adolescents and to establish appropriate programmes to respond to those needs” (UN 1994) (1).

India was a signatory to this PoA. Its efforts to operationalize the PoA began with the Reproductive and Child Health (RCH) program that was launched in 1997. In the following years, Adolescent Reproductive and Sexual Health (ARSH) was recognized as a top development priority in the country’s National Population Policy 2000 and as a health priority in the subsequent phase of the Reproductive and Child Health (RCH-II) program that was launched on the 1st April 2005 (2, 3). Under the National Rural Health Mission (NRHM) launched in the same year, States with weak health indicators and/or health infrastructure were identified. A package of promotive, preventive and curative health services to be delivered through a combination of static and outreach approaches, at each level of the public health system of these states, was formulated. The ARSH strategy that was launched in 2006, was set within these broader efforts of NRHM to strengthen the health system. Separate clinics with convenient working hours were planned for adolescents at secondary level health facilities, i.e. the community health centres (CHCs) and district hospitals (DHs).

Jharkhand is one of the poorest and least developed states in India. It has a population of about 32 million spread over 24 districts (Table 1). Twenty-two percent of the state population is in the adolescent age group of 10–19 years. Apart from awareness about HIV/AIDS, as compared to the national average, adolescents in the state – particularly girls – fare very poorly on all critical health indicators. Poverty, illiteracy, lack of awareness, autonomy and decision-making ability increase the vulnerability of adolescents (6). Further, a massive deficit in manpower and poor facilities in the government sector hinder provision of good quality services (7).

To address the poor health situation of adolescents, the government of Jharkhand launched a number of initiatives in partnership with non-governmental organisations (NGOs) and community-based organizations (CBOs). The Tarunya/ARSH project implemented in collaboration with EngenderHealth was one such initiative.

Tarunya Project In 2008, EngenderHealth launched the Tarunya Project with the financial support of the David and Lucile Packard Foundation. The overall objective of the project was to support the state government in improving the quality of services through cascading ARSH training and to strengthen its ability to increase knowledge about, generate demand for sustainable and good quality ARSH services and increase the utilization of these services by adolescents. At the community level, the project worked to strengthen links between ARSH services and outreach activities to enhance community engagement. At the block and district level it worked to strengthen CHCs and DHs to ensure the availability of and access to good quality ARSH services. And at the state level to institutionalize necessary changes in state policies. The project was initially launched in 12 districts and subsequently in 2011 on request from the state government it was scaled up to all the 24 districts of the state.

After 5 years of implementation, the David and Lucile Packard Foundation invited the Department of Reproductive Health and Research of the World Health Organization to lead an evaluation of the project’s efforts to build the capacity of the state of Jharkhand in implementing the state ARSH program.

The aim of the evaluation was to determine whether the Tarunya Project had implemented the interventions it set out to implement and achieved what it set out to achieve. The evaluation was also conducted to draw out the lessons learned from its 5 years of work, in order to strengthen the project’s operations in Jharkhand state and to apply these lessons in other states. The specific objectives of evaluation were to study the project’s design; implementation and monitoring; outputs in terms of strengthening the quality of health services, community support, adolescent awareness and adolescent demand, policies, planning, management and institutionalization; behavioral outcomes; and health impacts.

This paper focuses on one component of the evaluation, i.e. how the project working in conjunction with the state ARSH program contributed to improving the quality and utilization of ARSH services in the state. The questions it sought to answer were:

1. What was the project’s strategy to improve the quality and expand ARSH service provision to adolescents,

2. Did the project improve the quality and expand the provision of ARSH services to adolescents?

3. Did the project increase the utilization of health services by adolescents?

Materials and methods

Methods

The evaluation team secured ethical and technical clearance for the evaluation, from an ethical review committee set up by the evaluation team which also reviewed the evaluation protocol and tool kit.

To answer the first question of evaluation, the evaluation team reviewed relevant Tarunya Project documents, and interviewed an array of relevant stakeholders including: staff of EngenderHealth; program managers and staff of the Tarunya Project at the state and national levels; state and district level staff of the government of Jharkhand who were involved with the planning and management of the ARSH program; and medical officers-in-charge and other health service providers at health facilities.

To answer the second question, in addition to the above methods, the evaluation team observed health facilities and adolescent clientprovider interactions at these health facilities, and carried out exit interviews with adolescent clients. To answer the third question, the evaluation team first carried out surveys of adolescent girls and conducted focus group interviews with adolescent boys. The team then followed up with a supplementary survey of girls and boys. A more detailed description of the sampling and recruitment processes for all of these elements is described below.

• Districts: In total, 19 districts (Figure 1: Map of Jharkhand) out of the 24 districts were covered in the evaluation exercise. Five districts, four of which (Sahibgunj, Godda, Pakur and Dumka) are located in the north-eastern part of the state, were affected by an insurgency and were considered extremely unsafe. These along with Hazaribaug, another district considered unsafe, were therefore excluded from the list of districts considered for the evaluation.

• Health facilities: The Tarunya Project’s aim was to ensure that at least 79% of the health facilities (CHCs and DH) where ARSH clinics were located were ARSH ready^[1] by the end of the project period. The key unit of intervention and therefore of our analysis were the 194 health facilities (CHCs and DHs) in the state. As with many projects, due to resource and geographic constraints, not all facilities received the same degree of intervention. For the purpose of this evaluation, the facilities were classified into “High”, “Medium, and “Low” categories of intensity of interventions, based on an intervention intensity score^[2] devised by the evaluation team to reflect the level of intervention received by each facility.

• Populations studied

  Sample: Excluding the health facilities located in insurgency affected areas, from the sampling universe the evaluation team had a list of 149 health facilities. From this universe, a representative sample size of 34 was found to estimate p within ±0.12 with 95% confidence. To ensure adequate representation of health facilities with all three intervention intensity scores, 12 from “High”, 13 from “Medium” and nine from “Low” intensity of interventions categories were selected through the systematic, equal probability (epsem) method of sampling from the three categories of intervention intensity.

  At the level of these 34 facilities, the evaluation team conducted health facility assessments (n=34), interviewed three to four randomly chosen health service providers (n=121) and clients exiting these facilities (n=123) and observed client-provider interactions (n=123) for these clients. At facilities with low adolescent client caseload, mystery clients were employed; they were interviewed and their interactions with providers were observed.

  In addition, the evaluation team interviewed Tarunya program managers and EngenderHealth staff at the state, national and international levels (n=7) and senior state government officials (n=2) who were directly associated with the conceptualisation and implementation of the program.

  Finally, project reports, monitoring data and other relevant documents were reviewed to understand the project plan, implementation process and outcomes. The sample, selection process and tools used are presented in Table 2 (Appendix 1).

  The evaluation team carried out a household survey of 1288 adolescent girls. At the community level, the sample of adolescent girls with a 5% margin of error and a 95% confidence interval was estimated to be 400 in each of the categories of intensity of the project’s interventions – high, medium and low. Eighteen girls were selected through systematic random sampling from house lists of two randomly selected village level volunteers or Accredited Social Health Activists (ASHAs) under the jurisdiction of each of the 34 health facilities selected for the evaluation. In addition, the evaluation team conducted six focus group interviews with purposively selected boys, two in each category of intensity of interventions.

  Data was collected and managed by a team of 20 field investigators, two supervisors and two coordinators. All the data collection team members were graduates, fluent in the local language, had at least 2 years of experience and had received intensive training for 4 days. Data was collected using structured/semi-structured tools after obtaining requisite permissions from the state and informed consents from the respondents.

• Data entry and analysis: Quantitative data from the interviews was entered and analysed using software package – SPSS 22 (IBM, NY, USA). Data from in-depth interviews and FGIs was transcribed in English and analyzed manually. χ² tests were used to test statistical significance of findings.

• Composite index for quality service provision: The evaluation team developed a multi-component composite index of health facility and health worker performance in ARSH service provision (CI), in consultation with EngenderHealth. The index included 20 complementary indicators reflecting health facility and service provider characteristics through the perspective of different stakeholders (Appendix 2: Table 3). The evaluation team identified those that were critical and gave them twice the weightage as that given to others. Sources of information included observation of facilities and of client-provider interactions, and interviews with health facility managers, health service providers and adolescent clients exiting health facilities. The team added the scores on all 20 indicators for each health facility, and arrived at a total score for each of the 34 health facilities. Based on the CI score, the evaluation team assigned the 34 health facilities into one of categories of performance, “High performance” (top 11 facilities), “Medium performance” (12 facilities) and “Low performance” (last 11 facilities). The CI score of performance thus serves as a proxy for quality of health services.

• Hypothesis: The evaluation team’s hypothesis was that the project’s interventions were successful in helping make health facilities ARSH ready, in improving quality and in expanding services to adolescents and that the successes achieved were directly proportional to the project’s efforts, i.e. the intensity of interventions.

• Determinants of quality of services: The team assumed that the efforts of the Tarunya Project as reflected in the intensity of intervention, the rigorous monitoring carried out in select facilities and the period of exposure would have a positive effect on the quality of services provided at the health facilities and that this would result in a “High” CI score. Similarly, as the designation of facilities as “ARSH” ready was based on compliance with the state issued ARSH guidelines, the evaluation team assumed that the infrastructure, trained manpower and enabling environment were in place and that these health facilities would provide good quality services and secure “High” CI scores. Finally, the evaluation team assumed that those health facilities which provide good quality services and have “High” CI scores, would also be those which had more satisfied clients (than those with “Low” CI scores). To test these hypotheses, the determinants and expected relationships the evaluation team used the CI score of performance as a proxy for quality of health services.

Figure 1:

Map of Jharkhand.

: Districts in which evaluation took place.

Results

The evaluation team analyzed the findings to answer the three evaluation questions mentioned earlier.

1. What was the project’s strategy to improve the quality and expand ARSH service provision to adolescents?

   The project’s objectives and planned activities were well aligned. It had a clear and focussed strategy to improve the quality of ARSH services in government facilities. The project strengthened training and capacity building of health services providers through the development of technically correct and contextually appropriate training and educational materials in the local language. The government adopted its recommended cascade training model in which master trainers (MTs) were the foundation for training different cadres of health service providers. The project succeeded in creating a pool of trained master trainers. Project staff backstopped training of medical officers (MOs) at the regional level and front line workers (FLWs) and auxiliary nurse midwives (ANMs) at the district level. Master trainers trained by the project trained 363 MOs and 1936 FLWs/ANMs. All the FLWs/ANMs and 66% of MOs were reportedly trained by MTs who had been trained by the project.

   To bolster the supervisory and problem solving system, innovative approaches such as COPE^® (Client-Oriented, Provider Efficient services) (8), clinical monitoring (CM), supportive or facilitative supervision, and performance tracking of providers (PTS) were introduced. The project trained 63 supervisors in a facilitative approach to supervision.

   The project carried out focused monitoring in 76 of the ARSH facilities. The project staff conducted ARSH readiness assessments in these facilities and data on ARSH readiness of the remaining facilities was provided by the facility staff. Project staff developed tools such as client exit interview guides, ARSH readiness assessment checklists, clinical monitoring and performance to standards guides and to strengthen the reporting mechanism, they improved quality of the state government’s reporting format. The project appointed district coordinators played a vital role in monitoring and supporting the use of monitoring data to take remedial action. They supported district managers and health facility managers with monthly monitoring of the ARSH data and report writing, and participated in district review team missions and in monthly meetings of the district health team.

2. Did the project succeed in improving the quality and expanding the provision of ARSH service provision to adolescents?

   According to the project’s annual reports, COPE^® was used routinely by the staff in 70% of facilities where it was introduced, and CM in 31% of the facilities where it had been introduced. These efforts led to improvements in the quality of health facilities, and in the performance of service providers. The project’s internal assessment showed that 83% of the MOs and 69% of the FLWs whom staff had observed, correctly performed the critical skills required for provision of good quality ARSH services. The evaluation team’s assessment of quality found that almost 75% of the facilities (26 out of 34) provided quality health services as specified in the ARSH guidelines. The evaluation team used the CI score as a composite measure of quality health services. In relation to this score, the selected health facilities scored well, with minimal variation in the scores. The CI score ranged from 4.7 to 22 with the mean CI score of the 34 selected health facilities at 17.51 (out of 25). About 62% of the facilities had scores of 17.51 or more. When the health facilities were ranked as per the CI score, barring the last nine facilities all had scores above 16.5. The team’s assessment showed that 76% of the facilities (26 out of 34 facilities) had a CI score of more than 16.5 (i.e. 66.6% of the maximum composite score of 25), very close to the specified aim of the project.

   According to the project’s reports 34% of the 76 health facilities it monitored were “ARSH ready”. Of the health facilities assessed during the evaluation, 19 had been designated as “ARSH ready” by EngenderHealth team. Most of these indicators were also those which were either necessary for provision of quality health services. The 19 health facilities designated “ARSH ready” by the project had a mean CI score of 19.2 and seven of the 19 facilities ranked in first 10 as per our CI score based ranking. In contrast, the mean score for those 15 health facilities not yet considered to be “ARSH ready” was 15.4 and only three of them were in the first 10 ranks.

   Half of the sampled health facilities (n=34) had been exposed to the project’s interventions for a period of 3 years and the remaining 17 had been introduced into the project half-way through and were thus exposed for about 18 months. The evaluation team found that the mean score of health facilities exposed for the entire project duration of 3 years was 16.8 and three of the 17 facilities ranked in the first 10 facilities as per the CI score based ranking. Comparatively, the mean score for health facilities recently introduced into the project was 18.2 and seven of these 17 facilities ranked in first ten.

   Satisfaction with services across different health facilities was found to be similar, regardless of CI score. The majority of the clients (87%) during their exit interviews reported that they were satisfied with the services they received on the day of the visit. There was no relationship between the CI score based ranking of facilities and clients satisfaction at these facilities. Except for one facility in which three out of the four clients reported that they were not satisfied. This particular facility had “High” intensity of intervention, was amongst the 76 regularly and stringently monitored ones and had also been labeled as ‘ARSH ready” under the project.

   Type of health facilities in which the project was more successful: There appeared to be a direct but not perfect relationship between the CI score based ranking and the intensity of the project’s interventions. Out of the 12 facilities which had “High” intensity of intervention, five ranked within first ten in the CI score based ranking and the mean CI score at 19 was above the mean score for the selected 34 facilities. The mean CI scores for the “Medium” and “Low” intensity category facilities were 16 and 18, respectively. The proportion of facilities which ranked within first ten was two out of 13 in facilities with “Medium” and three out of nine in facilities with “Low” intensity of intervention.

   There was also a relationship between CI score based ranking and internal designation of facilities as “ARSH ready” by EngenderHealth. The majority of the project-supported health facilities performed at a relatively high level of ARSH readiness also had high CI scores (17/25 CI score or above). The period of exposure to interventions was however inversely related to the CI score based ranking. Facilities with a ‘long’ period of exposure to the project’s interventions did not have higher CI scores. While eight out of 11 facilities with “High” scores were “New”, i.e. had come into the project halfway through its timeline and had ‘short’ exposure to the intervention around 2011, 5 out of 11 facilities with “Low” CI scores were “New” (Appendix 3: Table 4).

3. Did the project’s effort succeed in improving the utilization of health services by adolescents?

   The Tarunya Project gathered data on the attendance and use of services by adolescents at ARSH clinics on a monthly basis. Data on selected indicators of service use by adolescents are graphically presented above (Graph 1). There were dramatic increases in service utilization in the first and second phases of the project. In the third phase – when an additional 12 districts were added in 2011 – increases were relatively less. By Phase V there was a visible decline in use of certain services at these clinics. The decline was specifically for maternal care and family planning services and prophylaxis for anemia.^[3]

   Data from the evaluation team’s survey of adolescent girls in the community showed that about 10% of girls surveyed reported that they had used any health services in 6 months prior to the survey. Of the girls who said that they had done so, 77% had sought services for menstrual problems, 8.3% for anemia and weakness, 6% for antenatal care, 3.6% for reproductive tract infections/sexually transmitted infections and 1.2% each for family planning, abortion services and for domestic violence. Only three out of the 1288 girls said that they had ever sought family planning methods and of these one was refused on the grounds of her not being eligible for receiving them. In line with the data gathered by the Tarunya Project, girls in the age group 15–19 years were more likely to have sought services than younger girls and boys, in the 6 months prior to the survey. More married and literate adolescent girls sought services than unmarried and illiterate, but these differences were not statistically significant.

   More girls from the areas of facilities with “High” scores said that they had sought services, from ARSH clinics (43.6%) and private clinics (33.3%) as compared to the girls in the area of facilities with “Low” CI scores (14.8%) for both ARSH clinics and private clinics.

   In focus group interviews conducted by the evaluation team, none of the boys had reported seeking care.

Graph 1:

Percent change in ARSH service utilization.

Source: Tarunya Project MIS.

Discussion

Implications

The multifaceted approach that the Tarunya Project used to improve health worker and health facility performance is in line with evidence-based best practice (9). The application of such approaches has shown to increase the quality of health service provision in a number of countries. Experience from a number of countries has shown that without concerted efforts to create community support for health service provision to adolescents and adolescent demand for their use, improving the quality of health provision alone will not lead to increased service utilization (10).

Given this, the evaluation points to clear actions for the Tarunya Project. It should continue to support cascade training by the government because while the training materials developed with the support of the project have been adopted by the government and the cascade model has been institutionalized in the government’s training system, handholding and back stopping by the project are still very much needed. It should also support the institutionalization of monitoring and data management. Finally, while the project should continue to support efforts to improve the quality of health service provision, it must step up support for increasing awareness of and demand for ARSH services.

The evaluation also points to clear actions for programs and projects elsewhere. Formative research is important to understand adolescents’ health service needs, and health care seeking practices and preferences, factors that hinder the provision and utilization of health services, and what could be done to overcome them.

The required standard of performance for health workers and health systems and a monitoring system should be set at the beginning of implementation. Health worker capacity and attitude building should employ complementary evidence-based approaches, and actions taken to make health services effective and more responsive. And quality should be assessed using data from multiple sources – provider readiness, health facility situation, and client satisfaction. Finally, alongside efforts to improve the quality of service provision, programs and projects should plan and implement complementary strategies to generate demand for health services.

Limitations

This evaluation should be viewed in light of a few limitations. There were no baseline data on most indicators, against which to compare our findings; thus, these results represent a post-intervention assessment only. Furthermore, due to resource constraints, the evaluation team was not able to compare these findings against a comparison group of facilities without intervention. The development of the intensity score was meant to serve as an internal proxy for comparing “dosage” of the intervention, in the absence of a comparison group. Secondly, as in many places, health providers were not present in some of the facilities (4/34). As a result, the planned sample size was not achieved.