Improving Emergency Packages Based on User Feedback
UNICEF Pakistan uses U-Report to monitor and strengthen programmes using community feedback
Due to seasonal flooding, earthquakes, droughts and conflict in Pakistan, the WASH emergency response included the need for water, sanitation and hygiene assistance. This led to UNICEF Pakistan to develop hygiene kits to be distributed to rural and urban communities, which have the highest risk of disease and infection. The design of the hygiene kit focused on enabling hygiene practices at critical times and reducing the risk of contamination and disease spreading.
To improve the effectiveness of the hygiene kit, only two feedback methods were available to beneficiaries; focus groups and written surveys. These methods, however, left a large gap towards gathering user experience, satisfaction, and kit effectiveness information. These conducted methods also did not penetrate a good sized population to generate a reliable sample and had no specific mechanism in place to allow every beneficiary to comment on the quality and relevance of the kit items received.
Past initial distribution, UNICEF had no way of determining whether each hygiene kit had reached its intended location. Essentially the hygiene kit distribution had deficits surrounding:
- Information received from the user and,
- Program oversight from UNICEF
As a result of these deficiencies and lack of input received from users, hygiene kits were not effectively improved.
To address these deficits, UNICEF Pakistan integrated UNICEF global innovations team’s U-Report, an SMS tool used for monitoring and strengthening UNICEF programs using community feedback to the WASH section. The U-Report is a software-based platform which sends and receives survey information from beneficiaries and consolidates their responses for post analysis.
At present, there has is no product that has allowed mass, automated, two-way communication with beneficiaries. This method has the potential to completely modernizes monitoring and evaluation, by automating data entry and analysis. Furthermore, the penetration rate for SMS feedback is much higher and further reaching, allowing the mass analysis of beneficiary accounts across gender, location, age and other factors such as profession, and minority status.
Prior to kit distribution, WASH specialists drafted questions on each kit item for users to respond to. These questions were then pre-tested on the field with a focus group in Rajanpur, Punjab. Once finalized, informative stickers were attached to the kit providing instructions to beneficiaries of how to trigger the survey for the Hygiene kit products.
After registration the respondents were asked four questions:
- Which item was most useful? A) Soap B) Bucket C) Menstrual Cloth D) Jerry Can
- Which item was least useful? A) Soap B) Bucket C) Menstrual Cloth D) Jerry Can
- Did anything arrive damaged? A) Soap B) Bucket C) Menstrual Cloth D) Jerry Can E) Nothing damaged
- What will improve this kit? (open answer)
UNICEF Pakistan distributed the hygiene kit together with the U-Report SMS innovation to districts in three provinces in Pakistan: Sindh, Punjab and Khyber Pakhtunkhwa.
The implementation was a success, resulting in the collection of real-time feedback from beneficiaries and encouraging results. Nearly nothing in the hygiene kit had arrived broken (~90%), and positive feedback was received about each items’ appropriateness and user preference.
Real-time response success
As the application of this project utilized SMS innovation in a programmatic capacity for the first time, there was no guarantee of uptake from the beneficiaries. Thus, the biggest highlight of this program was the success of the real-time monitoring component. Results show that nearly 2,500 beneficiaries out of a total of 20,000 distributed kits provided real-time feedback.
This represents a 12.3% response rate of all users. Moreover, the results were higher in areas of greater literacy and as a result, non-written methods of communicating the feedback survey such as Interactive Voice Recognition (IVR) will be considered in future in order to reach illiterate beneficiaries.
Areas for improvement
The negative score for the jerry could be due to a number of factors. Firstly, the most undesirable feature was the narrow entry point of the jerry as it was not particularly suitable to collect water on various water dispensary points. Secondly, the jerry was not big enough compared to other available options for beneficiaries and was deemed to be an inefficient vessel for water collection. Thirdly, the edges of the handle on the jerry were sharp. Lastly, after filling up the jerry, a significant weight is usually added on to it causing discomfort for beneficiaries during transportation.
Menstrual Hygiene Cloth
Results show that beneficiaries displayed a lack of clarity on how to use the menstrual cloth as well as numerous requests for alternative products by women. There were also communication gaps between intended design of items and beneficiaries’ understanding of their usage. As a result, the sanitary cloth tested negatively across all regions, for both genders (one case of indifference Men, Rajanpur, -0.03).
In closer detail, the sanitary cloth was designed for feminine hygiene purposes. However, including this to the kit required a minor altercation from its original state. The cloth included in the kit was a large sheet multiple times. Due to illiteracy issues, there were no written instructions on how to cut the cloth into sizes that meet the beneficiaries’ various needs. Moreover, some perceived the cloth as being a blanket and frequent instances of beneficiaries requesting commercial sanitary pads.
Currently, these findings have led the WASH section to further study household water storage and menstrual hygiene management during emergencies.
Implementation of this program provided constructive insights to the application of this technology towards distributing supplies.
The feedback regarding the potential confusion on the use and understanding of the cloth has led to investigating further the appropriateness of the menstrual cloth and the effectiveness of menstrual hygiene education.
More broadly this program provided best practice insights when conducting a remote survey. Firstly, the survey should be no longer than 7 questions, results showed that there was less engagement past 5 questions. Secondly, districts which received information mobilization from the NGO/implementing partner surrounding the innovation were more likely to respond.
This has been the first major rollout of program monitoring within Pakistan. Priority has been given to use U-Report to explore further applications within the UNICEF program portfolio. As this innovation has the potential to revolutionize program monitoring through effective community engagement.