Final Evaluation 

 Terms of Reference

Please note: A DBS /police check is required when an external consultant has either one-to-one contact, regular or frequent contact with children and/or young and vulnerable people in the course of their assignment. 

 Contents 

Background:                                                                            3

Purpose and Objectives                                                             4

Methodology                                                                            4

Schedule                                                                                 6

Dissemination plan                                                                   6

Annexure A: List of Stakeholders to be Interviewed                     7

Annexure B: List of Document for literature review                      7

Annex 3:  Evaluation Framework (OECD and TF Quality Standards)8

Acronyms

KII Key Informant Interview

OECDThe Organisation for Economic Co-operation 

DACDevelopment's Development Assistance Committee

TF Tearfund

FGD Focus Group Discussions

INGO International Non-Governmental Organisation

ToR Terms of Reference

SSWMBSindh Solid Waste Management Board

IRRCIntegrated Resource and Recovery Center 

AHKMTAkhter Hameed Khan Memorial Trust.

LFALogical Framework

TOCTheory of Change

EESEconomic and Environmental Sustainability

Terms of Reference 

1. Background:

1.1Programme Title:Turning plastic and other waste into green products, to improve the environment and provide poor people of Sindh improved health and livelihoods. 

1.2Background to the Programme:

The project ‘Turning plastic and other waste into green products, to improve the environment and provide poor people of Sindh improved health and livelihoods’ was designed incorporating learning of a smaller pilot project implemented in Islamabad and in line with the corporate priority of Tearfund “Economic and Environmental Sustainability”. The project is an outcome of a needs assessment conducted with various stakeholders and communities. The key problem identified was grossly inadequate solid waste management (SWM) in Sindh resulting in open burning and dumping, which is causing environmental degradation (CO2 emission, waste/plastics in water courses and oceans), negative impact on the health and wellbeing of the poor communities, and urban flooding. The causes determined were unsystematic and ad-hoc waste collection, existing solutions being too expensive and environmentally harmful, changes in functions, and lack of coordination between waste management and recycling actors and indigenous political economy actors including political parties, locally elected governments, and different service departments of governments, formal and informal private sector, influencing SWM services specifically in Karachi.

The project has undergone major adaptations during implementation based on the learnings and situational and contextual analysis. A community-based project was adopted as Public Private Partnership by signing a Memorandum of Agreement (MOA) with the Sindh Solid Waste Management Board (SSWMB), which is a key department of the local government of Sindh responsible for waste management in the Sindh province. The number of Haryali hubs reduced from 3 to 2, however, the capacity of two Haryali hubs increased from processing waste of 3,000 households per Hub to 12,000 households. Haryali Hubs are waste processing and recycling units where the household waste is sorted for value waste, and kitchen waste is processed in compost, and one-time-use plastic bags are recycled into pellets to be used as raw material. This waste is then sold to generate revenue to cover the operational cost of Haryali Hubs. 

The project is implemented in four union councils of two districts of Karachi. UC-30 Saeedabad, and UC-37 Naval Colony of district Kemari and UC-08 Moinabad, and UC-11 Murad Memon of district Malir in Karachi. A new community, UC-09 Sharafi Goth was added in the last year of the project at the request of SSWMB. 

Under this project, the targeted communities were mobilised and two household waste processing and recycling units were established, one in each district of Kemari and Malir. 

The aimed project outcome was a well-coordinated and sustainable waste management, recycling, and behavior change model in 4 target communities in 2 districts of Karachi, Sindh that can be adapted in the rest of Sindh, measured through four indicators given below.

1. Behaviour change Indicator: Number of households (individuals, % male/female) segregating kitchen waste from other household waste.

2. Sustainability indicator: number of Haryali Hubs whose revenue meets their operating expenditure  

3. Collaboration Indicator:  Enhanced stakeholder's skills and institutional capacity for operational waste management and the recycling model    

4. Learning Indicator: A resource bank of the Project's learning products established which is accessible (Tearfund Learn (a web portal),  learning documents) to a number of stakeholders both locally and internationally.        

Health research to assess the impacts of health & environmental indicators was designed and baseline and midline were conducted during the project period shall be a separate but connected piece of this evaluation   The proposal and reports are available in the relevant folder

2. Purpose and Objectives

2.1 Evaluation Goal and Objectives

The purpose of this evaluation is to identify lessons learned and examples of best practices. The evaluation aims to objectively assess the relevance, efficiency, effectiveness, impact, and sustainability of the project in light of its objectives and provide  learning and  recommendations for improvements in future programming. The evaluation also aims to develop an evidence-based lesson learned through independent research to encourage  local stakeholders for replicating the project model.  The evaluation will endeavour to serve the needs of a wide audience including Tearfund, its partners, local and international stakeholders, the beneficiaries, and the FCDO.

2.2 Key Objectives for Review

1. To assess project impact and an indication of the likely sustainability. of the project. 

2. To assess the relevance, efficiency, and effectiveness of approaches and interventions toward achieving higher-level goals. 

3. To assess whether the project was inclusive, participatory, and impactful for the vulnerable and marginalised communities.

4. To assess the coordination & collaboration with partners. 

5. To assess the alignment of the project towards the Economic and Environmental Sustainability priority of Tearfund. 

6. To assess the alignment of the project towards Tearfund’s quality standard namely: Behaviours empowerment, impartiality & Targeting, Resilience, Accountability, Protection, Gender & Technical Quality  

7. Provide evidence-based learnings, good practices, and recommendations for programming, community action, and advocacy. 

2.3 Review framework using OECD-DAC Criteria 

Guiding framework and key questions have been annexed for reference as Annex A .

3. Methodology

OECD-DAC Criteria of Relevance, Efficiency, Effectiveness, Coherence, Impact, and Sustainability will be the main guiding principles for the evaluation. Tearfund also uses its own standards around behaviors, namely; Empowerment; Impartiality & Targeting; Resilience; Accountability; Protection; Gender, and Technical Quality. The review will be carried out in a way to ensure that Tearfund's list of behaviors is picked up and also evaluated through the DAC criteria.

The methodology will be developed by the evaluator(s) in collaboration/discussion with the Tearfund Country Director and the Country Office team and should be appropriate and sensitive to the context and nature of the program/project which is being assessed. As this is an evaluation of a FCDO-funded project a further step to check that the methodology is acceptable to their MEAL manager will be taken. Tearfund evaluations are assessed using the Bond evidence principles (Voice and Inclusion, Appropriateness, Triangulation, and Contribution and Transparency) and consequently, evaluators should take these into account when devising a suitable methodology. The proposed methodology is expected to include:

• Desk review of key documents 

• In-country key informant interviews scheduled (online or in person) 

• Focus Group Discussions

• Tearfund and partner staff interviews.

• Direct observation/project visits by an in-country team.

• Haryali Hub progress data analysis 

• Secondary data where available 

• Discussion of initial draft report with Project, Country Office team, Regional team, and UK staff after submission (online / in person) to finalise any corrections and review findings.

A preliminary list of documents has been annexed as Annex B with TORs for reference. All interviews and focus groups are expected to be in in-person for stakeholders within the country. The evaluation team may facilitate remote focus group discussions (FGDs) and/or Key Informant Interviews (KII) only for stakeholders outside Pakistan. An initial stakeholder and potential interviewee list will be annexed as Annex C which includes a list of potential internal and external stakeholders.

 3.2    Ethics

The evaluator should adhere to Tearfund's approach to research ethics, outlined in this guidance: https://learn.tearfund.org/en/resources/tools-and-guides/doing-research-ethically,and work with the Tearfund team to complete an ethics risk assessment ahead of data collection. The proposal should outline key ethical considerations and how these will be addressed

4. Deliverables and Output 

 5. Evaluation Schedule

The evaluation is expected to be conducted in January 2024 and the final report is to be submitted latest by 29 March 2024 

6. Dissemination

The primary focus of the evaluation is to provide a final evaluation of the UK Aid Match funded project to the FCDO. Secondly, the evaluation would also serve Tearfund’s own learning within the regional and country office teams and thematic/corporate priority teams, and our partners in order to check our assumptions made regarding the appropriateness of the current program interventions and to inform future programming. The evaluation will also be shared with Tearfund Environment and Economic Sustainability Teams. 

Evaluation reports, especially learnings captured in the final evaluation, will be shared with the wider stakeholders both nationally and internationally via an online learning portal specifically designed under this project for the purpose of sharing the learnings with stakeholders.  (certain elements of the report may not be made public). 

7. Evaluation Leadership and Management 

Under the overall guidance of the Pakistan Country Director supported by the Country Office and ENA Regional Team, the evaluation will be conducted by an external evaluation team.  

External Evaluation Team: The evaluation will seek to have at least 1 local Karachi, Pakistan-based staff. The evaluation team may propose a detailed team configuration.  

Consultant/s Eligibility (for Evaluation Lead)

Tearfund will be responsible for:

• Reasonable assistance and support in pursuit of required security permissions to access project sites.

Evaluation Team: The evaluation team is expected to consist of at least 1-2 local staff..  

8. Proposal Submission:

All proposals should be submitted in hardcopy via courier/postage at the below address clearly indicating the “Call for Proposals for Project Final Evaluation”. Incomplete proposals will be rejected. The organisation may ask for all the proposals in soft copy as well. 

Address: Tearfund Country Office, House #15, Street # 158. Sector G-13/4, Islamabad, Pakistan

Contact No:  +92-51-2300863

The proposal should not be more than 10 pages. It must include the CV and references of key team members,

Timeline:  The proposal should reach the above-mentioned address maximum by COB 08 November 2023

Annex A:  Evaluation Framework (OECD and TF Quality Standards)

Please note that the following areas are only expected to be indicative and may be developed further and finalised by the evaluation team. 

Aspect 1 – Relevance

How relevant is the project to addressing the causes of the identified problem it intends to address?

Areas for consideration include:

• the validity of the project objectives;

• The beneficiary selection and engagement process

• the relevance of the project with the needs and vulnerabilities of the target group and the problem it intends to address; 

• the extent to which target beneficiaries felt involved in the process of design, implementation, and monitoring of the project;

• the ways in which the project design reflected lessons learned from previous experience;

• the use and enhancement by the project of the existing skills, knowledge and coping strategies of the target group;

• the consistency of the project with the vision, values, strategy, and resources of Tearfund;

• the major factors influencing the relevance of the project to the priorities and policies of the target group, the partner and Tearfund.

• flexibility of the response taking into account the changing context and how relevant the current intervention is in terms of geographical targeting

Aspect 2 – Efficiency

Has the project been efficient in achieving the intended objectives?

Areas for consideration include

• the cost-efficiency of project activities;

• the cost-efficiency of the Haryali Hub Model

• the output efficiency of the Haryali Hub in comparison to its intended output

• the achievement of objectives to time and to budget;

• comparison of the implementation of the project with alternatives;

• the major factors influencing the efficiency of the project.

Aspect 3 – Effectiveness

Has the project been effective in achieving the intended project objectives and outputs?

Areas for consideration include

• the achievement of the project outputs;

• the achievement of the project purpose;

• the contribution towards the project goal;

• the practices around a collection of data, evidence quality, monitoring and evaluation mechanisms, and their utilisation

• the major factors influencing the achievement / non-achievement of the project objectives and outputs.

• Use of learnings in updating design

Aspect 4 – Impact

What impact has the project had on beneficiaries and non-beneficiaries?

Areas for consideration include:

• the intended and unintended effects (social, physical, environmental, economic), both positive and negative, of the project on beneficiaries and non-beneficiaries. The views of beneficiaries about the impact should be sought.

• the major factors influencing the impact of the project on beneficiaries and non-beneficiaries.

Aspect 5 – Sustainability

Will the benefits of the project be sustained after the end of the project?

Areas for consideration include

• the continuation of Haryali Hubs Operations after funding by Tearfund has ceased;

• the ability of Hubs and projects to generate revenue;

• the ability to access funds and support from other sources (including the public & private sectors);

• the major factors influencing the sustainability of the project/Model.

• the major bottlenecks for sustainability at the moment?

• the major opportunities for sustainability

• the relevance of the exit strategy in terms of sustainability

• Recommendations for increasing sustainability.

Aspect 6 – Coordination and Coherence

How has the project been integrated with the activities and priorities of other agencies and organizations (including local and national government)?

Areas for consideration include:

• The linkages and relationships between the target group, Tearfund, and other agencies and organisations,

• The ability to manage multi-purpose partnerships,

• The coherence of the project with national and local policies, and departments

• The major factors influencing the coordination and coherence of the project with the activities and priorities of other agencies and organisations.

Aspect 7 - Beneficiary Participation and Satisfaction 

How has the project involved the intended participants (target group) in the design, implementation, and monitoring of the project? 

Annex B: List of Documents for literature review

Primary documents

1. Project Proposals & Budget

2. LFA and TOC (with all revisions)

3. MOA with SSWMB

4. Sustainability plan and post-project monitoring framework.

5. Project Quarterly & Annual reports submitted to MD. 

6. Assessments & survey reports by the Project team or by consultants

7. Monitoring reports

8. Minutes of Steering Committee Meetings

9. Records of Haryali Hubs Operations

10. Minutes of BiMonthly Hub Operational Meeting 

Secondary documents

11. EES Evidence Review Reports

12. Tearfund’s Quality Standards 

13. Partner organisational assessment, partner agreement

14. TF Guidance documents: DME Procedures, Safeguarding principles,  

15. TF Pakistan Country Strategy, Operational Plan, 

16. External literature on Waste Management Issues and Initiatives in Karachi Pakistan, 

Annex C:  List of Stakeholders to be Interviewed 

Internal

1. Country Director: Jonathan Johnson 

2. Project Director:Sajid Gulzar

3. MEL-Coordinator: Asher Loyal 

4. SCE-Coordinator:Syed Junaid Bashir 

5. Project Funding Manager:Katharina Raudzus Haque

6. Partner (MUET) Staff2

7. Haryali Hub Staff:6

External

1. Managing Director, SSWMB

2. Chairman, Haryali Hub Steering Committee 

3. Key Staff of Hangzhou (Contractor of SSWMB, responsible for door-to-door waste collection)

4. Beneficiaries Focus groups 

5. Community leaders

Quality Standards

 Tearfund’s Quality Standards

Tearfund aims to work to the highest possible standards with integrity and transparency, across all aspects of our work. We have identified a set of corporate Quality Standards in support of our vision and the delivery of our strategy. The eight standards summarise all of the relevant external and internal accountability and quality standards, codes, guidelines and principles to which we are committed. They exist to save time and ensure quality, meaning that new partners and staff can quickly get up to speed with Tearfund’s expectations and commitments to quality simply by familiarising themselves with these standards.

The eight commitments are non-negotiable; meaning that they are of primary importance in all Tearfund work and must be prioritised in any work we are doing: