This is the summary of conversation between Chanuka and Ricardo as part of the pre-workshop interaction. The responses depict views on issues related to evaluation of ICTs.
What constitutes credible evaluation and acceptable evidence in your context?
I have had limited experience in M&E, even with UNDP experience, I feel like a beginner in the context of IDRC. In this project we want to evaluate outcomes –rather than outputs.
Evidence: for example in India, instances where we gave rapid response and there were changes in policy. In Sri Lanka, when the government announced a tax on mobile phones, their research influenced the policy process and led to a reduced level of taxation.
How do you currently do M&E? What methods, approaches, etc. are you required / do you choose to use?
We are currently developing a method based on Key Performance Indicators (KPIs) and Key Result Indicators (KRIs). The idea is to have a set of quantifiable indicators that directly correlate to the performance of the organization, as well as individuals. For example: policy influence: the number of opportunities to voice their research at conferences. We will compare them with internal benchmarks.
What are the strengths and weaknesses of the way you currently do M&E?
Perhaps it is too early to comment as the process is still in the development stage. One strength is that we have already identified some indicators and we will now need to cut down to a select few.
How familiar you are with the following evaluation approaches?
GEM (Gender Evaluation Methodology)
No familiar at all.
OM (Outcome Mapping)
Have heard of it.
If you are interested in using parts of OM, do you want to use it for ongoing monitoring during the project or for summative evaluation?
How would you differentiate between outputs and outcomes?
Outputs is what we do (workshops, publications), while outcomes are about impact about the society at large.
Outcomes in their project are about policy influence.
MSC (Most Significant Change)
Not familiar at all.
What other approaches do you use?
UNDP approach, which I feel is similar to OM.
How much attention do you want us to give to LogFrames?
Provide brief review.
How much attention do you want us to give to ICT indicators?
Provide brief review, reference materials and a interactive session. I have already done a project on this and I am keen especially at the organizational level.
Preference for the reference materials to take home:
Both hard copies and electronic.
Provide your three main expectations for this workshop.
- To learn more about ICT indicators – specially in the organizational context.
- To become familiar with other M&E method and compare them with the one we intend to use.
- To make sure we are in the right path and learn something new to be added to current process.