Below are the categories we are using to organise all the submissions we’ve had to AMOSSHE’s Value & Impact Repository. After feedback the most popular option was to organise them by department area.
I’m excited because this means our members will be able to get to the tools they need as fast as possible.
I’d really welcome your comments and feedback! Please feel free to click comment or email me, at d.stoker@amosshe.org.uk with “VIP Categories Feedback” in the title.
I came across this last Wednesday:
The National Council for Voluntary Organisation: Here’s the Ultimate Impact Measurement Tool
Richard Piper’s suggestion is striking, namely: the human brain is the ultimate impact measurement tool!
The ultimate impact evaluation tool is the human brain. Every paper-based or online tool out there, every survey, every clever dial or star, every randomised control trial, every focus group, is an attempt to get close to the interpretative capability of the human brain. And many of these formal evaluation tools are ersatz versions of the brain, poor approximations of the real thing, chalk instead of bread flour, fizzy lager instead of real beer.
A declaration of interest here: I am building a repository of evaluation tools for leaders in Student Services to use to evaluate and improve what they do. Piper’s idea, in theory, could contradict my very job! Am I feeding people bread made with chalk, and sub-standard lager? Piper suggests he is not anti-tool per se:
I am not suggesting that the use of forms, surveys, dials, etc is necessarily wrong and that we should stop using them to evaluate impact. But I do feel that sometimes these tools are poor substitutes for our powerful, perceiving and interpreting brains. And I certainly think we should be using common sense ways of getting evidence, based on the perceptual and interpretative abilities of the brains at our disposal.
Good point, well made. The danger we face with tools is to switch off and over-rely on them. But I’m going to make my own counter argument:
Evaluation is a craft. As craftsmen we are refining what we do.
Evaluation is a science. What makes it so is our commitment to refining our knowledge and tools.
Can it be both? My own view is that both apply.
Below is my attempt to boil down Piper’s insights to some snippets that I think all of us working in evaluation can usefully reflect on:
I recently came across this article from the Philantopic blog, which has some really interesting thoughts on impact in the context of grant-applications in the not-for-profit sector that I think apply equally to student services. To this end I’ve cut out the key ideas for you; my credit to John Colburn for these. To begin;
“[I]t is clear that the advancement of knowledge and understanding occurs because a handful of practitioners persevered against the broader culture of practice and what can reasonably be “known” in order to elicit whole new understandings.
I totally agree! We must be brave in the learning process. He goes on to suggest:
[W]e should try to identify and avoid repeating the same mistakes that have yielded such limited results to date. Here are a few ways we can be smarter….
I think these four points are great!
a) Let’s separate accountability and compliance [...] – from impact assessment. The conflation of the two results in over-elaborate monitoring tools that distract from the impact-assessment process. […]
b) Let’s agree that simpler and inexact processes help move the ball on impact assessment. Often, we let the perfect be the enemy of the possible in impact assessment. […]
He suggests that we use “horse sense”, which I read as some sort of “intuition”. It’s worth remembering that intuition is actually the result of the totality of our experience – as scientists have studied.
c) Let’s agree that sharing our results — successes and failures — makes us all smarter. This means we need to begin to develop a common framework for describing our work, our goals, and our results. This allows [us] to learn from one another and avoid repeating each other’s mistakes. The boring and un-glamorous work of [evaluation tools] is an essential building block for developing and sharing knowledge.
The evaluation tools repository I’m working on will help Student Services managers in the UK to share information on best practice.
d) Let’s agree that there is complexity in impact assessment, but not let that stand in the way of seeking universal truths. Yes, context matters. […] And, yes, there are bound to be varying levels of quality in formulating and implementing impact assessment. Still, I am convinced that there are underlying commonalities to our work that allow us to learn from one another and begin to build a body of knowledge of what works and what doesn’t.
Some real wisdom I think, despite the slightly different context, and worth bearing in mind when approaching the difficult topic of measuring impact.
You can read the whole article here.
Let me introduce myself: My name’s David Stoker and I’m Value and Impact Project Officer at AMOSSHE, the Student Services Organisation.
You can find out more about AMOSSHE’s Value and Impact Project here: http://www.amosshe.org.uk/projects/vip
The Value and Impact Project is focused on improving our understanding and evaluations of student services. Its aim is to:
I am blogging to:
All views expressed on other projects are my own. I would love to chat with readers about value and impact via this blog!
Welcome to the Value and Impact blog. Watch this space.
