The outcome was a dataset of over 6,000 records: about 5,000 from non-HOL students and 1,000 from HOL participants. And the results are amazing.
FOR YEARS, Hands On Learning Australia has collected data about the dramatic impact the Hands On Learning (HOL) method has on students’ attendance and retention. However, while important, attendance and retention tell us very little about the quality of student experience.
Attendance data alone cannot distinguish between a kid who is depressed and miserable but turning up regularly, and another kid whose attitude has shifted from one of suffering, to hope. Through the lens of attendance these two children look the same, but their lived experiences are incredibly different.
We needed feedback that better reflects the way HOL works with kids, that captures the kinds of impact HOL is intended to have; this is far beyond whether they simply keep turning up or not. We wanted a measurement tool that could provide data to help us tell the HOL story, to collect data on things that we know matter at least as much, or arguably even more, than mere attendance.
The ARC project is particularly significant for us as it captures the idea of HOL as an enabling space for those kids who aren’t coping with school. It made it possible for us to use the framework of Connections, Capacities, and Meanings (CCM) to develop a set of questions to put to all secondary school students to establish a baseline to then compare with the responses of kids in HOL.
For us, the CCM framework represented three overlapping ideas, and we came up with questions for each part as shown in the diagram below:
Asking kids about their sense of inclusion fits nicely into the idea of Connection; skill development represents Capacity; and how much things matter maps well to Meaning. The overlap between skills and relating to others could be captured through communicating or feeling listened to (i.e. having good communication skills), that the overlap between skill and meaning was built into the sense of usefulness (individually having the skills to do something worthwhile), and the junction of meaning and connection was inherent in the idea of team effectiveness (people working well together to achieve things). At the very centre of all three lay the overarching concept of happiness.
This turned into the CCM survey, which generated responses from more than 20 HOL schools. We were also able to get three schools to administer it to all of their students at the end of every day for a week. We got kids to complete it multiple times, in order to average out their experience of school, given that some days are better than others. HOL kids also completed it at the end of each of their HOL days for two to three weeks to average out their experiences.
The outcome was a dataset of over 6,000 records: about 5,000 from non-HOL students and 1,000 from HOL participants. And the results are amazing.
The spread of non-HOL kids responses was a typical bell curve shape, while the HOL kids’ distributions were markedly skewed. Analysis confirmed that there were statistically significant differences between the two groups with a very high confidence level of p<0.001 for all seven questions. The main four results for Connections, Capacities, Meanings, and enjoyment are shown below.
We are delighted with the way this tool captures a slice of the reality of being in school. It tells the story of how schools are enabling spaces for many kids (those who cope well with the classroom), but it also highlights those kids who struggle with school. Importantly, it shows how the experience kids are having in HOL is dramatically different to their normal classroom experience. And, given HOL targets those kids who are not coping with school, it demonstrates the way in which HOL provides an enabling space within school for these young people. This explains why their attendance improves through HOL, because they really do feel that HOL is a place they belong, with people they belong to, doing real things, and having something of value to offer.
We hope other flexible learning organisations start to utilise the CCM framework to measure the impact of their own work with disadvantaged and struggling young people.
The Australian Research Council (ARC) national research project Building Futures For Young Australians, pioneers the concept of enabling spaces, and a new Connections, Capacities and Meanings (CCM) framework to measure the quality of student experience and what works to keep vulnerable kids at school.
The Linkage project was funded for 2011- 14 by the Australian Research Council. The ARC team constituted Johanna Wyn (CI) and Ani Wierenga from the Youth Research Centre at the University of Melbourne and Gavan McCarthy (CI), Mike Jones and Antonina Lewis from the eScholarship Research Centre and Simon Faivel (PI) and Duncan Peppercorn from Social Ventures Australia; Jo Taylor from Dusseldorp Forum; Ebeny Wood (PI) from Beacon Foundation; Richard O’Donovan (PI) from Hands on Learning Australia; Naomi Berman (PI) and Tessa Hillgrove (PI) from the Foundation for Young Australians; and Clare Shearman (PI) from the Southern Ethnic Advisory and Advocacy Council. Associate Professor Kitty te Riele was appointed chair of the project’s governance committee.
