How do you know if ICT is making a difference - and does it matter?
We are living in a digital world, we have plenty of anecdotal evidence that the use of ICTs can make a difference with learning, that it can engage and stimulate and extend the way our students learn. But how far do we need to show evidence that this is happening?
This was the topic of a lively discussion I had at a recent conference. Several teachers were in schools where 'proof' was required that ICTs could make a difference, before further investment would be made. There were concerns about assessing the use of the ICTs, as well as assessing within the intended learning area from the curriculum.
How do you know, when you are planning and then monitoring impact, whether the use of ICTs is making a difference? Does it matter? Should we be even trying to assess it?
Replies
- Annemarie Hyde (View all users posts) 15 Feb 2012 5:04pm ()
In my experience, the request for proof comes from the non-users.
School leaders are so aware that throwing money at ICTs doesn't mean instant pickup by staff, many who are still not engaged in the wonders of the internet, Web 2.0 tools and social networks - many of which are professional like this. There's nothing worse than finding that TELA laptop unused or the interactive whiteboard being used as a simple screen.
Unfortunately we need to convince ourselves that we are going to get bang for each buck as there are so few bucks to go round and other areas of the school need bucks too.
I know how I view technology (look at how I was when my VLN account wouldn't work). I have seen my students engaged with other students in other parts of the world in real time. I've seen nonreaders excited with their podcasts where they have been enabled to share their stories as others do. Can I show this on my standardised test results? Not always.
We also know that before ICTs are effective, effective teaching pedagogy has to be in place. How many times have I seen students playing games and the desktops/laptops playing the role of the baby sitting tool so that the teacher can work quietly with a group? Investment needs to go into professional development too.
At Ulearn last year, Graeme Aitkin, the Dean of Education at Auckland University, outlined the important things to measure when contemplating the use of ICTs: are they leading to greater achievement? Are they creating more motivation and interest in the learning? Do students have more confidence to learn? He shared the success of Steve Martin at Howick College who creates science lessons that encourage students to think for themselves, learn at their own pace and use ICTS which he says, "tricks" them into learning because they are having fun.
We can share this excitement with learning, by collating student voice and filming evidence to show others.
At the moment there are still a lot of people who just "don't get it" and seem to be hung up on the safety issues. Think about how the media twisted the message of Kevin Honeycutt's keynote at Learning@School this year!
- Moana Timoko (View all users posts) 15 Feb 2012 7:05pm ()
I've really had to think about that last question Karen - Should we be even trying to assess it? I'm thinking of how I've had to start using a Mac...an iphone...the VLN...in the past two weeks. I've learnt by having a tutu. I feel good about knowing the basics but realise that there are endless options for me to explore now that I've started. As a parent I I would be keen to see that my children are using different ICTs and then I would be wondering about what they can do in order to share their knowledge with me...one because I think my children know how to tutu better than I do and they're more than likely able to work at a faster pace because they've got smaller fingers lol! I honestly wouldn't care about being TOLD what they can do in some formal written report at a parent/teacher interview but... I'd rather prefer to be SHOWN what they can do and then be TAUGHT by them. My children have access to the internet via my lap top and iphone - and have been told that they have access at their schools - of which I am yet to check out! I think the important thing here is knowing that they have access to the tools of the present time in order to prepare them for the tools of the future...being an ICT tutu is a great start...ha ha l wish I could show someone what I've learnt over the past two weeks!!!!
- Karen Spencer (View all users posts) 19 Feb 2012 9:02pm ()
Thanks so much for your comments here, Annemarie and Moana, aku mihi nui ki a koe:-)
You both refer to the importance of seeing a difference, or students being able to demonstrate and share their learning, either with whānau or teachers. I like the examples you gave, Annemarie, from ULearn, and I remember the presentation well.
I thnk that, rather looking for a lift in test scores as 'proof' of the impact of ICT (too many variables), we are instead looking for evidence that technologies have enhanced or extended those behaviours that are the foundations for effective learning. In other words, we might observe the way - and ask students to record for themselves - how technologies have developed aspects of the Key Competencies, or extended the way we use effective pedagogical approaches.
Is that something that can be captured convincingly? How might it be done?
- annerotorua (View all users posts) 19 Feb 2012 9:41pm ()
How do we know if ICT's are making a difference. We note the following as examples:
1) We have engaged learners and the ICT's are helping us to engage learners. We could provide lots of examples from across all levels of the school - but highlight just a few:
An autistic child who has been helped, through using an app on the IPad, to move from being a very reluctant writer to being a very enthusiastic writer wanting to share his work.
Many students, including Maori and Pacific Island boys who "love" learning while using the Activboard. Note that I have said that they are using the board - not just watching the teacher use it!
Students enjoying the opportunity to participate in class forums, sharing/reflecting on their learning and providing peer feedback on our LMS.
Students love learning with other students from around the world - eg class blogs, travelling mascot blog, participating in online field trips, skyping other classes and resource people etc. We have the opportunity to offer authentic opportunities for learning
2) We have parents participating in the ongoing learning of their child through our LMS and class blogs. Parents can view work their child has done that day and provide some feedback to their child. They can also see what has been happening in class on a particular day or week or while students are at camp.
3) We are trying to make effective and efficient use of technology to keep parents well informed - Twitter for general reminders and notices, weekly newsletters posted on the website and emailed, emails used for contact with parents, parent surveys using survey monkey etc. If parents know what is happening and can communicate easily with the class teacher then we have better teacher/parent relationships and students coming prepared for learning eg bringing library books, swimming gear, returning notices for trips.
4) We have a number of teachers who find Twitter to be one of the best tools for PD. Of course it depends who you are following but it is very easy to share ideas and links, find assistance and see links to sites other have found particularly useful.
These are just a few examples. I'm sure others can provide many more.
We have reached the point where when we experience issues with any ICT tool we know about it really quickly and people want things back up and running asap - ICT is now an integral part of the everyday classroom learning.
- Enabling e-Learning (View all users posts) 21 Feb 2012 6:48am ()
Thank you for sharing these examples, St Mary's. You are clearly observing changes across your school and it is heartening to hear of the way individual students' learning has been positively affected:-)
Just out of interest, do you have any deliberate monitoring or evaluation processes of the different ways you are using ICTs (e.g. impact on parents)? Is that necessary, do you think?
- annerotorua (View all users posts) 21 Feb 2012 3:00pm ()
We have used survey monkey a number of times to survey parent
eg at the end of last year we surveyed parents about the use of the LMS from a parent perspective
eg we have surveyed parents about their access to the internet at home and at work
All teachers have an ICT goal as part of their appraisal and we monitor their progress and as part of this we have a competency checklist that teachers use to try and improve their Activboard skills. We expect to see that they are making progress and learning new skills.
- Karen Spencer (View all users posts) 22 Feb 2012 8:59pm ()
Thanks for sharing how you are trying to capture that evidence, St Mary's - it's clear that you have deliberate processes in place that other schools could also learn from. Deliberate actions of noticing and recording, planning and reviewing are all part of developing an inquiry approach.
Jeanette - kia ora, thanks for your comments, too - you also touch on the importance of that inquiry approach. Your comment about motivation is really important - this is true for both kaiako/teachers and ākonga/students. The use of technology needs to be part of a meaningful experience for all of us, authentic and relevant. And of course, this is true of effective teaching and learning, with or without technology:-) Have either of you used the e-Learning Planning Framework?
How else are people capturing or 'noticing' the impact of technology in their schools?
- Jeanette Hinerangi Eruera Murphy (View all users posts) 21 Feb 2012 8:33pm ()
How do you know if ICT is making a difference - and does it matter?
Over the past four or so years I have worked alongside a number of Y6-Y8 students as e-Learners and this has been one of those reflective questions we'd constantly ask ourselves as individuals and too as a roopu. The question would often be challenged by e-Learners and elaborated on to include... a difference... in what respect, to whom and for what purpose?
Many of the differences I'd observe as a Kaiako/Facilitator in an e-learning classroom did not come from the teaching, use of ICT or product. They came during the 'on-going' inquiry process as student ownership....the WITFM, what's in this for me statements... the bouts of creative and at times collaborative thinking.... 'aha and reflective moments'...and often lead on to review or the next steps of learning that students themselves decided upon.
I have since come to understand for myself that ICT is the merely the tool (cause). It is the user's intrinsic motivation to 'apply' the tool that will in due course initiate change and make or create a positive, negative or profound difference (ripple effect). Teens and cellphones are a classic example.
- Kathy Lauridsen (View all users posts) 01 Mar 2012 7:53pm ()
Kia ora
This is my first time with VLN and I'm overwhelmed by the knowledge here and ideas. Now I have changed schools this year going from 1:2 digital to 3 classroom computers. Sooooo I have started slowly with our class blog and I am amazed how engaged they are. At present we write the blogs together and design a photopeach or place photos on the post. They are so keen that after we have published I have parents in the classroom being dragged by their children to look at what they have been doing. Yay for our beautiful learners.
- Jane Armstrong Bos (View all users posts) 02 Mar 2012 8:33am ()
Kia ora Kathy, Its great to hear what you are doing in your classroom with blogs and how enthusiastic your students are about sharing their work with their whanau. There are some new school stories in the Enabling e-Learning site with teachers sharing how they are using blogs in their schools that might be interesting to you. These are in the Teaching Tools section and also on the e-Portfolios page. Blogs are a really valuable way of both engaging students, connecting with the community, creating an authentic audience for students writing as well as providing a tool for students to record their thinking, share their understandings and progress in their own learning journey.
Mike Wilson (View all users posts) 02 Mar 2012 3:12pm ()This is an excellent topic and one our school is discussing at the moment. Surveys do not tell you a lot about the influence on achievement (more how it motivates) and comparing classes or cohorts also has its problems both statistically and in the way ict is playing a role towards their achievement.
Can we assess this benefit? This depends on many factors and so many of these are NOT due to the technology.
1. It depends on the staffs willingness to think about how the tool is effecting their students learning (without this you are using a tool to entertain). Many teachers believe that a student who is quietly engaged in the "world" of their electronic device then they are learning. This is often not the case of course.
2. This links into: It involves more that providing the technology. Buy a class set of Ipads, cool! Buy some cows, Cool! Set up a BYOD initiative! Cool. This is still just like letting your class go free within a library with some questions. Benefits of having these tools come from the process leading up to their use and the structures of use put into place. Just because the student is swiping away at an Ipad does not mean that they are more engaged or actually learning anything. Can you do a summative test on the benefit of a ipad, yes. Would it be more effective that using a traditional style of learning? Maybe not.
3. That the assessment model relies on information and how this is processed by the students. We are past making work "prettier" and the effective teachers are now seeing that the real educational benefit is information and related to collaboration. This means providing something like an inquiry process that involves technology at some point in the early stages. Copy and paste of course does not lead to any beneficial outcomes with the students. NCEA has some limited standards that ICT could be of benefit but these are few and far between.
4. The use of LMS. I do believe these are powerful in terms of a teacher having the ability to guide the students learning outside the school grounds. Once again this relies on point 1. How do you assess the benefit? Without only allowing access to half a class or school it is very difficult. Through student voice you can find out that 84% find it of benefit but does this improve those students’ results in exams - hard to conclude.
A common discussion point that comes up is that ICT tools are just that, a tool and they should be thought of as such. Give a poor teacher an amazing tool and it will still be a poor lesson and this is correct. Introducing any tool to a teacher that does not know how to effectively use that tool will not improve the students learning (unless the student takes ownership and has the prior knowledge to use that tool).
So what does this mean? Are all these new initiatives costing schools hundreds of thousands of dollars’ worth it? Probably, but we have to focus again on the pedagogy of the teachers and put less emphasis on student voice (what, less focus on student voice!). Does a student know how to use a device? Yes. Does a student know how to effectively use a device to learn and achieve better results? Probably not?
- Karen Spencer (View all users posts) 06 Mar 2012 11:20am ()
Thank you for your thought-provoking post, Mike. You touch on much of the commonsense aspects of using ICT in education, making the excellent point that it's not about the tech so much as it's about effective use of pedagogy within curriculum content areas :-).
You're right, that it's tricky (and possibly a cul-de-sac) to try and assess the impact of ICT on its own. How, then, to describe the impact that e-learning (or indeed any other aspect of pedagogy) might be having on students' learning within a lesson, or series of lessons?
The Teaching and Learning dimension, in the e-Learning Planning Framework, states, at the Extending phase, that "learning activities integrate technologies appropriately to support authentic, higher order, collaborative learning." How might a teacher, or school, be reassured that this was happening? What information might they draw on?
- Nick Ford (View all users posts) 08 Mar 2012 3:31pm ()
As professional educators we do need to assess the effectiveness of any and all teaching & learning strategies we employ in our practice. In the context of the use of ICT's in teaching and learning the Ministry of Education has just released an annotated bibliography (Learners' Participation, Retention and Success in e-learning ) which provides an overview of the literature on e-learning in the tertiary sector in the context of learner participation, retention and success.
For me reading the key findings of this report (see below) demonstrates relevance to all education sectors (early childhood, primary, secondary, tertiary and vocational etc). It could be an interesting exercise to map the activities contained in this report to the e-Learning Planning Framework and also compare and contrast these with e-learning activities occurring in your school.
The key finding of this report are:
- Teaching practices and pedagogies, institutional support and student characteristics and attitudes are all critical in tertiary learners’ retention and success in e-learning. Of particular importance are appropriate teacher-student interactions. Courses need to be designed to incorporate e-learning’s strengths. This includes selecting appropriate technology and ensuring that e-learning is linked to assessments and authentic learning experiences.
- For best results, institutions need to provide ‘user-friendly’ systems, processes and appropriate pastoral and technical support. Students also need motivation, self-direction and independence as well as having prior experience in e-learning. Students with positive attitudes towards technology tend to do better in e-learning than learners with negative attitudes towards technology.
- e-Learning provides additional flexibility to traditional delivery by allowing students to study at a time, place and pace of their choosing. E-learning can also reduce isolation by better connecting learners to their peers, teachers and institutions – especially for learners studying part-time or through distance education.
- e-Learning can provide greater access to a wider range of resources and experts than is available through traditional delivery. The fact that all students can equally access these experts and resources is of benefit to non-mainstream learner groups e.g. disabled students.
- The evidence supporting younger learners being more successful than their older peers in e-learning is inconclusive. While some studies support the assertion that younger learners are more effective in e-learning, others do not.
- Moana Timoko (View all users posts) 08 Mar 2012 5:41pm ()
Oooo interesting reading!!! Once I've taken it all in I may add an extra comment...I may not....it's all there really!!! Thanks Nick
- Mary-Anne Murphy (View all users posts) 08 Mar 2012 5:54pm ()
Thanks for this post Nick.
If we were to change the word "student" to "learner", what implications would this have?
ie: If we were to place ourselves as "learners" what implications might that have for the way we work within our current positions? What implications might it have for the way we work with others and the way others work with us?
- Nick Ford (View all users posts) 09 Mar 2012 2:09pm ()
Nice question Mary-Anne :-)
It is interesting what Sharon Friesen has to say on this subject in her EdTalk 'New Roles for Teachers'.
- Annemarie Hyde (View all users posts) 09 Mar 2012 2:54pm ()
Thanks Nick - I found Sharon's thoughts really thought provoking. The fact that we might stand aside in order to be facilitators but need to be there, right in the thick of things, is something to take on board. We need to be the "weaver of the threads" who is able to correct the warped bits to keep the tapestry growing!
ICTs will only make a difference with direction. Every tapestry had a Master weaver who was hands on. What a wonderful metaphor!
- Tessa Gray (View all users posts) 28 Mar 2012 11:36am ()
I have been enjoying this thread. Thank you everybody!
It is a long-held view that if schools are going to spend so much money on e-learning tools, they might also want to evaluable the effectiveness of these resources. Becta had an indicator under one of their assessment of resources criteria which read, if a school was successful,
There is systematic widespread, monitoring and evaluation of the use of ICT resources. The outcomes underpin the on-going strategy and procurement and used to maximize access to learning opportunities.
Becta also had some guiding questions on the assessment of ICT capability, but unlike New Zealand, this is treated as a separate curriculum focus area. So what about the context of e-learning in New Zealand?
Like Annemarie says, do we undertake assessment of ICTs (making a difference) to guage the ‘bang for bucks’? Because as Mike mentions, this can be fraught with difficulty (too many other related variables). Or do we undertake to evaluate the benefits (or otherwise) to indicate impact on intended learning goals or outcome? Or as Karen implies, is it more about evaluating the use of e-learning tools to build on the Key Competencies – develop problem solving, reflective, social knowledge-builders and creators?
As already shared, collecting anecdotal responses (student voice), archiving experiences in blogs and reflective journals can provide a rich source of evidence showing the impact of e-learning. Like Jeanette has mentioned, knowing when and why to use ICTs within a inquiry cycle of learning enables students to:
“…use technologies appropriately in a continuous cycle to support the way they set their learning goals, manage life-long portfolios and work towards becoming self-regulated learners.” E-Learning Planning Framework (Teaching and Learning dimension: Assessment: Empowering)
Maybe a student survey could include references to e-learning tools (type, frequency, use) within a lesson sequence/s as well as evidence of effective teaching and learning practices. I’ve had a little go HERE and I’d love for people to add, tweak, grow a useful survey of some kind. This is a first crude draft, so feed-in/feedback please!
Other student surveys
https://www.surveymonkey.com/s/KESStudentsICT
http://www.horshamps.vic.edu.au/ict_survey.htm
Image sourced from http://www.freedigitalphotos.net/
Join this group to contribute to discussions.
