Log in

Evaluating e-learning implementation

  • Public
Started by Keir Whipp 04 Jun 2015 12:41pm () Replies (14)

I'd like to ask for any references (academic articles especially) on the evaluation of e-learning implementations in secondary schools.
Any models or theories on how to evaluate an e-learning innovation  much appreciated.

Kind regards



  • Tessa Gray (View all users posts) 04 Jun 2015 2:41pm ()

    Hi Keir, try these e-learning readings and research from Enabling e-Learning in TKI. http://elearning.tki.org.nz/Research-and-readings which has some references to e-learning and improved outcomes for students IE >>>

    IE: Digital readers at age 15: An overview of the PISA 2009 Electronic Reading Assessment
    The Programme for International Student Assessment (PISA) is an international study that assesses how well countries are preparing their 15-year-old students to meet real-life opportunities and challenges. In 2009, the main PISA study offered countries the option of assessing some of the participating students in reading using a computer based assessment — the Electronic Reading Assessment (ERA). The average (mean) digital reading literacy score for New Zealand 15-year-olds was significantly higher than the print reading literacy score. Compared with other high-performing countries or economies participating in the ERA, New Zealand had a relatively large proportion of students who demonstrated very advanced digital reading skills as well as a relatively high proportion of students who demonstrated poor skills.
    Author: Sarah Kirkham
    Published: June 2011


    PISA 2009 results: Students on line: Digital technologies and performance (volume VI)
    The Programme for International Student Assessment (PISA) is an international study. Students from 19 countries were tested on their ability to navigate and evaluate information on line. Key curriculum areas focused on in the report are reading, mathematics, and science. Findings show that, in every participating country, girls outperform boys in reading skills, boys outperform girls in mathematics, and in science gender differences tend to be small. Countries of similar prosperity can produce very different educational results. 
    Author: published on the responsibility of the secretary-general of the OECD
    Published: June 2011 

    as well as

    There is, a growing body of evidence to support the view that digital technologies have the potential to improve student outcomes and to enrich, if not transform, the learning experience of children. This report makes links to research demonstrating the increasing evidence of the positive impact of digital technologies in areas such as motivation, engagement, efficacy, and interaction.
    Author: Ministry of Education
    Published: July 2012

    You might also be interested in similar threaded conversation (but not necessarily specific to secondary contexts) >>>

    How do you know if ICT is making a difference - and does it matter ... with references to The Impact of Digital Technology on Learning" via DERN (the Digital Education Research Network).

    Hope that helps smiley

  • Tessa Gray (View all users posts) 05 Jun 2015 9:43am ()

    Hamish, you're on to a good debate here. Not surprising other community members have also discussed this here, How do you know if ICT is making a difference - and does it matter ... 

    The late, great Vince Ham used to say 'it's very hard to track or monitor the effectiveness of ICTs/e-tools when there are so many variables - #1 being effective teaching pedagogy.' Isolating the effectiveness of e-learning would probably mean analysing the difference between control groups on a single task. IE: Mindmappng - some students can do this using paper, coloured pencils etc, others can use mindmapping tools like https://bubbl.us/. Could we then collect data to show one tool was more powerful than the other and that the learning outcomes were better/higher/more advanced? What were we looking for anyway? If it was collaboration, then the web based collaborative tool would have the advantage surely?

    In terms of tracking specific learning outcomes it can get tricky. Some people use rubrics or matrixes to track ICT capabilities. Jury's still out for me on that, which is why I created this page: Alternatives to ICT skills checklists.

    What do you reckon? 



  • Tessa Gray (View all users posts) 05 Jun 2015 10:15am ()

    Hi Keir, while you're looking for more general view and look at models or theories on how to evaluate the implementation of innovation in a secondary school, here's a few more links that might help:

    'Leadership in digital technology: The challenge of decision making' (A Weijermars, 2012)

    BLURB: Masters Thesis – Master of Educational Leadership and Management, Unitec Institute of Technology

    This research investigated the perceptions of decision-making in the use of digital technologies by three secondary schools, leaders and teachers. In the use of digital technology teachers in these schools understood it to mean how data was captured, stored, manipulated, produced and distributed digitally as mass media. In the context of education digital technological devices offered a host of opportunities in the teachers’ repertoire of instructional tools. With so many innovative web-based ICT resources and DT devices being used in secondary schools, subject specific departments were literally free to pick and choose from a range of digital resources they saw fit for class instruction that enabled students to be captivated and engaged in their learning. For leaders charged with sanctioning budget requests for both ICT and digital technology resources a level of expertise, knowledge and jurisdiction in how these resources supported classroom instruction needed to be examined.

    The following may be applicable to secondary too:

    Effective Digital Learning Pedagogies & Environments

    BLURB: This is a site that provides summaries and resources related to Manaiakalani research and evaluation conducted by the Woolf Fisher Research Centre in partnership with the Manaiakalani schools and the Manaiakalani Education Trust . Useful to LCN Leaders, Teachers and Facilitators.

    Manaiakalani project is know for increasing students' access to digital technologies through BYOD (Bring/Buy your own device).

    Disrupting the boundaries of teaching and learning: How digital devices became a resource for transformative change in a time of crisis (Full research report) Te Toi Tupu, CORE Education, Greater Christchurch Schools Network, MoE, Louise Taylor, Tara Fagan, Merryn Dunmill

    BLURB: As a result of the Christchurch earthquakes, schools were facing closures, mergers and infrastructural changes.

    The report outlines how the project came about, who was involved, how the devices were integrated into schools, and the learning and change that occurred for teachers, students and the community as a result.

    Educational leaders have some credible research projects that relate directly to leadership and ICT. These can be found in this tag search.

    There's more in this original post here: Expanding our digital tools.

  • Tessa Gray (View all users posts) 17 May 2017 9:26am ()

    This is an older thread that's still relevant to us today where educators are asking, (and so they should) Where's the research to support digital technologies can have a positive impact on learning? 

    New research on Digital Technology in Schools released May 16th, 2017

    Eight-in-ten principals report that digital technologies are having a positive impact on student achievement, according to the 2016/17 ‘Digital Technologies in Schools’ survey, prepared by Research New Zealand for the 20/20 Trust. The survey was shorter and simpler than the 2014 survey and it was again open to all New Zealand schools. A total of 464 schools completed the survey.

    Summary: Digital technologies are having a positive impact on student achievement (4MB, pdf) – includes the 2017 report card, enabling schools to compare their position against the national average.

    Full report: Digital Technologies in Schools 2016-17 (3MB, pdf)

    Really exciting to read,

    Positively, more than three-quarters of principals reported that teachers in their school view integration of digital technologies in their teaching as being relevant, and more than half of principals felt that their school’s teachers have a good knowledge about what to do to achieve good integration of digital technologies in the teaching and learning process.

    This aligns to the research that implementing technologies effectively in the classroom really comes down to teacher beliefs


    Does this research match up with what's happening in your school/kura?

Join this group to contribute to discussions.