How Can I Use VLEs to Improve My Assessment of My Year 8 Class?

Originally written in 2010, with minor changes for the web.

Overview

This project uses the methodology of Action Research to look at the problem of how I can use Virtual Learning Environments (VLEs) to improve my assessment of my year 8 class. The project looks at:

  1. Introduction to the assignment

  2. Background to the problem

  3. The literature on VLEs

    1. Need for CAA

    2. Uses of CAA

    3. Advantages of CAA

    4. Problems with CAA

    5. Conclusions from the Literature

  4. The school experience of VLEs

    1. Current usage

    2. A teacher's experience

    3. My problem

  5. How do the literature and school experience compare?

  6. My Plan

I concluded that my plan should use the VLE to distribute materials to provide scaffolding for the pupils' work and then use the VLE to collect the work back in. This would provide many advantages while slowly evolving my practice to minimise the difficulty in implementing.

A computer lab filled with students sitting at desks with CRT monitors, focused on their screens.

Introduction to the assignment

This project is based on the methodology of Action Research. Action Research is "a fancy way of saying let's study what's happening at our school and decide how to make it a better place" (Calhoun in Hammond, 2009). More specifically (Hammond, 2009), it is an approach characterised by:

  • "research by teachers that addresses a problem that they have identified as being of concern to them."

  • "a process that involves problem identification, design, implementation evaluation and identification of new problems."

  • "typically involves collaboration between colleagues."

A circular diagram linking the following words iteratively: "Recognition of problem", "Analysis", "Design solutions", "Select Solution", "Implement", "Evaluate".

This project will only cover the first four sections of the action research cycle shown above.

Background to the problem

The problem I have identified in schools is the under-utilisation of Virtual Learning Environments (VLEs) for the use of assessment.

VLEs, sometimes called Course Delivery Systems, have varying features, but Ryan, Scott, Freeman & Patel (2000, pp141-143) summarises them with the following:

Student features Tutor features Course designer features
Course content Progress tracking A standard interface
Course conferencing system or bulletin board Timed & automatically graded quizzes Customisation
Synchronous communication or chat Student management Site management feature
E-mail
Notebook
White board
Objective tests
Student presentation areas
Grading information
Calendars

Although "every school should have integrated learning and management systems (a comprehensive suite of learning platform technologies)" (BECTA, 2006), many of their features seem to be ignored. During my pre-course observations in a state secondary school I noticed that the VLE was not mentioned in any lessons. In the cases where I have heard of VLEs being used, it was primarily for distributing resource files in lessons.

I think VLEs, when understood and fully utilised, can not only improve learning but also improve the experience of teaching. As Ryan, Scott, Freeman & Patel (2000) note, assessment is one of the most resource intensive aspects of teaching. Therefor, this project will look at how I can use VLEs to improve my assessment of my year 8 class — to improve learning and the teaching experience.

The literature on VLEs

The literature on assessment using VLEs is relatively scarce, with most authors generalising about VLEs in the abstract. Assessment is one of the functions often mentioned briefly, and it is possible to piece these fragments together to form a more substantial view. The literature often refers to using VLEs in assessment as "Computer Aided Assessment" (CAA). The fragments written about CAA can be grouped to identify the need for, uses of, advantages and problems with, and examples of CAA. I will cover these each in turn.

Need for CAA

Most of the literature seems to take CAA as an inevitable consequence of the digital age, failing to account for the necessity or justify the expense and re-training. Many advantages are raised, but there is little comparative writing to measure the worth of CAA over and above current methods.

An attempt by Anderson roots his argument in the benefits of increased formative assessment with the practical problem of increasing teacher workload.

“Strategies that are designed to provide formative feedback with minimal direct impact on teacher workload are most needed.”
— Anderson, 2004, p242

His suggested solution is a combination of VLEs, peer assessment and CAA, which he believes will allow increased assessment with minimal resource increase. As a bonus, Anderson (2004, p243) notes, "Quality online learning projects require students to gain new knowledge processing and management skills, increased social and negotiation skills, and process skills related to filtering and applying knowledge from diverse sources", which are "critical to the emerging information age."

Another, albeit brief, attempt by Ryan, Scott, Freeman and Patel (2000) similarly focuses on teacher workload. They see a need for Communications and Information Technology (CIT) to save time and resources.

“If CIT is to produce real savings in staff time and resources then [CAA] systems will play a key role”
— Ryan, Scott, Freeman and Patel, 2000, p125

As with many authors, CAA is stated as a time-saver without argument.

While I will later look at the advantage of saving time through the use of CAA, I will also look at the problem of CAA increasing workload. Therefore, it is not sufficient to assume that CAA is a time-saver, and the above arguments for the need for CAA are deeply floored.

This project will consider how using CAA can improve my assessment of my year 8 class, and I will have to consider the possibility that the answer is that it cannot.

Uses of CAA

Most of the literature encourages using CAA for formative assessment but shies away from summative assessment. BECTA claims' colleges typically do not use VLEs for summative assessment 'because of the risk of cheating' (BECTA, 2007, p35), but appends that using CAA within the school system may help with this. Ryan, Scott, Freeman and Patel (2000) add that not only a controlled environment but authentication procedures may be required.

I think this is sound advice. Whereas authentication works for credit cards, for example, because the owner does not want other people to use their card, this is not so when a pupil wants someone else to take their test. Any authentication based on something you know or something you have could be voluntarily passed onto the impersonator. This means the only authentication possible is based on something you are - your physical features. This would be expensive to set up remotely and could still be bypassed, with someone authenticating and then swapping places with the impersonator. The only way to cheaply and successfully authenticate a test-taker is for them to be in a controlled environment - such as a school.

Ryan, Scott, Freeman and Patel (2000) also point to the technical limitations of CAA implementation, meaning that sometimes results can be saved for summative assessment while other times the results are discarded. This is a consideration rather than a limitation when using CAA in the classroom, as there are many options for teachers of both varieties to choose from.

Gillespie, Boulton, Hramiak and Williamson (2007) suggest that schools initially only use CAA for formative assessment while staff and pupils are unfamiliar with the processes and interface. With this caveat, I see no reason why a properly implemented system in a controlled environment should not be used for both formative and summative assessments.

Aside from the exam-style summative assessment referred to above, e-portfolios are the CAA equivalent of coursework. Unlike exam-style assessment, there is very little detrimental difference to digitising coursework. The authentication problems in this instance are no different from the current paper-based system.

Advantages of CAA

There are many advantages to CAA over manual systems. I will briefly outline these benefits.

Flexible access

One of the most significant advantages of CAA is the flexibility of access in taking, marking and feeding back on assessments.

BECTA (2007) highlight the advantages to pupils in taking CAA at a time convenient to them, in a space convenient to them, with resources convenient to them. Elsewhere (BECTA, 2004), they highlight this benefit to absent or excluded pupils (and to absent teachers in providing CAA instructions to cover teachers). I think these seem genuine advantages, but they need to be weighed against the fairness of the assessment. Is a pupil working on a slow computer, when their parents have stopped using it at 10 pm, in a cramped room with a TV on in the background going to be at a disadvantage to a pupil able to work in more conducive conditions? This would be another argument for restricting CAA to the school site. Another advantage Gillespie, Boulton, Hramiak and Williamson (2007) mention is the flexibility of merging classwork and homework in CAA. With e-portfolios, there can be a fluid divide in when and where coursework can be completed. They also highlight the benefits of flexibility in marking and monitoring progress using e-portfolios, allowing the teacher to check coursework at any time from anywhere.

Ryan, Scott, Freeman and Patel (2000, p127) highlight the many benefits of flexible access to the results of assessments. Results "can be stored centrally and accessed by interested parties, such as staff and students" [BECTA (2004) add parents to this list]. Electronic storage of results can also allow flexible access by other software. This is a significant advantage as I have often experienced the mundane task of typing assessment data in by hand.

Reduced workload for teachers and pupils

If CAA is to be practical in the real world, then it must, at a minimum, only increase teacher workload in line with its advantages — preferably reducing teacher workload. In a real-life case study, BECTA (2004) noted a reduction in manual marking teachers had to do. As well as a reduction in marking, 'computerised correction and collection facilitation utilities can automatically keep records of results for individual students' (Ryan, Scott, Freeman and Patel, 2000, p127). So, teachers are saving time by reducing marking and then feeding that data into an automated pupil tracking and grading system. I think this could be one of the most potent ways CAA can save teachers time.

Gillespie, Boulton, Hramiak and Williamson (2007, p59) pick up on a possible future advantage of using e-portfolios for CAA. While "the current structure and nature of secondary education and the fact that is significantly influenced by the examinations system mean that opportunities for cross-curricular work are likely to be limited", they see a future where this may not be the case. I think many advantages of cross-curricular work can be introduced today, including reducing pupils' workload. If a piece of maths coursework can be contextualised within a piece of geography coursework and presented in a format suitable for ICT coursework, a pupil can gain accreditation in three subjects with a reduced workload. This will give them more time to improve the work they produce.

Increased differentiation

A considerable benefit of CAA is the ability to generate different assessment material for different pupils - possibly even automatically from previously collected CAA results. BECTA (2007, p44) unimaginatively suggests that teachers can "set assessments and target at individual students". Ryan, Scott, Freeman and Patel (2000) take this a lot further, suggesting that CAA in the form of a quiz can present every pupil with a unique series of questions. As pupils progress, the next question's difficulty level is selected based on their answer to the current question. Thus, every pupil has a quiz specifically differentiated for their level. I think this is close to the holy grail of CAA - decreasing teacher workload while simultaneously increasing the quality of pupils' learning and assessment.

Reduced resource use

One of my main concerns with manual assessment is the amount of paper wasted. Paper is often used to give pupils the task, who then use paper worksheets to complete the tasks and finally print their work off on paper. Arooj concludes:

“I liked [CAA] as I did not need to print out and use lots of paper”
— Arooj in Hammond, 2010

Ryan, Scott, Freeman and Patel (2000, p127) agree — paper can be saved by assessment being "created, collected, corrected, commented on, and returned entirely electronically".

Increased motivation

City of Sunderland College (in BECTA, 2007, p16) found that "the motivation of instant feedback for online assessments was a significant confidence-builder". Improved motivation and engagement through the use of ICT were also noted by Watts & Lloyd (in BECTA, 2004), Wilson & Whitelock (in BECTA, 2004) and the DfES [now Department for Education] (in Gillespie, Boulton, Hramiak and Williamson, 2007).

This aligns with my school experience with the frequent question, "are we going on the computers today?"

ICT skills are tested and improved

Watts and Lloyd (in BECTA, 2004) suggest that CAA can help pupils in their ICT skills. However, if the assessment is improving their ICT skills, you have to question if this reduces their grade for the task they completed using ICT.

Problems with CAA

There are many advantages to CAA over manual systems, but we must not discount the new problems that come with it. I will briefly outline these possible problems.

Cheating

As mentioned above, the main concern often cited is cheating. I have previously argued that CAA needs to be taken in a controlled environment if impersonation is a concern, but plagiarism is equally problematic in CAA.

“Teachers who are engaging in teaching and learning with pupils making multimedia resources to contribute to e-portfolios must have systems in place to make sure that work submitted by pupils as their own is that”
— Gillespie, Boulton, Hramiak and Williamson, 2007, p55

This applies to e-portfolios and any digitally submitted assessment, as the ease of copying is much greater than with hand-produced work. Gillespie, Boulton, Hramiak and Williamson go on to say this is as much a problem with inadvertent plagiarism as intentional. One of the solutions, then, is to make pupils aware of what plagiarism is and how to use other people's materials in their work appropriately. However, there is a growing industry in providing works to be plagiarised and detecting plagiarism in pupils' work.

Elsewhere Gillespie, Boulton, Hramiak and Williamson (2007) highlight the advantage of CAA automatically shuffling the order of questions to help reduce opportunities for cheating.

This is the biggest hurdle in adopting CAA, as the opportunity for cheating, especially in assessments taking place remotely or over time (such as in e-portfolios), is significantly increased. In my experience, secondary pupils are often not good at disguising plagiarism - a simple Google search of the first line often brings up the original source - but no pupil should get away with this. Over time it will be a cat and mouse game with increasingly sophisticated plagiarism and detection methods.

Difficulties using the technology

Especially when CAA has just been introduced, technical difficulties can impede assessment. This is one of the reasons many shy away from using CAA for summative assessment, for fear of harming pupils' grades.

When the University of Calgary trialled CAA with computer science undergraduates, pupils found disadvantages in "difficulty in contacting an instructor during the exam, despite contact information for the instructor being made available; inability to jot calculations in the margin (unlike the paper-based exam); confusion over time elapsing during the exam and uncertainty about how to set up the workstation; problems with home internet access" (Jacobson & Kremer, in BECTA, 2007, p23). If these problems were found with older, subject speciality pupils, I think this raises real concerns for secondary pupils.

When trialling CAA with secondary pupils, Arooj found he "had to teach the pupils had [sic] to upload as it was not very intuitive. The software was quite clunky" (Arooj, in Hammond, 2010).

In my own experience, how easily CAA can be adopted can vary hugely from class to class. However, I think the lesson is to introduce one aspect at a time and use CAA for formative assessment initially not to jeopardise summative assessment results being lowered or lost altogether.

What can be tested

One of the reasons the UK is cautious about adopting CAA is the question,

“Does CAA offer a sufficient range of assessment strategies or does it encourage the testing of superficial learning and knowledge acquisition?”
— Ryan, Scott, Freeman and Patel, 2000, p126

Computers are very efficient at marking fixed response assessments, such as multiple choice questions, but 'it is not possible to demonstrate the thought processes that lie behind an answer, and there is no opportunity for partial marks' (Jacobson & Kremer, in BECTA, 2007, p23). Software does exist to help mark essays, but computers struggle greatly to do this without primarily human decision-making.

I think that the main implementable gains in CAA are not in computer marking, although this has its place, but in computer assistance in marking: in submitting work, displaying it for marking by a teacher, providing feedback mechanisms and recording marks awarded. This increases the range of skills that can be tested instead of decreasing them.

Increased workload

As discussed above, the argument for CAA rests heavily on increasing assessment while maintaining or even reducing the workload on teachers. Anderson (2004) suggests that sharing CAA tools and material can go a long way to reducing this workload - something I have seen with the development of Interactive Whiteboards (IWBs). However, materials will often need adapting, as paper-based versions are, to match a class or course curriculum. Ryan, Scott, Freeman and Patel note:

“Construction of good objective tests requires skill and practice and so is initially time-consuming.”
— Ryan, Scott, Freeman and Patel, 2000, p126

I think it is yet to be seen if, in practice, CAA will increase or decrease teacher workload. It is my experience that in the long term, the workload is reduced compared to paper-based assessment.

Privacy/copyright

Anderson (2004) seems to be the only researcher to highlight the problem of privacy and copyright for work produced by pupils in CAA. While many worry about pupils plagiarising work, attention needs to be paid to pupils' work being plagiarised.

Both of these concerns arise from the digital format of pupils' submissions and the ease with which they can, legitimately, be transferred. For example, a teacher may email a pupil's work to their parents or another teacher for marking or to assist in them planning their scheme of work. However, as the government often finds with digital media, information has a habit of getting out.

For example, a teacher publishes their class' work on a website to provide feedback. They password-protect the homepage to restrict access to only pupils but fail to password-protect the files themselves. A pupil then puts a link to their work on their website. Google then follows the link to the class website and indexes all of the class' work. Their work is now publicly available through a search engine.

The example above may sound unlikely, but as more and more schools use CAA, there are more and more opportunities for the information to become publicly available in many different ways.

Because of the risks of pupils' work becoming available, Anderson (2004) suggests attention must be paid to the privacy and copyright of those pupils, removing identifying marks where privacy is needed and adding attribution information where copyright should be enforced.

Aside from Anderson, no one else seems to be writing about these issues, and I fear this problem will arise in the future, creating a more significant workload for teachers.

Distractions

Gillespie, Boulton, Hramiak and Williamson warn that:

“Pupils can get very distracted by the range of tools which many multimedia packages offer.”
— Gillespie, Boulton, Hramiak and Williamson, 2007, p52)

This is a problem in ICT classes in general. Since I am looking at a year 8 ICT class, this is not an added problem of CAA, but for teachers who are not ICT specialists, this may be a concern. Clear expectations and enforcement are needed.

Conclusions from the Literature

There are many examples given in the literature of CAA uses. This is partly due to the multifaceted nature of VLEs, each instance of a VLE being a combination of functions often bespoke. CAA examples are, therefore, a brainstorm of possible combinations - and teachers need to look at what combinations are available in their particular system - but there are several recurring examples of CAA to consider for my year 8 class:

Gillespie, Boulton, Hramiak and Williamson (2007) suggest using VLEs for feedback on assignments not submitted electronically, simple text submission and feedback (e.g. Essays), electronically marked quizzes, and e-portfolios.

Ryan, Scott, Freeman and Patel (2000) suggest assisted annotation of work, using a Bulletin Board System (BBS) for peer assessment, e-portfolios, electronically marked quizzes and tracking pupil ability.

As mentioned, computers do a much better job marking restricted response quizzes than any other form of grading, but these quizzes struggle to test higher-order skills. Using a VLE to assist peer marking can significantly reduce a teacher's workload and test higher order skills but is restricted by the higher order skills of the marker.

I think that, from the literature, the most significant gains can be made through the combined use of e-portfolios and quizzes to test low and high-order skills. It would appear this combination would allow flexible access, increase inclusion, increase differentiation, improve presentation, reduce resource use, provide quicker or instant feedback, increase motivation and test and improve ICT skills.

This combination, however, may provide problems with cheating, technology, privacy/copyright and distractions, but not the issue mentioned above of what can be tested as e-portfolios allow the testing of higher order skills.

The remaining question is whether teacher (and pupil) workload will be increased or decreased. The above theoretical advantages would appear to warrant a slight increase in workload for the gains to be had, but what is the real-life experience of teachers in schools?

The school experience of VLEs

Current usage

In my observations, I have seen e-portfolios in use for the DiDA key-stage 4 course. Pupils created files in many different software applications for their coursework and then linked to these files in their e-portfolios. The e-portfolios were already set up for the pupils and acted as a task list, with instructions on what to create and attach. The e-portfolios, however, were not linked with a VLE.

The main advantages I saw in this were:

  • pupils' work was kept organised

  • at any time, the pupil or the teacher could easily find the correct file for a particular task

  • the teacher could also easily keep track of who had completed which tasks

  • at the end of the project, the teacher didn't have to work out where pupils had saved their files.

I have seen VLEs with many of the features needed for assessment, but I have not seen them used for such.

A teacher's experience

I interviewed a trainee teacher to gather their experiences with VLEs [see appendix]. They were suitably placed for these questions as they had recently visited several schools and would therefore have a broader picture of current VLE usage than a teacher working in only one school.

The teacher saw current VLE usage as underused and a one-way stream from the teacher to the pupils. The VLEs were being used to deliver content to pupils but not to feed assessments or opinions back. In their experience at university, where VLEs have been used for longer, the VLE was used for tracking marks, sharing links and providing discussion forums.

The main advantage the teacher saw in using VLEs was that it opened the teacher up to pupils more, providing greater access. Instead of only communicating with the teacher during class, the pupils can reach out to them via e-mail or discussion forums when they need assistance.

The teacher was asked to consider how they could use VLEs more in their teaching. Regarding assessment, they said they would use the VLE for homework submission and formatively for computer-marked quizzes.

My problem

I have had one particularly insightful experience regarding VLEs and assessment. At the end of a term, I needed to mark two year 8 classes' website designs. The problem was collecting the files from the pupils. They had all saved their websites in their network areas, had often put them in the same folder as many other files, and had often not used sensible naming conventions.

With one class, I tried to collect the work electronically. At the end of the lesson, I got pupils to leave their work open on the computers, and I had planned to go around each computer and copy their work to my network space. I ran out of time and had to go back later, logging into each pupil's account and trying to find which files contained their work. This was very difficult with the problems mentioned above. It took a long time, and many websites didn't load their images correctly.

With the second class, I got them to print off their web pages and the coding. This was much easier to collect. However, it used lots of paper, and the webpages were often displayed differently when printed out than they had on the screen.

This is an example of the problems encountered by teachers that CAA will hopefully solve.

How does the literature and school experience compare?

Many of the theoretical advantages and problems of using VLEs for assessment are un-tested in the classroom due to the slow take-up by teachers. Even ICT departments have often been sidestepping VLEs. I think that a lot of this is due to the initial workload increase in researching, planning and implementing VLE solutions when the old systems seem to get teachers by. I think the many theoretical advantages make this initial outlay worth it. Another problem is that the many different VLEs in use make any investment in one system worthless when transferring schools.

It would seem that any implementation of VLEs for assessment needs to be:

  1. As easy as possible for teachers and pupils to learn.

  2. Where possible, an evolution of current practice to reduce new learning.

  3. Less time-consuming, in the long-term, than existing systems.

  4. VLE independent, relying on commonly available tools.

My Plan

Outline

This project aims to use the features of a VLE to improve my assessment of my year 8 class. From my research, I have found that implementation needs to be:

  1. easy for pupils and myself to learn

  2. an evolution of my current practice or that of the school

  3. less time-consuming than my current system

  4. use features common in most VLEs

As previously mentioned, the year 8 classes I have been working with have been working on creating websites. My method of collecting was either logging into pupils' accounts and searching for the correct files or getting pupils to print off their websites. I think I could improve this method significantly the next time I teach this unit.

I plan to:

  1. Create a website with relative links to where pupils should save their webpages (e.g./Page1.html, /Page2.html, /Page3.html) and links to help-sheets explaining the task, providing 'How To's and explaining how and where to save their webpages.

  2. Distribute the website in a .zip file to pupils via the VLE. Pupils will expand the .zip file somewhere in their network storage area.

  3. The pupils will create their website using the resources provided and save their pages with the specific filenames linked from the site (as in the example above).

  4. Upon completion of the project, pupils will copy the entire folder (including the provided website and their own) into a .zip file.

  5. Pupils will use the VLE to send the .zip files to me.

  6. I can then, at my convenience, open each student's web pages by following the links I initially provided the students.

Will it be easy to learn?

All the stages of my plan use skills and competencies I am familiar with and so will not require any new learning from myself.

Some pupils will be familiar with .zip files and how to save files in specific locations, but this will require new learning from most of my pupils. I hope all the needed actions can be performed step-by-step by following the instructions I will provide on the website. So if my instructions are clear enough, all that will be required is for the pupils to follow them precisely.

Is it an evolution of current practice?

The current practice in the school is for pupils to print off their websites. This is very easy for pupils, although they must be told how to print off the coding for their sites. My plan requires several new mechanisms, but most are a step on from the current methods in use. For example, pupils have previously been sent files by teachers using the VLE. It is only a step further for pupils to un-zip these files. The most significant jump, I think, will be in the use of the website to provide worksheets and instructions. Still, most pupils already have the skills they need from using the Internet.

Will it be less time-consuming?

There are a few things in this plan that will take up more time than the current methods — creating the website and teaching pupils how to receive and return the website. There will be gains, however, in the time it takes me to collect the work and mark it. Initially, as with many things in teaching, there will be more time lost than gained. The primary use of time will be in creating the website. However, once created, the class teacher can use this website each year. Also, most of the resources on the website will be ones I would be using anyway - for example, the help sheets. So, I think my plan, though initially time-consuming, will save time in the long term. I could also modify the same website for use with different units of teaching.

Does it use common VLE features?

I would be using two features of the VLE: delivering files to pupils and receiving them back again. These two features are common to almost every VLE I have come across. Even if I cannot collect the work through the VLE (VLEs are more likely to allow distribution of work rather than collection), I could easily use school e-mail systems.

Conclusion

I have concluded that my plan will require minimum extra learning, be an evolution of current systems, be less time-consuming in the long term, and be VLE independent. The main advantages will be in solving my problems of paper-wastage, website print-outs not being accurate, and the large amount of time taken in collecting work. I will also have the advantage of flexible access for marking.

References

Appendix — Trainee Teacher Interview

  • What is a VLE?
  • What can you tell me about VLE's usage?
  • What are VLEs used for in your experience?
  • Can you describe a positive experience of using a VLE?
  • Can you describe a negative experience of using a VLE?
  • Do you think VLEs add value to teaching?
  • How do you think VLE\'s can be used in your teaching?