Evaluation Planning

Motivation not Reaction

Kirkpatrick wanted to measure reactions (Kirkpatrick in Praslova, 2010). To put another way, how well the learners liked a particular learning process (Clark, 2012). However, the research in this area shows a weak link between learners reported reactions and job performance after training (Boehle in Clark, 2012). It is possible to have an enjoyable learning experience without meeting the learning objectives and equally possible to struggle through but meet objectives.

It is suggested instead to look at motivation (Clark, 2012) which can comprise other learner reportable criteria such as how important and doable the learning is perceived (Markus in Clark, 2012) and how much learners believe they have learned (Alliger et al. in Praslova, 2010).

Performance not Behavior

Kirkpatrick wanted to measure behaviour, What changes in job performance resulted from the learning process? (Clark, 2012), but Gilbert suggests that instead we should be looking at performance. Performance not only includes desired behaviours but also desired consequences (Gilbert in Clark, 2012).

Focusing too strongly on just behaviours can lead to blind robotic repetition, where as it is more desirable to root those behaviours in the context of their outcomes. A common example of this in the service industry is the training of staff to ask customers questions such as "Is everything OK with your meal?" but then giving a negative reaction to the customer if they do complain about something. If we were to just look at behaviour when evaluating the training (eg. 96% of staff remember to ask customers if their meal is OK) then we will miss the problematic consequences in our evaluation (eg. 80% of customers who responded with a complaint were unsatisfied with how it was dealt with).

The other problem with simply measuring behaviour in the classroom context is that you do not then know how learners will perform in the real world and you may end up teaching to the test instead preparing them for life tests (Halpern and Hakel in Praslova, 2010, p221). By equally focusing on performance consequences this danger is somewhat mitigated as the teacher is forced to find ways of evaluating realistic outcomes.

Flipped model

Perhaps the biggest change suggested by Clark is to flip the order of the steps and then use them as a planning model (Clark, 2012). The planning process now has eight stages that tightly link the learning activity design and the design of the evaluation.

1. Goals (Planning)
   1. Desired result: What is our organisational objective?
   2. Desired performance: What must the learner be able to do to achieve our desired result?
   3. Desired learning: What new knowledge and skills does the learner need for the desired performance?
   4. Desired motivation: What must the learner perceive in order to learn the desired learning?
2. Evaluation
   1. Actual motivation: Did they have the desired motivation?
   2. Actual learning: Did they learn the desired learning?
   3. Actual performance: Did they transfer their learning into practice for the desired performance?
   4. Actual result: Does the organisation see the desired result?

The activity is thus planned to be measurable, and any problems discovered in evaluation can be linked to a specific stage in the planning (Chyung in Clark, 2012). The stages can also be divided into internal to the learning activity (motivation and learning) and external (performance and result). Since the external stages are influenced by outside factors this can be taken into account when evaluating the learning activity on its own (Alliger et al. in Praslova, 2010) and thus help avoid blaming bad business management on good learning.

Actual motivation

What

I will evaluate how easy learners found the activity, how beneficial they felt the outcomes to themselves and how beneficial they felt the outcomes to their students.

Why

The learners must be motivated to meet the learning outcomes. "If the goal or task is judged as important and doable, then the learner is normally motivated to engage in it." (Markus in Clark, 2012)

Who

• Library colleagues
• Customers
  • academics
  • postgraduates
• Postgraduate Award Technology Enhanced Learning students

How

Self completion questionnaires. One built into activity and one online.

When

Immediately after the learning activity and 2 months later.

Actual learning

What

I will evaluate if learners are able to operate SMART Boards and if they are able to incorporate engagement, formative assessment and learning objectives into their use.

Why

Learns will need to have these skills and knowledge in order to perform well.

Who

• Library colleagues
• Customers
  • academics
  • postgraduates
• Postgraduate Award Technology Enhanced Learning students

How

Formative and summative assessment as pre and post tests provide the most direct measure of learning (Arthur in Praslova, 2010, p220).

When

Before and after the learning activity.

Actual performance

What

I will evaluate if learners design interactive whiteboards into their lectures and if they are used for increasing engagement, formative assessment and meeting learning objectives.

Why

If the desired result is to be achieved then learners will have to put their skills and knowledge into effective action.

Who

• Customers
  • academics
  • postgraduates
• Postgraduate Award Technology Enhanced Learning students

How

Self completion questionnaire.

When

1 term after learning activity.

Actual result

What

I will evaluate if there is an increase in student engagement, increased formative assessment and improved student outcome.

Why

If the desired result is not met than the purpose of the learning activity is undermined.

Who

• Customers
  • academics
  • postgraduates
• Postgraduate Award Technology Enhanced Learning students

How

Self completion questionnaire. This will be somewhat subjective as it will be based on the learners perceptions but will allow data to be collected practically.

When

1 term after learning activity.