Training Evaluation Tips

Evaluation of training within work settings can assist you in learning more about your organization. It is important to understand the purpose of your evaluation before planning it and choosing methods to do it. Some advantages of using evaluations are difficult to directly witness, but when done correctly they can impact organizations in positive ways.

Evaluation feedback assists in improving efficiency and effectiveness of:

• Training content and methods
• Use of organization dollars, personnel, and other resources
• Employee performance
• Organizational productivity

Through evaluation, trainers:

• Recognize the need for improvement in their teaching skills
• Are given suggestions from trainees for improving future training
• Can determine if training matches workplace needs

Email OMSHR for additional information on Training Evaluation Tips

Tip 1: Organizing Your Thinking About Evaluation

One way to think about evaluation is to use a model that provides concrete definitions of what can be learned from them. Kirkpatrick's four-level framework is one example that can be used.  This framework consists of four levels that progress in difficulty from 1 (the easiest to conduct) to 4 (the hardest).  When choosing the levels to include in your assessment, start by identifying the questions your evaluation needs to address.

KIRKPATRICK'S FOUR-LEVEL EVALUATION SCHEME

Examples:

Level 1: One way to assess trainee reactions and attitudes is to use a questionnaire.

• Questions can gather opinions about training methods, the instructor, the environment in which training took place, or other aspects of the training process.
• Pencil-and-paper surveys are convenient to use for trainees and the evaluator.

Level 2: Written or performance tests can assess change in knowledge/skills.

• The best way to measure changes in knowledge or skills is to test trainees before and after training.
• Note that even if a positive change is found, it is possible the trainees gained the new knowledge or skill from a source other than the training.
• If it is not possible to test trainees before training, their performance can be tested after training and they can be asked whether or not their understanding or skill came from the training session.

Level 3: Post-training testing or observations can assess use of skills on the job.

• This level must be completed outside of the classroom after trainees have had an opportunity to use what they have learned.
• This level is more difficult because it requires trainers or some other evaluator to follow-up months after training.

Level 4: Quantifiable measures are often used when assessing organizational impact.

• Some examples of measures that can be used are numbers regarding sales, injuries, or productivity.
• It can be difficult to determine the extent to which other factors besides training (i.e., economics of region) may have contributed to changes in organizational performance.

Key Points to Remember:

Different aspects of training can be evaluated.

• Level 1: Trainees’ perceptions
• Level 2: Knowledge/skills gained
• Level 3: Worksite implementation
• Level 4: Impact on organization

Remember to direct your evaluation to specific questions.

• How did the trainees react to the training?
• Do people seem to have increased their skills?
• Are new skills being used?
• Has the organization been impacted?

Reference:  Kirkpatrick, D. 2001. The four-level evaluation process. Ch. 12 in What Smart Trainers Know: The Secrets of Success from the World’s Foremost Experts, L.L. Ukens, ed. San Francisco: Jossey-Bass/Pfeiffer, pp. 122-132.

Tip 2: Planning with the Training Evaluation Worksheet

The training evaluation worksheet can be used to guide evaluation planning. The worksheet is a quick reference for issues to consider and allows information to be recorded in a clear and concise manner for future reference.

Instructions for using the worksheet

Plan the Evaluation

Question 1: Who will be interested in the results?

• Must be addressed by identifying all who will be interested in the results.
• Examples: trainers, managers, organizations, government agencies

Question 2: What questions will be answered?

• Identify questions that are of particular importance for this specific evaluation. These questions should be ones that you expect to answer upon completion of the evaluation.
• Examples: Have people increased their knowledge and/or skills?
                  Did the information that was learned in training transfer to the workplace?

Question 3: What resources are available for evaluating the training program?

• Determine what resources are currently available for use in the evaluation and/or what kinds of resources you can easily obtain if necessary.
• Examples: money, time, personnel, equipment, materials

Gather the Information

The table provided in this section is useful for specifying the methods you have chosen to use in the evaluation, and when each method will be used. It is important to clearly write the type of method you would like to use and then to circle the choice(s) provided for when the method will be used.

Example:

Tip 3: Ways to Gather Data

One of the most important aspects of conducting an evaluation is choosing the right ways to find information. There are questions you can ask before starting your evaluation process to help you chose the methods that are best for your situation.

Questions:

Who is interested in the evaluation results?

• Trainer?  Manager?  Organization?  Government Agency?

What questions do they want answered?

• Are skills/knowledge gained?  Did transfer occur?  Are there improvements?

What resources are available?

• Financial?  Time?  Personnel?  Equipment?  Materials?

There are many ways to gather information. Each has advantages and disadvantages.

Data Gathering Techniques

Questionnaires

• Advantage: Allows evaluators to quickly gather data from large groups.
• Disadvantage: Not always accurate because of factors such as people not responding honestly or accurately. Some reason for this are an uncomfortable testing environment, the desire to respond in a socially acceptable manner, and misunderstanding the instructions or questions.

Interviews

• Advantage: Allows the evaluator to gather data that is more accurate than that from questionnaires since the interviewer can verbally address any misunderstandings or questions. Can also ask for more in-depth information than is practical in written surveys.
• Disadvantage: Can be time-consuming and expensive if many questions are asked of many trainees. Analysis of in-depth data also takes a lot of time.

Facial expressions/Body language

• Advantage: Allows the evaluator to gather information without being intrusive.
• Disadvantage: One person’s perception of an expression may not be the same as another.

Performance tests

• Advantage: Can measure the skills of a worker in a real or simulated work environment.
• Disadvantage: Can be difficult to simulate a work environment. If the test is conducted in the actual work environment, then it must be scheduled with regard to production concerns.

Written tests

• Advantage: Often standardized and validated before use.  A reliable, valid test is able to consistently measure the same thing every time it is used.  Written tests are usually completed in a classroom setting where large groups can be evaluated at the same time.
• Disadvantage: If there are problems with the testing environment (i.e., the room is too hot, the chairs are uncomfortable), the test-takers may become distracted and not respond accurately. Literacy or language problems can also be an issue.

Workplace observations

• Advantage: Gives clearest data about whether or not training is being used in workplace.
• Disadvantage: Requires evaluation some time after training in the work environment. This may interfere with production.

Team Games

• Advantage: Can be a creative way to engage individuals and keep their attention.
• Disadvantage: Games make it difficult to “measure” or evaluate individual trainees.

Group discussion

• Advantage: Can be a great way to gather information about training or to answer questions by creating an open forum where individuals can interact and talk.
• Disadvantage: Individual differences exist between those that participate in the discussion, and this factor may influence the type of information received.  If some individuals are quieter than others, feel pressure to conform to what others are saying, or are disinterested, they may not share information or report how they really feel about the training.

Analysis of statistics

• Advantage: The use of numbers and statistics is highly regarded in providing and understanding information.
• Disadvantage: The numbers can sometimes be manipulated in such a way that data can be misleading.  Misleading data can lead to incorrect or inaccurate beliefs about the information gathered regarding training.

Conclusion:

If you are clear about the evaluation questions you need to address, you will be able to assess the pros and cons of using various data gathering methods and decide which are best for you.

 

 

Tip 4: Recognizing Mistakes in Survey Questions

One common way to do an evaluation is to use a survey.When using or developing a survey, it is important to be able to recognize and avoid writing bad questions. There are certain characteristics that you can look for to ensure that your questions are written properly.

Common Mistakes in Survey Questions:

1. Double-barreling

Double-barreling occurs when one question deals with two or more issues or ideas, which makes it impossible to decide which issue the response really addresses. It is important to address one issue at a time and to use one question for each issue.

2. Leading

Leading occurs when a question encourages a person to respond in a certain manner (positively or negatively). Survey questions should be written in a manner that allows the person to respond without being influenced.

3. Wrong assumptions

Making a wrong assumption occurs when a question is written based upon the idea that the person answering has prior knowledge that is necessary to respond, when they may not. The best approach to writing answerable questions is to keep them simple. When that isn’t possible you can expand them or create multiple questions when needed to ensure that each person can accurately respond.

4. Missing response options

The response options for questions must provide an option for every possibility that can accurately answer the question. When an item does not provide enough response options, you risk losing useful information about that which does not fall into the available categories.

5. Incorrect use of scales

One example of incorrect use of a scale occurs when a numerical scale with three or more response options is implemented for a question that can be better answered by only two options. Using appropriate scales will save you a considerable amount of time and effort, and your data will make more sense.

6. Negative behavior responses

If a question is written so that a person can only choose responses that incriminate them for negative behaviors, the response is seen as forced and poorly written. It is often difficult to obtain personal information from surveys, especially that which signifies some negative aspect about a person or their behavior. When seeking this type of information, it is important to phrase your question in a manner that is not intrusive.

Testing Survey Questions

The best way to know if your questions are going to get the answers you need is to test your survey before you use it to gather data. Have a person who has the same understanding of the issues being addressed as those people the survey is intended for review the survey and tell you: 1) if the questions are clear, 2) what he or she thinks each question is asking, 3) if there should be response options other than the ones you have listed, and 4) if they think valuable information will be obtained from these questions.

 

 

Tip 5: Writing Better Survey Questions

Good questions are needed for an effective survey. Poorly worded questions can confuse people and cause them to provide inaccurate information that will not be useful.

Recognizing Bad Questions:

Following are examples of poorly-written questions.  Read each one and think about why it is not a good question. After each question, the problem with it is explained and a better way to ask that question is suggested.

1.  Did you think this class was informative and enjoyable?   Yes     Somewhat     No

Question 1: This is an example of a double-barreled question, where two items (informative AND enjoyable) are combined in one question.  The trainee cannot, for example, respond “Yes” to informative and “No” to enjoyable.

Question 1 Improved:

1.1	Did you think this class was informative?	Yes	Somewhat	No
1.2	Did you think this class was enjoyable?	Yes	Somewhat	No

2. Did this class meet Part 46 annual refresher requirements?       Yes      No

Question 2: This question assumes everyone knows Part 46 requirements. Anyone who did not know these requirements would not be able to respond accurately.

Question 2 Improved:

2.1	Do you know what Part 46 requires for annual refresher training?	Yes	No
2.2	If you answered “Yes” above, do you think this class met those requirements?	Yes	No

3. Please give the following items a rating from 4 (most positive) to 1 (most negative).

	Positive			Negative
The instructor	4	3	2	1
The room	4	3	2	1
The course materials	4	3	2	1

Question 3: This question does not give enough information. 'Positive' and 'Negative' are only two options – different levels of each do not exist. The information that this question is trying to obtain may be more easily and accurately obtained by creating open-ended questions.

Question 3 Improved:

3.1	List one thing you liked about the instructor and one thing that could be improved.
	Liked: __________­­________ Improvement: ­­­__________­­________

3.2	List one thing you liked about the room and one thing that could be improved.
	Liked: __________­­________ Improvement: ­­­__________­­________

3.3	List one thing you liked about the course materials and one thing that could be improved.
	Liked: __________­­________ Improvement: ­­­__________­­________

4. Where do you work?

1. a mining company
2. a federal agency
3. a school
4. a contractor company

Question 4: This is an example of missing response options. If you worked somewhere other than the places listed or at more than one place, you would not be able to respond accurately.

Question 4 Improved:

4. Where do you work? (Circle all that apply.)

1. a mining company
2. a federal agency
3. a contractor company
4. Other (please specify): _____________________________________

5. When did you last drive over the speed limit on company property?

1. within the last week
2. more than a week, but less than a month ago
3. more than one month ago

Question 5: Choosing any of the options given for this question forces the person answering to admit to doing something wrong. If you wish to learn about negative behaviors, you must word your questions so they do not accuse the person of misconduct.

Question 5 Improved:

5. It has been suggested that employees driving over the speed limit on company property is a safety problem that should be addressed. When, if ever, was the last time you noticed others or yourself driving over the speed limit on company property?

1. within the last week
2. more than a week, but less than a month ago
3. more than one month ago
4. never

6. Should this important safety program be funded again next year?               Yes      No

Question 6: This is a leading question which encourages a positive response. The word “important” may influence people to respond positively regardless of whether or not they actually agree with funding the program.

Question 6 Improved: 6. Do you think this safety program should be funded again next year?              Yes      No

 

Training Evaluation Worksheet

Use this worksheet as a guide to help you plan the evaluation of your next training session (printer friendly version).

Plan the Evaluation

1.  Who will be interested in the results?

 

 

2.  What questions will be answered?

 

 

3.  What resources (personnel, time, money, equipment) are available for evaluating the training program?

 

 

Gather the Information

What method(s) will be used to gather information?

---