Building a better maintenance force — how knowledge testing simplifies training

The maintenance manager of any facility has as first priority the efficient and reliable operation of the organisation. But sometimes the individual employee assigned to a maintenance activity may not be equal to the task at hand.

The situation where maintenance mechanics possess vastly differing capabilities can occur in any plant. One reason that this situation could take place would be in a facility which has many differing job titles. Although a millwright or mechanical technician would be required for maintenance work, the plant may have promoted a welder or other specialist to a vacancy. Another cause could be replacement of an ageing workforce with employees trained or experienced in different areas. This could result in some incumbents being fully skilled while others are knowledgeable or skilled only in their original craft.

What does my skilled worker need to know?

A necessary starting point is determining what knowledge and skills are required parts of the job. In general, a job analysis should be performed to determine the knowledge and skill areas that are required for successful maintenance of the facility.

Information on what capabilities your personnel must have should be gathered from incumbent maintenance workers, supervisors and consumers of maintenance services. Data from these job experts may be gathered by interview, survey, online or in person. The job description should also be reviewed to determine the job activities and knowledge areas important to the job. These activities and knowledge areas must be related to functional tasks that workers perform.

It is equally important that the incumbents have the required basic skills (reading, writing and arithmetic) to be able to learn. In addition, they must also have the basic technical skills that are required (eg, basic AC/DC theory for electricians or hydraulics and pneumatics for mechanics). As part of the needs analysis it is very important to keep a record of what knowledge and skills each employee already possesses.

Lastly, when conducting the job analysis, knowledge areas that are most important to success should be annotated. These areas should be emphasised during the training needs analysis and conduct of training. A numeric ranking system can be used to help determine the importance of each knowledge area.

How can I determine what my people know and what they don’t?

After you have decided what the employee needs to know, the next step is to determine if they do have the knowledge needed. For instance, your job experts may have told you that knowledge of hydraulics and pneumatics is important, but knowledge of HVAC is not necessary for the job. You need to find a way to evaluate the knowledge that has been identified as important.

Professionally developed tests are generally accepted as an effective method to determine if the employee knows what is needed to be successful on the job. But, how do you find such a test? One way is to search the internet. For example, a search of ‘maintenance tests’ or ‘mechanical technician test’ may provide you with numerous sites to explore. Online information about commercially sold tests may give you enough information to narrow your choices. Find out if you can look at a sample copy of the test to determine if the test evaluates the same knowledge that you have identified as being important. Some industries have knowledge and skills that are so specialised that an off-the-shelf product may not be available, but many maintenance managers are able to find tests that work for them.

For example, if the needs analysis determines that there are eight knowledge and skills areas that are important for successful maintenance of the facility, a test with those eight knowledge and skills areas should be selected. The number of items in each section of the test should correlate with the importance of each of these knowledge areas to the success in maintenance of the facility.

Once a test is selected which reflects the knowledge and skills that are to be tested, the next step consists of testing both incumbents and supervisors. Supervisors should be tested so that they will have familiarity with the test, and be acquainted with what their employees are experiencing. The scores and statistics for the supervisors should be compiled separately from the incumbents in order to show a true picture of training needs of the workforce.

For each person tested, an individual scoring record can be developed, showing the number of items correct in each of the knowledge categories, the person’s percentile rank compared with the other employees in that particular locale, and the person’s percentile rank compared with a national database of other examinees. Most test publishers have an administration and scoring manual available to allow the employer to interpret these scores. Figure 1 is a sample of an individual scoring report.

Figure 1: Sample of an individual scoring report

This individual scoring sheet can be the start of the training needs analysis. As an example, you may decide that the individual does not need training in hydraulics, but does need instruction in the area of computers and PLCs.

Establishing a cutting (or passing) score is a helpful tool for determining deficiencies. For example, if the cutting score for the test is 70% correct, then any part of the test on which the individual scores below that percentage may be considered to be an area for remediation. Care should be taken not to confuse ‘percentile rank’ with ‘percent correct’. Most test publishers provide percentile rank in order to show comparison with other test takers. Percentile rank is the percentage of people that scored lower than the person whose individual scoring record is shown.

If the goal of testing is to determine each individual’s areas of strength and weakness, the individual scoring record is an important tool, however, you may be evaluating the overall needs of your entire maintenance staff to determine general areas of strength and weakness. Often it is beneficial to the employer to get aggregate data on test performance. This need not include individual scores by name if there is fear of recrimination. You should find out if the publisher of the test that you have used will provide a report that shows the local plant ranges, the local plant averages, and the local plant percentile ranks when compared with the national sample of maintenance workers. From this information you can easily determine where your workers have shown deficiencies and can determine a priority list of areas where training is needed.

Frequently there is a request by the union or individuals for anonymous testing. Employers complying with this request often will find undesirable consequences. Commonly, people don’t try to do their best when their name isn’t on the test. Sometimes employers may have a third party (the local community college or a human resources firm or staffing service) administer tests and either mail results to the test takers or give counselling sessions concerning remedial activities. That way a true test performance is obtained, and the individual retains his or her confidentiality.

Should I consider online testing?

Most commercially sold maintenance tests come in a paper and pencil format, but the newest method of test administration is online testing and the advantages to this method of test administration are numerous.

Testing online generates the largest benefits from simplicity of scoring. Because online tests are not hand scored, there is no possibility for human error, cheating or bias.

Results are often immediately available upon completion of the exam and because they are in digital format can be easily compared to other examinees or downloaded into spreadsheets for custom analysis.

The application of online testing to training is most beneficial in the comparison of testing results for an individual. In this case, multiple retakes of an exam can be compared to determine improvement and the effectiveness of the training. By setting deficiency scores it is possible to determine which areas need improvement (areas for remediation). Thus, many examinees may be assessed and compared.

Figure 2 is a sample training report that shows knowledge area deficiencies and the improvement after training.

Figure 2: Sample training report

The report in Figure 2 lists the initial testing of an individual (Assessment #1) to determine training needs (highlighted in red). This person was performing deficiently in knowledge of hydraulics and pneumatics, welding and rigging, and shop machines. The second assessment (Assessment #2) was completed after conducting training in these specified areas. The results are passing scores for hydraulics and pneumatics and shop machines. This report reveals that further training should be conducted in welding and rigging.

Online testing helps simplify the process of needs analysis by performing the numerical analysis and comparisons for you. By simplifying the process, there is an increased chance of success of the training process.

Implementing training strategies

Now that you know areas of strength and weakness, what now? A tabulation of the areas targeted for training can be made by the various departments or units. Decisions must be made as to whether or not training is to be focused on improving the weak areas of the entire staff or targeted specifically to help each individual achieve their fullest potential.

In many situations, when training is widely needed, no one person or group is singled out as lacking in skills. For the generic training, the local technical college can be a good resource.

As the areas where training is needed become more specialised, groups trained will become smaller and more detailed. As the company determines that training needs are linked to specific equipment, a vendor sponsored and run training session should be scheduled. Many organisations are eager to provide training that is specifically needed or matched to equipment and processes at the facility.

Figure 3: Training model flowchart

I have trained my employees. Am I done?

It is now important to retest in order to determine whether or not the training provided has successfully accomplished its task. If your employees have gone through extensive training over a long period of time, the same test originally administered can be retaken since it is unlikely that test takers would have memorised the questions. However, there may be other options. Many test publishers develop alternate forms of a test which would contain the same number of items, the same content areas and a similar level of difficulty. This retest will identify areas for further training.

*Roland T Ramsay, PhD, is the president of Ramsay Corporation, a human resources consulting services company that specialises in the development of knowledge, skills and aptitude assessments for skilled and technical workers. Ramsay Corporation has served over 4000 clients, including one third of the Fortune 100 companies, and is the market leader in test development for skilled workers. Dr Ramsay is also a licensed psychologist and has a Diploma in Industrial and Organizational Psychology from the American Board of Professional Psychology. He is the author of several guides to hiring and management and has more than 30 years of human resources experience.