It is hard to differentiate between the ‘good’, the ‘bad’ and the ‘ugly’ among the myriad of recruitment and selection services and tools and instruments offered today. Andreas Lohff, Founder and Director of the cut-e Group and now CEO of Aon’s Assessment Solutions, explains where online psychometrics can add the most value and how to avoid instrument selection pitfalls.
When is online assessment useful?
As technology and the Internet have spread, so has the application of online psychometric assessment. In some situations, however, online assessment is particularly useful and adds a lot of value while saving costs:
- When implementing consistent standards in local or dispersed selection processes.
- When speed is of the essence; for example, if competing for a select few candidates within a small talent pool.
- When you need a reliable way to sift out unsuitable candidates early on from a large number of people to save time and resources, enabling you to focus on those with real potential.
Let’s look at these situations in more detail.
Online diagnostics are often used when selection strategies are developed centrally yet executed locally; for example, when sales people, service staff or field technicians need to be selected at different locations.
With the help of online assessment, it is easier to set and implement specific selection standards. In other words, to control the selection process centrally, while allowing local departments to act autonomously. This simplifies organisational structures, saves resources and accelerates decisions.
Small pool of candidates
Online assessment can be very helpful whenever there are only few appropriate candidates for a vacancy, as often is the case with specialist profiles, eg, in the engineering and IT sectors. In order to engage interesting candidates, a fast reaction and an efficient selection process are necessary. Online assessment helps identify good candidates very quickly and helps recruiters to decide where to focus their time.
Finding a needle in a haystack
Online assessment plays an especially important and resource-saving role when screening large pools of applicants, such as graduates. The online assessment acts as a filter to detect and disqualify unsuitable candidates from a pre-selected candidate pool.
How do you know which instruments to choose?
Excellent psychometric instruments should meet key quality criteria. Objectivity, reliability and validity are the three key words - but what do they mean?
Objectivity refers to the extent to which the diagnostic results are independent of the investigator. Objectivity is achieved through standardisation - from the administration of the test through to scoring and interpreting the results.
Reliability, sometimes also termed accuracy, is the extent to which a diagnostic instrument shows consistent results in repeated application. In other words, does it measure the same thing again and again? Although there are no completely reliable instruments, realistically you shouldn’t accept a reliability value under 0.7. In the case of questionnaires, the ideal value lies between 0.75 and 0.85; for ability tests, it is between 0.8 and 0.9.
Validity is a statement of how far the test measures what it is designed to measure. To what degree of certainty can conclusions be drawn from the testing behaviour to the behaviour outside the test situation? A valid intelligence test, for example, provides a way of measuring the intelligence of a person and therefore allows a prediction of performance in situations in which intelligence is important.
An instrument with perfect validity would have a validity coefficient of 1.00. As with reliability, this value is unrealistically utopian. Nevertheless, diagnostic instruments can be considered good with a value of at least 0.3.
Additional criteria for online testing
It is essential that the procedures in online instruments are self-explanatory and have enough sequences of interactive examples. Hardware independence also plays a big role. For example, a candidate with a faster Internet connection must not have an advantage over another with a slower connection. Quality certifications from independent associations can be a good orientation guide for companies.
How cheat-proof are online tests?
We're often asked this question. Companies have two main concerns, online plagiarism and replacement cheating. Companies are often concerned that candidates will publish the test ‘items’, or test questions and corresponding answers, on the Internet. This makes them easily accessible to anyone needing to sit the test. This risk is mitigated when the tests, like all those designed by cut-e, are based on item generators. This means that a very large number of different items are available for each test. Each test is generated uniquely for each candidate, once only, at the start of the test. Another candidate applying for the same job at the same company taking the ‘same’ test would receive a completely different test consisting of other exercises. In this way, no sample solutions can be circulated and no repetition effect occurs.
Retesting to mitigate replacement cheating
Companies are also concerned that candidates taking tests unsupervised in their own home will ask a more talented friend to take the test in their place. As online tests are particularly effective in pre-selection where the objective is ‘negative’ selection - identifying non-suitable candidates to avoid wasted time and effort - successful candidates can be retested face-to face at a later stage. It becomes apparent at this point, if not earlier, when someone has asked a substitute to complete the first test. Warning messages (known as disclaimers) at the start of the online assessment warn candidates that they will be retested. I am happy to say that in reality, our clients and I have come across very few attempts to cheat during our test procedures.
The right business partner
So far, we have covered the situations in which psychometric assessment can be used, the importance of quality criteria and what to look out for when choosing your instruments – particularly concerning online testing. Often the real value of psychometric assessment comes from working with an experienced business partner to ensure that your business objectives are met in the best possible way. Consider whether your potential supplier is experienced and able to turn that experience into value for money for you. Can the supplier integrate the tests you need with any existing HR workflow or information systems? Is the supplier able to think out of the box and keep your specific challenge at the forefront? Can your corporate competency model, brand and values be integrated in the test reports - quickly and at a reasonable price? Can your supplier turnaround these changes as quickly as you need them without compromising quality? And last but not least, do the supplier’s products meet the key quality criteria.
Quality criteria for psychometric tests and questionnaires
- DIN 33430 establishes consistent standards for diagnostic instruments. It contains all important criteria and requirements concerning the instruments and the process.
- Similar guidelines exist from the International Test Commission (ITC: www.intestcom.org) and from the European Federation of Psychologists’ Associations (EFPA: www.efpa.be).
- A recommended book, DIN-Screen (Kersting, 2007), contains a detailed overview about how to use DIN 33430 with respect to the evaluation of test procedures
About the AuthorFollow on Linkedin Visit Website More Content by Aon's Assessment Solutions