Making situational judgement tests mobile

April 20, 2018 Richard Justenhoven

image chatAssess in use

The next step for Situational Judgement Tests

We’re looking forward to presenting at SIOP in Chicago. It’s an opportunity to share our research and findings with the wider organisational, occupation and business psychology world and to hear their thoughts on what we are doing. This year we are delighted to have a number of colleagues taking part in numerous panel discussions and presentations. I’ll showcase some of them in the coming weeks.

Together with Tara Johnson and Eleni Lobene from Aon, I shall be part of a panel of practitioners from some of the leading consulting organisations discussing the current and future state of mobile simulations or situational judgement tests.

We know that increasing numbers of organisations are adopting technology-enabled assessments including media-rich simulations. But creating these mobile-enabled simulations is a complex process, and there is little in the way of research or guidance in the literature.

We’ve been developing our capability over a number of years, finding out what works and what doesn’t. We know that Situational Judgement Tests (SJTs) are popular as an assessment tool because of their criterion-related validity (Lievens, Peeters, & Schollaert, 2008). However, the situations created are artificial to a certain extent, particularly when it comes to social interaction. This affects criterion-related validity (Lievens & Sackett, 2006). Gamification, i.e., incorporating gaming elements into the assessment process (Deterding, Dixon, Khaled, & Nacke, 2011) as well as transforming them into a mobile environment (King, et al., 2015) is likely to make candidates immerse in their role and make scenarios more realistic.

We have transformed situational judgement tests (or SJTs) into a mobile-first simulation called chatAssess. It’s customisable, and fully mobile and has received an excited and positive reaction by our clients. It simulates real-time instant messaging like WhatsApp. Participants take on the role of an employee working with a company and are given their colleagues’ profiles. They receive text and picture messages from different colleagues, but can also receive external links or invitations to additional ability tests. When candidates respond to messages, there is a response from the other person that the candidate can respond to again, which then creates a simulated instant messaging interaction flow in which lots of different elements such as tests or video interviews can be embedded. There are some limitations with regard to the current response pattern and the way forward is likely to be an open response format where candidate’s responses are run through a Natural Language Processing API which uses text analytics to understand the structure of sentences and their meaning and intention through statistical methods and machine learning. But that’s for the future.

Right now, chatAssess is already firmly embedded in the recruitment processes of some great clients. You can learn more about chatAssess here

We hope that our session at SIOP will highlight work that is currently being done in this area, as well as share key learnings, best practices, and discuss what the future holds in this area.

References
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From Game Design Elements to Gamefulness: Defining “Gamification”. MindTrek’11, Tampere, Finland.

King, D. D., Ryan, A. M., Kantrowitz, T., Grelle, D. & Dainis, A. (2015). Mobile Internet Testing: An analysis of equivalence, individual differences, and reactions. International Journal of Selection and Assessment, 23 (4), 382-394.

Lievens. F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: a review of recent research. Personnel Review, 37(4), 426-441.

Lievens, F. & Sackett, P. R. (2006). Video-Based Versus Written Situational Judgment Tests: A Comparison in Terms of Predictive Validity. Journal of Applied Psychology, 91(5), 1181–1188.

About the Author

Richard Justenhoven

Richard Justenhoven is the product development director within Aon's Assessment Solutions. A leading organizational psychologist, Richard is an acknowledged expert in the design, implementation and evaluation of online assessments and a sought after speaker about such topics.

Follow on Linkedin Visit Website More Content by Richard Justenhoven
Previous Article
The formula for successful assessment
The formula for successful assessment

Innovative formulas and models can tell you not only which assessments to choose but also how to utilise th...

NEXT FEATURE
The impact of gamified assessment
The impact of gamified assessment

Research: cut-e research into the impact of gamified assessment on test taker motivation, performance and p...

Subscribe to our talentNews

Subscribe