Understanding Talent Assessment Part 2: Assessment Blueprint Designing

December 28, 2020 Deepika Pant

Assessment Blueprint Designing

According to the estimates by the Society of Human Resource Management Studies [1], on average, employers spend an enormous amount of $4,129 per job in the United States for hiring. Comparable amount is spent across different geographies as well. This figure in turn indicates the importance of hiring decisions for an organization. Imagine spending such an exorbitant amount and energy on talent-hiring and getting incompetent hires.

Various researches [2] have shown that there is a huge knowledge gap as far as formal assessments are concerned. Many organization decision-makers and HR professionals have a misconception that the assessment space is highly technical and difficult to understand. The presence of a multitude of selection products makes the decision much more complex for them. Hence, it is of utmost importance that the organizations and HR professionals understand and realize the importance of scientifically proven assessments to make selection decisions and have a thorough understanding of assessment design.

The first blog of the series discussed the history of the talent assessments, their types, current challenges, and eventually explained how assessments became a crucial part of pre-employment testing.

With the second part, we aim to demystify the process of assessment design, by explaining the most fundamental step involved i.e. deciding the blueprint/construct for computer-based non adaptive tests. Our learning journey will cover the following points:

Selection Process Pipeline

A good assessment design is crucial, as it not only helps the organization to find the right candidate, but also leaves a positive impression on the person who is attempting the assessment. Most of the organizations follow multi-step processes before offering a position to any candidate. The selection process pipeline will give a better idea of where assessments fit in this process.
Most organizations rely on the funneling approach for selection. Generally, the process involves three steps:
a) Initial Screening: In the initial screening process, the pool of candidates is reduced to a manageable number who may then be put through a more extensive assessment process. Different organizations follow different approaches for initial screening, which may include reviewing resumes, excluding poor performers, obvious misfits, etc.
b) Assessments: The screened candidates are then evaluated on their knowledge, skills, abilities, and personality characteristics (KSAPC’s) by using a variety of testing methods
c) One to one Interactions: The apt candidates filtered through the assessment scores then undergo one to one interaction enabling the employer to analyze and evaluate their potential and job-fitment.

We will focus on the Assessments part of the funnel, and discuss different formal assessment methods, how these can be linked to the job analysis, and eventually will uncover the assessment blue-print concept along with different points(length, duration etc.) that should be kept in mind while designing different assessments.

Job Analysis and Assessments:

The foremost step in developing or selecting an assessment method for a given job is to analyze and understand what knowledge, skills, abilities, and personal characteristics (KSAPC’s) an individual must possess to perform the job effectively. This is done with the help of job analysis. A job analysis contains the following portions:

  • Task-Based Job Analysis: This portion of job analysis consists of a comprehensive list of work/tasks that the individuals are required to perform on the job
  • Worker Oriented/KSAPC’s Based Job Analysis: This portion of job analysis consists of the KSAPC’s a worker must possess to effectively perform the on- job tasks

The table below shows a sample task list for an Analyst along with related KSAPC’s required for performing the tasks. The basic purpose of an assessment is to measure the most critical tasks or KSAPC’s identified through job-analysis.

Once KSAPC’s are identified, assessment methods are selected to identify candidate’s underlying abilities to perform important work tasks. Different organizations use different assessment methods for their hiring. Table 2 shows which assessment methods are predominantly task-based and which are predominantly KSAPC based.


Essentially an assessment creator will map KSAPC’s listed in Table 1 with Assessment methods listed in Table 2 to identify which Assessment method will be apt to evaluate the required skills. For example, consider the below table which shows the mapping of KSAPC’s with assessment methods for an Analyst profile.

 After understanding the process of identifying KSAPC’s and their linking to relevant assessment methods, let us discuss how an assessment blueprint is designed.

How is an assessment blueprint designed?

Once the assessment creator has identified which assessment methods are to be used, they need to map the KSAPC’s to the sections, topics and sub-topics that need to be covered in the assessment. For example, an Analyst should have an eye to detail implying that assessment should have questions on “Attention to Details”. Once the knowledge aspects that are to be covered within each test are identified, a test blueprint is created. Type of questions (Multiple choice, subjective type etc.) to be used in the test are also finalized. A basic blueprint defines the question allocation per section and sub-section with the assigned combination of easy, moderate, and difficult questions. It can also include the details of marks allocation (section wise/overall) and time duration (section wise/ overall).

For example, for an Analyst profile, the below assessments will be used collectively to measure the required skills and competencies:

As per the listed KSAPC’s we can figure out that the Cognitive Assessment required should have elements of Logical Reasoning and Quantitative Aptitude. The knowledge assessment should be focused on SQL. The communication assessment should be able to measure candidate’s ability of reading, writing and language proficiency. The Personality test should have elements that measure aspects of multi-tasking, flexibility, accountability etc.

After identifying the sections to be covered under each assessment, relevant topics and question types are identified for each assessment. Table 4 below shows a blueprint of the Analytical section of the Cognitive paper for Analyst, with a comprehensive description of the subsections covered.

The difficulty level to the questions is assigned in accordance with various standards, one of them being the “Bloom’s Taxonomy” which is a multi-tiered model of classifying thinking according to different levels of complexity [3]. The figure below shows the different levels in Bloom’s taxonomy:

In simple terms, an easy question will be one that can be answered by recalling facts and basic concepts, while a difficult question will be one which requires making judgements about the information, validity of ideas, quality of work based on a set of criteria.

We will have more insights on difficulty levels, question types, question creation and other related guidelines in the next blog of this series.

In a nutshell, the process of assessment blueprint design can be summarized as:

Final steps in Designing:

While keeping a check on competencies and abilities that should be a part of the required assessment, it is important to ensure that the assessment is engaging, has less drop-off rate and have an overall pleasant candidate experience. To maintain the same, consideration of the following points is important:

The length and duration of the assessment depend on the purpose of the assessment; for ex: recruitment and admissions/ entrances. Generally, recruitment assessments are relatively shorter with a lesser number of questions, commonly used duration being 45 to 60 minutes, an entrance/ admission-based assessment might be as long as 120-180 minutes, with a greater number of questions.

Understanding of formal assessment methods and to be able to identify and select the apt assessment method that can enhance the quality and productivity of an organization’s workforce is the need of the hour. This blog aimed at equipping the organizational decision-makers with the basic understanding of different assessment methods, fundamentals of blueprint designing and a few pointers like length and duration of the assessment that should be kept in mind while selecting a formal assessment.

To summarize, the steps involved in the assessment creation after Job Analysis are:

  1. Identifying the assessment methods to be used
  2. Designing the blueprint of the assessments
  3. Creating questions as per the required construct/ Pick relevant questions from the already created of questions
  4. Performing benchmarking, reliability, and validity exercises to ensure test is reliable and valid.

Wouldn’t it be interesting to understand the fundamentals and principles of question designing? The next blog of the series will aim to discuss the procedures and guidelines that are followed while designing questions for different types of assessments.

Have feedback on the write-up and/or any questions related to the topic?
Feel free to write to us on deepika.pant@aon.com.

References:

[1] Harvard Business Review. 2020. Your Approach to Hiring Is All Wrong. [online] Available at: https://hbr.org/2019/05/recruiting
[2] Shrm.org. 2020. [online] Available at: https://www.shrm.org/hr-today/trends-and-forecasting/special-reports-and-expert-views/Documents/Selection-Assessment-Methods.pdf
[3] D41.org. 2020. [online] Available at: https://www.d41.org/cms/lib/IL01904672/Centricity/Domain/422/BloomsTaxonomy.pdf

About the Author

Deepika Pant

Deepika Pant is working with Aon Assessment Solutions as a Cognitive expert and has earned her bachelor's degree in Electrical Engineering. Her expertise lies in Consulting, Research, Product Design, and providing hiring solutions to clients for Cognitive assessments.

More Content by Deepika Pant
Previous Article
Aon enables Ocado Group to Improve Diversity in Early Talent Recruitment Process Using AI
Aon enables Ocado Group to Improve Diversity in Early Talent Recruitment Process Using AI

Aon has implemented a transformative graduate recruitment process to help Ocado Group in the UK hire young ...

NEXT FEATURE
Client Spotlight - Talent Innovators: Siemens Energy's Simon Roberts
Client Spotlight - Talent Innovators: Siemens Energy's Simon Roberts

We interview Simon Roberts, regional head of talent acquisition for Siemens Eneergy, to find out which tale...

Subscribe to our talentNews

Subscribe