Are you choosing the right assessment?

March 31, 2020 Rishabh Saxena

Part 1: Validity: concept, types and applicability 

Using assessment tools in organizations has become a common trend now. Given a scientific assessment’s objective and efficient nature, almost every mature and growth-focused organization uses assessments for making their hiring and post-hiring decisions. These assessments may be developed in-house with the help of Subject-Matter Experts, or through a partnership with an assessment expert organization. The questions that often remain unanswered are:

  • Are the assessments cost-efficient?
  • How accurate are the results of the assessment?
  • Do assessments fulfil the purpose for which they were chosen?

All these questions help answer the one main concern of the organization: Are the assessments credible enough to allow organizations to base their personnel decisions on them?

The scientific properties of an assessment provide a clear answer to these concerns. Before finalizing their choice of assessment, the organization must investigate its psychometric properties like validity, reliability, fairness etc. In this blog, I attempt to introduce one of the assessment properties, validity, with its interpretation and applicability in actual scenarios.

Concept of validity

Validity refers to how well the assessment measures the underlying outcome of interest. If an assessment measures what it claims to measure, and the results closely correspond to real-world values, then it can be considered valid.

Validity can be measured through multiple approaches resulting in different types of validity. There are three broad types, and their interpretations (in a simplified form) can be expressed as follows:

  1. Construct validity:
    Does the assessment measure the skill/ ability/ knowledge for which it was designed?
  2. Content validity:
    Is the assessment able to measure each aspect of the skill or knowledge which it aims to measure?
  3. Criterion validity:
    • Concurrent validity: Are the scores/result of assessment similar to scores/ result of different assessment assessing same skill?
    • Predictive validity: Do the scores of the assessment predict the on-job performance of assessed candidates?

However, each type of validity has its applicability and associated challenges. The table below explains the same.

Type

Desired Outcome

Required Data

Challenges

Construct Validity

To check if the assessment measures target competencies or not

Performance of the same candidates in an assessment with a similar construct and one with a completely different construct

The Same set of candidates giving multiple assessments which can be used for validation

Content Validity

To check if the assessment measures the intended competencies comprehensively

Subject matter experts’ inputs on the assessments

SMEs for niche assessments

Criterion Validity

To check if assessment results can predict the desired outcome

Data of end outcome to correlate with assessment scores

Unavailability of well-defined outcome data

 

Let’s look at each type one by one to get an idea of how these are used in actual scenarios.
I have also tried to relate them with a couple of examples for better understanding.

  1. Construct validity

Construct validity evaluates whether the assessment tool really represents the outcome we are interested in measuring. It is about ensuring that the assessment is in-line with the construct intended to be measured. If an assessment is developed to measure learnability, you need to know if the questionnaire really targets aspects of cognitive ability or is it measuring the respondent’s knowledge, or something else?

Example:
If an assessment is supposed to assess cognitive ability, then the construct should not be measuring the technical knowledge of the candidate.

Construct validity is usually calculated by correlating scores of concerned assessments with scores of assessments having similar construct (convergent validity) and different construct (divergent validity). The correlation should be strong in the former case while a weak correlation should exist for divergent validity.

The other types of validity described below can all be considered as evidence for construct validity.
 

  1. Content validity

Content validity tells whether an assessment has all the aspects of the construct.
To produce valid results, the content of an assessment must cover all relevant parts of the outcome. If some aspects are missing from the assessment (or if irrelevant aspects are included), the validity is threatened.

Example:
An algebra assessment is developed which should cover every form of algebra that was taught in the class. If some subtopics of algebra are missed out, then the results may not be an accurate indication candidate’s understanding. Similarly, if the assessment has questions that are not related to algebra, the scores of the assessment would not be a valid measure of algebra knowledge.

The most accessible method to check content validity is through a Subject Matter Expert’s (SME) review on the items/ questions and topics of the assessment.
 

  1. Criterion validity

Criterion validity is of two types i.e. concurrent validity and predictive validity.

Concurrent validity: It evaluates how closely the results of an assessment correspond to the results of a different assessment that has already been established valid for the required outcome.

Example:
An assessment to measure candidates’ English writing ability will have high concurrent validity if it has results very similar to the results of an established valid assessment. Mandatory criteria for doing this exercise is that the same group of candidates or group with similar background must have taken both the assessments.  

Predictive validity: It is drawn by the correlation between assessment scores and the desired outcome metric.

Example:
Consider an assessment intended to measure the sales ability of the employees. If the correlation between the sales revenue of employees and their assessment scores is high, then the assessment is said to have high predictive validity.

In the next blog of the series, we will be addressing the reliability of assessments.
 


Would you like to learn more about our assessment solutions?
Read our case study "Supporting Autonomous Development with Self-Assessment at Open Grid".

About the Author

Rishabh Saxena

Rishabh Saxena is Senior Business Analyst at Aon. He graduated from IIT (BHU) Varanasi and did an Executive Program focused in Business Analytics from IIM Calcutta. Rishabh has a background in insights generation and data visualization through Tableau, PowerBI and Market research.

More Content by Rishabh Saxena
Previous Video
Webinar: Talent Strategies in Disruptive Times - Essential Retail, eCommerce and Delivery Services Talent in North America
Webinar: Talent Strategies in Disruptive Times - Essential Retail, eCommerce and Delivery Services Talent in North America

Aon’s on-demand 30-minute webinar looks at what essential retailers can do in today’s disrupted world to ke...

NEXT FEATURE
Mind the Employability Gap
Mind the Employability Gap

Admissions are going down year on year. Those graduating are not being considered employable enough to join...

Subscribe to our talentNews

Subscribe