Are you Choosing a Fair Assessment?

May 29, 2020 Shloka Goyal

Part 3: Fairness: Concept and Applications

Imagine the results of an assessment wherein all the high scorers have similar demographic characteristics. Suppose the female participants are scoring lower than males just because the problem statements are more male-oriented. Another possibility could be of the results being unfavorable towards the higher age-group participants just because they could not comprehend some trending internet slangs.

 

Hence comes in the question of fairness and objectivity in assessment.

After talking about the validity and reliability of an assessment, one major characteristic that people often miss out on is Fairness. Standards for Educational and Psychological Testing* claims Fairness to be one of the three foundational factors. This blog aims to explain what this dimension is all about and how to ensure that you are choosing a fair assessment.

 

Why to look for ‘fairness’ in assessment?

Individual Differences is an undisputable part of human nature. On an organizational level, these differences give rise to diversity in workforce and work environment. Therefore, when an organisation decides to assess its current or prospective employees on some trait, skill, or competency, it is only fair to ensure that their chosen assessment tool does not discriminate among these individuals on the basis of their gender, ethnicity, age, social media exposure, ideologies or any other characteristic which does not form a part of the intended construct.

The practice of testing and assessment as a selection criterion gained its popularity due to its objective nature. It aimed at eradicating the assessor’s/interviewer’s subjectivity and errors pertaining to their personal biases. Thus, without a doubt, Fairness is a characteristic of high importance, ensuring the credibility of an assessment.

The main idea is to promote equality of opportunity for all.

 

The Four Perspectives to Fairness

Fairness in an assessments can essentially be looked for in four main areas:

1. The assessment’s content

The content and language of an assessment plays a crucial role in ensuring fairness. An assessment may be biased towards certain minority groups, language community, cultures, gender, or age groups simply because of the way some items are framed, or the terminology used in questions.

Fairness in the assessment content can be ensured by eliminating culturally rooted references or social and political inclinations. Furthermore, it is important to avoid any irrelevant information in the items and assessment stimuli.

2. The administrative procedure

Administrative context also plays a major part in reducing assessment biases and increasing fairness. The instructions provided to all participants should be standard and uniform. The room arrangement, the time limit, use of proctors and security measures are also required to be kept constant. The physical environment during the administration needs to be kept in check to exhibit minimum variability, including the lights, ventilation, as well as factors like time of the day, which might affect the internal or external state of the participant.

3. The response elicited

The responses which may be produced by an assessment need to be kept in check. Question items and assessment stimuli must not illicit irrelevant responses which do not align with the intended construct.

A very common way to tackle this could be to use a close-ended response approach wherever possible. For example, providing dichotomous response options or multiple-choice question responses to participants to narrow-down the variability in case of knowledge-based assessments.

4. Access to construct

Learning opportunity and past exposure of the participants are also important variability factors. For example, using historical facts as a premise for a question item in an intelligence test will be negatively biased towards individuals who have not had a formal education in the subject.

Since coming up with an assessment with a “Universal Design” is not always possible in such scenarios, a more efficient practice would be to develop differential reference norms and standards of comparisons and score interpretation for different sub-groups assessed.

 
   

What do the Assessment Users need to know?

Although as an assessment user you cannot always control the fairness characteristic, what you can do is ensure that the assessment you are choosing follows international standards and fairness guidelines. Aon’s Assessment Solutions (AAS), for example, follows exhaustive guidelines pertaining to development and executions of their assessment tools.

To simplify the whole process, below is a brief checklist explaining what aspects to keep in mind and where to look for them before choosing an assessment tool.

1.Sub-group Differences

Sub-group differences are revealed by the difference in the distribution of results/scores of various groups when they take the same assessment. Check for the demographic information of the sample on which the properties of the assessment were measured. Lower sub-group differences reveal a fairer assessment.

For example, the Written English Test by AAS presents an extremely marginal difference between the scores of male and female participants, with the small effect size of 0.1 between the two groups.

2. Norms

In certain cases (often pertaining to the nature of the assessment or trait being measured) it may not be possible to completely get rid of the sub-group differences. For such an assessment, separate norms must be present for different groups.

For example, a personality test like ADEPT-15 provides its users country/region-based norms for comparison along with a global norm. The results obtained by the participant should be compared to the appropriate norm in order to reach a credible interpretation.

3. Translations are not enough

If you are choosing an assessment which was originally not in the language of your requirement, simply getting it translated into a local language is not enough. Apart from ensuring the quality of translation, one must also keep a check on language-sensitive words and symbols.

For example, several technical terms and official names (like ‘software’ or ‘United Nations’) lose their meaning if translated literally from one language to another. In such cases, AAS experts often opt for transliteration, along with translation, of those particular terms.

4. Development process

The methodology and process used to develop an assessment must also be investigated. It must be ensured that the assessment developers have used a standard process with pre-set guidelines and the guidelines have a sound scientific base.

For example, an assessment ideally should be accompanied by its technical documentation marking the development, validation, and application of that assessment. Assessment users have the right to request for such documents and ensuring the authenticity of the assessment.

5. Nature of assessment items

The item development process may also be looked into. The major concern here is to ensure that assessment items do not have culturally sensitive references or provoke any religious or political inclinations.

For example, in certain countries, the star shape holds a strong symbolic meaning and often relates to people’s religious sentiments and thus should not be used in prompts/question items in assessment administered in those countries.

6. Guidelines of administration

The assessment must have well documented guidelines on how to administer it as well as requirement and special considerations of administration, if any. This also includes the acceptable mode of administration, standard instructions and troubleshoot guidelines.

For example, AAS’ assessment tools come with standard instructions which are displayed to all candidates. The technical documents have clearly laid out administrative procedure and guidelines along with details on usage and application of each assessment.

To conclude, fairness can be viewed as an absence of assessment bias towards any gender, age group, ethnicity, culture, etc. To ensure that, the major areas of concern can be broadly divided into assessment context, administrative context, item responses, and past exposure and learning opportunities. These, however, are only a few of the major guidelines and standards. Considerations to ensure fairness and objectivity at each stage and level of assessment stand to be of immense importance.

*The Standards for Educational and Psychological Testing is a set of assessment standards curated jointly by the American Educational Research Association, American Psychological Association (APA), and the National Council on Measurement in Education.

 

Have feedback on the write-up and/or any questions related to the topic?

Feel free to write to us on shloka.goyal@cocubes.com

 

About the Author

Shloka Goyal

Shloka is an I/O Psychologist working with Aon's Assessment Solutions and has earned her masters' degree in I/O Psychology. Her expertise lies in the area of psychometric testing and assessments. Her work is majorly focused on knowledge development and providing scientific, research-based assessment solutions.

Follow on Linkedin More Content by Shloka Goyal
Previous Article
4 Must-haves of a Successful Graduate Application Process
4 Must-haves of a Successful Graduate Application Process

When great talent is in high demand, you need to set yourself apart from the competition. You need a hiring...

NEXT FEATURE
Building Resilience by Shifting Focus and Taking Control
Building Resilience by Shifting Focus and Taking Control

During uncertain times, it is important that leaders instill resilience and assurance in their workforce.

Subscribe to our talentNews

Subscribe